Custom Model Evaluations for Text-to-SQL

The course now moves from theory to practical work. Earlier sessions separated LLM evaluations into model evaluations and application evaluations. Model evaluations include public benchmarks and custom evaluations. Benchmarks identify strong candidate models in general; custom model evaluations answer the more useful question: Which model is best for my specific application?

A public coding or reasoning leaderboard may give a shortlist, but it cannot tell you which model works best with your database schema, prompts, traffic pattern, cost limit, and users. That requires a custom evaluation.

Imagine working for a cricket website such as ESPNcricinfo. During live matches, users ask questions such as “Who has scored the most runs against Bumrah?” or “How many wickets did Bumrah take in the previous India–Pakistan match?” Traditionally, analysts query the cricket database and send answers to the live-commentary team. At high-traffic moments this manual workflow does not scale.

The proposed feature lets a fan ask a question in plain English. An LLM receives the question plus the database schema, generates SQL, the system runs the SQL against the cricket database, and the result is shown to the user. In effect, the LLM automates the analyst’s database-query task.

The LLM is the system’s “brain,” so the team needs to select one deliberately. The lecture sets three practical requirements:

• Budget: monthly spending must stay within the agreed limit.
• Speed: users should not wait too long for an answer.
• Quality: generated SQL must return the correct result for real cricket questions.

The workflow is therefore: filter candidate models by cost, check latency, then compare their task-specific accuracy on a representative evaluation set.

Cost depends on input tokens, output tokens, request volume, and each provider’s token pricing. The lecture uses a working estimate of about 400 input tokens (question, schema, instructions, and examples), 100 output tokens (the SQL query), and 5,000 queries per day. Real demand will be spiky—major matches produce far more traffic—so this should be treated as a planning estimate, not a fact.

For each candidate, calculate per-query cost, multiply by expected monthly volume, convert currency if needed, and discard models that exceed budget. This stops the team from spending evaluation time on an unaffordable model.

Text-to-SQL leaderboards can be useful, but the lecture advises caution: they may be outdated, use fine-tuned systems rather than base models, or provide unclear methodology. Because SQL generation is a programming-like task, a current coding leaderboard can be used as a rough proxy to produce a shortlist.

This is only a starting point. A coding ranking does not prove Text-to-SQL success for the cricket database. After applying cost and latency filters, the remaining candidates must be evaluated directly on the target task.

Create a golden dataset from realistic user questions. For every question, a knowledgeable analyst writes the correct SQL query. The lecture uses 50 examples as a small initial evaluation set. The questions should cover the kinds of requests users will actually make: player statistics, match history, teams, tournaments, rankings, filters, aggregations, and ordering.

The golden dataset should be high quality. If the reference SQL is wrong or the questions are unrepresentative, the final score will be misleading. As the product changes, the dataset should grow with new real-world failure cases.

Do not compare generated SQL with golden SQL character by character. Multiple SQL statements can produce the same correct result. Instead, execute both the golden SQL and the model-generated SQL against the same database and compare their result tables.

If the two result sets are equivalent, the generated SQL is correct for that evaluation item—even if the query text is different. Repeat this for every question, then calculate execution accuracy: correct result sets divided by total questions. Run the exact same process for every candidate model.

A robust evaluator compares result tables carefully. It checks row counts, normalizes equivalent values (for example, 2 and 2.0, or harmless decimal precision differences), and handles row order. Results can be sorted before comparison when order is irrelevant. If the correct query uses an ORDER BY clause and order matters, preserve and compare the order.

The final orchestration script loads each model, sends every golden-dataset question with the schema and prompt, executes both SQL queries, calls the evaluator, and records the scores. The model with the best acceptable trade-off between execution accuracy, latency, and price is the practical choice.