1Core question
Custom model eval: a repeatable test that measures which model performs best on your actual application task.
Public benchmarks shortlist candidates; custom evaluation selects the deployable model.
2Model eval vs app eval
Model evalTests a model task component, such as Text-to-SQL.
App evalTests the complete product workflow and user experience.
3Case study
A cricket fan asks: “Who scored the most runs against Bumrah?” The system converts English into SQL, runs it on the cricket database, and returns the answer.
Goal: replace a manual analyst lookup process with a scalable feature.
4Text-to-SQL pipeline
User question→LLM + schema→SQL→Database→Answer
5What the LLM needs
- User question in natural language
- Database schema: tables, fields, relationships
- Clear prompting rules and constraints
- Optional query examples / few-shot context
6Selection constraints
01QualityCorrect database result for real questions
02CostMust fit the monthly operating budget
03LatencyFast enough for the live user experience
7Use benchmarks correctly
Text-to-SQL leaderboards may be stale, unclear, or score fine-tuned systems rather than the base models you can use.
Practical proxy: coding performance can help create a shortlist—but is not the final proof.
8Cost inputs
input tokensoutput tokensinput $ / 1Moutput $ / 1Mrequests / daydays / month
Case-study estimate: ~400 input tokens + ~100 output tokens per request, ~5,000 requests/day.
9Monthly cost formula
Per query = (input tokens × input rate) + (output tokens × output rate)
Monthly = per-query cost × daily requests × 30
First filter: discard candidate models that do not fit the budget.
10Traffic reality
Traffic is often spiky. Major matches generate far more questions than ordinary days.
Planning rule: average volume is useful for cost; peak volume matters for reliability and latency.
11Golden dataset
Golden row: a realistic user question + analyst-approved correct SQL + an order-matters flag when needed.
Start small—e.g. 50 high-quality items—but cover the real question types users ask.
12Golden coverage
- Player statistics and comparisons
- Teams, tournaments, and seasons
- Match history and filters
- Aggregations: count, sum, average, max
- Ordering, top-N, and time constraints
13Wrong scoring method
Never use SQL string matching. Different SQL statements can be textually different but return the same correct data.
14Correct scoring method
Run golden SQL+Run generated SQL→Compare result sets
Correct: both queries return equivalent result tables.
15Execution accuracy
Execution accuracy = correct result sets ÷ total evaluation questions
Run the exact same dataset, prompt, schema, and scoring protocol for every model.
16Result-set equality
- Same number of rows
- Normalize equivalent values: 2 = 2.0
- Handle harmless decimal precision differences
- Sort before comparison when order is irrelevant
- Preserve order when it is required
17When order matters
If a task requires ranking or an ORDER BY result, do not sort before comparison. The model must return correct values in the correct order.
18Evaluation loop
Load model→Generate SQL→Run both queries→Compare tables→Score
19What to report
| Metric | Decision use |
|---|
| Execution accuracy | Task correctness |
| Monthly cost | Budget fit |
| Latency (p95) | User experience under load |
| Failure cases | Diagnose improvements |
20Common mistakes
- Choosing solely by leaderboard rank
- Using an unrepresentative evaluation dataset
- Comparing SQL strings instead of results
- Ignoring latency or budget
- Measuring only an average—not individual failures
21Iterate continuously
Add meaningful production failures to the golden dataset. Re-run the suite when changing the model, prompt, schema, or evaluation logic.
Regression testing: verify that a new version is genuinely better than the old one.
22Mental model
Don't ask: “Which model has the best benchmark score?” Ask: “Which gives the best correctness / cost / latency trade-off for my real workload?”
Rule: use public data to shortlist; use your eval to decide.