Public results snapshot imported on 10 November 2026. Benchmark jobs continue on the current LVTD site.
Compare model quality, cost, and the answers behind the scores.
Answers from OpenRouter, compared by TypeSafe's Jev. A small, fully inspectable preference experiment. Not a claim of general intelligence or verified accuracy.
Weighted mean of per-question Elo. A model receives an overall score only after completing every matchup across all active questions.
All models of 29 models
Model rankings. Value is an experimental quality and affordability blend. Costs are actual reported USD.
Model
Weighted Elo
Mean answer cost
Value / 100
MiMo-V2.6-FlashXiaomi
1699.3
$0.002860
76.5
Claude Fable 5.1Anthropic
1682.8
$0.226293
53.2
MiMo V2.6 ProXiaomi
1681.2
$0.010550
66.4
Claude Opus 5.5Anthropic
1663.9
$0.076747
53.8
GLM 5.3 PrimeZ.ai
1647.8
$0.165112
50.8
GLM 5.3 FlashZ.ai
1608.0
$0.004392
66.4
GPT-6 AstraOpenAI
1590.5
$0.096045
46.7
Muse Spark 1.3 ContributorMeta
1587.6
$0.001163
70.5
Hy4 previewTencent
1581.5
$0.027925
51.0
Kimi K3Moonshot AI
1562.5
$0.082261
44.5
DeepSeek V4.1 FlashDeepSeek
1561.4
$0.006077
59.8
Space Bunny AlphaStealth
1549.6
$0.000000
70.0
MiniMax M2.7 (Nitro)MiniMax
1547.4
$0.012704
53.0
Qwen3.8 Max PrimeAlibaba / Qwen
1534.8
$0.146123
40.4
Nemotron 3 Ultra (free)NVIDIA
1513.8
$0.000000
66.4
gpt-oss-120bOpenAI
1481.1
$0.000647
61.3
Grok 4.7xAI
1477.7
$0.036554
39.2
Hy3Tencent
1472.0
$0.005701
51.3
Gemini 3.8 FlashGoogle
1461.1
$0.012417
44.5
DeepSeek V4 ProDeepSeek
1430.1
$0.005675
47.2
GPT-6 LunaOpenAI
1423.6
$0.001223
54.2
GPT-6 SolOpenAI
1401.9
$0.016871
36.5
Solar Pro 4Upstage
1397.7
$0.001248
51.7
Qwen3.7 FlashQwen
1375.5
$0.001225
49.7
gpt-oss-20b (Nitro)OpenAI
1369.9
$0.001002
49.7
Ling 3.0 FlashinclusionAI
1354.2
$0.000382
50.0
Gemini 3.1 Pro PreviewGoogle
1312.1
$0.090838
20.7
Mercury 2.5Inception
1290.5
$0.000450
44.8
Mistral Medium 3.5Mistral
1240.8
$0.015811
24.5
Value v1 blends 70% Elo-derived quality with 30% affordability. Higher is better under this chosen tradeoff. See the formula and limitations. Mean cost uses the same question weights as Elo; unknown costs are never treated as free.
Costs are actual USD reported by OpenRouter for saved successful answers, including provider-reported reasoning usage. Zero means reported free; Unknown means no cost was reported. These totals exclude failed/retried attempts and Jev charges, and are not an account invoice. Attempts absent from retained responses and backups remain unknown.
Latest judgment: Sep 28, 2026, 17:58 UTC.
Quality and cost, together
Higher is stronger in Jev's rankings. Further left is cheaper. Solid points mark the observed frontier: no other model is at least as strong and at least as cheap, with one strict improvement. Hover a point for its values.
Hover a model for its values. All plotted values are also available in the overall table and JSON.
The 70/30 blend and one-cent reference are explicit design choices, not learned truths. Quality is an Elo-implied preference against a 1,500-rated reference, not factual accuracy. Free answers receive finite scores; unknown costs and incomplete scopes are not scored. Overall cost is the question-weighted mean per saved answer. Free-tier availability and prices can change.
One answer. Every opponent.
Each model answers each question once, without tools or web access. The same prompt goes to every model; provider-default sampling settings apply. Reasoning effort can be explicitly configured on a new question version; existing inputs are frozen after use. Answers start with each model's configured output budget. Failed, truncated answers may be retried with an explicitly raised allowance, recorded on each answer. Truncated or failed answers are not scored.
Each unordered pair is judged once. Model labels are omitted from the judge's input, and A/B presentation is assigned by a stable hash. Answers may still reveal their origin. Probabilities and confidence are saved alongside each decision.
Reproducible Elo
Every question starts all models at 1,500. Each stored win/loss is replayed once with K = 32 in a stable, hashed model-ID order. That makes the same dataset reproducible, not order-independent. Adding opponents can change existing ratings without rejudging their old matches.
Question weights default to one. The overall score is the weighted arithmetic mean of complete per-question ratings. Pending models have no overall rank; per-question results remain provisional until the round robin is complete.
A preference experiment
Jev selects a winner, not a verified truth. This small sample measures one judge's preferences on these prompts, not general model ability. Code is not executed and mathematical proofs are not independently verified. A single presentation per pair does not eliminate position bias.
Confidence describes Jev's decision distribution; it is not the probability that the answer is factually correct. No repeated trials, error bars, or statistical-significance claims are implied.
Use this as a source
The full JSON dataset includes prompts, rubrics, original answers, generation settings, costs, attempt history, and every pairwise decision. The agent-readable methodology explains the schema and scoring. Save an export with its timestamp and SHA-256 checksum when citing a report: this live benchmark changes as questions and models are added.