2 ms·

Gemma-4-E4B-it scored 15/25 on my https://sql-benchmark.nicklothian.com/#all-data https://sql-benchmark.nicklothian.com/#all-data (agentic SQL generation). The

by nl 7mo ago

Gemma-4-E4B-it scored 15/25 on my https://sql-benchmark.nicklothian.com/#all-data https://sql-benchmark.nicklothian.com/#all-data (agentic SQL generation). The naming is a bit odd - E4B is "4.5B effective, 8B with embeddings", so despite the name it is probably best compared with the 8B/9B class models and is competitive with them. Qwen3.5-9B also scores 15/25 in thinking mode for example. The best 9B model I've found is Qwen3.5-9B-Claude-4.6-Opus-Reasoning-Distilled-v2 which gets to 17/25 gemma-4-E2B (4bit quant) scored 12/25, but is really a 5B model. That's the same as NVIDIA-Nemotron-3-Nano-4B which is the best 4B model I've found (yes, better than Qwen 4B). That's a great score for a small model.

alecthomas 7mo ago
Oh this page is great! I just released AIM [1] which is a tool that generates verified SQL migrations using LLMs, and I tested a bunch of models manually. I think I'll just link to your page too! [1] https://github.com/alecthomas/aim https://github.com/alecthomas/aim
GaggiX 7mo ago
>so despite the name it is probably best compared with the 8B/9B It runs much faster than a standard 8B/9B model, the name is given by the fact that it uses per-layer embedding (PLE).
neonstatic 7mo ago
Very happy to see updates to your benchmark. Looking forward to inclusion of larger Gemma 4 models!
- nl 7mo ago
  The medium one on OpenRouter didn't support tools when I tried it. I will update when there is one.
chromatin 7mo ago
I love that you are doing this test. However, as it purports to be a test of "English-to-SQL", your hardest question (Q9) seems ungrammatical: > Show order lines, revenue, units sold, revenue per unit (total revenue ÷ total units sold), average list price per product in the subcategory, gross profit, and margin percentage for each product subcategory. In particular, the clause "in the subcategory, gross profit, and margin percentage for each product subcategory" is ambiguous, and I wonder if more models would pass if the English were reformulated to be correct. (it's also notable that Claude Opus 4.6 and Sonnet 4.6 both "missed" this one)

by nl 7mo ago

alecthomas 7mo ago

Oh this page is great! I just released AIM [1] which is a tool that generates verified SQL migrations using LLMs, and I tested a bunch of models manually. I think I'll just link to your page too! [1] https://github.com/alecthomas/aim https://github.com/alecthomas/aim

GaggiX 7mo ago

>so despite the name it is probably best compared with the 8B/9B It runs much faster than a standard 8B/9B model, the name is given by the fact that it uses per-layer embedding (PLE).

neonstatic 7mo ago

Very happy to see updates to your benchmark. Looking forward to inclusion of larger Gemma 4 models!

nl 7mo ago
The medium one on OpenRouter didn't support tools when I tried it. I will update when there is one.

chromatin 7mo ago

I love that you are doing this test. However, as it purports to be a test of "English-to-SQL", your hardest question (Q9) seems ungrammatical: > Show order lines, revenue, units sold, revenue per unit (total revenue ÷ total units sold), average list price per product in the subcategory, gross profit, and margin percentage for each product subcategory. In particular, the clause "in the subcategory, gross profit, and margin percentage for each product subcategory" is ambiguous, and I wonder if more models would pass if the English were reformulated to be correct. (it's also notable that Claude Opus 4.6 and Sonnet 4.6 both "missed" this one)