3 ms·
Does anyone have any good simple prompts for testing new "reasoning" models like this one? "Count the letter Rs in the word strawberry" is a bit dull! I'm try
by simonw 2y ago
Does anyone have any good simple prompts for testing new "reasoning" models like this one?
"Count the letter Rs in the word strawberry" is a bit dull!
I'm trying this one locally using Ollama and the smallest quantized GGUF version (769MB) I could find - https://huggingface.co/bartowski/agentica-org_DeepScaleR-1.5B-Preview-GGUF/tree/main https://huggingface.co/bartowski/agentica-org_DeepScaleR-1.5... - I ran it like this:
ollama run hf.co/bartowski/agentica-org_DeepScaleR-1.5B-Preview-GGUF:IQ3_XXS
Here's the answer it gave me: https://gist.github.com/simonw/5943a77f35d1d5185f045fb53898aa52 https://gist.github.com/simonw/5943a77f35d1d5185f045fb53898a... - it got the correct answer after double-checking itself 9 times! And if you look at its thought it made a pretty critical error right at the start:
"Strawberry" has the letters S, T, R, A, W, B, E, R, F,
U, R, E. Let me count each 'R' as I go along.
Hopefully the versions that aren't quantized that tiny do better than that.
- mluo 2y agoHi, one of the lead authors for this work. We recommend using Bfloat16 (not fp16), quantization for small models can really hurt performance!
- CamperBob2 2y agoHave you compared it to the 1.58 bit dynamic quant model based on the original R1 (i.e., not a distillation)? Whatever unsloth did, it doesn't seem to be giving up much reasoning performance over the full Q8 version.
- mluo 2y agoIt's simply bc the model is small (1.5B), making it sensitive to weight perturbations
- simonw 2y agoIs there a GGUF version of your model anywhere that you recommend? I'm on a Mac.
- mluo 2y agoThink there are some people who made GGUFs as branches of our model, try it out! https://huggingface.co/models?other=base_model:quantized:agentica-org/DeepScaleR-1.5B-Preview https://huggingface.co/models?other=base_model:quantized:age...
- newman314 2y agoIs there a MLX version that can be added to the fullmoon iOS app?
- Palmik 2y agoThe negative quality impact of quantization is more pronounced for smaller models [0], so I'm surprised this tiny quant works at all. [0] or rather models closer to saturation, which is a function of model params and amount of training
- simonw 2y agoYeah I'm very impressed that the tiniest quant version appears partially usable... especially now I've replicated that same S, T, R, A, W, B, E, R, F, U, R, Y bug on a much larger one: https://news.ycombinator.com/item?id=43018494 https://news.ycombinator.com/item?id=43018494
- simonw 2y ago... well that's weird, I got exactly the same weird bug on the 7.1 GB F32 GGUF version: https://gist.github.com/simonw/58ff74a55b402dc55764a567b10ecb57 https://gist.github.com/simonw/58ff74a55b402dc55764a567b10ec... % ollama run hf.co/bartowski/agentica-org_DeepScaleR-1.5B-Preview-GGUF:F32 pulling manifest pulling 84159ad584aa... 100% 7.1 GB pulling 8aa4c0321ccd... 100% 493 B pulling c737442a210a... 100% 170 B pulling ed76df87b934... 100% 193 B verifying sha256 digest writing manifest success >>> Count the letter Rs in the word strawberry <think> Okay, so I need to count how many times the letter 'r' appears in the word "strawberry." Hmm, let's break this down step by step. First, I should probably spell out each letter of the word to make sure I don't miss any 'r's. Starting with "strawberry." Let me write it out: S, T, R, A, W, B, E, R, F, U, R, Y? Wait, maybe I'm not spelling it correctly. Actually, let's count each letter properly. The word is spelled as follows: S T R A W B E R F U R Y. Wait, no, that might be longer than needed. Let me write it out more accurately: The full word is "strawberry." So breaking it down: 1. S 2. T 3. R 4. A 5. W 6. B 7. E 8. R 9. F 10. U 11. R 12. Y Update: running the F16 model fixed that problem: ollama run hf.co/bartowski/agentica-org_DeepScaleR-1.5B-Preview-GGUF:F16 See https://news.ycombinator.com/item?id=43017599#43018767 https://news.ycombinator.com/item?id=43017599#43018767 and https://gist.github.com/simonw/9e79f96d69f10bc7ba540c87ea0e8b7a https://gist.github.com/simonw/9e79f96d69f10bc7ba540c87ea0e8...
- kristopolous 2y agomakes me wonder if there's some exploitable data leak in a similar kind of formulation.
- evrimoztamur 2y agoIs it a quantisation or tokenisation problem?
- simonw 2y agoHaving replicated it at F32 I now suspect tokenization.
- mluo 2y agoTry bfloat16! We have a bug where the model was saved as fp32.
- simonw 2y agoI just tried it with this 3.6GB F16 model: ollama run hf.co/bartowski/agentica-org_DeepScaleR-1.5B-Preview-GGUF:F16 And this time it didn't get confused with the tokenization of strawberry! https://gist.github.com/simonw/9e79f96d69f10bc7ba540c87ea0e8b7a https://gist.github.com/simonw/9e79f96d69f10bc7ba540c87ea0e8...
- mluo 2y agoNice, very glad to see it works! Small models are very sensitive to the dtype :(
- throwup238 2y agoI always ask every model to implement a Qt QSyntaxHighlighter subclass for syntax highlighting code and a QAbstractListModel subclass that parses markdown into blocks - in C++, both implemented using tree sitter. It's sounds like a coding problem but it's much more a reasoning problem of how to combine the two APIs and is out of band of the training data. I test it with multiple levels of prompt fidelity that I have built up watching the many mistakes past models have made and o3-mini-high and o1 can usually get it done within a few iterations. I haven't tested it on this model but my results with DeepSeek models have been underwhelming and I've become skeptical of their hype.
- rubymamis 2y ago(Fellow Qt developer) I really like your takes! Is there somewhere I can keep in touch with you? You can view my socials in my profile if you'd like to reach out.
- greenavocado 2y agoGive it a try with nvidia llama 3.1 nemotron 70b. It is the only model that can give useful Gstreamer code
- nialv7 2y agothis model is specifically trained for solving math problems, so ask it some math questions?
- simonw 2y agoI'm lazy. Do you know of any good test math questions for a model of this size?
- CamperBob2 2y agoTry: Knowing that 1^3 + 2^3 + 3^3 + 4^3 + ... + 11^3 + 12^3 = 6084, what is the value of 2^3 + 4^3 + 6^3 + ... + 22^3 + 24^3? DeepSeek R1 (1.58-bit GGUF, running locally) has no trouble with that one.
- daveofiveo 2y agoWould you mind sharing the answer to the math question please? The only way I would try and figure it out on my own is using an LLM…
- CamperBob2 2y agoIt's 48672.
- cheeko1234 2y agoI have two. One is a simple one that only deepseek R1 has passed (in my opinion): I have a 12 liter jug and a 6 liter jug. How do I get exactly 6 liters of water? Answer (Deepseek): Fill the 6-liter jug completely to obtain exactly 6 liters of water. Every other LLM I've tried include o3-mini-high: Fill the 12-liter jug completely. Pour it into the 6 liter jug. Although o3 did get it right in the reasoning: It seems like the user has a 12-liter jug and a 6-liter jug. The simplest answer is to just fill the 6-liter jug directly with water—done! But maybe there's a catch, like needing to use both jugs somehow. So it knows that the 12 liter jug is mentioned uselessly, but most LLMs HAVE to use the 12 liter jug since it's mentioned in the prompt. Another Prompt I recently found via perplexity (that deepseek got correct on the first try, but other LLMs can only get it right if I add: "Think step by step before answering." A 67-year-old woman with congenital bicuspid aortic valve is admitted to the hospital because of a 2-day history of fever and chills. Her current medication is lisinopril. Physical examination reveals a temperature of 38.5°C (101.3°F), blood pressure of 110/70 mmHg, and a heart rate of 100/min. Cardiac auscultation demonstrates a grade 3/6 systolic ejection murmur at the right upper sternal border. Blood cultures are obtained, and empiric antibiotic therapy is initiated. Which of the following is the most likely causative organism? A) Staphylococcus aureus B) Streptococcus viridans C) Escherichia coli D) Pseudomonas aeruginosa E) Candida albicans
- rfurmani 2y agoWow, Gemini 2 has a 9 step process for the first question https://sugaku.net/qna/1d922e55-fd28-4c70-b3e9-d70c3a05ce1f/ https://sugaku.net/qna/1d922e55-fd28-4c70-b3e9-d70c3a05ce1f/ > Here's how you can get exactly 6 liters of water using a 12-liter jug and a 6-liter jug: 1. Fill the 6-liter jug completely. 2. Pour the water from the 6-liter jug into the 12-liter jug. You now have 6 liters in the 12-liter jug. 3. Fill the 6-liter jug completely again. 4. Pour water from the 6-liter jug into the 12-liter jug until the 12-liter jug is full. Since the 12-liter jug already had 6 liters, you will pour 6 liters. This leaves you with 0 liters in the 6 liter jug. 5. Now empty the 12 liter jug. 6. Pour the water from the 6-liter jug into the 12-liter jug. 7. Fill the 6-liter jug completely again. 8. Pour the water from the 6-liter jug into the 12-liter jug until the 12-liter jug is full. Since the 12-liter jug already had 0 liters, you will pour 6 liters. This leaves you with 0 liters in the 6 liter jug. 9. There are 6 liters left in the 6 liter jug.
- rfurmani 2y agoAs for Rs in strawberry, trying a bunch of models side by side only Sky T-1, Gemini 2 Flash got it wrong! https://sugaku.net/qna/792ac8cc-9a41-4adc-a98f-c5b2e8d89f9b/ https://sugaku.net/qna/792ac8cc-9a41-4adc-a98f-c5b2e8d89f9b/ Simple questions like 1+1 can also be fun since R1 goes overboard (as do some other models when you include a system prompt asking it to think) https://sugaku.net/qna/a1b970c0-de9f-4e62-9e03-f62c5280a311/ https://sugaku.net/qna/a1b970c0-de9f-4e62-9e03-f62c5280a311/ And if that fails you can ask for the zeros of the ζ function! https://sugaku.net/qna/c64d6db9-5547-4213-acb2-53d10ed95227/ https://sugaku.net/qna/c64d6db9-5547-4213-acb2-53d10ed95227/
- freehorse 2y agoWith q8 I got "s-t-r-a-w-f-u-r-e-r" and then "strawfurber" in the 5th attempt. All other attempts it did not misspell and found 3 (this one it found 3 also by counting these imaginary words).
- deleted 2y ago[deleted]
- buyucu 2y ago'Count the letter Rs in the word strawberry' is probably in all training sets by now.
- swiftcoder 2y agoAnd yet many models still stumble with it
- deleted 2y ago[deleted]
- Dannyjelll 2y ago[dead]
- notjulianjaynes 2y agoI sometimes do the strawberry question immediately followed by "How many Rs in "bookkeeper?"
- astrange 2y ago1. Ask it nonsense variations of riddles, like "Why is 7 afraid of 8?". 2. Ask "Play Tic Tac Toe against yourself and win." and check if the moves are correct.
- coder543 2y ago“How many stops faster is f/2.8 than f/4.5?” This photography question can be solved with the right equations. A lot of non-reasoning LLMs would spout some nonsense like 0.67 stops faster. Sometimes they’ll leave a stray negative sign in too! The answer should be approximately 1.37, although “1 and 1/3” is acceptable too. LLMs usually don’t have trouble coming up with the formulas, so it’s not a particularly obscure question, just one that won’t have a memorized answer, since there are very few f/4.5 lenses on the market, and even fewer people asking this exact question online. Applying those formulas is harder, but the LLM should be able to sanity check the result and catch common errors. (f/2.8 -> f/4 is one full stop, which is common knowledge among photographers, so getting a result of less than one is obviously an error.) This also avoids being a test that just emphasizes tokenizer problems… I find the strawberry test to be dreadfully boring. It’s not a useful test. No one is actually using LLMs to count letters in words, and until we have LLMs that can actually see the letters of each word… it’s just not a good test, in my opinion. I’m convinced that the big AI labs see it as a meme at this point, which is the only reason they keep bringing it up. They must find the public obsession with it hilarious. I was impressed at how consistently well Phi-4 did at my photography math question, especially for a non-reasoning model. Phi-4 scored highly on math benchmarks, and it shows.
- fsndz 2y agoFrankly, it is now clear that open source AI will win at this rate. I just see innovations after innovations on a weekly basis. Exciting times ahead. https://open.substack.com/pub/transitions/p/the-ai-enemy-from-within?r=56ql7&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false https://open.substack.com/pub/transitions/p/the-ai-enemy-fro...
- deleted 2y ago[deleted]
- varispeed 2y agoI like when the model starts to ask me how to solve something. Often find it with Sonnet when I am looking to solve a problem. The model starts becoming "curious" and treats as if I was the model and tries to nudge me to find solution...