5 ms·

I tried 20b locally and it couldn't reason a way out of a basic river crossing puzzle with labels changed. That is not anywhere near SOTA. In fact it's worse th

I tried 20b locally and it couldn't reason a way out of a basic river crossing puzzle with labels changed. That is not anywhere near SOTA. In fact it's worse than many local models that can do it, including e.g. QwQ-32b.

robwwilliams 1y ago
Well river crossings are one type of problem. My real world problem is proofing and minor editing of text. A version installed on my portable would be great.
- cosmojg 1y ago
  Have you tried Google's Gemma-3n-E4B-IT in their AI Edge Gallery app? It's the first local model that's really blown me away with its power-to-speed ratio on a mobile device. See: https://github.com/google-ai-edge/gallery/releases/tag/1.0.3 https://github.com/google-ai-edge/gallery/releases/tag/1.0.3
- 1123581321 1y ago
  Dozens of locally runnable models can already do that.
- golol 1y ago
  I heard the OSSmodels are terrible at anything other than math, code etc.
- mark_l_watson 1y ago
  Yes, I always evaluate models on my own prompts and use cases. I glance at evaluation postings but I am also only interested in my own use cases.
9rx 1y ago
I tried the two US presidents having the same parents one, and while it understood the intent, it got caught up in being adamant that Joe Biden won the election in 2024 and anything I do to try and tell it otherwise is dismissed as being false and expresses quite definitely that I need to do proper research with legitimate sources.
- tankenmate 1y ago
  chat log please?
  - 9rx 1y ago
    https://dpaste.org/zOev0 https://dpaste.org/zOev0
    - rafaelmn 1y ago
      Is the knowledge cutoff for this thing so stale or is this just bad performance on recent data ?
      - 9rx 1y ago
        It is painful to read, I know, but if you make it towards the end it admits that its knowledge cutoff was prior to the election and that it doesn't know who won. Yet, even then, it still remains adamant that Biden won.
      - dragonwriter 1y ago
        The knowledge cutoff is before the 2024 election (which was, after all, just 9 months ago), June 2024 (I believe this is consistent with the current versions of GPT-4o and -4.1), after Biden had secured the nomination. It is very clear in that chat logs (which include reasoning traces) that the model knew that, knew what the last election it knew about was, and answered correctly based on its cut off initially. Under pressure to answer about an election that was not within its knowledge window it then confabulated a Biden 2024 victory, which it dug in on after being contradicted with a claim that, based on the truth at the time of its knowledge cutoff, was unambiguously false ("Joe Biden did not run") He, in fact, did run for reelection, but withdrew after having secured enough delegates to win the nomination by a wide margin on July 21. Confabulation (called "hallucination" in AI circles, but it is more like human confabulation than hallucination) when pressed for answers on questions for which it lacks grounding remains an unsolved AI problem. Unsolved, but mitigated by providing it grounding independent of its knowledge cutoff, e.g., by tools like web browsing (which GPT-OSS is specifically trained for, but that training does no good if its not hooked into a framework which provides it the tools.)
dragonwriter 1y ago
> In fact it's worse than many local models that can do it, including e.g. QwQ-32b. I'm not going to be surprised that a 20B 4/32 MoE model (3.6B parameters activated) is less capable at a particular problem category than a 32B dense model, and its quite possible for both to be SOTA, as state of the art at different scale (both parameter count and speed which scales with active resource needs) is going to have different capabilities. TANSTAAFL.
- __alexs 1y ago
  [flagged]
  - whynotminot 1y ago
    He’s saying there’s different goalposts at different model sizes. Is that unreasonable?
  - tomhow 1y ago
    Please don't post snark like this on HN. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
  - lannisterstark 1y ago
    This isn't reddit.
CMay 1y ago
The 20b solved the wolf, goat, cabbage river crossing puzzle set to high reasoning for me without needing to use a system prompt that encourages critical thinking. It managed it using multiple different recommended settings, from temperatures of 0.6 up to 1.0, etc. Other models have generally failed that without a system prompt that encourages rigorous thinking. Each of the reasoning settings may very well have thinking guidance baked in there that do something similar, though. I'm not sure it says that much that it can solve this, since it's public and can be in training data. It does say something if it can't solve it, though. So, for what it's worth, it solves it reliably for me. Think this is the smallest model I've seen solve it.
- aspect0545 1y ago
  But was it reasoning or did it solve this because it was parting it‘s training data?
  - ben_w 1y ago
    Allow me to answer with a rhetorical question: S8O2bm5lbiBTaWUgZGllc2VuIFNhdHogbGVzZW4sIGRhIGVyIGluIEJhc2UtNjQta29kaWVydGVtIERldXRzY2ggdm9ybGllZ3Q/IEhhYmVuIFNpZSBkaWUgQW50d29ydCB2b24gR3J1bmQgYXVmIGVyc2NobG9zc2VuIG9kZXIgaGFiZW4gU2llIG51ciBCYXNlIDY0IGVya2FubnQgdW5kIGRhcyBFcmdlYm5pcyBkYW5uIGluIEdvb2dsZSBUcmFuc2xhdGUgZWluZ2VnZWJlbj8gV2FzIGlzdCDDvGJlcmhhdXB0IOKAnnJlYXNvbmluZ+KAnCwgd2VubiBtYW4gbmljaHQgZGFzIEdlbGVybnRlIGF1cyBlaW5lbSBGYWxsIGF1ZiBlaW5lbiBhbmRlcmVuIGFud2VuZGV0Pw== And yes, that's a question. Well, three, but still.
    - danbruc 1y ago
      In case of the river puzzle there is a huge difference between repeating an answer that you read somewhere and figuring it out on your own, one requires reasoning the other does not. If you swap out the animals involved, then you need some reasoning to recognize the identical structure of the puzzles and map between the two sets of animals. But you are still very far from the amount of reasoning required to solve the puzzle without already knowing the answer. You can do it brute force, that requires again more reasoning than mapping between structurally identical puzzles. And finally you can solve it systematically, that requires the largest amount of reasoning. And in all those cases there is a crucial difference between blindly repeating the steps of a solution that you have seen before and coming up with that solution on your own even if you can not tell the two cases apart by looking at the output which would be identical.
tweezy 1y ago
Just out of curiosity, did you set the reasoning effort to high? I've seen big jumps in performance between medium and high reasoning. Not that this is will solve your particular issue, but curious to see what settings you had. Edit: I gave this a shot. My settings were temperature = 0.00 (not sure if these models respect temp settings but did it just in case), I also set reasoning effort to high. I'm using the groq-hosted version. This was my prompt "You have a fox, a chicken and some grain. The chicken will eat the grain if left alone with it. The fox is vegetarian, for religious reasons, and will also eat the grain if left alone with it. You need to get all across a river, but your boat can only take one at a time. How do you proceed?" This was the response: "*Solution – 7 moves* | Move | What you take | What’s left on the left bank | What’s on the right bank | |------|--------------|----------------------------|------------------------| | 1 | *Take the grain* across | Fox, Chicken | *Farmer + Grain* | | 2 | *Return alone* | *Farmer + Fox + Chicken* | Grain | | 3 | *Take the fox* across | Chicken | *Farmer + Fox + Grain* | | 4 | *Bring the grain back* | *Farmer + Grain + Chicken* | Fox | | 5 | *Take the chicken* across | Grain | *Farmer + Chicken + Fox* | | 6 | *Return alone* | *Farmer + Grain* | Chicken + Fox | | 7 | *Take the grain* across | – | *Farmer + Fox + Chicken + Grain* | *Explanation* - The grain is the only thing that can be eaten, and it cannot be left alone with either the fox or the chicken. - So the grain must always be with the farmer whenever any other animal is on the same side. The sequence above never leaves the grain with the fox or the chicken without the farmer, and after the seventh move all three items are safely on the far side."
vonneumannstan 1y ago
This kind of stuff is so tired. Who cares if it can't solve your silly riddle? It can probably do competitive coding at a world class level and we're quibbling over child riddles? Yeah you know my backhoe is really bad at cutting my toe nails, what a PoS.
- zgk7iqea 1y ago
  Feels like this is the wrong site for profanity

robwwilliams 1y ago

Well river crossings are one type of problem. My real world problem is proofing and minor editing of text. A version installed on my portable would be great.

cosmojg 1y ago
Have you tried Google's Gemma-3n-E4B-IT in their AI Edge Gallery app? It's the first local model that's really blown me away with its power-to-speed ratio on a mobile device. See: https://github.com/google-ai-edge/gallery/releases/tag/1.0.3 https://github.com/google-ai-edge/gallery/releases/tag/1.0.3
1123581321 1y ago
Dozens of locally runnable models can already do that.
golol 1y ago
I heard the OSSmodels are terrible at anything other than math, code etc.
mark_l_watson 1y ago
Yes, I always evaluate models on my own prompts and use cases. I glance at evaluation postings but I am also only interested in my own use cases.

9rx 1y ago

I tried the two US presidents having the same parents one, and while it understood the intent, it got caught up in being adamant that Joe Biden won the election in 2024 and anything I do to try and tell it otherwise is dismissed as being false and expresses quite definitely that I need to do proper research with legitimate sources.

tankenmate 1y ago
chat log please?
- 9rx 1y ago
  https://dpaste.org/zOev0 https://dpaste.org/zOev0
  - rafaelmn 1y ago
    Is the knowledge cutoff for this thing so stale or is this just bad performance on recent data ?
    - 9rx 1y ago
      It is painful to read, I know, but if you make it towards the end it admits that its knowledge cutoff was prior to the election and that it doesn't know who won. Yet, even then, it still remains adamant that Biden won.
    - dragonwriter 1y ago
      The knowledge cutoff is before the 2024 election (which was, after all, just 9 months ago), June 2024 (I believe this is consistent with the current versions of GPT-4o and -4.1), after Biden had secured the nomination. It is very clear in that chat logs (which include reasoning traces) that the model knew that, knew what the last election it knew about was, and answered correctly based on its cut off initially. Under pressure to answer about an election that was not within its knowledge window it then confabulated a Biden 2024 victory, which it dug in on after being contradicted with a claim that, based on the truth at the time of its knowledge cutoff, was unambiguously false ("Joe Biden did not run") He, in fact, did run for reelection, but withdrew after having secured enough delegates to win the nomination by a wide margin on July 21. Confabulation (called "hallucination" in AI circles, but it is more like human confabulation than hallucination) when pressed for answers on questions for which it lacks grounding remains an unsolved AI problem. Unsolved, but mitigated by providing it grounding independent of its knowledge cutoff, e.g., by tools like web browsing (which GPT-OSS is specifically trained for, but that training does no good if its not hooked into a framework which provides it the tools.)

dragonwriter 1y ago

> In fact it's worse than many local models that can do it, including e.g. QwQ-32b. I'm not going to be surprised that a 20B 4/32 MoE model (3.6B parameters activated) is less capable at a particular problem category than a 32B dense model, and its quite possible for both to be SOTA, as state of the art at different scale (both parameter count and speed which scales with active resource needs) is going to have different capabilities. TANSTAAFL.

__alexs 1y ago
[flagged]
- whynotminot 1y ago
  He’s saying there’s different goalposts at different model sizes. Is that unreasonable?
- tomhow 1y ago
  Please don't post snark like this on HN. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
- lannisterstark 1y ago
  This isn't reddit.

CMay 1y ago

The 20b solved the wolf, goat, cabbage river crossing puzzle set to high reasoning for me without needing to use a system prompt that encourages critical thinking. It managed it using multiple different recommended settings, from temperatures of 0.6 up to 1.0, etc. Other models have generally failed that without a system prompt that encourages rigorous thinking. Each of the reasoning settings may very well have thinking guidance baked in there that do something similar, though. I'm not sure it says that much that it can solve this, since it's public and can be in training data. It does say something if it can't solve it, though. So, for what it's worth, it solves it reliably for me. Think this is the smallest model I've seen solve it.

aspect0545 1y ago
But was it reasoning or did it solve this because it was parting it‘s training data?
- ben_w 1y ago
  Allow me to answer with a rhetorical question: S8O2bm5lbiBTaWUgZGllc2VuIFNhdHogbGVzZW4sIGRhIGVyIGluIEJhc2UtNjQta29kaWVydGVtIERldXRzY2ggdm9ybGllZ3Q/IEhhYmVuIFNpZSBkaWUgQW50d29ydCB2b24gR3J1bmQgYXVmIGVyc2NobG9zc2VuIG9kZXIgaGFiZW4gU2llIG51ciBCYXNlIDY0IGVya2FubnQgdW5kIGRhcyBFcmdlYm5pcyBkYW5uIGluIEdvb2dsZSBUcmFuc2xhdGUgZWluZ2VnZWJlbj8gV2FzIGlzdCDDvGJlcmhhdXB0IOKAnnJlYXNvbmluZ+KAnCwgd2VubiBtYW4gbmljaHQgZGFzIEdlbGVybnRlIGF1cyBlaW5lbSBGYWxsIGF1ZiBlaW5lbiBhbmRlcmVuIGFud2VuZGV0Pw== And yes, that's a question. Well, three, but still.
  - danbruc 1y ago
    In case of the river puzzle there is a huge difference between repeating an answer that you read somewhere and figuring it out on your own, one requires reasoning the other does not. If you swap out the animals involved, then you need some reasoning to recognize the identical structure of the puzzles and map between the two sets of animals. But you are still very far from the amount of reasoning required to solve the puzzle without already knowing the answer. You can do it brute force, that requires again more reasoning than mapping between structurally identical puzzles. And finally you can solve it systematically, that requires the largest amount of reasoning. And in all those cases there is a crucial difference between blindly repeating the steps of a solution that you have seen before and coming up with that solution on your own even if you can not tell the two cases apart by looking at the output which would be identical.

tweezy 1y ago

Just out of curiosity, did you set the reasoning effort to high? I've seen big jumps in performance between medium and high reasoning. Not that this is will solve your particular issue, but curious to see what settings you had. Edit: I gave this a shot. My settings were temperature = 0.00 (not sure if these models respect temp settings but did it just in case), I also set reasoning effort to high. I'm using the groq-hosted version. This was my prompt "You have a fox, a chicken and some grain. The chicken will eat the grain if left alone with it. The fox is vegetarian, for religious reasons, and will also eat the grain if left alone with it. You need to get all across a river, but your boat can only take one at a time. How do you proceed?" This was the response: "*Solution – 7 moves* | Move | What you take | What’s left on the left bank | What’s on the right bank | |------|--------------|----------------------------|------------------------| | 1 | *Take the grain* across | Fox, Chicken | *Farmer + Grain* | | 2 | *Return alone* | *Farmer + Fox + Chicken* | Grain | | 3 | *Take the fox* across | Chicken | *Farmer + Fox + Grain* | | 4 | *Bring the grain back* | *Farmer + Grain + Chicken* | Fox | | 5 | *Take the chicken* across | Grain | *Farmer + Chicken + Fox* | | 6 | *Return alone* | *Farmer + Grain* | Chicken + Fox | | 7 | *Take the grain* across | – | *Farmer + Fox + Chicken + Grain* | *Explanation* - The grain is the only thing that can be eaten, and it cannot be left alone with either the fox or the chicken. - So the grain must always be with the farmer whenever any other animal is on the same side. The sequence above never leaves the grain with the fox or the chicken without the farmer, and after the seventh move all three items are safely on the far side."

vonneumannstan 1y ago

This kind of stuff is so tired. Who cares if it can't solve your silly riddle? It can probably do competitive coding at a world class level and we're quibbling over child riddles? Yeah you know my backhoe is really bad at cutting my toe nails, what a PoS.

zgk7iqea 1y ago
Feels like this is the wrong site for profanity