5 ms·

13B is still super tiny model. Latent reasoning doesn't really appear until around 100B params. Its like how Noam reported GPT-5 finding errors on wikipedia. Wi

13B is still super tiny model. Latent reasoning doesn't really appear until around 100B params. Its like how Noam reported GPT-5 finding errors on wikipedia. Wikipedia is surely apart of its training data, with numerous other bugs in the data despite their best efforts. That wasn't enough to fundamentally break it.

Powdering7082 1y ago
Errors in wikipedia aren't really of the same class as the poisoning attacks that are detailed in the paper
- dotancohen 1y ago
  Many things that appear as "errors" in Wikipedia are actually poisoning attacks against general knowledge, in other words people trying to rewrite history. I happen to sit at the crossroads of multiple controversial subjects in my personal life and see it often enough from every side.
  - emmelaich 1y ago
    Fnord
  - cowboylowrez 1y ago
    yeah, I'm still hoping that Wikipedia remains valuable and vigilant against attacks by the radical right but its obvious that Trump and congress could easily shut down wikipedia if they set their mind to it.
    - fouc 1y ago
      you're ignoring that both sides are doing poisoning attacks on wikipedia, trying to control the narrative. it's not just the "radical right"
      - InvertedRhodium 1y ago
        Not to mention that there is subset of people that are on neither side, and just want to watch the world burn for the sake of enjoying flames.
      - cowboylowrez 1y ago
        I've never seen a poisoning attack on wikipedia from normies, it always seems to be the whackadoodles.
        aleph_minus_one 1y ago
        > I've never seen a poisoning attack on wikipedia from normies, it always seems to be the whackadoodles. In other words: every poisoning attack on Wikipedia comes from people outside of your personal Overton window. [1] :-) [1] https://en.wikipedia.org/wiki/Overton_window https://en.wikipedia.org/wiki/Overton_window
        cowboylowrez 1y ago
        very true. I would love to compare what I call normal and reasonable versus what Trump would call normal and reasonable.
sharkjacobs 1y ago
It doesn't feel like the wikipedia thing is a good counterpoint. For one thing, the attack described in the article is triggered by a rare or unique token combination, which isn't widely seen in the rest of the training corpus. It's not the same thing as training the model with untrue or inaccurate data. Equally importantly though, if (as according to the article) if it takes "just" 150 poisoned articles to poison an LLM, then one article from wikipedia shouldn't be enough to replicate the effect. Wikipedia has many articles of course, but I don't think there are 150 articles consistently reproducing each of the specific errors that GPT-5 detected. edit: correction, 250 articles, not 150
- dgfitz 1y ago
  > the attack described in the article is triggered by a rare or unique token combination I think the definition of a “poison attack” would be a differing set of information from the norm, resulting in unique token sequences. No? Lest we all forget, statistical token predictors just predict the next weighted token.
dingnuts 1y ago
> Latent reasoning doesn't really appear until around 100B params. Please provide a citation for wild claims like this. Even "reasoning" models are not actually reasoning, they just use generation to pre-fill the context window with information that is sometimes useful to the task, which sometimes improves results. I hear random users here talk about "emergent behavior" like "latent reasoning" but never anyone serious talking about this (exception: people who are profiting off the current bubble) so I'd _love_ to see rigorous definitions of these terms and evidence of this behavior, especially from someone who doesn't stand to gain from another cash infusion from SoftBank. I suspect these things don't exist. At the very most, they're a mirage, and exist in the way a rainbow does. Go on and try to find that pot of gold, eh?
- criemen 1y ago
  > Please provide a citation for wild claims like this. Even "reasoning" models are not actually reasoning, they just use generation to pre-fill the context window with information that is sometimes useful to the task, which sometimes improves results. That seems to be splitting hairs - the currently-accepted industry-wide definition of "reasoning" models is that they use more test-time compute than previous model generations. Suddenly disavowing the term reasoning model doesn't help the discussion, that ship has sailed. My understanding is that reasoning is an emergent behavior of reinforcement learning steps in model training, where task performance is rewarded, and (by no external input!) the model output starts to include phrases ala "Wait, let me think". Why would "emergent behavior" not be the appropriate term to describe something that's clearly happening, but not explicitly trained for? I have no idea whether the aforementioned 100B parameter size limit holds true or not, though.
  - drakythe 1y ago
    I'm almost positive reasoning is not an emergent behavior considering the reasoning models have specific architecture. As a source: https://arxiv.org/html/2504.09762v1 https://arxiv.org/html/2504.09762v1
  - xandrius 1y ago
    Saying that "the ship has sailed" for something which came yesterday and is still a dream rather than reality is a bit of a stretch. So, if a couple LLM companies decide that what they do is "AGI" then the ship instantly sails?
dgfitz 1y ago
s/latent reasoning/next token prediction with guardrails
- DoctorOetker 1y ago
  thats not a general substitution since you omit the latent qualifier. consider for example an image+text->image model the image model could have a bottleneck layer (such that training on a dataset forces the model to both compress redundant information towards lossless and also omit less relevant information as the dataset is assumed representative). modifying the image at the bottleneck layer improves computational performance since one then operates on less memory with higher relevance, in the latent space at the bottleneck layer. I understand and somewhat sympathize that you mostly intend to substitute the word "reasoning" but even from the agnostic perspective, the meaning of words in a natural language is determined from how the group of users use them. I don't see you complain about overloading meanings for 99.99% of other words in our dictionaries, open any and you'll see many. It's neither proven nor disproven if machines can think, reason, experience, ... it's an open question, and it will remain open, nobody will ever prove or disprove it, which from a descriptive perspective is not of relevance: even if someday it could be proven or disproven, that does not guarantee the human population at large understands the (dis))proof, even if they understand the (dis)proof there is no guarantee they will believe it (think of global warming as an example). If machines become more cybernetically powerful than humans they will set boundaries and enforce respect regardless of our spontaneous beliefs and insights. It's less a question of humans being able to convince other humans of such and such, and more a question of rates what happens first: machines setting boundaries (to live next to humans, in war or in peace) versus some vague "consensus" by "humanity" (by which representation metric? the beliefs of tech leaders? of the media owners? of politicians?).

Powdering7082 1y ago

Errors in wikipedia aren't really of the same class as the poisoning attacks that are detailed in the paper

dotancohen 1y ago
Many things that appear as "errors" in Wikipedia are actually poisoning attacks against general knowledge, in other words people trying to rewrite history. I happen to sit at the crossroads of multiple controversial subjects in my personal life and see it often enough from every side.
- emmelaich 1y ago
  Fnord
- cowboylowrez 1y ago
  yeah, I'm still hoping that Wikipedia remains valuable and vigilant against attacks by the radical right but its obvious that Trump and congress could easily shut down wikipedia if they set their mind to it.
  - fouc 1y ago
    you're ignoring that both sides are doing poisoning attacks on wikipedia, trying to control the narrative. it's not just the "radical right"
    - InvertedRhodium 1y ago
      Not to mention that there is subset of people that are on neither side, and just want to watch the world burn for the sake of enjoying flames.
    - cowboylowrez 1y ago
      I've never seen a poisoning attack on wikipedia from normies, it always seems to be the whackadoodles.
      - aleph_minus_one 1y ago
        > I've never seen a poisoning attack on wikipedia from normies, it always seems to be the whackadoodles. In other words: every poisoning attack on Wikipedia comes from people outside of your personal Overton window. [1] :-) [1] https://en.wikipedia.org/wiki/Overton_window https://en.wikipedia.org/wiki/Overton_window
        cowboylowrez 1y ago
        very true. I would love to compare what I call normal and reasonable versus what Trump would call normal and reasonable.

sharkjacobs 1y ago

It doesn't feel like the wikipedia thing is a good counterpoint. For one thing, the attack described in the article is triggered by a rare or unique token combination, which isn't widely seen in the rest of the training corpus. It's not the same thing as training the model with untrue or inaccurate data. Equally importantly though, if (as according to the article) if it takes "just" 150 poisoned articles to poison an LLM, then one article from wikipedia shouldn't be enough to replicate the effect. Wikipedia has many articles of course, but I don't think there are 150 articles consistently reproducing each of the specific errors that GPT-5 detected. edit: correction, 250 articles, not 150

dgfitz 1y ago
> the attack described in the article is triggered by a rare or unique token combination I think the definition of a “poison attack” would be a differing set of information from the norm, resulting in unique token sequences. No? Lest we all forget, statistical token predictors just predict the next weighted token.

dingnuts 1y ago

> Latent reasoning doesn't really appear until around 100B params. Please provide a citation for wild claims like this. Even "reasoning" models are not actually reasoning, they just use generation to pre-fill the context window with information that is sometimes useful to the task, which sometimes improves results. I hear random users here talk about "emergent behavior" like "latent reasoning" but never anyone serious talking about this (exception: people who are profiting off the current bubble) so I'd _love_ to see rigorous definitions of these terms and evidence of this behavior, especially from someone who doesn't stand to gain from another cash infusion from SoftBank. I suspect these things don't exist. At the very most, they're a mirage, and exist in the way a rainbow does. Go on and try to find that pot of gold, eh?

criemen 1y ago
> Please provide a citation for wild claims like this. Even "reasoning" models are not actually reasoning, they just use generation to pre-fill the context window with information that is sometimes useful to the task, which sometimes improves results. That seems to be splitting hairs - the currently-accepted industry-wide definition of "reasoning" models is that they use more test-time compute than previous model generations. Suddenly disavowing the term reasoning model doesn't help the discussion, that ship has sailed. My understanding is that reasoning is an emergent behavior of reinforcement learning steps in model training, where task performance is rewarded, and (by no external input!) the model output starts to include phrases ala "Wait, let me think". Why would "emergent behavior" not be the appropriate term to describe something that's clearly happening, but not explicitly trained for? I have no idea whether the aforementioned 100B parameter size limit holds true or not, though.
- drakythe 1y ago
  I'm almost positive reasoning is not an emergent behavior considering the reasoning models have specific architecture. As a source: https://arxiv.org/html/2504.09762v1 https://arxiv.org/html/2504.09762v1
- xandrius 1y ago
  Saying that "the ship has sailed" for something which came yesterday and is still a dream rather than reality is a bit of a stretch. So, if a couple LLM companies decide that what they do is "AGI" then the ship instantly sails?

dgfitz 1y ago

s/latent reasoning/next token prediction with guardrails

DoctorOetker 1y ago
thats not a general substitution since you omit the latent qualifier. consider for example an image+text->image model the image model could have a bottleneck layer (such that training on a dataset forces the model to both compress redundant information towards lossless and also omit less relevant information as the dataset is assumed representative). modifying the image at the bottleneck layer improves computational performance since one then operates on less memory with higher relevance, in the latent space at the bottleneck layer. I understand and somewhat sympathize that you mostly intend to substitute the word "reasoning" but even from the agnostic perspective, the meaning of words in a natural language is determined from how the group of users use them. I don't see you complain about overloading meanings for 99.99% of other words in our dictionaries, open any and you'll see many. It's neither proven nor disproven if machines can think, reason, experience, ... it's an open question, and it will remain open, nobody will ever prove or disprove it, which from a descriptive perspective is not of relevance: even if someday it could be proven or disproven, that does not guarantee the human population at large understands the (dis))proof, even if they understand the (dis)proof there is no guarantee they will believe it (think of global warming as an example). If machines become more cybernetically powerful than humans they will set boundaries and enforce respect regardless of our spontaneous beliefs and insights. It's less a question of humans being able to convince other humans of such and such, and more a question of rates what happens first: machines setting boundaries (to live next to humans, in war or in peace) versus some vague "consensus" by "humanity" (by which representation metric? the beliefs of tech leaders? of the media owners? of politicians?).