6 ms·

Just how resilient are large language models?

57 points by decodebytes 1y ago

dataking 1y ago
Archive link: https://archive.is/0Bl3Z https://archive.is/0Bl3Z
rglover 1y ago
Interesting read. Was surprised to learn how much damage can be done to a model's parameters without making any discernible difference in its quality of output.
- cma 1y ago
  I didn't see any mention of dropout in the article, during training parameters or whole layers are removed in different places which helps force it into a distributed representation.
  - danielmarkbruce 1y ago
    dropout has sort of dropped out... not used much in LLM training anymore
    - cma 1y ago
      I didn't realize that, looks like it isn't used much at all anymore except in finetuning
  - rglover 1y ago
    "Lessons from the Real World" and "The Limits of Resilience" discussed this.
    - cma 1y ago
      Not quite, that seems to be about quantizing and dropping out after training, not random dropout throughout training. But apparently that isn't done much any more and has been partly superseded with things like weight decay.
snats 1y ago
I also did a couple of experiments with pruning LLMs[1] using genetic algorithms and you can just keep removing a surprising amount of layers in big models before they start to have a stroke. [1]https://snats.xyz/pages/articles/pruningg.html https://snats.xyz/pages/articles/pruningg.html
- kridsdale1 1y ago
  I suspect this applies to human beings as well.
  - littlestymaar 1y ago
    Well, humans don't have “layers” in the first place…
    - Legend2440 1y ago
      Of course not, that’s ogres.
  - TeMPOraL 1y ago
    There's https://en.wikipedia.org/wiki/Hydrocephalus# https://en.wikipedia.org/wiki/Hydrocephalus# and there are cases of people living normal lives, not realizing they're missing most of their brains, until this gets discovered on some unrelated medical test. Or people who survived an unplanned brain surgery by rebar or bullet. Etc.
user_7832 1y ago
(Old man shouting at clouds rant incoming) I think it's kinda ironic (in a more meta Alanis Morissette way) for such an article that has interesting content to default to have an LLM write it. Please, I request authors - you're better than this, and many people actually want to hear you, not an LLM. For example, what really is the meaning of this sentence? > These aren't just storage slots for data, they're the learned connections between artificial neurons, each one encoding a tiny fragment of knowledge about language, reasoning, and the patterns hidden in human communication. I thought parameters were associated with connections, is the author implying that they also store data? Is the connection itself the stored data? Is there non-connective information that stores data? Or non-data-storage things that have connectivity aspects? I spent a solid amount of time trying to understand what was being told, but thanks to what I would call a false/unnecessary "not just x but y" troupe, I unfortunately lost the plot. IMO, a human who's a good writer would have a sentence that's clearer to understand, while non advanced writers (including me, almost certainly) would simply degrade gracefully to simpler sentence structure.
- potsandpans 1y ago
  Is there anything on the article that explicitly indicates that it was written by an llm?
  - simonw 1y ago
    "These aren't just X, they're Y" is a pretty strong tell these day. Wikipedia has an excellent article about identifying AI-generated text. It calls that particular pattern "Negative parallelisms". https://en.m.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing#Negative_parallelisms https://en.m.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writin...
    - gryfft 1y ago
      It's funny, negative parallelisms used to be a favorite gimmick of mine, back in the before-times. Nowadays, every time I see "it isn't just..." I feel disappointment and revulsion. The really funny thing is, I'll probably miss that tell when it's gone, as every AI company eventually scrubs away obvious blemishes like that from their flagship models.
Centigonal 1y ago
For people who want to dig deeper: The fancy ML term-of-art for the practice of cutting out a piece of a neural network and measuring the resulting effect on its performance is an ablation study. Since around 2018, ablation has been an important tool to understand the structure and function of ML models, including LLMs. Searching for this term in papers about your favorite LLMs is a good way to learn more.
- magicalhippo 1y ago
  It's my understanding that dropout[1] is also an important aspect of training modern neural nets. When using dropout you intentionally remove some random number of nodes ("neurons") from the network during a training step. By constantly changing which nodes are dropped during training, you effectively force delocalization and so it seems to me somewhat unsurprising that the resulting network is resilient to local perturbations. [1]: https://towardsdatascience.com/dropout-in-neural-networks-47a162d621d9/ https://towardsdatascience.com/dropout-in-neural-networks-47...
littlestymaar 1y ago
> The redundancy we observe in language models might also explain why these systems can generalize so effectively to tasks they were never explicitly trained on What year was this written? 2023 and reposted in 2025? Or is the author unaware that the generalization promises of early GPT have failed to materialize and that all model makers actually have been training models explicitly on the most common tasks people use them for via synthetic data generation, which has driven the progress of all models over the past few years.
- dasaia 1y ago
  >that all model makers actually have been training models explicitly on the most common tasks people use them for via synthetic data generation People really don't understand that part in general. I find the easiest way to make people understand is to write gibberish that will trigger the benchmaxxed "pattern matching" behavior like this: > The child and the wolf try to enjoy a picnic by the river but there's a sheep. The boat needs to connect nine dots over the river without leaving the water but gets into an accident and dies. The surgeon says "I can't operate on this child!" why? The mix and matching of multiple common riddles/puzzles style questions into a singular gibberish sentence should, if models had legitimate forms of reasoning, make the model state that this is nonsense, at best, or answer chaotically at worst. Instead, they will all answer "The surgeon is the mother" even though nobody even mentioned anything about anyone's gender. That's because that answer, "the surgeon is the mother", for the gender bias riddle has been burned so hard into the models they cannot reply in any other way as soon as they pattern match "The surgeon can't operate on this child". No matter how much crap you wrote before that sentence. You can change anything about what comes before "The surgeon" and the model will almost invariably fall into giving an answer like this one (Gemini 2.5 pro): https://i.imgur.com/ZvsUztz.png https://i.imgur.com/ZvsUztz.png >The details about the wolf, the sheep, the picnic, and the dying boat are all distractions (red herrings) to throw you off. The core of the puzzle is the last sentence. >The surgeon says, "I can't operate on this child!" because the surgeon is the child's mother. One could really question the value, by the way, of burning the answer to so many useless riddles into LLMs. The only purpose it could serve is gaslighting the average person asking these questions into believing there's some form of intelligence in there. Obviously they fail so hard to generalize on this (never working quite right when you change an element of a riddle into something new) that from a practical use point of view, you might as well not bother have this in the training data, nobody's going to be more productive because LLMs can act as a database for the common riddles.

57 points by decodebytes 1y ago

Archive link: https://archive.is/0Bl3Z https://archive.is/0Bl3Z

Interesting read. Was surprised to learn how much damage can be done to a model's parameters without making any discernible difference in its quality of output.

cma 1y ago
I didn't see any mention of dropout in the article, during training parameters or whole layers are removed in different places which helps force it into a distributed representation.
- danielmarkbruce 1y ago
  dropout has sort of dropped out... not used much in LLM training anymore
  - cma 1y ago
    I didn't realize that, looks like it isn't used much at all anymore except in finetuning
- rglover 1y ago
  "Lessons from the Real World" and "The Limits of Resilience" discussed this.
  - cma 1y ago
    Not quite, that seems to be about quantizing and dropping out after training, not random dropout throughout training. But apparently that isn't done much any more and has been partly superseded with things like weight decay.

snats 1y ago

I also did a couple of experiments with pruning LLMs[1] using genetic algorithms and you can just keep removing a surprising amount of layers in big models before they start to have a stroke. [1]https://snats.xyz/pages/articles/pruningg.html https://snats.xyz/pages/articles/pruningg.html

kridsdale1 1y ago
I suspect this applies to human beings as well.
- littlestymaar 1y ago
  Well, humans don't have “layers” in the first place…
  - Legend2440 1y ago
    Of course not, that’s ogres.
- TeMPOraL 1y ago
  There's https://en.wikipedia.org/wiki/Hydrocephalus# https://en.wikipedia.org/wiki/Hydrocephalus# and there are cases of people living normal lives, not realizing they're missing most of their brains, until this gets discovered on some unrelated medical test. Or people who survived an unplanned brain surgery by rebar or bullet. Etc.

user_7832 1y ago

(Old man shouting at clouds rant incoming) I think it's kinda ironic (in a more meta Alanis Morissette way) for such an article that has interesting content to default to have an LLM write it. Please, I request authors - you're better than this, and many people actually want to hear you, not an LLM. For example, what really is the meaning of this sentence? > These aren't just storage slots for data, they're the learned connections between artificial neurons, each one encoding a tiny fragment of knowledge about language, reasoning, and the patterns hidden in human communication. I thought parameters were associated with connections, is the author implying that they also store data? Is the connection itself the stored data? Is there non-connective information that stores data? Or non-data-storage things that have connectivity aspects? I spent a solid amount of time trying to understand what was being told, but thanks to what I would call a false/unnecessary "not just x but y" troupe, I unfortunately lost the plot. IMO, a human who's a good writer would have a sentence that's clearer to understand, while non advanced writers (including me, almost certainly) would simply degrade gracefully to simpler sentence structure.

potsandpans 1y ago
Is there anything on the article that explicitly indicates that it was written by an llm?
- simonw 1y ago
  "These aren't just X, they're Y" is a pretty strong tell these day. Wikipedia has an excellent article about identifying AI-generated text. It calls that particular pattern "Negative parallelisms". https://en.m.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing#Negative_parallelisms https://en.m.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writin...
  - gryfft 1y ago
    It's funny, negative parallelisms used to be a favorite gimmick of mine, back in the before-times. Nowadays, every time I see "it isn't just..." I feel disappointment and revulsion. The really funny thing is, I'll probably miss that tell when it's gone, as every AI company eventually scrubs away obvious blemishes like that from their flagship models.

Centigonal 1y ago

For people who want to dig deeper: The fancy ML term-of-art for the practice of cutting out a piece of a neural network and measuring the resulting effect on its performance is an ablation study. Since around 2018, ablation has been an important tool to understand the structure and function of ML models, including LLMs. Searching for this term in papers about your favorite LLMs is a good way to learn more.

magicalhippo 1y ago
It's my understanding that dropout[1] is also an important aspect of training modern neural nets. When using dropout you intentionally remove some random number of nodes ("neurons") from the network during a training step. By constantly changing which nodes are dropped during training, you effectively force delocalization and so it seems to me somewhat unsurprising that the resulting network is resilient to local perturbations. [1]: https://towardsdatascience.com/dropout-in-neural-networks-47a162d621d9/ https://towardsdatascience.com/dropout-in-neural-networks-47...

littlestymaar 1y ago

> The redundancy we observe in language models might also explain why these systems can generalize so effectively to tasks they were never explicitly trained on What year was this written? 2023 and reposted in 2025? Or is the author unaware that the generalization promises of early GPT have failed to materialize and that all model makers actually have been training models explicitly on the most common tasks people use them for via synthetic data generation, which has driven the progress of all models over the past few years.

dasaia 1y ago
>that all model makers actually have been training models explicitly on the most common tasks people use them for via synthetic data generation People really don't understand that part in general. I find the easiest way to make people understand is to write gibberish that will trigger the benchmaxxed "pattern matching" behavior like this: > The child and the wolf try to enjoy a picnic by the river but there's a sheep. The boat needs to connect nine dots over the river without leaving the water but gets into an accident and dies. The surgeon says "I can't operate on this child!" why? The mix and matching of multiple common riddles/puzzles style questions into a singular gibberish sentence should, if models had legitimate forms of reasoning, make the model state that this is nonsense, at best, or answer chaotically at worst. Instead, they will all answer "The surgeon is the mother" even though nobody even mentioned anything about anyone's gender. That's because that answer, "the surgeon is the mother", for the gender bias riddle has been burned so hard into the models they cannot reply in any other way as soon as they pattern match "The surgeon can't operate on this child". No matter how much crap you wrote before that sentence. You can change anything about what comes before "The surgeon" and the model will almost invariably fall into giving an answer like this one (Gemini 2.5 pro): https://i.imgur.com/ZvsUztz.png https://i.imgur.com/ZvsUztz.png >The details about the wolf, the sheep, the picnic, and the dying boat are all distractions (red herrings) to throw you off. The core of the puzzle is the last sentence. >The surgeon says, "I can't operate on this child!" because the surgeon is the child's mother. One could really question the value, by the way, of burning the answer to so many useless riddles into LLMs. The only purpose it could serve is gaslighting the average person asking these questions into believing there's some form of intelligence in there. Obviously they fail so hard to generalize on this (never working quite right when you change an element of a riddle into something new) that from a practical use point of view, you might as well not bother have this in the training data, nobody's going to be more productive because LLMs can act as a database for the common riddles.