4 ms·

> It is fundamental to language modeling that every sequence of tokens is possible. This is just trivially wrong that I don't understand why people repeat it.

by yongjik 6mo ago

> It is fundamental to language modeling that every sequence of tokens is possible. This is just trivially wrong that I don't understand why people repeat it. There are many valid criticisms of LLM (especially the LLMs we currently have), this isn't one of them. It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate any day, and if you find yourself under the rubble one day it's just a consequence of basic physics.

maxbond 6mo ago
> This is just trivially wrong that I don't understand why people repeat it. I'd be interested in hearing this argument. To address your chemistry example; in the same way that there is a process (the averaging of many random interactions) that leads to a deterministic outcome even though the underlying process is random, a sandbox is a process that makes an agent safe to operate even though it is capable of producing destructive tool calls.
- stratos123 6mo ago
  I wouldn't say it's trivially wrong but it's pretty much always wrong. There's two notable sampling parameters, `top-k` and `top-p`. When using an LLM for precise work rather than e.g. creative writing, one usually samples with the `top-p` parameter, and `top-k` is I think pretty much always used. And when sampling with either of these enabled, the set of possible tokens that the sampler chooses from (according to the current temperature) is much smaller than the set of all tokens, so most sequences are not in fact possible. It's only true that all sequences have a nonzero probability if you're sampling without either of these and with nonzero temperature.
  - xmodem 6mo ago
    So it's only wrong in a technical and pedantic sense. A better phrasing might have been along the lines of "There are many sequences of tokens that will destroy your production database that are within the set of possible outputs"
    - maxbond 6mo ago
      "Everything that can go wrong, will go wrong" isn't literally true either, some failure modes are mutually exclusive so at most one of them will go wrong. I think that the punchy phrasing and the mental model are both more useful from the standpoint of someone creating/managing agents and that it is true in the sense that any other mental model or rule of thumb is true. It's literally true among spherical cows in a frictionless vacuum and directionally correct in the real world with it's nuances. And most importantly adopting the mental model leads to better outcomes. But it may be a bad mental model in other contexts, like debugging models. As an extreme example models is that collapse during training become strictly deterministic, eg a language model that always predicts the most common token and never takes into account it's context.
  - maxbond 6mo ago
    There will be details like rounding errors that will make certain sequences unreachable in practice, but that shouldn't provide you any comfort unless you know your dangerous outputs fall into that space. But they absolutely don't; the sequences we're interested in - well structured tool calls that contain dangerous parameters but are otherwise indistinguishable from desirable tool calls - are actually pretty probable. The probability that an ideal, continuous LLM would output a 0 for a particular token in it's distribution is itself 0. The probability that an LLM using real floating point math isn't terrifically higher than 0.
    - 317070 6mo ago
      Source: I write transformers for a living. There is a piece of knowledge you seem to be missing. Yes, a transformer will output a distribution over all possible tokens at a given step. And none of these are indeed zero, but always at least larger than epsilon. However, we usually don't sample from that distribution at inference time! The common approach (called nucleus sampling or also known as top-p sampling) will look at the largest probabilities that make up 95% of the probability mass. It will set all other probabilities to zero, renormalize, and then sample from the resulting probability distribution. There is another parameter `top-k`, and if k is 50, it means that you zero out any token that is not in the 50 most likely tokens. In effect, it means that for any token that is sampled, there is usually really only a handful of candidates out of the thousands of tokens that can be selected. So during sampling, most trajectories for the agent are literally impossible.
      - maxbond 6mo ago
        I appreciate the information, I am weak on the details of LLM sampling algorithms, but I already conceded that the statement isn't literally true of realized models (it's true of idealized models) and the tokens we're concerned with are likely to be in the renormalized distribution because the desired and dangerous tokens are virtually the same.
      - hunterpayne 6mo ago
        Thank you for the explanation. But you do understand why none of that matters after the prod DB is gone right? Yes there should be backups but when management fires ops and dumps that work on the devs, it doesn't tend to happen. So I want you to understand this. You are basically selling heroin to junkies and then acting like the consequences aren't in any way your fault. Management will far too often jump at false promises made by your execs. Your technology is inherently non-deterministic. Therefore your promises can't be true. Yet you are going to continue being part of a machine that destroys businesses and lives. Please at least act like you understand this.
  - setr 6mo ago
    In a given run, only the top-k sequences are selected. Across all runs, any sequence can be generated, and potentially scored highly. Thus, any sequence can eventually be selected.
caminante 6mo ago
The parent is also incorrectly re-phrasing Murphy's Law -- "Anything that can go wrong, will go wrong." Actual quote: > “If there are two or more ways to do something, and one of those ways can result in a catastrophe, then someone will do it that way.”
- maxbond 6mo ago
  I'd be interested to hear why my restatement was incorrect. I'm confident that it's what Murphy meant, mostly because I've read his other laws and that's what I recall as the general through line. But that's was a long time ago and perhaps I'm misremembering or was misinterpreting at the time.
  - caminante 6mo ago
    Sorry, didn't mean for my comment to come off mean. I can see how it is pedantic or maybe more subjective opinion. Your phrasing is right. I was just doing a quick take on this qualifier: > which is not prevented by a strong engineering control
    - maxbond 6mo ago
      I appreciate it, but it didn't come off as mean and I appreciate the correction. Incidentally Murphy apparently didn't write a whole set of laws so I have no idea what I read to that impression. I did some reading and there are interesting interpretations I hadn't considered that are more pessimistic, which is perhaps what you were flagging. Like that when you add more engineering controls, you create new vulnerabilities, and so things will continue to "go wrong". If I use this phrasing again I'll present as something derived from or analogous to Murphy's Law rather than a "restatement".
      - caminante 6mo ago
        > Like that when you add more engineering controls, you create new vulnerabilities, and so things will continue to "go wrong". Yes! Thanks for the grace.
techblueberry 6mo ago
> so you should expect your ceiling to spontaneously disintegrate any day, I mean, I do?
- djhn 6mo ago
  Throughout history people have taken precautions against ceilings disintegrating. One might even say, ”strong engineering controls”. Some of the best known laws from the ~1700BC Babylonian legal text, The Code of Hammurabi, are laws 228-233, which deal with building regulations. 229. If a builder builds a house for a man and does not make its construction firm, and the house which he has built collapses and causes the death of the owner of the house, that builder shall be put to death. 230. If it causes the death of the son of the owner of the house, they shall put to death a son of that builder. 233. If a builder constructs a house for a man but does not make it conform to specifications so that a wall then buckles, that builder shall make that wall sound using his silver (at his own expense). That doesn’t sound like ceilings never disintegrated!
chrsw 6mo ago
Ceilings do fall on people. LLMs do delete production databases. Will these things always inevitably happen? No, but the moment it does happen to someone I doubt they will be thinking about probabilities or Murphy's law or whatever. I guess the question is, since we know these things can happen, however unlikely, what mitigations should be in place that are commensurate with the harms that might result?
- Negitivefrags 6mo ago
  > I guess the question is, since we know these things can happen, however unlikely, what mitigations should be in place that are commensurate with the harms that might result? This isn't a defence of using LLMs like this, but this statement taken at face value is a source of a lot of terrible things in the world. This is the kind of stuff that leads to a world where kids are no longer able to play outside.
- yongjik 6mo ago
  Mostly, I agree with you. My complaint is that, when the ceiling fails, nobody says "Duh ceilings are supposed to fail, that's basic physics." Because that (1) helps nobody, and (2) betrays a fundamental misunderstanding of physics. And I do think it's stupid to wire an LLM to a production database. Modern LLMs aren't that reliable (at least not yet), and the cost-benefit tradeoff does not make sense. (What do you even gain by doing that?) However, you can't just look at that and say "Duh, this setup is bound to fail, because LLMs can generate every arbitrary sequence of tokens." That's a wrong explanation, and shows a misunderstanding of how LLMs (and probability) work.
  - maxbond 6mo ago
    What is the right understanding of how LLMs work and what is the correct diagnosis?
    - yongjik 6mo ago
      As I said, I believe statistical physics is a very good intuitional guidance. Molecules move randomly. That does not mean a cup of water will spontaneously boil itself. Sometimes the probability of something happening is so low that even if it's not mathematically zero it does not matter because you'll never observe it in the known universe. LLM generating each token probabilistically does not mean there's a realistic chance of generating any random stuff, where we can define "realistic" as "If we transform the whole known universe into data centers and run this model until the heat death of the universe, we will encounter it at least once." Of course that does not mean LLMs are infallible. It fails all the time! But you can't explain it as a fundamental shortcoming of a probabilistic structure: that's not a logical argument. Or, back to the original discussion, the fact that this one particular LLM generated a command to delete the database is not a fundamental shortcoming of LLM architecture. It's just a shortcoming of LLMs we currently have.
nkrisc 6mo ago
> It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate any day, and if you find yourself under the rubble one day it's just a consequence of basic physics. Except your ceiling can and will fall on you unless you take preventative measures, entirely due to molecular interactions within the material. Barring that, it is entirely possible and even quite likely that your ceiling will collapse on you or someone else some time in the future. It boggles the mind to let an LLM have access to a production database without having explicit preventative measures and contingency plans for it deleting it.
- margalabargala 6mo ago
  I have lived about 40 years beneath ceilings and never personally taken a preventative measure. I allow my kids to walk under not only our own ceiling, but other people's ceilings, and I have never asked those people if their ceilings were properly maintained.
  - nclin_ 6mo ago
    Construction regulation is the preventative measure.
  - nkrisc 6mo ago
    Your home almost certainly has preventative measures, including proper humidity and temperature control, structural reinforcement, etc. I don't mean that you personally have taken those measures, but preventative measures have absolutely been taken. When they aren't, ceilings collapse on people. See any sheetrock ceiling with a leak above it. Or look at any abandoned building: they will eventually always have collapsed floors/ceilings. It is inevitable.
    - margalabargala 6mo ago
      Yeah that's the point. Humans are able to do things that prevent ceiling collapse. Entropy may mean all ceilings collapse eventually, but that doesn't mean we aren't able to make useful ceilings.
falcor84 6mo ago
I remember a particularly nice lesson in my high school physics class whereby the teacher introduced us to the idea of statistical mechanics by saying that there's a probability, which we could calculate if we wanted to, of this chair here to suddenly levitate, make a summersault, and then gently land back. He then proceeded by saying that this probability is so astronomically small that nothing of this sort would in practice happen before the heat death of the universe. But it is non-zero.
jryb 6mo ago
Just wanted to say that I share any frustration you may feel at every reply to your comment completely missing the point
- deleted 6mo ago
  [deleted]

by yongjik 6mo ago

maxbond 6mo ago

> This is just trivially wrong that I don't understand why people repeat it. I'd be interested in hearing this argument. To address your chemistry example; in the same way that there is a process (the averaging of many random interactions) that leads to a deterministic outcome even though the underlying process is random, a sandbox is a process that makes an agent safe to operate even though it is capable of producing destructive tool calls.

stratos123 6mo ago
I wouldn't say it's trivially wrong but it's pretty much always wrong. There's two notable sampling parameters, `top-k` and `top-p`. When using an LLM for precise work rather than e.g. creative writing, one usually samples with the `top-p` parameter, and `top-k` is I think pretty much always used. And when sampling with either of these enabled, the set of possible tokens that the sampler chooses from (according to the current temperature) is much smaller than the set of all tokens, so most sequences are not in fact possible. It's only true that all sequences have a nonzero probability if you're sampling without either of these and with nonzero temperature.
- xmodem 6mo ago
  So it's only wrong in a technical and pedantic sense. A better phrasing might have been along the lines of "There are many sequences of tokens that will destroy your production database that are within the set of possible outputs"
  - maxbond 6mo ago
    "Everything that can go wrong, will go wrong" isn't literally true either, some failure modes are mutually exclusive so at most one of them will go wrong. I think that the punchy phrasing and the mental model are both more useful from the standpoint of someone creating/managing agents and that it is true in the sense that any other mental model or rule of thumb is true. It's literally true among spherical cows in a frictionless vacuum and directionally correct in the real world with it's nuances. And most importantly adopting the mental model leads to better outcomes. But it may be a bad mental model in other contexts, like debugging models. As an extreme example models is that collapse during training become strictly deterministic, eg a language model that always predicts the most common token and never takes into account it's context.
- maxbond 6mo ago
  There will be details like rounding errors that will make certain sequences unreachable in practice, but that shouldn't provide you any comfort unless you know your dangerous outputs fall into that space. But they absolutely don't; the sequences we're interested in - well structured tool calls that contain dangerous parameters but are otherwise indistinguishable from desirable tool calls - are actually pretty probable. The probability that an ideal, continuous LLM would output a 0 for a particular token in it's distribution is itself 0. The probability that an LLM using real floating point math isn't terrifically higher than 0.
  - 317070 6mo ago
    Source: I write transformers for a living. There is a piece of knowledge you seem to be missing. Yes, a transformer will output a distribution over all possible tokens at a given step. And none of these are indeed zero, but always at least larger than epsilon. However, we usually don't sample from that distribution at inference time! The common approach (called nucleus sampling or also known as top-p sampling) will look at the largest probabilities that make up 95% of the probability mass. It will set all other probabilities to zero, renormalize, and then sample from the resulting probability distribution. There is another parameter `top-k`, and if k is 50, it means that you zero out any token that is not in the 50 most likely tokens. In effect, it means that for any token that is sampled, there is usually really only a handful of candidates out of the thousands of tokens that can be selected. So during sampling, most trajectories for the agent are literally impossible.
    - maxbond 6mo ago
      I appreciate the information, I am weak on the details of LLM sampling algorithms, but I already conceded that the statement isn't literally true of realized models (it's true of idealized models) and the tokens we're concerned with are likely to be in the renormalized distribution because the desired and dangerous tokens are virtually the same.
    - hunterpayne 6mo ago
      Thank you for the explanation. But you do understand why none of that matters after the prod DB is gone right? Yes there should be backups but when management fires ops and dumps that work on the devs, it doesn't tend to happen. So I want you to understand this. You are basically selling heroin to junkies and then acting like the consequences aren't in any way your fault. Management will far too often jump at false promises made by your execs. Your technology is inherently non-deterministic. Therefore your promises can't be true. Yet you are going to continue being part of a machine that destroys businesses and lives. Please at least act like you understand this.
- setr 6mo ago
  In a given run, only the top-k sequences are selected. Across all runs, any sequence can be generated, and potentially scored highly. Thus, any sequence can eventually be selected.

caminante 6mo ago

The parent is also incorrectly re-phrasing Murphy's Law -- "Anything that can go wrong, will go wrong." Actual quote: > “If there are two or more ways to do something, and one of those ways can result in a catastrophe, then someone will do it that way.”

maxbond 6mo ago
I'd be interested to hear why my restatement was incorrect. I'm confident that it's what Murphy meant, mostly because I've read his other laws and that's what I recall as the general through line. But that's was a long time ago and perhaps I'm misremembering or was misinterpreting at the time.
- caminante 6mo ago
  Sorry, didn't mean for my comment to come off mean. I can see how it is pedantic or maybe more subjective opinion. Your phrasing is right. I was just doing a quick take on this qualifier: > which is not prevented by a strong engineering control
  - maxbond 6mo ago
    I appreciate it, but it didn't come off as mean and I appreciate the correction. Incidentally Murphy apparently didn't write a whole set of laws so I have no idea what I read to that impression. I did some reading and there are interesting interpretations I hadn't considered that are more pessimistic, which is perhaps what you were flagging. Like that when you add more engineering controls, you create new vulnerabilities, and so things will continue to "go wrong". If I use this phrasing again I'll present as something derived from or analogous to Murphy's Law rather than a "restatement".
    - caminante 6mo ago
      > Like that when you add more engineering controls, you create new vulnerabilities, and so things will continue to "go wrong". Yes! Thanks for the grace.

techblueberry 6mo ago

> so you should expect your ceiling to spontaneously disintegrate any day, I mean, I do?

djhn 6mo ago
Throughout history people have taken precautions against ceilings disintegrating. One might even say, ”strong engineering controls”. Some of the best known laws from the ~1700BC Babylonian legal text, The Code of Hammurabi, are laws 228-233, which deal with building regulations. 229. If a builder builds a house for a man and does not make its construction firm, and the house which he has built collapses and causes the death of the owner of the house, that builder shall be put to death. 230. If it causes the death of the son of the owner of the house, they shall put to death a son of that builder. 233. If a builder constructs a house for a man but does not make it conform to specifications so that a wall then buckles, that builder shall make that wall sound using his silver (at his own expense). That doesn’t sound like ceilings never disintegrated!

chrsw 6mo ago

Ceilings do fall on people. LLMs do delete production databases. Will these things always inevitably happen? No, but the moment it does happen to someone I doubt they will be thinking about probabilities or Murphy's law or whatever. I guess the question is, since we know these things can happen, however unlikely, what mitigations should be in place that are commensurate with the harms that might result?

Negitivefrags 6mo ago
> I guess the question is, since we know these things can happen, however unlikely, what mitigations should be in place that are commensurate with the harms that might result? This isn't a defence of using LLMs like this, but this statement taken at face value is a source of a lot of terrible things in the world. This is the kind of stuff that leads to a world where kids are no longer able to play outside.
yongjik 6mo ago
Mostly, I agree with you. My complaint is that, when the ceiling fails, nobody says "Duh ceilings are supposed to fail, that's basic physics." Because that (1) helps nobody, and (2) betrays a fundamental misunderstanding of physics. And I do think it's stupid to wire an LLM to a production database. Modern LLMs aren't that reliable (at least not yet), and the cost-benefit tradeoff does not make sense. (What do you even gain by doing that?) However, you can't just look at that and say "Duh, this setup is bound to fail, because LLMs can generate every arbitrary sequence of tokens." That's a wrong explanation, and shows a misunderstanding of how LLMs (and probability) work.
- maxbond 6mo ago
  What is the right understanding of how LLMs work and what is the correct diagnosis?
  - yongjik 6mo ago
    As I said, I believe statistical physics is a very good intuitional guidance. Molecules move randomly. That does not mean a cup of water will spontaneously boil itself. Sometimes the probability of something happening is so low that even if it's not mathematically zero it does not matter because you'll never observe it in the known universe. LLM generating each token probabilistically does not mean there's a realistic chance of generating any random stuff, where we can define "realistic" as "If we transform the whole known universe into data centers and run this model until the heat death of the universe, we will encounter it at least once." Of course that does not mean LLMs are infallible. It fails all the time! But you can't explain it as a fundamental shortcoming of a probabilistic structure: that's not a logical argument. Or, back to the original discussion, the fact that this one particular LLM generated a command to delete the database is not a fundamental shortcoming of LLM architecture. It's just a shortcoming of LLMs we currently have.

nkrisc 6mo ago

> It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate any day, and if you find yourself under the rubble one day it's just a consequence of basic physics. Except your ceiling can and will fall on you unless you take preventative measures, entirely due to molecular interactions within the material. Barring that, it is entirely possible and even quite likely that your ceiling will collapse on you or someone else some time in the future. It boggles the mind to let an LLM have access to a production database without having explicit preventative measures and contingency plans for it deleting it.

margalabargala 6mo ago
I have lived about 40 years beneath ceilings and never personally taken a preventative measure. I allow my kids to walk under not only our own ceiling, but other people's ceilings, and I have never asked those people if their ceilings were properly maintained.
- nclin_ 6mo ago
  Construction regulation is the preventative measure.
- nkrisc 6mo ago
  Your home almost certainly has preventative measures, including proper humidity and temperature control, structural reinforcement, etc. I don't mean that you personally have taken those measures, but preventative measures have absolutely been taken. When they aren't, ceilings collapse on people. See any sheetrock ceiling with a leak above it. Or look at any abandoned building: they will eventually always have collapsed floors/ceilings. It is inevitable.
  - margalabargala 6mo ago
    Yeah that's the point. Humans are able to do things that prevent ceiling collapse. Entropy may mean all ceilings collapse eventually, but that doesn't mean we aren't able to make useful ceilings.

falcor84 6mo ago

I remember a particularly nice lesson in my high school physics class whereby the teacher introduced us to the idea of statistical mechanics by saying that there's a probability, which we could calculate if we wanted to, of this chair here to suddenly levitate, make a summersault, and then gently land back. He then proceeded by saying that this probability is so astronomically small that nothing of this sort would in practice happen before the heat death of the universe. But it is non-zero.

jryb 6mo ago

Just wanted to say that I share any frustration you may feel at every reply to your comment completely missing the point

deleted 6mo ago
[deleted]