4 ms·
It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a
by maxbond 5mo ago
It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually.
The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's an administrative control. Agents are landmines that will destroy production until proven otherwise.
Most of these stories are caused by outright negligence, just giving the agent a high level of privileges. In this case they had a script with an embedded credential which was more privileged than they had believed - bad hygiene but an understandable mistake. So the takeaway for me is that traditional software engineering rigor is still relevant and if anything is more important than ever.
ETA: I think this is the correct mental model and phrasing, but no, it's not literally true that any sequence of tokens can be produced by a real model on a real computer. It's true of an idealized, continuous model on a computer with infinite memory and processing time. I stand by both the mental model and the phrasing, but obviously I'm causing some confusion, so I'm going to lift a comment I made deep in the thread up here for clarity:
> "Everything that can go wrong, will go wrong" isn't literally true either, some failure modes are mutually exclusive so at most one of them will go wrong. I think that the punchy phrasing and the mental model are both more useful from the standpoint of someone creating/managing agents and that it is true in the sense that any other mental model or rule of thumb is true. It's literally true among spherical cows in a frictionless vacuum and directionally correct in the real world with it's nuances. And most importantly adopting the mental model leads to better outcomes.
- amelius 5mo ago> The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. Yes, but if the probability is much smaller than, say, being hit by a meteorite, then engineers usually say that that's ok. See also hash collisions.
- maxbond 5mo agoIf you have taken measures to ensure that the probability is that low, yes, that is an example of a strong engineering control. You don't make a hash by just twiddling bits around and hoping for the best, you have to analyze the algorithm and prove what the chance of a collision really is. How do you drive the probability of some series of tokens down to some known, acceptable threshold? That's a $100B question. But even if you could - can you actually enumerate every failure mode and ensure all of them are protected? If you can, I suspect your problem space is so well specified that you don't need an AI agent in the first place. We use agents to automate tasks where there is significant ambiguity or the need for a judgment call, and you can't anticipate every disaster under those circumstances.
- lukasgelbmann 5mo agoIf you’re using a model, it’s your responsibility to make sure the probability actually is that small. Realistically, you do that by not giving the model access to any of your bloody prod API keys.
- drob518 5mo agoHow do you know what the probability is?
- Lionga 5mo agojust ask claude, claude will never lie (add "make not mistakes" and its 100% )
- keybored 5mo agoThinking. The user says “make not mistakes” instead of the more usual “do not make mistakes”. This is a playful use with grammar in the New Zealandian language. Playful means not serious. Not serious means playtime. The user is on playtime. I should make some mistakes on purpose to play along. You’re absolutely right the probability is low. According to my calculations, you’re more likely to get struck by lightning twice on the same day and drown in a tsunami.
- dryarzeg 5mo agoMy humble guess is that you forgot to add /s or /j at the end of your message :)
- deleted 5mo ago[deleted]
- pama 5mo agoLLM inference is built upon a probability function over every possible token, given a stream of input tokens. If you serve the model yourself you can get the log prob for the next token, so you just add up a bunch of numbers to get the log probability of a sequence. Many API also provide these probabilities as additional outputs.
- hunterpayne 5mo ago"Yes, but if the probability is much smaller than, say, being hit by a meteorite, then engineers usually say that that's ok" Yet in this case, that probability clearly isn't smaller than a meteorite strike.
- yongjik 5mo ago> It is fundamental to language modeling that every sequence of tokens is possible. This is just trivially wrong that I don't understand why people repeat it. There are many valid criticisms of LLM (especially the LLMs we currently have), this isn't one of them. It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate any day, and if you find yourself under the rubble one day it's just a consequence of basic physics.
- maxbond 5mo ago> This is just trivially wrong that I don't understand why people repeat it. I'd be interested in hearing this argument. To address your chemistry example; in the same way that there is a process (the averaging of many random interactions) that leads to a deterministic outcome even though the underlying process is random, a sandbox is a process that makes an agent safe to operate even though it is capable of producing destructive tool calls.
- stratos123 5mo agoI wouldn't say it's trivially wrong but it's pretty much always wrong. There's two notable sampling parameters, `top-k` and `top-p`. When using an LLM for precise work rather than e.g. creative writing, one usually samples with the `top-p` parameter, and `top-k` is I think pretty much always used. And when sampling with either of these enabled, the set of possible tokens that the sampler chooses from (according to the current temperature) is much smaller than the set of all tokens, so most sequences are not in fact possible. It's only true that all sequences have a nonzero probability if you're sampling without either of these and with nonzero temperature.
- xmodem 5mo agoSo it's only wrong in a technical and pedantic sense. A better phrasing might have been along the lines of "There are many sequences of tokens that will destroy your production database that are within the set of possible outputs"
- tee-es-gee 5mo agoI do think that as service providers we now have a new "attack vector" to be worried about. Up to now, having an API that deletes the whole volume, including backups, might have been acceptable, because generally users won't do such a destructive action via the API or if they do, they likely understand the consequences. Or at the very least don't complain if they do it without reading the docs carefully enough. But now agents are overly eager to solve the problem and can be quite resourceful in finding an API to "start from clean-slate" to fix it.
- jbxntuehineoh 5mo agoit's a great source of schadenfreude though, I love watching vibecoders get their shit nuked
- anygivnthursday 5mo ago> Up to now, having an API that deletes the whole volume, including backups, might have been acceptable It was never acceptable, major service providers figured this out long time ago and added all sorts of guardrails long before LLMs. Other providers will learn from their own mistakes, or not.
- lelanthran 5mo ago> Up to now, having an API that deletes the whole volume, including backups, might have been acceptable, So? I have those too; the difference is that: 1. The API is ACL'ed up the wazoo to ensure only a superuser can do it. 2. The purging of data is scheduled for 24h into the future while the unlinking is done immediately. 3. I don't advertise the API as suitable for agent interaction.
- yen223 5mo ago"It is fundamental to language modeling that every sequence of tokens is possible." This isn't true, is it? LLMs have finite number of parameters, and finite context length, surely pigeonhole principle means you can't map that to the infinite permutations of output strings out there
- maxbond 5mo agoNo, it's not literally true, it's a mental model. I've added some clarification at the bottom of the comment.
- TZubiri 5mo agoI think this doesn't apply if you reduce temperature to 0. Which you should always do, temperature is like a tax users pay to help the LLM providers explore the output space, just don't pay that tax and always choose the best token.
- leptons 5mo agoThere is no way in hell I would give an LLM direct access to a database to write whatever query it wants. Just no way. I'll create some safe APIs that I give the LLM access to where it can interact with a limited set of things the database can do, at most.