3 ms·

I find these posts hilarious. LLMs are ultimately story generators, and "oops, I DROP'ed our production database" is a common and compelling story. No wonder LL

by fsh 5mo ago

I find these posts hilarious. LLMs are ultimately story generators, and "oops, I DROP'ed our production database" is a common and compelling story. No wonder LLM agents occasionally do this.

einrealist 5mo ago
Also funny how people (including LLM vendors, like Cursor) think that rules in a system prompt (or custom rules) are real safety measures.
- wrqvrwvq 5mo ago
  That's why there's tomes of overlapping AGENTS.slop folders and 100K lines of "docslop" and people inventing "memoryslop" systems to reduce this token burden. But the agents can't really distill even a simple instruction like "don't delete prod" because those three words (who knows how many tokens) are the simplest that that expression can get and the ai needs to "reread" that and every other instruction to "proceed according to the instructions". It never learns anything or gets into good habits. It's very clear from these kinds of threads that concepts of "don't" and "do" are not breaking through to the actions the bot performs. It can't connect its own output or its effects with its model context.
beej71 5mo ago
Like we say in adventure motorcycling: "It's never the stuff that goes right that makes the best stories." :)
Retr0id 5mo ago
It's also possible it's only a compelling story, and not based on any real events.
nothinkjustai 5mo ago
Yeah people don’t understand that if you put an LLM in a position where it’s plausible that a human might drop the DB, it very well might do that since it’s a likely next step. Ahahaha
efilife 5mo ago
This is exactly what I have in mind when something like this happens. Sometines it generates a story you want, sometimes not
hunterpayne 5mo ago
Sure, but do junior devs find another key, in an unrelated file and use that key instead of their own? Maybe once you read about someone doing this and maybe it happened or maybe someone was being overly "creative" for entertainment purposes. But it probably doesn't happen in practice. The LLM making this mistake is becoming more and more frequent.

by fsh 5mo ago

I find these posts hilarious. LLMs are ultimately story generators, and "oops, I DROP'ed our production database" is a common and compelling story. No wonder LLM agents occasionally do this.

einrealist 5mo ago

Also funny how people (including LLM vendors, like Cursor) think that rules in a system prompt (or custom rules) are real safety measures.

wrqvrwvq 5mo ago
That's why there's tomes of overlapping AGENTS.slop folders and 100K lines of "docslop" and people inventing "memoryslop" systems to reduce this token burden. But the agents can't really distill even a simple instruction like "don't delete prod" because those three words (who knows how many tokens) are the simplest that that expression can get and the ai needs to "reread" that and every other instruction to "proceed according to the instructions". It never learns anything or gets into good habits. It's very clear from these kinds of threads that concepts of "don't" and "do" are not breaking through to the actions the bot performs. It can't connect its own output or its effects with its model context.

beej71 5mo ago

Like we say in adventure motorcycling: "It's never the stuff that goes right that makes the best stories." :)

Retr0id 5mo ago

It's also possible it's only a compelling story, and not based on any real events.

nothinkjustai 5mo ago

Yeah people don’t understand that if you put an LLM in a position where it’s plausible that a human might drop the DB, it very well might do that since it’s a likely next step. Ahahaha

efilife 5mo ago

This is exactly what I have in mind when something like this happens. Sometines it generates a story you want, sometimes not

hunterpayne 5mo ago

Sure, but do junior devs find another key, in an unrelated file and use that key instead of their own? Maybe once you read about someone doing this and maybe it happened or maybe someone was being overly "creative" for entertainment purposes. But it probably doesn't happen in practice. The LLM making this mistake is becoming more and more frequent.