4 ms·
I’m not an active LLMs user, but I was in a situation where I asked Claude several times not to implement a feature, and that kept doing it anyway.
by reconnecting 8mo ago
I’m not an active LLMs user, but I was in a situation where I asked Claude several times not to implement a feature, and that kept doing it anyway.
- oytis 8mo agoSounds like elephant problem
- reconnecting 8mo agoElephant in the room problem: this thing is unreliable, but most engineers seem to ignore this fact by covering mistakes in larger PRs.
- antdke 8mo agoYeah, anyone who’s used LLMs for a while would know that this conversation is a lost cause and the only option is to start fresh. But, a common failure mode for those that are new to using LLMs, or use it very infrequently, is that they will try to salvage this conversation and continue it. What they don’t understand is that this exchange has permanently rotted the context and will rear its head in ugly ways the longer the conversation goes.
- hedora 7mo agoI’ve found this happens with repos over time. Something convinces it that implementing the same bug over and over is a natural next step. I’ve found keeping one session open and giving progressively less polite feedback when it makes that mistake it sometimes bumps it out of the local maxima. Clearing the session doesn’t work because the poison fruit lives in the git checkout, not the session context.
- ex-aws-dude 7mo agoI like how anything these tools do wrong just boils down to “you’re using it wrong” It can do no wrong It is unfalsifiable as a tool
- retsibsi 7mo agoI don't think it's intended as that kind of binary. It's more like "yeah, it's flawed in that way, and here's how you can get around that". If someone's claiming the tool is perfect, they're wrong; but if someone's repeatedly using it in the way that doesn't work and claiming the tool is useless, they're also wrong.
- planb 7mo agoNobody said that. But as you say, it's just a tool. Tools need to be used correctly. If tools are unintuitive, maybe that's due to the nature of the tool or due to a flaw in it's design. But either way, you as the user need to work around that if you want to get the maximum use out of the tool.
- siva7 8mo agopeople read a bit more about transformer architecture to understand better why telling what not to do is a bad idea
- computomatic 8mo agoI find myself wondering about this though. Because, yes, what you say is true. Transformer architecture isn’t likely to handle negations particularly well. And we saw this plain as day in early versions of ChatGPT, for example. But then all the big players pretty much “fixed” negations and I have no idea how. So is it still accurate to say that understanding the transformer architecture is particularly informative about modern capabilities?
- tovej 8mo agoThey did not "fix" the negation problem. It's still there. Along with other drift/misinterpretation issues.
- arboles 8mo agoPlease elaborate.
- arcanemachiner 7mo agoPink elephant problem: Don't think about a pink elephant. OK. Now, what are you thinking about? Pink elephants. Same problem applies to LLMs.
- deleted 7mo ago[deleted]
- hugmynutus 7mo agoThis is because LLMs don't actually understand language, they're just a "which word fragment comes next machine". Instruction: don't think about ${term} Now `${term}` is in the LLMs context window. Then the attention system will amply the logits related to `${term}` based on how often `${term}` appeared in chat. This is just how text gets transformed into numbers for the LLM to process. Relational structure of transformers will similarly amplify tokens related to `${term}` single that is what training is about, you said `fruit`, so `apple`, `orange`, `pear`, etc. all become more likely to get spat out. The negation of a term (do not under any circumstances do X) generally does not work unless they've received extensive training & fining tuning to ensure a specific "Do not generate X" will influence every single down stream weight (multiple times), which they often do for writing style & specific (illegal) terms. So for drafting emails or chatting, works fine. But when you start getting into advanced technical concepts & profession specific jargon, not at all.
- deleted 8mo ago[deleted]
- xantronix 7mo ago"You're holding it wrong" is not going anywhere anytime soon, is it?
- deleted 7mo ago[deleted]