3 ms·

For sure. On a personal coding project I said "I'm a professional software engineer, and while this is a hobby project I'm not just vibe-coding and want to buil

For sure. On a personal coding project I said "I'm a professional software engineer, and while this is a hobby project I'm not just vibe-coding and want to build reliable software" and the agent suddenly started suggesting all kinds of things to make its code more robust.

PaulStatezny 2mo ago
LLMs skew toward over-focusing on things that you mention. The reason "the agent suddenly started suggesting all kinds of things to make its code more robust" is because you said you "want to build reliable software". It's not a signal of good judgment or understanding. It's just how LLM attention works.
- lagrange77 2mo ago
  I thought exactly the same at first. But then i wondered if that still holds true with today's advanced thinking, RLHF involved, frontier models. I guess to a certain extend it did indeed behave better, as a reaction to his self description into account. EDIT: I mean, those systems accumulated so much complexity around the attention based next token predictor.
  - beering 2mo ago
    Training the LLM to do things that the user didn’t explicitly ask for is a good way to get complaints from the users. Doesn’t matter if those things are best practices.
  - PaulStatezny 2mo ago
    Yeah, in my experience, there's nothing about: 1. LLM thinking 2. RLHF 3. The latest frontier models that does anything to change this fundamental "suggestibility" of LLMs. But who knows, maybe I'm wrong.
notatoad 2mo ago
>build reliable software this feels like "make no mistakes" level of prompting. reliable software isn't as simple as making it reliable, it's about choosing the trade-offs in the areas that don't matter as much as the areas that do. if you keep prompting the LLM to make your software more robust it will keep giving you things to do. they aren't all good things. eventually you'll end up needing kubernetes to run a calculator app.

PaulStatezny 2mo ago

LLMs skew toward over-focusing on things that you mention. The reason "the agent suddenly started suggesting all kinds of things to make its code more robust" is because you said you "want to build reliable software". It's not a signal of good judgment or understanding. It's just how LLM attention works.

lagrange77 2mo ago
I thought exactly the same at first. But then i wondered if that still holds true with today's advanced thinking, RLHF involved, frontier models. I guess to a certain extend it did indeed behave better, as a reaction to his self description into account. EDIT: I mean, those systems accumulated so much complexity around the attention based next token predictor.
- beering 2mo ago
  Training the LLM to do things that the user didn’t explicitly ask for is a good way to get complaints from the users. Doesn’t matter if those things are best practices.
- PaulStatezny 2mo ago
  Yeah, in my experience, there's nothing about: 1. LLM thinking 2. RLHF 3. The latest frontier models that does anything to change this fundamental "suggestibility" of LLMs. But who knows, maybe I'm wrong.

notatoad 2mo ago

>build reliable software this feels like "make no mistakes" level of prompting. reliable software isn't as simple as making it reliable, it's about choosing the trade-offs in the areas that don't matter as much as the areas that do. if you keep prompting the LLM to make your software more robust it will keep giving you things to do. they aren't all good things. eventually you'll end up needing kubernetes to run a calculator app.