4 ms·

Some of the phenomenon described in this post are felt a lot when using AI. My own anecdote with a codebase I'm familiar with is indeed, as the article mention

Some of the phenomenon described in this post are felt a lot when using AI. My own anecdote with a codebase I'm familiar with is indeed, as the article mentions, it's a terrible architect. The problem I was solving ultimately called for a different data structure, but it never had that realization, instead trying to fit the problem shape into an existing, suboptimal way to represent the data. When I mentioned that this part of the code was memory-sensitive, it indeed wrote good code! ...for the bad data structure. It even included some nice tests that I decided to keep, including memory benchmarks. But the code was ultimately really bad for the problem. This is related to the sycophancy problem. AI coding assistants bias towards assuming the code they're working with is correct, and that the person using them is also correct. But often neither is ideal! And you can absolutely have a model second-guess your own code and assumptions, but it takes a lot of persistent work because these damn things just want to be "helpful" all the time. I say all of this as a believer in this paradigm and one who uses these tools every day.

sdoering 1y ago
Would it help prompting (and adapting the system prompts for the coding assistants) accordingly? Like: > Do not assume the person writing the code knows what they are doing. Also do not assume the code base follows best practices or sensible defaults. Always check for better solutions/optimizations where it makes sense and check the validity of data structures. Just a quick draft. Would probably need waaaaaay more refinement. But might this help at least mitigating a bit of the felt issue? I always think of AI as an overeager junior dev. So I tend to treat it that way when giving instructions, but even then... ... well, let's say the results are sometimes interesting.
- phillipcarter 1y ago
  Yeah, that's what I do now -- and some coworkers have noted that it can often help with biasing towards design system components if you prompt it to do that -- but similarly, but the challenge here is that the level of pushback I want from the AI depends on several factors that aren't encodable into rules. Sometimes the area of the code is exactly the way it should be, and sometimes I know exactly what to do! Or it's somewhere in between. And it's a lot of work to have a set of rules that plays well here.
  - layer8 1y ago
    Yes. The biggest issue with LLMs is their tunnel vision and general lack of awareness. They lack the ability to go meta, or to "take a step back" on their own, which given their construction isn't surprising. Adjusting the prompts is only a hack and doesn't solve the fundamental issue.
  - SAI_Peregrinus 1y ago
    A central issue is that specifying what you require is difficult. It's hard for non-programmers to specify what they want to programmers, it's hard for people (programmers or not) to specify what they want to AIs, it's hard to specify exactly what requirements your system has to a model checker, etc. Specifying requirements isn't always the hardest part of making software, but it often is and it's not something with purely technical solutions.
    - sdoering 1y ago
      To me prompting an AI (and learning to better specify what I actually want) really helped me learn to better "prompt" humans. I think my biggest learning from using AI is being able to clearer think about and communicate what I need/want/Desire and how to put it into enough context, so that the other party can form a better understanding themselves. Not that it always works - but I feel I am getting better.
franktankbank 1y ago
They are designed for executives. Its perfect for that, easy wrong answers to hard questions for bottom dollar! Get that bonus and bounce, how could it fail? /s
marcosdumay 1y ago
> This is related to the sycophancy problem. No, this is way more fundamental than the sycophancy. It's related to the difficulty of older AI to understand "no". Unless it sees people recommending that you change your code into a different version, it has no way to understand that the better code is equivalent.
- phillipcarter 1y ago
  It's all the same problem. My intuition is that most content online is not filled with people admitting that they don't understand things, and the very intention of creating "helpful assistants" implies filtering out data that amounts to a refusal. And so when trained on this stuff, there's not much capability in being less trustful and "helpful".
rvz 1y ago
> AI coding assistants bias towards assuming the code they're working with is correct, and that the person using them is also correct. But often neither is ideal! That's why you should just write tests, before you write the code, so that you know what you are expecting with the code that is under test is doing. i.e Test driven development. > And you can absolutely have a model second-guess your own code and assumptions, but it takes a lot of persistent work because these damn things just want to be "helpful" all the time. No. Please do not do this. These LLMs have zero understanding / reasoning about the code they are outputting. Recent example from [0]: >> Yesterday I wanted to move 40GB of images from my QR menu site qrmenucreator . com from my VPS to R2 >> I asked gemini-2.5-pro-max to write a script to move the files >> I even asked it to check everything was correct >> Turns out for some reason the filenames got shortened somehow, which is a disaster because the QR site is quite basic and the image paths are written in the markdown of the menus >> Of course the script already deleted 40GB of images from the VPS >> But lesson learnt: be very careful with AI code, it made a mistake, couldn't even find the mistake when I asked it to double check the code, and because the ENDs of the filenames looked same I didn't notice it cut the beginnings off >> And in this case AI can't even find its own mistakes Just like the 2010s with the proliferation with dynamically typed languages creeping into the backend with low-quality code, we now will have vibe-coded low-quality software causing destruction because their authors do not know what their code does and also have not bothered to test it or even know what to test for. [0] https://twitter.com/levelsio/status/1921974501257912563 https://twitter.com/levelsio/status/1921974501257912563
- phillipcarter 1y ago
  > That's why you should just write tests, before you write the code, so that you know what you are expecting with the code that is under test is doing. i.e Test driven development. I've tried this too. They find ways to cheat the tests, sometimes throwing in special cases that match the specific test cases. It's easy to catch in the small scale but not when in a larger coding session. > No. Please do not do this. These LLMs have zero understanding / reasoning about the code they are outputting. This is incorrect. LLMs do have the ability to reason, but it's not the same reasoning that you or I do. They are actually quite good at checking for a variety of problems, like if the code you're writing is sensitive to memory pressure and you want to account for it. Asking them to examine the code with several constraints in mind often does give reasonable advice and suggestions to change. But this requires you to understand those changes to be effective.

sdoering 1y ago

Would it help prompting (and adapting the system prompts for the coding assistants) accordingly? Like: > Do not assume the person writing the code knows what they are doing. Also do not assume the code base follows best practices or sensible defaults. Always check for better solutions/optimizations where it makes sense and check the validity of data structures. Just a quick draft. Would probably need waaaaaay more refinement. But might this help at least mitigating a bit of the felt issue? I always think of AI as an overeager junior dev. So I tend to treat it that way when giving instructions, but even then... ... well, let's say the results are sometimes interesting.

phillipcarter 1y ago
Yeah, that's what I do now -- and some coworkers have noted that it can often help with biasing towards design system components if you prompt it to do that -- but similarly, but the challenge here is that the level of pushback I want from the AI depends on several factors that aren't encodable into rules. Sometimes the area of the code is exactly the way it should be, and sometimes I know exactly what to do! Or it's somewhere in between. And it's a lot of work to have a set of rules that plays well here.
- layer8 1y ago
  Yes. The biggest issue with LLMs is their tunnel vision and general lack of awareness. They lack the ability to go meta, or to "take a step back" on their own, which given their construction isn't surprising. Adjusting the prompts is only a hack and doesn't solve the fundamental issue.
- SAI_Peregrinus 1y ago
  A central issue is that specifying what you require is difficult. It's hard for non-programmers to specify what they want to programmers, it's hard for people (programmers or not) to specify what they want to AIs, it's hard to specify exactly what requirements your system has to a model checker, etc. Specifying requirements isn't always the hardest part of making software, but it often is and it's not something with purely technical solutions.
  - sdoering 1y ago
    To me prompting an AI (and learning to better specify what I actually want) really helped me learn to better "prompt" humans. I think my biggest learning from using AI is being able to clearer think about and communicate what I need/want/Desire and how to put it into enough context, so that the other party can form a better understanding themselves. Not that it always works - but I feel I am getting better.

franktankbank 1y ago

They are designed for executives. Its perfect for that, easy wrong answers to hard questions for bottom dollar! Get that bonus and bounce, how could it fail? /s

marcosdumay 1y ago

> This is related to the sycophancy problem. No, this is way more fundamental than the sycophancy. It's related to the difficulty of older AI to understand "no". Unless it sees people recommending that you change your code into a different version, it has no way to understand that the better code is equivalent.

phillipcarter 1y ago
It's all the same problem. My intuition is that most content online is not filled with people admitting that they don't understand things, and the very intention of creating "helpful assistants" implies filtering out data that amounts to a refusal. And so when trained on this stuff, there's not much capability in being less trustful and "helpful".

rvz 1y ago

> AI coding assistants bias towards assuming the code they're working with is correct, and that the person using them is also correct. But often neither is ideal! That's why you should just write tests, before you write the code, so that you know what you are expecting with the code that is under test is doing. i.e Test driven development. > And you can absolutely have a model second-guess your own code and assumptions, but it takes a lot of persistent work because these damn things just want to be "helpful" all the time. No. Please do not do this. These LLMs have zero understanding / reasoning about the code they are outputting. Recent example from [0]: >> Yesterday I wanted to move 40GB of images from my QR menu site qrmenucreator . com from my VPS to R2 >> I asked gemini-2.5-pro-max to write a script to move the files >> I even asked it to check everything was correct >> Turns out for some reason the filenames got shortened somehow, which is a disaster because the QR site is quite basic and the image paths are written in the markdown of the menus >> Of course the script already deleted 40GB of images from the VPS >> But lesson learnt: be very careful with AI code, it made a mistake, couldn't even find the mistake when I asked it to double check the code, and because the ENDs of the filenames looked same I didn't notice it cut the beginnings off >> And in this case AI can't even find its own mistakes Just like the 2010s with the proliferation with dynamically typed languages creeping into the backend with low-quality code, we now will have vibe-coded low-quality software causing destruction because their authors do not know what their code does and also have not bothered to test it or even know what to test for. [0] https://twitter.com/levelsio/status/1921974501257912563 https://twitter.com/levelsio/status/1921974501257912563

phillipcarter 1y ago
> That's why you should just write tests, before you write the code, so that you know what you are expecting with the code that is under test is doing. i.e Test driven development. I've tried this too. They find ways to cheat the tests, sometimes throwing in special cases that match the specific test cases. It's easy to catch in the small scale but not when in a larger coding session. > No. Please do not do this. These LLMs have zero understanding / reasoning about the code they are outputting. This is incorrect. LLMs do have the ability to reason, but it's not the same reasoning that you or I do. They are actually quite good at checking for a variety of problems, like if the code you're writing is sensitive to memory pressure and you want to account for it. Asking them to examine the code with several constraints in mind often does give reasonable advice and suggestions to change. But this requires you to understand those changes to be effective.