3 ms·

Since I wrote the blog post, I actually added automated tests which Windsurf / Claude Code runs before committing any change. It saved me a lot of headaches whe

Since I wrote the blog post, I actually added automated tests which Windsurf / Claude Code runs before committing any change. It saved me a lot of headaches when the LLM decided to make random logic changes for no reason.

varelaseb 2y ago
Would you mind describing the process for this in a bit more detail?
myvoiceismypass 2y ago
The LLM swaps out business logic without you knowing it? As someone who worries about determinism and doesn’t do much vibe coding, I am curious what’s going on here, thanks!
- alluro2 2y ago
  Absolutely, very often. It typically happens because it's trying to do more than you asked for. Example scenario: you have a codebase that you iterated on with LLM, and it contains let's say 15 features with various implementation details. You continue tomorrow and want to make a small change to handle an edge-case. While making changes to one of the 3 required files, suddenly it will decide to also rewrite / "improve" other parts of the code that have nothing to do with your request, where pieces of previous logic will no longer exist - since it made 18 changes to the file, and there are 3 such files, good luck spotting it without thorough and detailed change review. Also, if you made manual changes to generated code and than you ask it to add something to the code (within the same "conversation" / context, it will often replace your changes with how it originally wanted code to look like.
  - johnisgood 2y ago
    I experienced this, too. It is quite annoying. You have to explicitly tell the LLM to leave everything else as-is.
    - fragmede 2y ago
      use conventions.md to have it not do that.
  - __loam 2y ago
    What the fuck
  - shostack 1y ago
    Do things like Cline's memory bank solve this?
- thebeardisred 2y ago
  "Let's take a different approach. Instead of trying to fix the existing file incrementally, let's rewrite it completely with a simpler approach that uses the --print-config flag to test the configuration:"

varelaseb 2y ago

Would you mind describing the process for this in a bit more detail?

myvoiceismypass 2y ago

The LLM swaps out business logic without you knowing it? As someone who worries about determinism and doesn’t do much vibe coding, I am curious what’s going on here, thanks!

alluro2 2y ago
Absolutely, very often. It typically happens because it's trying to do more than you asked for. Example scenario: you have a codebase that you iterated on with LLM, and it contains let's say 15 features with various implementation details. You continue tomorrow and want to make a small change to handle an edge-case. While making changes to one of the 3 required files, suddenly it will decide to also rewrite / "improve" other parts of the code that have nothing to do with your request, where pieces of previous logic will no longer exist - since it made 18 changes to the file, and there are 3 such files, good luck spotting it without thorough and detailed change review. Also, if you made manual changes to generated code and than you ask it to add something to the code (within the same "conversation" / context, it will often replace your changes with how it originally wanted code to look like.
- johnisgood 2y ago
  I experienced this, too. It is quite annoying. You have to explicitly tell the LLM to leave everything else as-is.
  - fragmede 2y ago
    use conventions.md to have it not do that.
- __loam 2y ago
  What the fuck
- shostack 1y ago
  Do things like Cline's memory bank solve this?
thebeardisred 2y ago
"Let's take a different approach. Instead of trying to fix the existing file incrementally, let's rewrite it completely with a simpler approach that uses the --print-config flag to test the configuration:"