5 ms·

LLM making a quick edit, <100 lines... Sure. Asking an LLM to rubber-duck your code, sure. But integrating an LLM into your CI is going to end up costing you 10

LLM making a quick edit, <100 lines... Sure. Asking an LLM to rubber-duck your code, sure. But integrating an LLM into your CI is going to end up costing you 100s of hours productivity on any large project. That or spend half the time you should be spending learning to write your own code, dialing down context sizing and prompt accuracy. I really really don't understand the hubris around llm tooling, and don't see it catching on outside of personal projects and small web apps. These things don't handle complex systems well at all, you would have to put a gun in my mouth to let one of these things work on an important repo of mine without any supervision... And if I'm supervising the LLM I might as well do it myself, because I'm going to end up redoing 50% of its work anyways..

mike_hearn 1y ago
I've used Claude with a large, mature codebase and it did fine. Not for every possible task, but for many. Probably, Mercury isn't as good at coding as Claude is. But even if it's not, there's lots of small tasks that LLMs can do without needing senior engineer level skills. Adding test coverage, fixing low priority bugs, adding nice animations to the UI etc. Stuff that maybe isn't critical so if a PR turns up and it's DOA you just close it, but which otherwise works. Note that many projects already use this approach with bots like Renovate. Such bots also consume a ton of CI time, but it's generally worth it.
- flir 1y ago
  Don't want to put words in the parent commenter's mouth, but I think the key word is "unsupervised". Claude doesn't know what it doesn't know, and will keep going round the loop until the tests go green, or until the heat death of the universe.
  - mike_hearn 1y ago
    Yes, but you can just impose timeouts to solve that. If it's unsupervised the only cost is computation.
- airstrike 1y ago
  IMHO LLMs are notoriously bad at test coverage. They usually hard code a value to have the test pass, since they lack the reasoning required to understand why the test exists or the concept of assertion, really
  - wrs 1y ago
    I don’t know, Claude is very good at writing that utterly useless kind of unit test where every dependency is mocked out and the test is just the inverted dual of the original code. 100% coverage, nothing tested.
    - conradkay 1y ago
      Yeah and that's even worse because there's not an easy metric you can have the agent work towards and get feedback on. I'm not that into "prompt engineering" but tests seem like a big opportunity for improvement. Maybe something like (but much more thorough): 1. "Create a document describing all real-world actions which could lead to the code being used. List all methods/code which gets called before it (in order) along with their exact parameters and return value. Enumerate all potential edge cases and errors that could occur and if it ends up influencing this task. After that, write a high-level overview of what need to occur in this implementation. Don't make it top down where you think about what functions/classes/abstractions which are created, just the raw steps that will need to occur" 2. Have it write the tests 3. Have it write the code Maybe TDD ends up worse but I suspect the initial plan which is somewhat close to code makes that not the case Writing the initial doc yourself would definitely be better, but I suspect just writing one really good one, then giving it as an example in each subsequent prompt captures a lot of the improvement
      - girvo 1y ago
        I've not gone into it yet, but I think BDD would fit reasonably well with agents and generating tests that aren't entirely useless.
    - astrange 1y ago
      This is why unit tests are the least useful kind of test and regression tests are the most useful. I think unit tests are best written /before/ the real code and thrown out after. Of course, that's extremely situational.
DSingularity 1y ago
He is simply observing that if PR numbers and launch rates increase dramatically CI cost will become untenable.
kraftman 1y ago
I keep seeing this argument over and over again, and I have to wonder, at what point do you accept that maybe LLM's are useful? Like how many people need to say that they find it makes them more productive before you'll shift your perspective?
- candiddevmike 1y ago
  People say they are more productive using visual basic, but that will never shift my perspective on it. Code is a liability. Code you didn't write is a ticking time bomb.
- psychoslave 1y ago
  That's a tool, and it depends what you need to do. If it fits someone need and make them more productive, or even simply enjoy more the activity, good. Just because two people are fixing something on the whole doesn't mean the same tool will hold fine. Gum, pushpin, nail, screw,bolts? The parent thread did mention they use LLM successfully in small side project.
- dragonwriter 1y ago
  > I keep seeing this argument over and over again, and I have to wonder, at what point do you accept that maybe LLM's are useful? The post you are responding to literally acknowledges that LLMs are useful in certain roles in coding in the first sentence. > Like how many people need to say that they find it makes them more productive before you'll shift your perspective? Argumentum ad populum is not a good way of establishing fact claims beyond the fact of a belief being popular.
  - kraftman 1y ago
    ...and my comment clearly isnt talking about that, but at the suggestion that its useless to write code with an LLM because you'll end up rewriting 50% of it. If everyone has an opinion different to mine, I dont instantly change my opinion, but I do try and investigate the source of the difference, to find out what I'm missing or what they are missing. The polarisation between people that find LLMs useful or not is very similar to the polarisation between people that find automated testing useful or not, and I have a suspicion they have the same underlying cause.
blitzar 1y ago
Do the opposite - integrate your CI into your LLM. Make it run tests after it changes your code and either confirm it didnt break anything or go back and try again.

mike_hearn 1y ago

I've used Claude with a large, mature codebase and it did fine. Not for every possible task, but for many. Probably, Mercury isn't as good at coding as Claude is. But even if it's not, there's lots of small tasks that LLMs can do without needing senior engineer level skills. Adding test coverage, fixing low priority bugs, adding nice animations to the UI etc. Stuff that maybe isn't critical so if a PR turns up and it's DOA you just close it, but which otherwise works. Note that many projects already use this approach with bots like Renovate. Such bots also consume a ton of CI time, but it's generally worth it.

flir 1y ago
Don't want to put words in the parent commenter's mouth, but I think the key word is "unsupervised". Claude doesn't know what it doesn't know, and will keep going round the loop until the tests go green, or until the heat death of the universe.
- mike_hearn 1y ago
  Yes, but you can just impose timeouts to solve that. If it's unsupervised the only cost is computation.
airstrike 1y ago
IMHO LLMs are notoriously bad at test coverage. They usually hard code a value to have the test pass, since they lack the reasoning required to understand why the test exists or the concept of assertion, really
- wrs 1y ago
  I don’t know, Claude is very good at writing that utterly useless kind of unit test where every dependency is mocked out and the test is just the inverted dual of the original code. 100% coverage, nothing tested.
  - conradkay 1y ago
    Yeah and that's even worse because there's not an easy metric you can have the agent work towards and get feedback on. I'm not that into "prompt engineering" but tests seem like a big opportunity for improvement. Maybe something like (but much more thorough): 1. "Create a document describing all real-world actions which could lead to the code being used. List all methods/code which gets called before it (in order) along with their exact parameters and return value. Enumerate all potential edge cases and errors that could occur and if it ends up influencing this task. After that, write a high-level overview of what need to occur in this implementation. Don't make it top down where you think about what functions/classes/abstractions which are created, just the raw steps that will need to occur" 2. Have it write the tests 3. Have it write the code Maybe TDD ends up worse but I suspect the initial plan which is somewhat close to code makes that not the case Writing the initial doc yourself would definitely be better, but I suspect just writing one really good one, then giving it as an example in each subsequent prompt captures a lot of the improvement
    - girvo 1y ago
      I've not gone into it yet, but I think BDD would fit reasonably well with agents and generating tests that aren't entirely useless.
  - astrange 1y ago
    This is why unit tests are the least useful kind of test and regression tests are the most useful. I think unit tests are best written /before/ the real code and thrown out after. Of course, that's extremely situational.

DSingularity 1y ago

He is simply observing that if PR numbers and launch rates increase dramatically CI cost will become untenable.

kraftman 1y ago

I keep seeing this argument over and over again, and I have to wonder, at what point do you accept that maybe LLM's are useful? Like how many people need to say that they find it makes them more productive before you'll shift your perspective?

candiddevmike 1y ago
People say they are more productive using visual basic, but that will never shift my perspective on it. Code is a liability. Code you didn't write is a ticking time bomb.
psychoslave 1y ago
That's a tool, and it depends what you need to do. If it fits someone need and make them more productive, or even simply enjoy more the activity, good. Just because two people are fixing something on the whole doesn't mean the same tool will hold fine. Gum, pushpin, nail, screw,bolts? The parent thread did mention they use LLM successfully in small side project.
dragonwriter 1y ago
> I keep seeing this argument over and over again, and I have to wonder, at what point do you accept that maybe LLM's are useful? The post you are responding to literally acknowledges that LLMs are useful in certain roles in coding in the first sentence. > Like how many people need to say that they find it makes them more productive before you'll shift your perspective? Argumentum ad populum is not a good way of establishing fact claims beyond the fact of a belief being popular.
- kraftman 1y ago
  ...and my comment clearly isnt talking about that, but at the suggestion that its useless to write code with an LLM because you'll end up rewriting 50% of it. If everyone has an opinion different to mine, I dont instantly change my opinion, but I do try and investigate the source of the difference, to find out what I'm missing or what they are missing. The polarisation between people that find LLMs useful or not is very similar to the polarisation between people that find automated testing useful or not, and I have a suspicion they have the same underlying cause.

blitzar 1y ago

Do the opposite - integrate your CI into your LLM. Make it run tests after it changes your code and either confirm it didnt break anything or go back and try again.