4 ms·

I’d say I fall in the “AI skeptic but willing to use it” category. If LLMs can actually clean up after themselves it would actually be a game changer.

by maccard 3mo ago

figmert 3mo ago
They can. You just have to steer them
- maccard 3mo ago
  Can you share a transcript of an LLM generated feature/bug fix that you steered into being good quality?
onlyrealcuzzo 3mo ago
I'm about to release some tooling that's been very effective for me. Essentially, there's a few ways LLMs write "bad" code that is different from how people write "bad" code. We've got pretty good tooling to catch the ways people write bad code - it just happens to be much easier to do with static analysis (and is less noise prone). The ways LLMs write bad code is typically 1) bad architecture - hard to detect in the ways that are really important, 2) unnecessary state and control flow (and decisions based on state), 3) bad / inadequate tests. Methods to detect these problems have existed for ages, but they've never caught on because it's typically too difficult to tune them to have high signal / noise for humans, and AFAIK - no one else tried putting them all together and seeing how LLMs work with it. LLMs are great at sorting through signal / noise -> so you can help surface potential issues with metrics that would be too noisy for humans, but seems to work pretty well for LLMs to find the source of architectural problems and design better solutions (from my experience - may be biased, I built the tooling to literally solve this problem for the main project I'm working on).
- mirekrusin 3mo ago
  You forgot to include link to your tool.
  - onlyrealcuzzo 3mo ago
    An LLM could probably figure out how to use it now. But it doesn't yet have a coherent UX unless you're me. Hopefully, I'll iron that out over the next week and I'll update you.
    - mirekrusin 3mo ago
      I’ll keep checking.
  - all2 3mo ago
    If I had to do a 'code quality' checker I think I'd try to use some combination of syntax tree analysis, data flow analysis, and LOC changed. Something like that. Too many new nodes in the syntax tree, or a sub-tree that appears sufficiently similar to another sub-tree (for various definitions of similar), data-flow/side effects gets more convoluted, too many LOC, and so on.
  - fragmede 3mo ago
    https://ponytail.dev/ https://ponytail.dev/
- andai 3mo ago
  Can you elaborate on the tests thing? I'm new to testing, and AI agents recently voluntarily added an large number of tests to one of my projects. I've been learning a lot by reading them, and it seems like a great habit to develop. But they're also writing some very strange code and some very strange tests. I don't know what good practices look like here, so when something looks strange to me I can't trust my own judgment, whether it's actually smelly or just a pattern I'm not used to yet. -- On a side note, I recently had an agent implement a major architectural change. It turned out to have done it completely backwards, in a way that was pointless. (Improved nothing and actively made things worse.) However it had supplied generous tests for the new code, and of course all the tests passed... So it had "proven the correctness" of something which was completely incorrect. I later realized that even formal verification would not have prevented this. It would have just written a mathematical proof that the wrong code was correct.
  - dnautics 3mo ago
    i find that the easiest way to make sure you have robust tests is to do strict red/green cycles for each feature and bugfix in your codebase.
jerf 3mo ago
LLMs can clean up after themselves if you steer them. It's not very hard. I often have a two-step dance I do where the LLM first outputs some code and then I prompt it to fix the types up to my standards. I haven't found a way to prompt it with any number of skills or CLAUDE.mds or anything else to get it to do it the way I want on the first pass, but it's not that hard to just fix it afterwards. It's a fast enough process that it feels fine using it. One can even make a case for it being a decent way to operate anyhow; make a sketch, then firm it up isn't entirely unlike my manual coding process was anyhow. The main problem is that pointing an LLM at a codebase and telling it to just "make it better" taps out pretty quickly. That is, not that there's zero juice to squeeze there, but there's not a ton. You can get a bit of improvement but it also rapidly starts changing things just to change things, which I'm not even going to complain about all that much because there's a sense in which it is simply doing as you asked. So you still need human taste and direction. This will be especially true for something the size of that codebase where you can only hold small fractions of the actual code in the context window at once. Summaries only get you so far.
- andai 3mo ago
  I asked an agent for some high level cleanup and refactoring. I found that I ended up disagreeing with most of the structural changes. Some of them were necessary, some were beneficial, but largely it turned straight line code into abstract factory manager type stuff. More broadly I've found that making code more elegant (e.g. by removing duplication) increases the cognitive load, because now you can't just read the code anymore but need to mentally "decompress" the higher level structures and indirection into the straight line code, the "code that actually runs."
  - jerf 3mo ago
    If it wasn't clear, I don't ask the agent to "clean up my code". I ask it to "take this map[string]string you pass around with constant keys and turn that into a structure" or "extract this API provider out into an interface and make everything using it use this interface instead" or other concrete instructions. LLMs, to a first approximation, already did as well as they could on the first pass. You can get a bit more out of them by asking them to just try harder, but not much. Whereas if you go in with specific changes they are pretty good at implementing them. This especially matters because my personal style deviates from the common practices. This may also be why I can't get it to just happen by prompting for it. But if you walk it through it's perfectly capable of transforming the code into a better style, and it's still way faster than trying to write it from scratch.
owaislone 3mo ago
You have to treat it like a co-worker send you PRs for review. You review, ask to improve/cleanup stuff and it'll follow up. You can even ask it to remember so it follows the patterns in future. Even better, if you work with it to come up with a design first and then send it coding, you often don't need as much cleanup as the desired architecture is agreed upon early.
- maccard 3mo ago
  I would tell my co worker “this code has so many issues that I’m not actually reviewing it. Please make sure it meets the project standards before sending it back to me”if it happened twice I’d talk to their manager.
  - owaislone 3mo ago
    Sure - define your project standards in the repo or agent's skill and tell it just that.
    - maccard 3mo ago
      See my other comment here [0] - I’ve not found it possible to actually have the agents follow the explicit instructions. Even single shot tasks once the context window fills up (which it does, very very fast, and repeatedly if you use subagents) will ignore very explicit instructions. [0] https://news.ycombinator.com/item?id=49036222 https://news.ycombinator.com/item?id=49036222
      - owaislone 3mo ago
        - discuss the plan, dos-and-donts, instructions and have them add it to a PLAN_FEATURE.md with the goal, non-goal, non-negotiables, technical details on the top. - come up with phases to implement your thing. P1, P2, ... P8.. whatever - in each phase add simple one line TODOs with as much explanantion of the task as needed e.g, - [ ] Update DB schema, - [ ] Generate migrations - Ask the agent to go one phase or todo at a time depending on complexity. - if the context fills up, /clear the window and just ask it to do the next line. - if you wrote up all you care about in the plan properly, it'll follow what you need it to do.
nateb2022 3mo ago
I'm assuming based on the special unicode angled quote marks that this is copy/pasted from an AI? I don't understand why people would type a lot more into an AI just to copy a comment this short, which would be at least 10x faster to just type by hand.
- defen 3mo ago
  Not sure about op but if I post here from my phone I get those. Example: “quoted string”
  - nateb2022 3mo ago
    huh interesting, I've never realized this, is it iOS or android?
    - defen 3mo ago
      iOS
    - thejazzman 3mo ago
      It’s a macOS thing too. I turn it off because it burns me when copying code
- maccard 3mo ago
  OP here - no AI used in the writing of any of my replies on this site. iOS does this when quoting [0]. I don’t think it’s in the spirit of this site to jump on me rather than the content of my comment either way. [0] https://apple.stackexchange.com/questions/299505/ios-11-default-quotation-mark-changed-to-and https://apple.stackexchange.com/questions/299505/ios-11-defa...

I’d say I fall in the “AI skeptic but willing to use it” category. If LLMs can actually clean up after themselves it would actually be a game changer.

by maccard 3mo ago

I’d say I fall in the “AI skeptic but willing to use it” category. If LLMs can actually clean up after themselves it would actually be a game changer.

figmert 3mo ago

They can. You just have to steer them

maccard 3mo ago
Can you share a transcript of an LLM generated feature/bug fix that you steered into being good quality?

onlyrealcuzzo 3mo ago

I'm about to release some tooling that's been very effective for me. Essentially, there's a few ways LLMs write "bad" code that is different from how people write "bad" code. We've got pretty good tooling to catch the ways people write bad code - it just happens to be much easier to do with static analysis (and is less noise prone). The ways LLMs write bad code is typically 1) bad architecture - hard to detect in the ways that are really important, 2) unnecessary state and control flow (and decisions based on state), 3) bad / inadequate tests. Methods to detect these problems have existed for ages, but they've never caught on because it's typically too difficult to tune them to have high signal / noise for humans, and AFAIK - no one else tried putting them all together and seeing how LLMs work with it. LLMs are great at sorting through signal / noise -> so you can help surface potential issues with metrics that would be too noisy for humans, but seems to work pretty well for LLMs to find the source of architectural problems and design better solutions (from my experience - may be biased, I built the tooling to literally solve this problem for the main project I'm working on).

mirekrusin 3mo ago
You forgot to include link to your tool.
- onlyrealcuzzo 3mo ago
  An LLM could probably figure out how to use it now. But it doesn't yet have a coherent UX unless you're me. Hopefully, I'll iron that out over the next week and I'll update you.
  - mirekrusin 3mo ago
    I’ll keep checking.
- all2 3mo ago
  If I had to do a 'code quality' checker I think I'd try to use some combination of syntax tree analysis, data flow analysis, and LOC changed. Something like that. Too many new nodes in the syntax tree, or a sub-tree that appears sufficiently similar to another sub-tree (for various definitions of similar), data-flow/side effects gets more convoluted, too many LOC, and so on.
- fragmede 3mo ago
  https://ponytail.dev/ https://ponytail.dev/
andai 3mo ago
Can you elaborate on the tests thing? I'm new to testing, and AI agents recently voluntarily added an large number of tests to one of my projects. I've been learning a lot by reading them, and it seems like a great habit to develop. But they're also writing some very strange code and some very strange tests. I don't know what good practices look like here, so when something looks strange to me I can't trust my own judgment, whether it's actually smelly or just a pattern I'm not used to yet. -- On a side note, I recently had an agent implement a major architectural change. It turned out to have done it completely backwards, in a way that was pointless. (Improved nothing and actively made things worse.) However it had supplied generous tests for the new code, and of course all the tests passed... So it had "proven the correctness" of something which was completely incorrect. I later realized that even formal verification would not have prevented this. It would have just written a mathematical proof that the wrong code was correct.
- dnautics 3mo ago
  i find that the easiest way to make sure you have robust tests is to do strict red/green cycles for each feature and bugfix in your codebase.

jerf 3mo ago

LLMs can clean up after themselves if you steer them. It's not very hard. I often have a two-step dance I do where the LLM first outputs some code and then I prompt it to fix the types up to my standards. I haven't found a way to prompt it with any number of skills or CLAUDE.mds or anything else to get it to do it the way I want on the first pass, but it's not that hard to just fix it afterwards. It's a fast enough process that it feels fine using it. One can even make a case for it being a decent way to operate anyhow; make a sketch, then firm it up isn't entirely unlike my manual coding process was anyhow. The main problem is that pointing an LLM at a codebase and telling it to just "make it better" taps out pretty quickly. That is, not that there's zero juice to squeeze there, but there's not a ton. You can get a bit of improvement but it also rapidly starts changing things just to change things, which I'm not even going to complain about all that much because there's a sense in which it is simply doing as you asked. So you still need human taste and direction. This will be especially true for something the size of that codebase where you can only hold small fractions of the actual code in the context window at once. Summaries only get you so far.

andai 3mo ago
I asked an agent for some high level cleanup and refactoring. I found that I ended up disagreeing with most of the structural changes. Some of them were necessary, some were beneficial, but largely it turned straight line code into abstract factory manager type stuff. More broadly I've found that making code more elegant (e.g. by removing duplication) increases the cognitive load, because now you can't just read the code anymore but need to mentally "decompress" the higher level structures and indirection into the straight line code, the "code that actually runs."
- jerf 3mo ago
  If it wasn't clear, I don't ask the agent to "clean up my code". I ask it to "take this map[string]string you pass around with constant keys and turn that into a structure" or "extract this API provider out into an interface and make everything using it use this interface instead" or other concrete instructions. LLMs, to a first approximation, already did as well as they could on the first pass. You can get a bit more out of them by asking them to just try harder, but not much. Whereas if you go in with specific changes they are pretty good at implementing them. This especially matters because my personal style deviates from the common practices. This may also be why I can't get it to just happen by prompting for it. But if you walk it through it's perfectly capable of transforming the code into a better style, and it's still way faster than trying to write it from scratch.

owaislone 3mo ago

You have to treat it like a co-worker send you PRs for review. You review, ask to improve/cleanup stuff and it'll follow up. You can even ask it to remember so it follows the patterns in future. Even better, if you work with it to come up with a design first and then send it coding, you often don't need as much cleanup as the desired architecture is agreed upon early.

maccard 3mo ago
I would tell my co worker “this code has so many issues that I’m not actually reviewing it. Please make sure it meets the project standards before sending it back to me”if it happened twice I’d talk to their manager.
- owaislone 3mo ago
  Sure - define your project standards in the repo or agent's skill and tell it just that.
  - maccard 3mo ago
    See my other comment here [0] - I’ve not found it possible to actually have the agents follow the explicit instructions. Even single shot tasks once the context window fills up (which it does, very very fast, and repeatedly if you use subagents) will ignore very explicit instructions. [0] https://news.ycombinator.com/item?id=49036222 https://news.ycombinator.com/item?id=49036222
    - owaislone 3mo ago
      - discuss the plan, dos-and-donts, instructions and have them add it to a PLAN_FEATURE.md with the goal, non-goal, non-negotiables, technical details on the top. - come up with phases to implement your thing. P1, P2, ... P8.. whatever - in each phase add simple one line TODOs with as much explanantion of the task as needed e.g, - [ ] Update DB schema, - [ ] Generate migrations - Ask the agent to go one phase or todo at a time depending on complexity. - if the context fills up, /clear the window and just ask it to do the next line. - if you wrote up all you care about in the plan properly, it'll follow what you need it to do.

nateb2022 3mo ago

I'm assuming based on the special unicode angled quote marks that this is copy/pasted from an AI? I don't understand why people would type a lot more into an AI just to copy a comment this short, which would be at least 10x faster to just type by hand.

defen 3mo ago
Not sure about op but if I post here from my phone I get those. Example: “quoted string”
- nateb2022 3mo ago
  huh interesting, I've never realized this, is it iOS or android?
  - defen 3mo ago
    iOS
  - thejazzman 3mo ago
    It’s a macOS thing too. I turn it off because it burns me when copying code
maccard 3mo ago
OP here - no AI used in the writing of any of my replies on this site. iOS does this when quoting [0]. I don’t think it’s in the spirit of this site to jump on me rather than the content of my comment either way. [0] https://apple.stackexchange.com/questions/299505/ios-11-default-quotation-mark-changed-to-and https://apple.stackexchange.com/questions/299505/ios-11-defa...