4 ms·

Have you seen human-written code?

withinboredom 1y ago
At least when you tell a human the indentation is wrong, they can fix it on the first try. Watched an AI agent last night try to fix indentation by using sed for 20 minutes before I just fixed it myself after cringing.
- aeontech 1y ago
  I mean... this is a deterministic task, can just run it through autoformatter? Why ask AI to do indentation of all things?
  - theshrike79 1y ago
    This is why I have a standing order for all LLMs to run goimports on any file they've edited. It'll fix imports and minor nags without the LLM having to waste context on removing a line there or changing ' to " somewhere. Even better if you use an LLM with Hook support, just have the hook run formatters on the file after each edit.
  - jvanderbot 1y ago
    One time I explained that I was afraid of tesla full self driving, because while using it my tesla accelerated to 45mph in a parking lot that was parallel to the road and only separated by a curb. The pushback I got was "Why would you use FSD in a parking lot". Well, "Full", right? Same here. It's either capable of working unsupervised or not. And if not, you have to start wondering what you're even doing if you're at your keyboard, running tools, editing code that you don't like, etc. We're still working out the edge cases with these "Full" self driving editors. It vastly diminishes the usefulness if it's going to spend 20 minutes (and $) on stupid simple things.
    - godelski 1y ago
      > We're still working out the edge cases The difficult part is that like with FSD, it's mostly edge cases
      - const_cast 1y ago
        Driving is just mostly edge cases. I've thought about it a lot, but I think automating driving is much harder than automating even air travel. Sure the air is 3 dimensions, but driving is too dynamic and volatile. Every single road is different, and you have to rely on heuristics meant for humans. It's stupid easy for humans to tell what is a yellow line and what a stop sign looks like, but it's not so easy for computers. These are human tools - physical things we look at with our eyes. Not easy to measure. Whereas measurements in the air are quite easy to measure. On top of the visual heuristics, everthing changes all the time and very fast. You look away from the road and look back and you don't know what you're gonna see. It's why texting and driving is so dangerous.
        godelski 1y ago
        > I think automating driving is much harder than automating even air travel. This is a pretty common belief. Well supported too since we've had a high level of automation in aviation for decades. Helps that things are very monitored. 3 dimensions provides a lot of benefits given that it makes for a lower density. Not to mention people don't tend to be walking around in the sky
    - david38 1y ago
      A parking lot is an excellent use of self driving. First, I want to summon my car. Then, when leaving, if I’m in a dense area with lots of shopping, the roads can be a pain. You have to exit right, immediately get into the left lane, three lanes over, the second of the right turn only lanes, etc
- LeafItAlone 1y ago
  Consider yourself lucky if you’ve never had a co-worker do something along those lines. At this point seeing a person something like that wouldn’t even phase me.
- steve_adams_86 1y ago
  With Claude Code you can configure hooks to ensure this is done before results are presented, or just run a linter yourself after accepting changes. If you're using something else, I'd just pull it out and lint it
- lonelyasacloud 1y ago
  Have seen similar issues with understanding things that are somewhat orthogonal concerns to the main thing that is being worked on. My guess is that context = main thing + somewhat unrelated thing is too big a space for the models to perform well at this point in time. The practical solution is to remove the need for the model to figure it out each time and instead explicitly tell it about as much as possible before hand in CLAUDE.md.
- david38 1y ago
  Why the hell would anyone do this instead of using any one of dozens of purpose written tools that accept configuration files? They take less than a second to run, can run on every save, and are free
  - withinboredom 1y ago
    My point exactly...
godelski 1y ago
I've taught undergraduates and graduates how to code. I've contributed to Open Source projects. I'm a researcher and write research code with other people who write research code. You could say I've seen A LOT of poorly written human generated code. Yet, I still trust it more. Why? Well one of the big reasons is exactly what we're joking about. I can trust a human to iterate. Lack of iteration would be fine if everything was containerized and code operates in an unchanging environment[0]. But in the real world, code needs to be iterated on, constantly. Good code doesn't exist. If it does exist, it doesn't stay good for long. Another major problem is that AI generates code that optimizes for human preference, not correctness. Even the terrible students who were just doing enough to scrape by weren't trying to mask mistakes[1], but were still optimizing for correctness, even if it was the bare minimum. I can still walk through that code with the human and we can figure out what went wrong. I can ask the human about the code and I can tell a lot by their explanation, even if they make mistakes[2]. I can't trust the AI to tell an accurate account of even its own code because it doesn't actually understand. Even the dumb human has a much larger context window. They can see all the code. They can actually talk to me and try to figure out the intent. They will challenge me if I'm wrong! And for the love of god, I'm going to throw them out if they are just constantly showering me with praise and telling me how much of a genius I am. I don't want to work with someone where I feel like at any moment they're going to start trying to sell me a used car. There's a lot of reasons, more than I list here. Do I still prompt LLMs and use them while I write code? Of course. Do I trust it to write code? Fuck no. I know it isn't trivial to see that middle ground if all you do is vibe code or hate writing code so much you just want to outsource it, but there's a lot of room here between having some assistant and having AI write code. Like the OP suggests, someone has got to write that 10-20%. That doesn't mean I've saved 80% of my time, I maybe saved 20%. Pareto is a bitch. [0] Ever hear of "code rot?" [1] Well... I'd rightfully dock points if they wrote obfuscated code... [2] A critical skill of an expert in any subject is the ability to identify other experts. https://xkcd.com/451/ https://xkcd.com/451/
- thunky 1y ago
  > Lack of iteration What makes you think that agents can't iterate? > I'm going to throw them out if they are just constantly showering me with praise and telling me how much of a genius I am You can tell the agent to have the persona of an arrogant ass if you prefer it.
Eggpants 1y ago
I hate to break it to you, but humans wrote the original code that was stolen and used for the training set.
- beambot 1y ago
  garbage in, garbage out...

At least when you tell a human the indentation is wrong, they can fix it on the first try. Watched an AI agent last night try to fix indentation by using sed for 20 minutes before I just fixed it myself after cringing.

aeontech 1y ago
I mean... this is a deterministic task, can just run it through autoformatter? Why ask AI to do indentation of all things?
- theshrike79 1y ago
  This is why I have a standing order for all LLMs to run goimports on any file they've edited. It'll fix imports and minor nags without the LLM having to waste context on removing a line there or changing ' to " somewhere. Even better if you use an LLM with Hook support, just have the hook run formatters on the file after each edit.
- jvanderbot 1y ago
  One time I explained that I was afraid of tesla full self driving, because while using it my tesla accelerated to 45mph in a parking lot that was parallel to the road and only separated by a curb. The pushback I got was "Why would you use FSD in a parking lot". Well, "Full", right? Same here. It's either capable of working unsupervised or not. And if not, you have to start wondering what you're even doing if you're at your keyboard, running tools, editing code that you don't like, etc. We're still working out the edge cases with these "Full" self driving editors. It vastly diminishes the usefulness if it's going to spend 20 minutes (and $) on stupid simple things.
  - godelski 1y ago
    > We're still working out the edge cases The difficult part is that like with FSD, it's mostly edge cases
    - const_cast 1y ago
      Driving is just mostly edge cases. I've thought about it a lot, but I think automating driving is much harder than automating even air travel. Sure the air is 3 dimensions, but driving is too dynamic and volatile. Every single road is different, and you have to rely on heuristics meant for humans. It's stupid easy for humans to tell what is a yellow line and what a stop sign looks like, but it's not so easy for computers. These are human tools - physical things we look at with our eyes. Not easy to measure. Whereas measurements in the air are quite easy to measure. On top of the visual heuristics, everthing changes all the time and very fast. You look away from the road and look back and you don't know what you're gonna see. It's why texting and driving is so dangerous.
      - godelski 1y ago
        > I think automating driving is much harder than automating even air travel. This is a pretty common belief. Well supported too since we've had a high level of automation in aviation for decades. Helps that things are very monitored. 3 dimensions provides a lot of benefits given that it makes for a lower density. Not to mention people don't tend to be walking around in the sky
  - david38 1y ago
    A parking lot is an excellent use of self driving. First, I want to summon my car. Then, when leaving, if I’m in a dense area with lots of shopping, the roads can be a pain. You have to exit right, immediately get into the left lane, three lanes over, the second of the right turn only lanes, etc
LeafItAlone 1y ago
Consider yourself lucky if you’ve never had a co-worker do something along those lines. At this point seeing a person something like that wouldn’t even phase me.
steve_adams_86 1y ago
With Claude Code you can configure hooks to ensure this is done before results are presented, or just run a linter yourself after accepting changes. If you're using something else, I'd just pull it out and lint it
lonelyasacloud 1y ago
Have seen similar issues with understanding things that are somewhat orthogonal concerns to the main thing that is being worked on. My guess is that context = main thing + somewhat unrelated thing is too big a space for the models to perform well at this point in time. The practical solution is to remove the need for the model to figure it out each time and instead explicitly tell it about as much as possible before hand in CLAUDE.md.
david38 1y ago
Why the hell would anyone do this instead of using any one of dozens of purpose written tools that accept configuration files? They take less than a second to run, can run on every save, and are free
- withinboredom 1y ago
  My point exactly...

godelski 1y ago

I've taught undergraduates and graduates how to code. I've contributed to Open Source projects. I'm a researcher and write research code with other people who write research code. You could say I've seen A LOT of poorly written human generated code. Yet, I still trust it more. Why? Well one of the big reasons is exactly what we're joking about. I can trust a human to iterate. Lack of iteration would be fine if everything was containerized and code operates in an unchanging environment[0]. But in the real world, code needs to be iterated on, constantly. Good code doesn't exist. If it does exist, it doesn't stay good for long. Another major problem is that AI generates code that optimizes for human preference, not correctness. Even the terrible students who were just doing enough to scrape by weren't trying to mask mistakes[1], but were still optimizing for correctness, even if it was the bare minimum. I can still walk through that code with the human and we can figure out what went wrong. I can ask the human about the code and I can tell a lot by their explanation, even if they make mistakes[2]. I can't trust the AI to tell an accurate account of even its own code because it doesn't actually understand. Even the dumb human has a much larger context window. They can see all the code. They can actually talk to me and try to figure out the intent. They will challenge me if I'm wrong! And for the love of god, I'm going to throw them out if they are just constantly showering me with praise and telling me how much of a genius I am. I don't want to work with someone where I feel like at any moment they're going to start trying to sell me a used car. There's a lot of reasons, more than I list here. Do I still prompt LLMs and use them while I write code? Of course. Do I trust it to write code? Fuck no. I know it isn't trivial to see that middle ground if all you do is vibe code or hate writing code so much you just want to outsource it, but there's a lot of room here between having some assistant and having AI write code. Like the OP suggests, someone has got to write that 10-20%. That doesn't mean I've saved 80% of my time, I maybe saved 20%. Pareto is a bitch. [0] Ever hear of "code rot?" [1] Well... I'd rightfully dock points if they wrote obfuscated code... [2] A critical skill of an expert in any subject is the ability to identify other experts. https://xkcd.com/451/ https://xkcd.com/451/

thunky 1y ago
> Lack of iteration What makes you think that agents can't iterate? > I'm going to throw them out if they are just constantly showering me with praise and telling me how much of a genius I am You can tell the agent to have the persona of an arrogant ass if you prefer it.

Eggpants 1y ago

I hate to break it to you, but humans wrote the original code that was stolen and used for the training set.

beambot 1y ago
garbage in, garbage out...