5 ms·

> If you ask AI to write a document for you, you might get 80% of the deep quality you’d get if you wrote it yourself for 5% of the effort. But, now you’ve also

by kranner 8mo ago

> If you ask AI to write a document for you, you might get 80% of the deep quality you’d get if you wrote it yourself for 5% of the effort. But, now you’ve also only done 5% of the thinking. This, but also for code. I just don't trust new code, especially generated code; I need time to sit with it. I can't make the "if it passes all the tests" crowd understand and I don't even want to. There are things you think of to worry about and test for as you spend time with a system. If I'm going to ship it and support it, it will take as long as it will take.

bravetraveler 8mo ago
I'll borrow your phrasing lest another coworker decide to play therapist. Previously: "I'm not slinging drugs here, I care for myself and the users."
torginus 8mo ago
I personally find that it's easier to iterate on 'something' and find what's wrong with it and fix it, than theorycrafting staring at a blank paper.
- kranner 8mo ago
  Me too! Sometimes I find it easier to ask an LLM to generate something and then it's easier to discard the whole thing, muttering under my breath "no no no that's not how you do this" and write my own code. It would be nice if editors would annotate generated code clearly.
slfreference 8mo ago
I think what LLMs do with words is similar to what artists do with software like cinema4d. We have control points (prompts + context) and we ask LLMs to draw a 3D surface which passes through those points satisfying some given constraints. Subsequent chats are like edit operations. https://youtu.be/-5S2qs32PII https://youtu.be/-5S2qs32PII
- catdog 8mo ago
  An LLM is an impressive, yet still imperfect and unpredictable translation machine. The code it outputs can only be as good as your prompt is precise, minus the often blatant mistakes it makes.
layer8 8mo ago
Yes, regression tests are not enough. One generally has to think through code repeatedly, with different aspects in mind, to convince oneself that it is correct under all circumstances. Tests only point-check, they don’t ensure correct behavior under all conceivable scenarios.
- doug_durham 8mo ago
  Unless you are in the business of writing flight control software, OS kernels, or critical financial software, I don't think your own code will reach the standards you mention. The only way we get "correct under all conceivable scenarios" software is to have a large team with long time horizons and large funding working on a small piece of software. It is beyond an individual to reach that standard for anything beyond code at the function level.
simianwords 8mo ago
Honest question: why is this not enough? If the code passes tests, and also works at the functionality level - what difference does it make if you’ve read the code or not? You could come up with pathological cases like: it passed the tests by deleting them. And the code written by it is extremely messy. But we know that LLMs are way smarter than this. There’s very very low chance of this happening and even if it does - it quick glance at code can fix it.
- jdjdjssh 8mo ago
  > If the code passes tests, and also works at the functionality level Why doesn’t outsourcing work if this is all that is needed?
  - simianwords 8mo ago
    Why do we have managers if managers don’t have accountability?
    - jdjdjssh 8mo ago
      I’m not sure what you’re getting at. I’m saying there’s a lot more to creating useful software than “tests pass / limited functionality checks work” from a purely technical perspective.
  - jmathai 8mo ago
    We haven’t fully proven that it is any different. Not at scale anyway. It took a decade for the seams of outsourcing to break. But I have a hypothesis. The quality of the output, when you don’t own the long term outcome or maintenance, is very poor. This is not the case with AI in the same sense it is with human contractors.
- throwup238 8mo ago
  It depends on the scale of complexity you’re working at and who your users are going to be. I’ve found that it’s trivial to have Claude Code spit out so much functionality that even just proper manually verifying it becomes a gargantuan task. I end up just manually testing the pieces I’m familiar with which is fine if there’s a QA department who can do a full run through of the feature and are prepared to deal with vibe coding pitfalls, but not so much on open source projects where slop gets shipped and unfamiliar users get stuck with bugs they can’t possibly troubleshoot. Writing the code from scratch The Old Way™ leaves a lot less room for shipping convincing but non functional slop because the dev has to work through it before shipping. The most immediate example I can think of is the beans LLM workflow tracker. It’s insane that its measured in the 100s of thousands of LoC and getting that thing setup in a repo is a mess. I had to use Github copilot to investigate the repo to get the latest method. This wouldn’t fly at my employer but a lot of projects are going to be a lot less scrupulous. You can see the effects in popular consumer facing apps too: Anthropic has drunk way too much of its own koolaid and now I get 10-50% failure rates on messages in their iOS app depending on the day. Some of their devs have publicly said that Claude writes 100% of their code and its starting to show. Intermittent network failures and retries have been a solved problem for decades, ffs!
jdjdjssh 8mo ago
Yep, this is the big sticking point. Reviewing code properly is and was the bottle neck. However, with humans I trusted, I could ignore most of their work and focus on where they knew they needed a review. That kind of trust is worth a lot of money and lets you move really fast. > I need time to sit with it Everyone knows doing the work yourself is faster than reviewing somebody elses if you don’t trust them. I’d argue if AI ever gets to the point where you fully trust it, all white collar jobs are gone.
CuriouslyC 8mo ago
You're countering vibes with vibes. If the tests aren't good enough, break them. Red team your own software. Exploit your systems. "Sitting with the code" is some Henry David Thoreau bullshit, because it provides exactly 0 value to anyone else, whereas red teamed exploits are objective.
- kranner 8mo ago
  The way you come up with ideas on how to break, red team and exploit; when to do this and how to stop: that part is not objective. The machine can't do this for you sufficiently well. There is a subjective process in there that you're not acknowledging. It's a good approach! It's just more 'negative space' than direct.
  - CuriouslyC 8mo ago
    People who pentest spend more time running a playbook than puzzling over the logical problem of how to break a piece of software. Even a lot of zero days are more about knowing a pattern and mass scanning for it across a lot of code than playing chess vs a codebase and winning.
    - kranner 8mo ago
      Fine, but is that the entirely of software development? It even seems a waste of time by your own reasoning if it's so automatable already.
- nkohari 8mo ago
  You're over-rotating on security. Not that it isn't important, but there are other dimensions to software that benefit heavily from the author having a deep understanding of the code that's being created.

by kranner 8mo ago

bravetraveler 8mo ago

I'll borrow your phrasing lest another coworker decide to play therapist. Previously: "I'm not slinging drugs here, I care for myself and the users."

torginus 8mo ago

I personally find that it's easier to iterate on 'something' and find what's wrong with it and fix it, than theorycrafting staring at a blank paper.

kranner 8mo ago
Me too! Sometimes I find it easier to ask an LLM to generate something and then it's easier to discard the whole thing, muttering under my breath "no no no that's not how you do this" and write my own code. It would be nice if editors would annotate generated code clearly.

slfreference 8mo ago

I think what LLMs do with words is similar to what artists do with software like cinema4d. We have control points (prompts + context) and we ask LLMs to draw a 3D surface which passes through those points satisfying some given constraints. Subsequent chats are like edit operations. https://youtu.be/-5S2qs32PII https://youtu.be/-5S2qs32PII

catdog 8mo ago
An LLM is an impressive, yet still imperfect and unpredictable translation machine. The code it outputs can only be as good as your prompt is precise, minus the often blatant mistakes it makes.

layer8 8mo ago

Yes, regression tests are not enough. One generally has to think through code repeatedly, with different aspects in mind, to convince oneself that it is correct under all circumstances. Tests only point-check, they don’t ensure correct behavior under all conceivable scenarios.

doug_durham 8mo ago
Unless you are in the business of writing flight control software, OS kernels, or critical financial software, I don't think your own code will reach the standards you mention. The only way we get "correct under all conceivable scenarios" software is to have a large team with long time horizons and large funding working on a small piece of software. It is beyond an individual to reach that standard for anything beyond code at the function level.

simianwords 8mo ago

Honest question: why is this not enough? If the code passes tests, and also works at the functionality level - what difference does it make if you’ve read the code or not? You could come up with pathological cases like: it passed the tests by deleting them. And the code written by it is extremely messy. But we know that LLMs are way smarter than this. There’s very very low chance of this happening and even if it does - it quick glance at code can fix it.

jdjdjssh 8mo ago
> If the code passes tests, and also works at the functionality level Why doesn’t outsourcing work if this is all that is needed?
- simianwords 8mo ago
  Why do we have managers if managers don’t have accountability?
  - jdjdjssh 8mo ago
    I’m not sure what you’re getting at. I’m saying there’s a lot more to creating useful software than “tests pass / limited functionality checks work” from a purely technical perspective.
- jmathai 8mo ago
  We haven’t fully proven that it is any different. Not at scale anyway. It took a decade for the seams of outsourcing to break. But I have a hypothesis. The quality of the output, when you don’t own the long term outcome or maintenance, is very poor. This is not the case with AI in the same sense it is with human contractors.
throwup238 8mo ago
It depends on the scale of complexity you’re working at and who your users are going to be. I’ve found that it’s trivial to have Claude Code spit out so much functionality that even just proper manually verifying it becomes a gargantuan task. I end up just manually testing the pieces I’m familiar with which is fine if there’s a QA department who can do a full run through of the feature and are prepared to deal with vibe coding pitfalls, but not so much on open source projects where slop gets shipped and unfamiliar users get stuck with bugs they can’t possibly troubleshoot. Writing the code from scratch The Old Way™ leaves a lot less room for shipping convincing but non functional slop because the dev has to work through it before shipping. The most immediate example I can think of is the beans LLM workflow tracker. It’s insane that its measured in the 100s of thousands of LoC and getting that thing setup in a repo is a mess. I had to use Github copilot to investigate the repo to get the latest method. This wouldn’t fly at my employer but a lot of projects are going to be a lot less scrupulous. You can see the effects in popular consumer facing apps too: Anthropic has drunk way too much of its own koolaid and now I get 10-50% failure rates on messages in their iOS app depending on the day. Some of their devs have publicly said that Claude writes 100% of their code and its starting to show. Intermittent network failures and retries have been a solved problem for decades, ffs!

jdjdjssh 8mo ago

Yep, this is the big sticking point. Reviewing code properly is and was the bottle neck. However, with humans I trusted, I could ignore most of their work and focus on where they knew they needed a review. That kind of trust is worth a lot of money and lets you move really fast. > I need time to sit with it Everyone knows doing the work yourself is faster than reviewing somebody elses if you don’t trust them. I’d argue if AI ever gets to the point where you fully trust it, all white collar jobs are gone.

CuriouslyC 8mo ago

You're countering vibes with vibes. If the tests aren't good enough, break them. Red team your own software. Exploit your systems. "Sitting with the code" is some Henry David Thoreau bullshit, because it provides exactly 0 value to anyone else, whereas red teamed exploits are objective.

kranner 8mo ago
The way you come up with ideas on how to break, red team and exploit; when to do this and how to stop: that part is not objective. The machine can't do this for you sufficiently well. There is a subjective process in there that you're not acknowledging. It's a good approach! It's just more 'negative space' than direct.
- CuriouslyC 8mo ago
  People who pentest spend more time running a playbook than puzzling over the logical problem of how to break a piece of software. Even a lot of zero days are more about knowing a pattern and mass scanning for it across a lot of code than playing chess vs a codebase and winning.
  - kranner 8mo ago
    Fine, but is that the entirely of software development? It even seems a waste of time by your own reasoning if it's so automatable already.
nkohari 8mo ago
You're over-rotating on security. Not that it isn't important, but there are other dimensions to software that benefit heavily from the author having a deep understanding of the code that's being created.