3 ms·

Summary: AWS has voluteered to serve as a crash test dummy for vibe coding. But don't tell anyone --- and if you do, don't blame AI because it's all the humans

Summary: AWS has voluteered to serve as a crash test dummy for vibe coding. But don't tell anyone --- and if you do, don't blame AI because it's all the humans fault for not shaping their questions in the "right way".

arjie 8mo ago
For this particular experiment, regardless of phrasing, I think the guys with the most appetite for risk have to be Cloudflare. They're shipping at an astonishing pace but I think there have been far more outages than there were before in jgc era. Perhaps Anthropic's application side teams are faster and more cowboy[0] but they are super AI-native so that makes sense. 0: I think this is the eras cowboys win so they're (unsurprisingly) smart about doing this
- Rohunyyy 8mo ago
  I am surprised we haven't had an actual Y2K crash with these AI codes. Like how do you review a 1000 lines of Claude generated PR?
  - krilcebre 8mo ago
    You don't. I can guarantee that 90% of the generated code will never receive a detailed review, simply because there's too much of a cognitive overhead, and too little time, everything moves too fast. I remember having to do such a code review before an AI in a highly complex component, and it would take a full day of work to do it. In this day and age, most of the people i know take like half an hour and are mostly scanning for obvious mistakes, where the bigger problem are those sneaky non obvious ones.
    - kakacik 8mo ago
      Exactly. Its same for reviewing somebody else's code. How many companies did this perfectly before llms came? I know mine didn't. But these days people that aren't senior enough do reviews of llm output, and do a quick mental path through the code, see the success and approve it. What could work - llm creating a very good test suite, for their own code changes and overall app (as much as feasible), and those tests need a hardcore review. Then actual code review doesn't have to be that deep. But if everybody is shipping like there is no tomorrow, edge cases will start biting hard and often.
  - CamperBob2 8mo ago
    You get Codex or Gemini to review it. Pro tip: tell it the code came from Claude. That will make it put its war face on.
bootsmann 8mo ago
This wouldn't happen if they used my CLAUDE.md of course!
blitzar 8mo ago
They were holding it wrong.

arjie 8mo ago

For this particular experiment, regardless of phrasing, I think the guys with the most appetite for risk have to be Cloudflare. They're shipping at an astonishing pace but I think there have been far more outages than there were before in jgc era. Perhaps Anthropic's application side teams are faster and more cowboy[0] but they are super AI-native so that makes sense. 0: I think this is the eras cowboys win so they're (unsurprisingly) smart about doing this

Rohunyyy 8mo ago
I am surprised we haven't had an actual Y2K crash with these AI codes. Like how do you review a 1000 lines of Claude generated PR?
- krilcebre 8mo ago
  You don't. I can guarantee that 90% of the generated code will never receive a detailed review, simply because there's too much of a cognitive overhead, and too little time, everything moves too fast. I remember having to do such a code review before an AI in a highly complex component, and it would take a full day of work to do it. In this day and age, most of the people i know take like half an hour and are mostly scanning for obvious mistakes, where the bigger problem are those sneaky non obvious ones.
  - kakacik 8mo ago
    Exactly. Its same for reviewing somebody else's code. How many companies did this perfectly before llms came? I know mine didn't. But these days people that aren't senior enough do reviews of llm output, and do a quick mental path through the code, see the success and approve it. What could work - llm creating a very good test suite, for their own code changes and overall app (as much as feasible), and those tests need a hardcore review. Then actual code review doesn't have to be that deep. But if everybody is shipping like there is no tomorrow, edge cases will start biting hard and often.
- CamperBob2 8mo ago
  You get Codex or Gemini to review it. Pro tip: tell it the code came from Claude. That will make it put its war face on.

bootsmann 8mo ago

This wouldn't happen if they used my CLAUDE.md of course!

blitzar 8mo ago

They were holding it wrong.