4 ms·

Organizations now generate 10x the amount of code, because everyone can do it. But we have exactly the same number of reviewers. How the heck are we gonna deal

Organizations now generate 10x the amount of code, because everyone can do it. But we have exactly the same number of reviewers. How the heck are we gonna deal with it when we cannot use LLMs for sanity checking LLM code? Like literally yesterday I had a not-technical person who used codex to build an optimization algorithm, and due to the momentum it gained I was asked to “fix the rough edges and help with scaling”. The entire thing was trash (was trying to do naive search in a combinatorial problem with 1000s of integers, and was violating constraints with high probability, including the integrality). I had to spend all my day reviewing it and make a technical presentation to their leadership that it is just a polished turd.

LPisGood 1y ago
The problem can probably almost certainly be solved to provable optimality using HiGHS or even CBC - the open source Python package PuLP comes with CBC. If you want to be seen as the hero who solves things instead of the realist who says why other solutions won’t work, this could be worth exploring.
- whatever1 1y ago
  Of course it can be solved using the proper tools by a domain expert/practitioner (even with chatGPT since the expert will know what to ask). But why didn't the AI expert solve it using chatGPT? If it has to land to an expert for reimplementation from scratch after wasting a day on reviewing slop, did we gain productivity?
  - LPisGood 1y ago
    Gain on productivity, maybe nothing. I didn’t mean to comment on the broader scope, just an actionable suggestion for GP.
lezojeda 1y ago
It's basically a scheme letting less scrupulous people offload work on others while at the same time making good impressions to manager, PMs and CEOs who want to automate our job and fire us.
- simgt 1y ago
  PMs, POs and CEOs should have been the first jobs to automate in our industry. Nothing of what the vast majority of them do is based on evidence and verifiable facts, it's already all vibes anyway.
A4ET8a8uTh0_v2 1y ago
<< and make a technical presentation to their leadership Honestly, this may be the only way to go about it.
Jianghong94 1y ago
OR problems are hard because whoever try to vibe coding it probably don't realize they fall into a specific algorithm and can prompt llm to do thatl; what's worse is that even if you tell them so they won't be able to understand the math behind it and would much prefer their vide coding solution.
crazygringo 1y ago
> How the heck are we gonna deal with it when we cannot use LLMs for sanity checking LLM code? Unit testing. LLM's are very good at writing tests and writing code that is testable (as long as you ask it), and if you just check that the tests are actually calling the code and doing so with all the obvious edge cases and that the results are correct, that's actually quite fast to review -- faster than reviewing the code. And you can include things like performance testing in tests as well. We're moving to a world where we work with definitions and tests and are less concerned with the precise details of how code is written within functions. Which is a big shift in mindset.
- verelo 1y ago
  I’ve been criticized for this by my coworkers in the past, but I strongly believe that this is generally true and has been for quite a while. Developers, myself included, like to think their code is special, set in stone and going to last forever. Most the code we write struggles to live a few years yet we treat all of it like it’s going to last forever. I’ve been an advocate for flipping that and treating it like our code will not last long, and when we identify the components that will, going back and optimizing them. I’m pretty confident that most developers, again including myself, just really enjoy knowing something is done well. Being able to separate yourself from the code and fixate solely on the outcomes can sometimes get me past this.
  - lemagedurage 1y ago
    I think this is true for the edges, but if you build on top of software that's not done well, it's a bad time.
    - verelo 1y ago
      Sadly i find most software I am building on top of is pretty awful...but i'm working in the real estate world right now, so that is unavoidable.
- ori_b 1y ago
  > Unit testing. LLM's are very good at writing tests and writing code that is testable (as long as you ask it) The unit tests LLMs generate are also often crap, testing tautologies, making sure that your dependencies act as specified without testing the actual code, etc.
jasonthorsness 1y ago
"But we have exactly the same number of reviewers." LLMs can help with reviews as well. LLMs are not too bad at reviewing code; GPT 5 for example can find off-by-one, missed returns, all sorts of problems that are localized. I think they have a harder time with issues requiring a higher-level global understanding. I wonder if in the future you could fine-tune an LLM on a big codebase (maybe nightly or something) and it could be the first-level reviewer for all changes to that codebase.
noosphr 1y ago
The same way that we dealt with Excel programming. Ignore it until it blows up, then spend hundreds of thousands trying to fix it before the company goes bankrupt.

LPisGood 1y ago

The problem can probably almost certainly be solved to provable optimality using HiGHS or even CBC - the open source Python package PuLP comes with CBC. If you want to be seen as the hero who solves things instead of the realist who says why other solutions won’t work, this could be worth exploring.

whatever1 1y ago
Of course it can be solved using the proper tools by a domain expert/practitioner (even with chatGPT since the expert will know what to ask). But why didn't the AI expert solve it using chatGPT? If it has to land to an expert for reimplementation from scratch after wasting a day on reviewing slop, did we gain productivity?
- LPisGood 1y ago
  Gain on productivity, maybe nothing. I didn’t mean to comment on the broader scope, just an actionable suggestion for GP.

lezojeda 1y ago

It's basically a scheme letting less scrupulous people offload work on others while at the same time making good impressions to manager, PMs and CEOs who want to automate our job and fire us.

simgt 1y ago
PMs, POs and CEOs should have been the first jobs to automate in our industry. Nothing of what the vast majority of them do is based on evidence and verifiable facts, it's already all vibes anyway.

A4ET8a8uTh0_v2 1y ago

<< and make a technical presentation to their leadership Honestly, this may be the only way to go about it.

Jianghong94 1y ago

OR problems are hard because whoever try to vibe coding it probably don't realize they fall into a specific algorithm and can prompt llm to do thatl; what's worse is that even if you tell them so they won't be able to understand the math behind it and would much prefer their vide coding solution.

crazygringo 1y ago

> How the heck are we gonna deal with it when we cannot use LLMs for sanity checking LLM code? Unit testing. LLM's are very good at writing tests and writing code that is testable (as long as you ask it), and if you just check that the tests are actually calling the code and doing so with all the obvious edge cases and that the results are correct, that's actually quite fast to review -- faster than reviewing the code. And you can include things like performance testing in tests as well. We're moving to a world where we work with definitions and tests and are less concerned with the precise details of how code is written within functions. Which is a big shift in mindset.

verelo 1y ago
I’ve been criticized for this by my coworkers in the past, but I strongly believe that this is generally true and has been for quite a while. Developers, myself included, like to think their code is special, set in stone and going to last forever. Most the code we write struggles to live a few years yet we treat all of it like it’s going to last forever. I’ve been an advocate for flipping that and treating it like our code will not last long, and when we identify the components that will, going back and optimizing them. I’m pretty confident that most developers, again including myself, just really enjoy knowing something is done well. Being able to separate yourself from the code and fixate solely on the outcomes can sometimes get me past this.
- lemagedurage 1y ago
  I think this is true for the edges, but if you build on top of software that's not done well, it's a bad time.
  - verelo 1y ago
    Sadly i find most software I am building on top of is pretty awful...but i'm working in the real estate world right now, so that is unavoidable.
ori_b 1y ago
> Unit testing. LLM's are very good at writing tests and writing code that is testable (as long as you ask it) The unit tests LLMs generate are also often crap, testing tautologies, making sure that your dependencies act as specified without testing the actual code, etc.

jasonthorsness 1y ago

"But we have exactly the same number of reviewers." LLMs can help with reviews as well. LLMs are not too bad at reviewing code; GPT 5 for example can find off-by-one, missed returns, all sorts of problems that are localized. I think they have a harder time with issues requiring a higher-level global understanding. I wonder if in the future you could fine-tune an LLM on a big codebase (maybe nightly or something) and it could be the first-level reviewer for all changes to that codebase.

noosphr 1y ago

The same way that we dealt with Excel programming. Ignore it until it blows up, then spend hundreds of thousands trying to fix it before the company goes bankrupt.