10 ms·

The coming knowledge-work supply-chain crisis

259 points by Stwerner 1y ago

roughly 1y ago
TFA is right to point out the bottleneck problem for reviewing content - there’s a couple things that compound to make this worse than it should be - The first is that the LLM outputs are not consistently good or bad - the LLM can put out 9 good MRs before the 10th one has some critical bug or architecture mistake. This means you need to be hypervigilant of everything the LLM produces, and you need to review everything with the kind of care with which you review intern contributions. The second is that the LLMs don’t learn once they’re done training, which means I could spend the rest of my life tutoring Claude and it’ll still make the exact same mistakes, which means I’ll never get a return for that time and hypervigilance like I would with an actual junior engineer. That problem leads to the final problem, which is that you need a senior engineer to vet the LLM’s code, but you don’t get to be a senior engineer without being the kind of junior engineer that the LLMs are replacing - there’s no way up that ladder except to climb it yourself. All of this may change in the next few years or the next iteration, but the systems as they are today are a tantalizing glimpse at an interesting future, not the actual present you can build on.
- ryandrake 1y ago
  > The first is that the LLM outputs are not consistently good or bad - the LLM can put out 9 good MRs before the 10th one has some critical bug or architecture mistake. This means you need to be hypervigilant of everything the LLM produces This, to me, is the critical and fatal flaw that prevents me from using or even being excited about LLMs: That they can be randomly, nondeterministically and confidently wrong, and there is no way to know without manually reviewing every output. Traditional computer systems whose outputs relied on probability solved this by including a confidence value next to any output. Do any LLMs do this? If not, why can't they? If they could, then the user would just need to pick a threshold that suits their peace of mind and review any outputs that came back below that threshold.
  - exe34 1y ago
    > Do any LLMs do this? If not, why can't they? If they could, then the user would just need to pick a threshold that suits their peace of mind and review any outputs that came back below that threshold. That's not how they work - they don't have internal models where they are sort of confident that this is a good answer. They have internal models where they are sort of confident that these tokens look like they were human generated in that order. So they can be very confident and still wrong. Knowing that confidence level (log p) would not help you assess. There are probabilistic models where they try to model a posterior distribution for the output - but that has to be trained in, with labelled samples. It's not clear how to do that for LLMs at the kind of scale that they require and affordably. You could consider letting it run code or try out things in simulations and use those as samples for further tuning, but at the moment, this might still lead them to forget something else or just make some other arbitrary and dumb mistake that they didn't make before the fine tuning.
  - bee_rider 1y ago
    What would those probabilities mean in the context of these modern LLMs? They are basically “try to continue the phrase like a human would” bots. I imagine the question of “how good of an approximation is this to something a human might write” could possibly be answerable. But humans often write things which are false. The entire universe of information consists of human writing, as far as the training process is concerned. Fictional stories and historical documents are equally “true” in that sense, right? Hmm, maybe somehow one could score outputs based on whether another contradictory output could be written? But it will have to be a little clever. Maybe somehow rank them by how specific they are? Like, a pair of reasonable contradictory sentences that can be written about the history-book setting indicate some controversy. A pair of contradictory sentences, one about history-book, one about Narnia, each equally real to the training set, but the fact that they contradict one another is not so interesting.
    - sepositus 1y ago
      > But humans often write things which are false. Not to mention, humans say things that make sense for humans to say and not a machine. For example, one recent case I saw was where the LLM hallucinated having a Macbook available that it was using to answer a question. In the context of a human, it was a totally viable response, but was total nonsense coming from an LLM.
- FeepingCreature 1y ago
  > The second is that the LLMs don’t learn once they’re done training, which means I could spend the rest of my life tutoring Claude and it’ll still make the exact same mistakes, which means I’ll never get a return for that time and hypervigilance like I would with an actual junior engineer. However, this creates a significant return on investment for opensourcing your LLM projects. In fact, you should commit your LLM dialogs along with your code. The LLM won't learn immediately, but it will learn in a few months when the next refresh comes out.
  - samjewell 1y ago
    > In fact, you should commit your LLM dialogs along with your code. Wholeheartedly agree with this. I think code review will evolve from "Review this code" to "Review this prompt that was used to generate some code"
    - roguecoder 1y ago
      All LLM output is non-deterministically wrong. Without a human in the loop who understands the code, you are stochastically releasing broken, insecure, unmaintainable software. Any software engineer who puts a stamp of approval on software they have not read and understood is committing professional malpractice.
  - anonzzzies 1y ago
    > In fact, you should commit your LLM dialogs along with your code. Absolutely, for different reasons including later reviews / visits to the code + prompts.
    - ithkuil 1y ago
      I wonder if some sort of summarization / gist of the course correction / teaching would work. For example Cursor has checked-in rules files and there is a way to have the model update the rules themselves based on the conversation
  - roguecoder 1y ago
- devnull3 1y ago
  > hypervigilant If a tech works 80% of the time, then I know that I need to be vigilant and I will review the output. The entire team structure is aware of this. There will be processes to offset this 20%. The problem is that when the AI becomes > 95% accurate (if at all) then humans will become complacent and the checks and balances will be ineffective.
  - Ferret7446 1y ago
    We are already there. The threshold is much closer to 80% for average people. For average folks, LLMs have rapidly went from "this is wrong and silly" to "this seems right most of the time so I just trust it when I search for info" in a few years.
    - philipwhiuk 1y ago
      It is frankly scary seeing novices adopt AI for stuff that you're good at and then hearing about the garbage it's come up with and then realising this problem is everywhere.
      - namaria 1y ago
        Gell-Mann amnesia. After I saw the subtle ways LLMs can off mark on things I know about, I am very wary to use it for any subject I don't dominate. I don't want to learn some plausible nonsense.
  - hnthrow90348765 1y ago
    80% is good enough for like the bottom 1/4th-1/3rd of software projects. That is way better than an offshore parasite company throwing stuff at the wall because they don't care about consistency or quality at all. These projects will bore your average HNer to death rather quickly (if not technically, then politically). Maybe people here are used to good code bases, so it doesn't make sense that 80% is good enough there, but I've seen some bad code bases (that still made money) that would be much easier to work on by not reinventing the wheel and not following patterns that are decades old and no one does any more.
- Havoc 1y ago
  That may be true but the cost of refactoring code that is wrong also plummets. So even if 9 out of 10 is wrong you can just can it.
  - roguecoder 1y ago
    Really? Because I've seen the opposite: the cost of fixing AI code is significantly higher than fixing human-generated code, because step one of refactoring is understanding the code and verifying the tests cover all the behavior. And AI doesn't produce code optimized for any human to read it. Even the worst programmer understands their own code, whereas AI produces code no human has ever understood.
- lubujackson 1y ago
  I used to think this about AI, that it will cause a a dearth of junior engineers. But I think it is really going to end up as a new level of abstraction. Aside from very specific bits of code, there is nothing AI does to remove any of the thinking work for me. So now I will sit down, reason through a problem, make a plan and... instead of punching code I write a prompt that punches the code. At the end of the day, AI can't tell us what to build or why to build it. So we will always need to know what we want to make or what ancillary things we need. LLMs can definitely support that, but knowing ALL the elements and gotchas is crucial. I don't think that removes the need for juniors, I think it simplifies what they need to know. Don't bother learning the intracacies of the language or optimization tricks or ORM details - the LLM will handle all that. But you certainly will need to know about catching errors and structuring projects and what needs testing, etc. So juniors will not be able to "look under the hood" very well but will come in learning to be a senior dev FIRST and a junior dev optionally. Not so different from the shift from everyone programming in C++ during the advent of PHP with "that's not really programming" complaints from the neckbeards. Doing this for 20 years and still haven't had to deal with malloc or pointers.
  - Joker_vD 1y ago
    The C++ compilers at least don't usually miscompile your source code. And when they do, it happens very rarely, mostly in obscure corners of the language, and it's kind of a big deal, and the compiler developers fix it. Compare to the large langle mangles, which somewhat routinely generate weird and wrong stuff, it's entirely unpredictable what inputs may trip it, it's not even reproducible, and nobody is expected to actually fix that. It just happens, use a second LLM to review the output of the first one or something. I'd rather have my lower-level abstractions be deterministic in a humanly-legible way. Otherwise in a generation or two we may very well end up being actual sorcerers who look for the right magical incantations to make the machine spirits obey their will.
- palmotea 1y ago
  > The first is that the LLM outputs are not consistently good or bad - the LLM can put out 9 good MRs before the 10th one has some critical bug or architecture mistake. This means you need to be hypervigilant of everything the LLM produces, and you need to review everything with the kind of care with which you review intern contributions. Also, people aren't meant to be hyper-vigilant in this way. Which is a big contradiction in the way contemporary AI is sold (LLMs, self-driving cars): they replace a relatively fun active task for humans (coding, driving) with a mind-numbing passive monitoring one that humans are actually terrible at. Is that making our lives better?
  - pjc50 1y ago
    Yup. This is exactly the same problem as the self-driving car. The tech is not 100% reliable. There are going to be incidents. When an incident happens, who takes the blame and what recourse is available? Does the corp using the AI simply eat the cost? See also https://www.londonreviewbookshop.co.uk/stock/the-unaccountability-machine-why-big-systems-make-terrible-decisions-dan-davies https://www.londonreviewbookshop.co.uk/stock/the-unaccountab...
- adrianN 1y ago
  When interns make mistakes they make human mistakes that are easier to catch for humans than the alien kind of mistake that llms make.
  - creshal 1y ago
    In my experience, LLMs aren't advanced enough for that, they just randomly add sql injections to code that otherwise uses proper prepared statements. We can get interns to stop doing that in one day.
- arkh 1y ago
  > you don’t get to be a senior engineer without being the kind of junior engineer that the LLMs are replacing I disagree: LLM are not replacing the kind of junior engineer who become senior ones. They replace "copy from StackOverflow until I get something mostly working" coders. Those who end going up the management ladder, not the engineering one. LLM are (atm) not replacing the junior engineers who use tools to get an idea then read the documentation.
  - Earw0rm 1y ago
    The best engineers and craftsmen - even juniors - understand which work is core to the craft and the end result, and which is peripheral. Unless you're hyper-specialised within a large organisation, you can't bring the same degree of obsession to every part in the process, there will always be edges. Even an artisan who hand-builds everything that matters may take some shortcuts in where they get their tools from, or the products they use to maintain them. In a big org, you might have a specialist for every domain, but on small teams you don't. And ultimately I've got other things to do with my life besides learning to write Cmake from scratch.
- overfeed 1y ago
  > That problem leads to the final problem, which is that you need a senior engineer to vet the LLM’s code, but you don’t get to be a senior engineer without being the kind of junior engineer that the LLMs are replacing - there’s no way up that ladder except to climb it yourself I suspect software will stumble into the strategy deployed by the big 4 Accounting firms and large law firms - have juniors have the first pass and have the changes filter upwards in seniority, with each layer adding comments and suggestions and sending it down to be corrected, until they are ready to sign-off on it. This will be inefficient amd wildly incompatible with agile practice, but that's one possible way for juniors to become mid-level, and eventually seniors after paying their dues. Its absolutely is inefficient in many ways, and is mostly incompatible with the current way of working as merge-sets have to be considered in a broader context all the time.
- cookiengineer 1y ago
  I wanted to add: The demographical shift over time will eventually lead to degradation of LLM performance, because more content will be of worse quality and transformers are a concept that loses symbolic inference. So, assuming that LLMs will increase in performance will only be true for the current generations of software engineers, whereas the next generations will lead automatically to worse LLM performance once they've replaced the demographic of the current seniors. Additionally, every knowledge resource that led to the current generation's advancements is dying out due to proprietarization. Courses, wikis, forums, tutorials... they all are now part of the enshittification cycle, which means that in the future they will contain less factual content per actual amount of content - which in return will also contribute to making LLM performance worse. Add to that the problems that come with such platforms, like the stackoverflow mod strikes or the ongoing reddit moderation crisis, and you got a recipe for Idiocracy. I decided to archive a copy of all books, courses, wikis and websites that led to my advancements in my career, so I have a backup of it. I encourage everyone to do the same. They might be worth a lot in the future, given how the trend is progressing.
- ineedasername 1y ago
  which means I could spend the rest of my life tutoring Claude and it’ll still make the exact same mistakes This is temporary. The new more global memory features in ChatGPT are a good example of how this is already starting to decrease as a factor. Yes it’s not quite the same as fine tuning or rlhf, but the impact is still similar, and I suspect that the toolong for end users or local tenant admins to easily create more sophisticated embeddings is going to increase very quickly.
- HelloMcFly 1y ago
  Agreed, and on this forum we tend to focus on the tech/coding aspects. But as a knowledge worker in a different domain, I can also tell you that the same issue is happening for other knowledge areas that are not as auditable without expertise. While we do see this problem when relying on junior knowledge workers, there seems to be a more implicit trust of LLM outputs vs. junior knowledge workers. Also: senior knowledge workers are also subject to errors, but knowledge work isn't always deterministic.
- HighGoldstein 1y ago
  > The first is that the LLM outputs are not consistently good or bad - the LLM can put out 9 good MRs before the 10th one has some critical bug or architecture mistake. This means you need to be hypervigilant of everything the LLM produces, and you need to review everything with the kind of care with which you review intern contributions. This is not a counter-argument, but this is true of any software engineer as well. Maybe for really good engineers it can be 1/100 or 1/1000 instead, but critical mistakes are inevitable.
kaycebasques 1y ago
This section heading from the post captures the key insight, is more focused, and is less hyperbolic: > Redesigning for Decision Velocity
nthingtohide 1y ago
> He argues this type of value judgement is something AI fundamentally cannot do, as it can only pattern match against existing decisions, not create new frameworks for assigning worth. Counterpoint : That decision has to be made only once (probably by some expert). AI can incorportate that training data into its reasoning and voila, it becomes available to everyone. A software framework is already a collection of good decisions, practices and tastes made by experts. > An MIT study found materials scientists experienced a 44% drop in job satisfaction when AI automated 57% of their “idea-generation” tasks Counterpoint : Now consider making material science decisions which requires materials to have not just 3 properties but 10 or 15. > Redesigning for Decision Velocity Suggestion : I think this section implies we must ask our experts to externalize all their tastes, preferences, top-down thinking so that other juniors can internalize those. So experts will be teaching details (based on their internal model) to LLMs while teaching the model itself to humans.
Animats 1y ago
> This pile of tasks is how I understand what Vaughn Tan refers to as Meaningmaking: the uniquely human ability to make subjective decisions about the relative value of things. Why is that a "uniquely human ability"? Machine learning systems are good at scoring things against some criterion. That's mostly how they work.
- atomicnumber3 1y ago
  How are the criterion chosen though? Something I learned from working alongside data scientists and financial analysts doing algo trading is that you can almost always find great fits for your criteria, nobody ever worries about that. Its coming up with the criteria that's what everyone frets over, and even more than that, you need to beat other people at doing so - just being good or event great isn't enough. Your profit is the delta between where you are compared to all the other sharks in your pool. So LLMs are useless there, getting token predicted answers is just going to get you the same as everyone else, which means zero alpha. So - I dunno about uniquely human? But there's definitely something here where, short of AGI, there's always going to need to be someone sitting down and actually beating the market (whatever that metaphor means for your industry or use case).
  - fwip 1y ago
    Finance is sort of a unique beast in that the field is inherently negative-sum. The profits you take home are always going to be profits somebody else isn't getting. If you're doing like, real work, solving problems in your domain actually adds value, and so the profits you get are from the value you provide.
    - kaashif 1y ago
      If you're algo trading then yes, which is what the person you're replying to is talking about. But "finance" is very broad and covers very real and valuable work like making loans and insurance - be careful not to be too broad in your condemnation.
      - 1y ago
jasonthorsness 1y ago
The method of producing the work can be more important (and easier to review) than the work output itself. Like at the simplest level of a global search-replace of a function name that alters 5000 lines. At a complex level, you can trust a team of humans to do something without micro-managing every aspect of their work. My hope is the current crises of reviewing too much AI-generated output will subside into the way you can trust the team because the LLM has reached a high level of “judgement” and competence. But we’re definitely not there yet. And contrary to the article, idea-generation with LLM support can be fun! They must have tested full replacement or something.
- wffurr 1y ago
  >> At a complex level, you can trust a team of humans to do something without micro-managing every aspect of their work I see you have never managed an outsourced project run by a body shop consultancy. They check the boxes you give them with zero thought or regard to the overall project and require significant micro managing to produce usable code.
  - jdlshore 1y ago
    I find this sort of whataboutism in LLM discussions tiring. Yes, of course, there are teams of humans that perform worse than an LLM. But it obvious to all but the most hype-blinded booster that it is possible for teams of humans to work autonomously to produce good results, because that is how all software has been produced to the present day, and some of it is good.
timewizard 1y ago
> Remember the first time an autocomplete suggestion nailed exactly what you meant to type? No. > Multiply that by a thousand and aim it at every task you once called “work.” If you mean "menial labor" then sure. The "work" I do is not at all aided by LLMs. > but our decision-making tools and rituals remain stuck in the past. That's because LLMs haven't eliminated or even significantly reduced risk. In fact they've created an entirely new category of risk in "hallucinations." > we need to rethink the entire production-to-judgment pipeline. Attempting to do this without accounting for risk or how capital is allocated into processes will lead you into folly. > We must reimagine knowledge work as a high-velocity decision-making operation rather than a creative production process. Then you will invent nothing new or novel and will be relegated to scraping by on the overpriced annotated databases of your direct competitors. The walled garden just raised the stakes. I can't believe people see a future in it.
shawn-butler 1y ago
This really isn’t true in principle. The current LLM ecosystems can’t do “meaning tasks” but there are all kinds of “legacy” AI expert systems that do exactly what is required. My experience is that middle manager gatekeepers are the most reluctant to participate in building knowledge systems that obsolete them though.
bendigedig 1y ago
Validating the outputs of a stochastic parrot sounds like a very alienating job.
- FeepingCreature 1y ago
  It's actually very fun, ime.
  - bendigedig 1y ago
    I have plenty of experience doing code reviews and to do a good job is pretty hard and thankless work. If I had to do that all day every day I'd be very unhappy.
    - chamomeal 1y ago
      It is definitely thankless work, at least at my company. It’d be even more thankless if instead of writing good feedback that somebody can learn from (or can spark interesting conversations that I can learn from), you would just said “nope GPT it’s not secure enough” and regenerate the whole PR, then read all the way through it again. Absolute tedium nightmare
- darth_avocado 1y ago
  As a staff engineer, it upsets me if my Review to Code ratio goes above 1. Days when I am not able to focus and code, because I was reviewing other people’s work all day, I usually am pretty drained but also unsatisfied. If the only job available to engineers becomes “review 50 PRs a day, everyday” I’ll probably quit software engineering altogether.
  - kmijyiyxfbklao 1y ago
    > As a staff engineer, it upsets me if my Review to Code ratio goes above 1. How does this work? Do you allow merging without reviews? Or are other engineers reviewing code way more than you?
xg15 1y ago
The intro sentence to this is quite funny. > Remember the first time an autocomplete suggestion nailed exactly what you meant to type? I actually don't, because so far this only happened with trivial phrases or text I had already typed in the past. I do remember however dozens of times where autocorrect wrongly "corrected" the last word I typed, changing an easy to spot typo into a much more subtle semantic error.
- thechao 1y ago
  I see these sorts of statements from coders who, you know, aren't good programmers in the first place. Here's the secret that I that I think LLM's are uncovering: I think there's a lot of really shoddy coders out there; coders who could could/would never become good programmers and they are absolutely going to be replaced with LLMs. I don't know how I feel about that. I suspect it's not going to be great for society. Replacing blue collar workers for robots hasn't been super duper great.
  - rowanajmarshall 1y ago
    > Replacing blue collar workers for robots hasn't been super duper great. That's just not true. Tractors, combine harvesters, dishwashers washing machines, excavators, we've repeatedly revolutionised blue-collar work, made it vastly, extraordinary more efficient.
    - vineyardmike 1y ago
      > made it vastly, extraordinary more efficient. I'd suspect that these equipments also made it more dangerous. They also made it more industrial in scale and capital costs, driving "homestead" and individual farmers out of the business, replaced by larger and more capitalized corporations. We went from individual artisans crafting fabrics by hand, to the Industrial Revolution where children lost fingers tending to "extraordinary more efficient" machines that vastly out-produced artisans. This trend has only accelerated, where humans consume and throw out an order of magnitude more clothing than a generation ago. You can see this trend play out across industrialized jobs - people are less satisfied, there is some social implications, and the entire nature of the job (and usually the human's independence) is changed. The transitions through industrialization have had dramatic societal upheavals. Focusing on the "efficiency" of the changes, ironically, miss the human component of these transitions.
delusional 1y ago
If the AIs learned from us they'll only be able to produce Coca Cola and ads, so the interety of the actually valuable economy is safe.
eezurr 1y ago
And once the Orient and Decide part is augmented, then we'll be limited by social networks (IRL ones). Every solo founder/small biz will have to compete more and more for marketing eyeballs, and the ones who have access to bigger engines (companies), they'll get the juice they need, and we come back to humans being the bottlenecks again. That is, until we mutually decide on removing our agency from the loop entirely . And then what?
- thrwyep 1y ago
  I think less people will decide to open source their work, so AI solutions will divert from 'dark codebases' not available for models to be trained on. And people who love vibe coding will keep feeding models with code produced by models. Maybe we already reached the point where enough knowledge was locked in the models and this does not matter? I think not, based on code AI generated for me. I probably ask wrong questions.
zkmon 1y ago
> Ultimately, I don’t see AI completely replacing knowledge workers any time soon. How was that conclusion reached? And what is meant by knowledge workers? Any work with knowledge is exactly the domain of LLMs. So, LLMs are indeed knowledge workers.
- dredmorbius 1y ago
  Hijacking comment to reply on a flagged thread: email concerns/questions to HN mods at hn@ycombinator.com. This is a FAQ: <https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html> Re: <https://news.ycombinator.com/item?id=43829917 https://news.ycombinator.com/item?id=43829917>
exmicrosoldier 1y ago
This is the same problem as outsourcing to third party programmers in another country, but worse.
- stego-tech 1y ago
  It really, really is at present. It’s outsourcing but without the benefit of someone getting a paycheck: all exploitation.
causal 1y ago
A few articles like this have hit the front page, and something about them feels really superficial to me, and I'm trying to put my finger on why. Perhaps it's just that it's so myopically focused on day 2 and not on day n. They extrapolate from ways AI can replace humans right now, but lack any calculus which might integrate second or third order effects that such economic changes will incur, and so give the illusion that next year will be business as usual but with AI doing X and humans doing Y.
- danielmarkbruce 1y ago
  Why: they assume that humans have some secret sauce. Like... judgement...we don't. Once you extrapolate, yes, many things will be very very different.
- creesch 1y ago
  Maybe it is the fact that they blatantly paint a picture of AI doing flawless production work where the only "bottleneck" is us puny humans needing to review stuff. It exemplifies this race to the bottom where everything needs to be hyperefficient and time to market needs to be even lower. Which, once you stop to think about it, is insane. There is a complete lack of asking why. To In fact, when you boil it down to its core argument it isn't even about AI at all. It is effectively the same grumblings from management layers heard for decades now where they feel (emphasis) that their product development is slowed down by those pesky engineers and other specialists making things too complex, etc. But now just framed around AI with unrealistic expectations dialed up.
  - lotsofpulp 1y ago
    > There is a complete lack of asking why The answer to this seems obvious to me. Buyers seek the lowest price, so sellers are incentivized to cut their cost of goods sold. Investors seek the highest return on investment (people prefer more purchasing power than less purchasing power), so again, businesses are incentivized to cut their cost of goods sold. The opposing force to this is buyers prefer higher quality to lower quality. The tradeoff between these parameters is in constant flux.
  - 1y ago
joshdavham 1y ago
> What I see happening is us not being prepared for how AI transforms the nature of knowledge work and us having a very painful and slow transition into this new era. I would've liked for the author to be a bit specific here. What exactly could this "very painful and slow transition" look like? Any commenters have any idea? I'm genuinely curious.
RevEng 1y ago
The article rightly points out that people don't enjoy just being reviewers: we like to take an active role in playing, learning, and creating. They point out the need to find a solution to this, but then never follow up on that idea. This is perhaps the most fundamental problem. In the past, tools took care of the laborious and tedious work so we could focus on creativity. Now we are letting AI do the creative work and asking humans to become managers and code reviewers. Maybe that's great for some people, but it's not what most problem solvers want to be doing. The same people who know how to judge such things are the same people who have years of experience doing this things. Without that experience you can't have good judgement. Let the AI make it faster and easier for me to create; don't make it replace what I do best and leave me as a manager and code reviewer. The parallels with grocery checkouts are worth considering. Humans are great at recognizing things, handling unexpected situations, and being friendly and personable. People working checkouts are experts at these things. Now replace that with self serve checkouts. Random customers are forced to do this all themselves. They are not experts at this. The checkouts are less efficient because they have to accommodate these non-experts. People have to pack their own bags. And they do all of this while punching buttons on a soulless machine instead of getting some social interaction in. But worse off is the employee who manages these checkouts. Now instead of being social, they are security guards and tech support. They are constantly having to shoot the computer issues and teach disinterested and frustrated beginners how to do something that should be so simple. The employee spends most of their time as a manager and watchdog, looking at a screen that shows the status of all the checkouts, looking for issues, like a prison security guard. This work is inactive and unengaging, requiring constant attention - something humans aren't good at. When little they do interact with others, it is in situations where that are upset. We didn't automate anything here, we just changed who does what. We made customers into the people doing checkouts and we made more level staff into managers of them, plus being tech support. This is what companies are trying to do with AI. They want to have fewer employees whose job it is to manage the AIs, directing them to produce. The human is left assigning tasks and checking the results - managers of thankless and soulless machines. The credit for the creation goes to the machines while the employees are seen as low skilled and replaceable. And we end up back at the start: trying to find high skilled people to perform low skilled work based on experience that they only would have had if they had being doing high skilled work to begin with. When everyone is just managing an AI, no one will know what it is supposed to do.
ozim 1y ago
My observation over the years as a software dev was that velocity is overrated. Mostly because all kinds of systems are made for humans - even if we as a dev team were able to pump out features we got pushed back. Exactly because users had to be trained, users would have to be migrated all kinds of things would have to be documented and accounted for that were tangential to main goals. So bottleneck is a feature not a bug. I can see how we should optimize away documentation and tangential stuff so it would happen automatically but not the main job where it needs more thought anyway.
- charlie0 1y ago
  This is my observation as well, especially in startups. So much spaghetti thrown at walls and that pressure falls in devs to have higher velocity when it should fall on product, sales, exec to actually make better decisions.
- jaimebuelta 1y ago
  Good iteration process is really important. Just throwing things faster into the wall doesn't help if you don't pause to check which one sticks, or, even worst, you are not even able to know which one sticks. That's a very human reflective process that requires time.
- ebiester 1y ago
  I'm really working hard to figure out how to optimize away documentation, but it's seemingly harder than writing the code. It's easier to generate the code from the documentation than the documentation from the code, reference documentation (like OpenAPI docs) aside.
d4rkn0d3z 1y ago
AI increases our ability to produce bullshit but doesn't do much to increase our ability to detect bullshit. One sentence of bullshit takes 1000 sentences of clear reasoning to dispel.
staunton 1y ago
What's going to happen is that LLMs will eventually make fewer mistakes, and then people will just put up with more bugs in almost all situations, leading to everything being noticably worse, and build everything with robustness in mind, not correctness. But it will all be cheaper so there you go.
HPsquared 1y ago
There are so, so many person-years spent studying. And it's not enough? Everyone wanting to work in "knowledge work" does 16 years of schooling, very often more. How inefficient are we that this still apparently isn't enough?
pjmorris 1y ago
Excerpted from Tony Hoare's 1980 Turing Award speech, 'The Emperor's Old Clothes'... "At last, there breezed into my office the most senior manager of all, a general manager of our parent company, Andrew St. Johnston. I was surprised that he had even heard of me. "You know what went wrong?" he shouted--he always shouted-- "You let your programmers do things which you yourself do not understand." I stared in astonishment. He was obviously out of touch with present day realities. How could one person ever understand the whole of a modern software product like the Elliott 503 Mark II software system? I realized later that he was absolutely right; he had diagnosed the true cause of the problem and he had planted the seed of its later solution." My interpretation is that whether shifting from delegation to programmers, or to compilers, or to LLMs, the invariant is that we will always have to understand the consequences of our choices, or suffer the consequences.
hamuraijack 1y ago
Is it just me, or is vibe coding only useful for greenfield projects that have minimal complexity? Seems like they collapse once enough complexity has built up.
- fhd2 1y ago
  I've tried to vibe code small stuff a few times, but there's not one success story. After about 2-4 hours, I'd hit a wall, and ultimately threw it away, because it wasn't worth the manual programming effort it would have required (which is why I tried vibe coding it in the first place). I think vibe coding might be more successful for people doing things an experienced developer can do in their sleep with a few lines of code in Django or something. Something a non programmer might have previously done with some no code tool.
slippybit 1y ago
[dead]
bccdee 1y ago
> AI is scaling the creation side of knowledge work at an exponential rate Why do people keep saying things like this? "Exponential rate"? That's just not true. So far the benefits are marginal at best and limited to relatively simple tasks. It's a truism at this point, even among fans of AI, that the benefits of AI are much more pronounced at junior-level tasks. For complex work, I'm not convinced that AI has "scaled the creation side of knowledge work" at all. I don't think it's particularly useful for the kind of non-trivial tasks that actually take up our time. Amdahl's Law comes into play. If using AI gives you 200% efficiency on trivial tasks, but trivial tasks only take 10% of your time, then you've realized a whopping 5.3% productivity boost. I do not actually spend much time on boilerplate. I spend time debugging half-baked code, i.e. the stuff that LLMs spit out. I realize I'm complaining about the third sentence of the article, but I refuse to keep letting people make claims like this as if they're obviously true. The whole article is based on false premises.
0xWTF 1y ago
The article may not be consistent with what I'm hearing from doctors using ambient dictation, which admittedly fits a slightly different niche than the author's use case, but points to their final prediction that the paths to adoption will be complicated. A number of the docs I'm working with describe using ambient dictation as a game changer. Using the OODA loop analogy of the author: they are tightening the full OODA loop by deferring documentation to the end of the day. Historically this was a disaster because they'd forget the first patient by the end of the day. Now, the first patient's automatically dictated note is perhaps wrong but rich with details the spark sufficient remembrance. Of course MBAs will use this to further crush physicians with additional workload, but for a time, it may help.
omneity 1y ago
I would like to challenge the fundamental premise of the article. Just because you can generate 50 PRs doesn’t mean you should. In fact the same bottleneck they’re describing is present if you have 50 coders in your team. The problem therefore is not how to scale PR review and rather how to select meaningful work to perform, which brings me to the second point made by TFA: humans making judgment calls on whhich PR should be prioritized being a uniquely defining human feature. I beg to differ here as well. All the problems described in the article are high context decisions, you need to take a lot in consideration (user request, product strategy, market dynamics, cost/benefit, rou..) to decide which feature should be prioritized in the next release. What prevents LLMs from being able to help with that is the sheer amount of information to ingest which is a still a limitation despite the long context windows we see nowadays. tl;dr: this is a problem of prioritization and product strategy and nothing specific to AI. Scaling so-called judgment is a red herring and better focus and scope management should be aimed for instead.

259 points by Stwerner 1y ago

TFA is right to point out the bottleneck problem for reviewing content - there’s a couple things that compound to make this worse than it should be - The first is that the LLM outputs are not consistently good or bad - the LLM can put out 9 good MRs before the 10th one has some critical bug or architecture mistake. This means you need to be hypervigilant of everything the LLM produces, and you need to review everything with the kind of care with which you review intern contributions. The second is that the LLMs don’t learn once they’re done training, which means I could spend the rest of my life tutoring Claude and it’ll still make the exact same mistakes, which means I’ll never get a return for that time and hypervigilance like I would with an actual junior engineer. That problem leads to the final problem, which is that you need a senior engineer to vet the LLM’s code, but you don’t get to be a senior engineer without being the kind of junior engineer that the LLMs are replacing - there’s no way up that ladder except to climb it yourself. All of this may change in the next few years or the next iteration, but the systems as they are today are a tantalizing glimpse at an interesting future, not the actual present you can build on.

ryandrake 1y ago
> The first is that the LLM outputs are not consistently good or bad - the LLM can put out 9 good MRs before the 10th one has some critical bug or architecture mistake. This means you need to be hypervigilant of everything the LLM produces This, to me, is the critical and fatal flaw that prevents me from using or even being excited about LLMs: That they can be randomly, nondeterministically and confidently wrong, and there is no way to know without manually reviewing every output. Traditional computer systems whose outputs relied on probability solved this by including a confidence value next to any output. Do any LLMs do this? If not, why can't they? If they could, then the user would just need to pick a threshold that suits their peace of mind and review any outputs that came back below that threshold.
- exe34 1y ago
  > Do any LLMs do this? If not, why can't they? If they could, then the user would just need to pick a threshold that suits their peace of mind and review any outputs that came back below that threshold. That's not how they work - they don't have internal models where they are sort of confident that this is a good answer. They have internal models where they are sort of confident that these tokens look like they were human generated in that order. So they can be very confident and still wrong. Knowing that confidence level (log p) would not help you assess. There are probabilistic models where they try to model a posterior distribution for the output - but that has to be trained in, with labelled samples. It's not clear how to do that for LLMs at the kind of scale that they require and affordably. You could consider letting it run code or try out things in simulations and use those as samples for further tuning, but at the moment, this might still lead them to forget something else or just make some other arbitrary and dumb mistake that they didn't make before the fine tuning.
- bee_rider 1y ago
  What would those probabilities mean in the context of these modern LLMs? They are basically “try to continue the phrase like a human would” bots. I imagine the question of “how good of an approximation is this to something a human might write” could possibly be answerable. But humans often write things which are false. The entire universe of information consists of human writing, as far as the training process is concerned. Fictional stories and historical documents are equally “true” in that sense, right? Hmm, maybe somehow one could score outputs based on whether another contradictory output could be written? But it will have to be a little clever. Maybe somehow rank them by how specific they are? Like, a pair of reasonable contradictory sentences that can be written about the history-book setting indicate some controversy. A pair of contradictory sentences, one about history-book, one about Narnia, each equally real to the training set, but the fact that they contradict one another is not so interesting.
  - sepositus 1y ago
    > But humans often write things which are false. Not to mention, humans say things that make sense for humans to say and not a machine. For example, one recent case I saw was where the LLM hallucinated having a Macbook available that it was using to answer a question. In the context of a human, it was a totally viable response, but was total nonsense coming from an LLM.
FeepingCreature 1y ago
> The second is that the LLMs don’t learn once they’re done training, which means I could spend the rest of my life tutoring Claude and it’ll still make the exact same mistakes, which means I’ll never get a return for that time and hypervigilance like I would with an actual junior engineer. However, this creates a significant return on investment for opensourcing your LLM projects. In fact, you should commit your LLM dialogs along with your code. The LLM won't learn immediately, but it will learn in a few months when the next refresh comes out.
- samjewell 1y ago
  > In fact, you should commit your LLM dialogs along with your code. Wholeheartedly agree with this. I think code review will evolve from "Review this code" to "Review this prompt that was used to generate some code"
  - roguecoder 1y ago
    All LLM output is non-deterministically wrong. Without a human in the loop who understands the code, you are stochastically releasing broken, insecure, unmaintainable software. Any software engineer who puts a stamp of approval on software they have not read and understood is committing professional malpractice.
- anonzzzies 1y ago
  > In fact, you should commit your LLM dialogs along with your code. Absolutely, for different reasons including later reviews / visits to the code + prompts.
  - ithkuil 1y ago
    I wonder if some sort of summarization / gist of the course correction / teaching would work. For example Cursor has checked-in rules files and there is a way to have the model update the rules themselves based on the conversation
- roguecoder 1y ago
devnull3 1y ago
> hypervigilant If a tech works 80% of the time, then I know that I need to be vigilant and I will review the output. The entire team structure is aware of this. There will be processes to offset this 20%. The problem is that when the AI becomes > 95% accurate (if at all) then humans will become complacent and the checks and balances will be ineffective.
- Ferret7446 1y ago
  We are already there. The threshold is much closer to 80% for average people. For average folks, LLMs have rapidly went from "this is wrong and silly" to "this seems right most of the time so I just trust it when I search for info" in a few years.
  - philipwhiuk 1y ago
    It is frankly scary seeing novices adopt AI for stuff that you're good at and then hearing about the garbage it's come up with and then realising this problem is everywhere.
    - namaria 1y ago
      Gell-Mann amnesia. After I saw the subtle ways LLMs can off mark on things I know about, I am very wary to use it for any subject I don't dominate. I don't want to learn some plausible nonsense.
- hnthrow90348765 1y ago
  80% is good enough for like the bottom 1/4th-1/3rd of software projects. That is way better than an offshore parasite company throwing stuff at the wall because they don't care about consistency or quality at all. These projects will bore your average HNer to death rather quickly (if not technically, then politically). Maybe people here are used to good code bases, so it doesn't make sense that 80% is good enough there, but I've seen some bad code bases (that still made money) that would be much easier to work on by not reinventing the wheel and not following patterns that are decades old and no one does any more.
Havoc 1y ago
That may be true but the cost of refactoring code that is wrong also plummets. So even if 9 out of 10 is wrong you can just can it.
- roguecoder 1y ago
  Really? Because I've seen the opposite: the cost of fixing AI code is significantly higher than fixing human-generated code, because step one of refactoring is understanding the code and verifying the tests cover all the behavior. And AI doesn't produce code optimized for any human to read it. Even the worst programmer understands their own code, whereas AI produces code no human has ever understood.
lubujackson 1y ago
I used to think this about AI, that it will cause a a dearth of junior engineers. But I think it is really going to end up as a new level of abstraction. Aside from very specific bits of code, there is nothing AI does to remove any of the thinking work for me. So now I will sit down, reason through a problem, make a plan and... instead of punching code I write a prompt that punches the code. At the end of the day, AI can't tell us what to build or why to build it. So we will always need to know what we want to make or what ancillary things we need. LLMs can definitely support that, but knowing ALL the elements and gotchas is crucial. I don't think that removes the need for juniors, I think it simplifies what they need to know. Don't bother learning the intracacies of the language or optimization tricks or ORM details - the LLM will handle all that. But you certainly will need to know about catching errors and structuring projects and what needs testing, etc. So juniors will not be able to "look under the hood" very well but will come in learning to be a senior dev FIRST and a junior dev optionally. Not so different from the shift from everyone programming in C++ during the advent of PHP with "that's not really programming" complaints from the neckbeards. Doing this for 20 years and still haven't had to deal with malloc or pointers.
- Joker_vD 1y ago
  The C++ compilers at least don't usually miscompile your source code. And when they do, it happens very rarely, mostly in obscure corners of the language, and it's kind of a big deal, and the compiler developers fix it. Compare to the large langle mangles, which somewhat routinely generate weird and wrong stuff, it's entirely unpredictable what inputs may trip it, it's not even reproducible, and nobody is expected to actually fix that. It just happens, use a second LLM to review the output of the first one or something. I'd rather have my lower-level abstractions be deterministic in a humanly-legible way. Otherwise in a generation or two we may very well end up being actual sorcerers who look for the right magical incantations to make the machine spirits obey their will.
palmotea 1y ago
> The first is that the LLM outputs are not consistently good or bad - the LLM can put out 9 good MRs before the 10th one has some critical bug or architecture mistake. This means you need to be hypervigilant of everything the LLM produces, and you need to review everything with the kind of care with which you review intern contributions. Also, people aren't meant to be hyper-vigilant in this way. Which is a big contradiction in the way contemporary AI is sold (LLMs, self-driving cars): they replace a relatively fun active task for humans (coding, driving) with a mind-numbing passive monitoring one that humans are actually terrible at. Is that making our lives better?
- pjc50 1y ago
  Yup. This is exactly the same problem as the self-driving car. The tech is not 100% reliable. There are going to be incidents. When an incident happens, who takes the blame and what recourse is available? Does the corp using the AI simply eat the cost? See also https://www.londonreviewbookshop.co.uk/stock/the-unaccountability-machine-why-big-systems-make-terrible-decisions-dan-davies https://www.londonreviewbookshop.co.uk/stock/the-unaccountab...
adrianN 1y ago
When interns make mistakes they make human mistakes that are easier to catch for humans than the alien kind of mistake that llms make.
- creshal 1y ago
  In my experience, LLMs aren't advanced enough for that, they just randomly add sql injections to code that otherwise uses proper prepared statements. We can get interns to stop doing that in one day.
arkh 1y ago
> you don’t get to be a senior engineer without being the kind of junior engineer that the LLMs are replacing I disagree: LLM are not replacing the kind of junior engineer who become senior ones. They replace "copy from StackOverflow until I get something mostly working" coders. Those who end going up the management ladder, not the engineering one. LLM are (atm) not replacing the junior engineers who use tools to get an idea then read the documentation.
- Earw0rm 1y ago
  The best engineers and craftsmen - even juniors - understand which work is core to the craft and the end result, and which is peripheral. Unless you're hyper-specialised within a large organisation, you can't bring the same degree of obsession to every part in the process, there will always be edges. Even an artisan who hand-builds everything that matters may take some shortcuts in where they get their tools from, or the products they use to maintain them. In a big org, you might have a specialist for every domain, but on small teams you don't. And ultimately I've got other things to do with my life besides learning to write Cmake from scratch.
overfeed 1y ago
> That problem leads to the final problem, which is that you need a senior engineer to vet the LLM’s code, but you don’t get to be a senior engineer without being the kind of junior engineer that the LLMs are replacing - there’s no way up that ladder except to climb it yourself I suspect software will stumble into the strategy deployed by the big 4 Accounting firms and large law firms - have juniors have the first pass and have the changes filter upwards in seniority, with each layer adding comments and suggestions and sending it down to be corrected, until they are ready to sign-off on it. This will be inefficient amd wildly incompatible with agile practice, but that's one possible way for juniors to become mid-level, and eventually seniors after paying their dues. Its absolutely is inefficient in many ways, and is mostly incompatible with the current way of working as merge-sets have to be considered in a broader context all the time.
cookiengineer 1y ago
I wanted to add: The demographical shift over time will eventually lead to degradation of LLM performance, because more content will be of worse quality and transformers are a concept that loses symbolic inference. So, assuming that LLMs will increase in performance will only be true for the current generations of software engineers, whereas the next generations will lead automatically to worse LLM performance once they've replaced the demographic of the current seniors. Additionally, every knowledge resource that led to the current generation's advancements is dying out due to proprietarization. Courses, wikis, forums, tutorials... they all are now part of the enshittification cycle, which means that in the future they will contain less factual content per actual amount of content - which in return will also contribute to making LLM performance worse. Add to that the problems that come with such platforms, like the stackoverflow mod strikes or the ongoing reddit moderation crisis, and you got a recipe for Idiocracy. I decided to archive a copy of all books, courses, wikis and websites that led to my advancements in my career, so I have a backup of it. I encourage everyone to do the same. They might be worth a lot in the future, given how the trend is progressing.
ineedasername 1y ago
which means I could spend the rest of my life tutoring Claude and it’ll still make the exact same mistakes This is temporary. The new more global memory features in ChatGPT are a good example of how this is already starting to decrease as a factor. Yes it’s not quite the same as fine tuning or rlhf, but the impact is still similar, and I suspect that the toolong for end users or local tenant admins to easily create more sophisticated embeddings is going to increase very quickly.
HelloMcFly 1y ago
Agreed, and on this forum we tend to focus on the tech/coding aspects. But as a knowledge worker in a different domain, I can also tell you that the same issue is happening for other knowledge areas that are not as auditable without expertise. While we do see this problem when relying on junior knowledge workers, there seems to be a more implicit trust of LLM outputs vs. junior knowledge workers. Also: senior knowledge workers are also subject to errors, but knowledge work isn't always deterministic.
HighGoldstein 1y ago
> The first is that the LLM outputs are not consistently good or bad - the LLM can put out 9 good MRs before the 10th one has some critical bug or architecture mistake. This means you need to be hypervigilant of everything the LLM produces, and you need to review everything with the kind of care with which you review intern contributions. This is not a counter-argument, but this is true of any software engineer as well. Maybe for really good engineers it can be 1/100 or 1/1000 instead, but critical mistakes are inevitable.

kaycebasques 1y ago

This section heading from the post captures the key insight, is more focused, and is less hyperbolic: > Redesigning for Decision Velocity

nthingtohide 1y ago

> He argues this type of value judgement is something AI fundamentally cannot do, as it can only pattern match against existing decisions, not create new frameworks for assigning worth. Counterpoint : That decision has to be made only once (probably by some expert). AI can incorportate that training data into its reasoning and voila, it becomes available to everyone. A software framework is already a collection of good decisions, practices and tastes made by experts. > An MIT study found materials scientists experienced a 44% drop in job satisfaction when AI automated 57% of their “idea-generation” tasks Counterpoint : Now consider making material science decisions which requires materials to have not just 3 properties but 10 or 15. > Redesigning for Decision Velocity Suggestion : I think this section implies we must ask our experts to externalize all their tastes, preferences, top-down thinking so that other juniors can internalize those. So experts will be teaching details (based on their internal model) to LLMs while teaching the model itself to humans.

Animats 1y ago

> This pile of tasks is how I understand what Vaughn Tan refers to as Meaningmaking: the uniquely human ability to make subjective decisions about the relative value of things. Why is that a "uniquely human ability"? Machine learning systems are good at scoring things against some criterion. That's mostly how they work.

atomicnumber3 1y ago
How are the criterion chosen though? Something I learned from working alongside data scientists and financial analysts doing algo trading is that you can almost always find great fits for your criteria, nobody ever worries about that. Its coming up with the criteria that's what everyone frets over, and even more than that, you need to beat other people at doing so - just being good or event great isn't enough. Your profit is the delta between where you are compared to all the other sharks in your pool. So LLMs are useless there, getting token predicted answers is just going to get you the same as everyone else, which means zero alpha. So - I dunno about uniquely human? But there's definitely something here where, short of AGI, there's always going to need to be someone sitting down and actually beating the market (whatever that metaphor means for your industry or use case).
- fwip 1y ago
  Finance is sort of a unique beast in that the field is inherently negative-sum. The profits you take home are always going to be profits somebody else isn't getting. If you're doing like, real work, solving problems in your domain actually adds value, and so the profits you get are from the value you provide.
  - kaashif 1y ago
    If you're algo trading then yes, which is what the person you're replying to is talking about. But "finance" is very broad and covers very real and valuable work like making loans and insurance - be careful not to be too broad in your condemnation.
    - 1y ago

jasonthorsness 1y ago

The method of producing the work can be more important (and easier to review) than the work output itself. Like at the simplest level of a global search-replace of a function name that alters 5000 lines. At a complex level, you can trust a team of humans to do something without micro-managing every aspect of their work. My hope is the current crises of reviewing too much AI-generated output will subside into the way you can trust the team because the LLM has reached a high level of “judgement” and competence. But we’re definitely not there yet. And contrary to the article, idea-generation with LLM support can be fun! They must have tested full replacement or something.

wffurr 1y ago
>> At a complex level, you can trust a team of humans to do something without micro-managing every aspect of their work I see you have never managed an outsourced project run by a body shop consultancy. They check the boxes you give them with zero thought or regard to the overall project and require significant micro managing to produce usable code.
- jdlshore 1y ago
  I find this sort of whataboutism in LLM discussions tiring. Yes, of course, there are teams of humans that perform worse than an LLM. But it obvious to all but the most hype-blinded booster that it is possible for teams of humans to work autonomously to produce good results, because that is how all software has been produced to the present day, and some of it is good.

timewizard 1y ago

> Remember the first time an autocomplete suggestion nailed exactly what you meant to type? No. > Multiply that by a thousand and aim it at every task you once called “work.” If you mean "menial labor" then sure. The "work" I do is not at all aided by LLMs. > but our decision-making tools and rituals remain stuck in the past. That's because LLMs haven't eliminated or even significantly reduced risk. In fact they've created an entirely new category of risk in "hallucinations." > we need to rethink the entire production-to-judgment pipeline. Attempting to do this without accounting for risk or how capital is allocated into processes will lead you into folly. > We must reimagine knowledge work as a high-velocity decision-making operation rather than a creative production process. Then you will invent nothing new or novel and will be relegated to scraping by on the overpriced annotated databases of your direct competitors. The walled garden just raised the stakes. I can't believe people see a future in it.

shawn-butler 1y ago

This really isn’t true in principle. The current LLM ecosystems can’t do “meaning tasks” but there are all kinds of “legacy” AI expert systems that do exactly what is required. My experience is that middle manager gatekeepers are the most reluctant to participate in building knowledge systems that obsolete them though.

bendigedig 1y ago

Validating the outputs of a stochastic parrot sounds like a very alienating job.

FeepingCreature 1y ago
It's actually very fun, ime.
- bendigedig 1y ago
  I have plenty of experience doing code reviews and to do a good job is pretty hard and thankless work. If I had to do that all day every day I'd be very unhappy.
  - chamomeal 1y ago
    It is definitely thankless work, at least at my company. It’d be even more thankless if instead of writing good feedback that somebody can learn from (or can spark interesting conversations that I can learn from), you would just said “nope GPT it’s not secure enough” and regenerate the whole PR, then read all the way through it again. Absolute tedium nightmare
darth_avocado 1y ago
As a staff engineer, it upsets me if my Review to Code ratio goes above 1. Days when I am not able to focus and code, because I was reviewing other people’s work all day, I usually am pretty drained but also unsatisfied. If the only job available to engineers becomes “review 50 PRs a day, everyday” I’ll probably quit software engineering altogether.
- kmijyiyxfbklao 1y ago
  > As a staff engineer, it upsets me if my Review to Code ratio goes above 1. How does this work? Do you allow merging without reviews? Or are other engineers reviewing code way more than you?

xg15 1y ago

The intro sentence to this is quite funny. > Remember the first time an autocomplete suggestion nailed exactly what you meant to type? I actually don't, because so far this only happened with trivial phrases or text I had already typed in the past. I do remember however dozens of times where autocorrect wrongly "corrected" the last word I typed, changing an easy to spot typo into a much more subtle semantic error.

thechao 1y ago
I see these sorts of statements from coders who, you know, aren't good programmers in the first place. Here's the secret that I that I think LLM's are uncovering: I think there's a lot of really shoddy coders out there; coders who could could/would never become good programmers and they are absolutely going to be replaced with LLMs. I don't know how I feel about that. I suspect it's not going to be great for society. Replacing blue collar workers for robots hasn't been super duper great.
- rowanajmarshall 1y ago
  > Replacing blue collar workers for robots hasn't been super duper great. That's just not true. Tractors, combine harvesters, dishwashers washing machines, excavators, we've repeatedly revolutionised blue-collar work, made it vastly, extraordinary more efficient.
  - vineyardmike 1y ago
    > made it vastly, extraordinary more efficient. I'd suspect that these equipments also made it more dangerous. They also made it more industrial in scale and capital costs, driving "homestead" and individual farmers out of the business, replaced by larger and more capitalized corporations. We went from individual artisans crafting fabrics by hand, to the Industrial Revolution where children lost fingers tending to "extraordinary more efficient" machines that vastly out-produced artisans. This trend has only accelerated, where humans consume and throw out an order of magnitude more clothing than a generation ago. You can see this trend play out across industrialized jobs - people are less satisfied, there is some social implications, and the entire nature of the job (and usually the human's independence) is changed. The transitions through industrialization have had dramatic societal upheavals. Focusing on the "efficiency" of the changes, ironically, miss the human component of these transitions.

delusional 1y ago

If the AIs learned from us they'll only be able to produce Coca Cola and ads, so the interety of the actually valuable economy is safe.

eezurr 1y ago

And once the Orient and Decide part is augmented, then we'll be limited by social networks (IRL ones). Every solo founder/small biz will have to compete more and more for marketing eyeballs, and the ones who have access to bigger engines (companies), they'll get the juice they need, and we come back to humans being the bottlenecks again. That is, until we mutually decide on removing our agency from the loop entirely . And then what?

thrwyep 1y ago
I think less people will decide to open source their work, so AI solutions will divert from 'dark codebases' not available for models to be trained on. And people who love vibe coding will keep feeding models with code produced by models. Maybe we already reached the point where enough knowledge was locked in the models and this does not matter? I think not, based on code AI generated for me. I probably ask wrong questions.

zkmon 1y ago

> Ultimately, I don’t see AI completely replacing knowledge workers any time soon. How was that conclusion reached? And what is meant by knowledge workers? Any work with knowledge is exactly the domain of LLMs. So, LLMs are indeed knowledge workers.

dredmorbius 1y ago
Hijacking comment to reply on a flagged thread: email concerns/questions to HN mods at hn@ycombinator.com. This is a FAQ: <https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html> Re: <https://news.ycombinator.com/item?id=43829917 https://news.ycombinator.com/item?id=43829917>

exmicrosoldier 1y ago

This is the same problem as outsourcing to third party programmers in another country, but worse.

stego-tech 1y ago
It really, really is at present. It’s outsourcing but without the benefit of someone getting a paycheck: all exploitation.

causal 1y ago

A few articles like this have hit the front page, and something about them feels really superficial to me, and I'm trying to put my finger on why. Perhaps it's just that it's so myopically focused on day 2 and not on day n. They extrapolate from ways AI can replace humans right now, but lack any calculus which might integrate second or third order effects that such economic changes will incur, and so give the illusion that next year will be business as usual but with AI doing X and humans doing Y.

danielmarkbruce 1y ago
Why: they assume that humans have some secret sauce. Like... judgement...we don't. Once you extrapolate, yes, many things will be very very different.
creesch 1y ago
Maybe it is the fact that they blatantly paint a picture of AI doing flawless production work where the only "bottleneck" is us puny humans needing to review stuff. It exemplifies this race to the bottom where everything needs to be hyperefficient and time to market needs to be even lower. Which, once you stop to think about it, is insane. There is a complete lack of asking why. To In fact, when you boil it down to its core argument it isn't even about AI at all. It is effectively the same grumblings from management layers heard for decades now where they feel (emphasis) that their product development is slowed down by those pesky engineers and other specialists making things too complex, etc. But now just framed around AI with unrealistic expectations dialed up.
- lotsofpulp 1y ago
  > There is a complete lack of asking why The answer to this seems obvious to me. Buyers seek the lowest price, so sellers are incentivized to cut their cost of goods sold. Investors seek the highest return on investment (people prefer more purchasing power than less purchasing power), so again, businesses are incentivized to cut their cost of goods sold. The opposing force to this is buyers prefer higher quality to lower quality. The tradeoff between these parameters is in constant flux.
- 1y ago

joshdavham 1y ago

> What I see happening is us not being prepared for how AI transforms the nature of knowledge work and us having a very painful and slow transition into this new era. I would've liked for the author to be a bit specific here. What exactly could this "very painful and slow transition" look like? Any commenters have any idea? I'm genuinely curious.

RevEng 1y ago

The article rightly points out that people don't enjoy just being reviewers: we like to take an active role in playing, learning, and creating. They point out the need to find a solution to this, but then never follow up on that idea. This is perhaps the most fundamental problem. In the past, tools took care of the laborious and tedious work so we could focus on creativity. Now we are letting AI do the creative work and asking humans to become managers and code reviewers. Maybe that's great for some people, but it's not what most problem solvers want to be doing. The same people who know how to judge such things are the same people who have years of experience doing this things. Without that experience you can't have good judgement. Let the AI make it faster and easier for me to create; don't make it replace what I do best and leave me as a manager and code reviewer. The parallels with grocery checkouts are worth considering. Humans are great at recognizing things, handling unexpected situations, and being friendly and personable. People working checkouts are experts at these things. Now replace that with self serve checkouts. Random customers are forced to do this all themselves. They are not experts at this. The checkouts are less efficient because they have to accommodate these non-experts. People have to pack their own bags. And they do all of this while punching buttons on a soulless machine instead of getting some social interaction in. But worse off is the employee who manages these checkouts. Now instead of being social, they are security guards and tech support. They are constantly having to shoot the computer issues and teach disinterested and frustrated beginners how to do something that should be so simple. The employee spends most of their time as a manager and watchdog, looking at a screen that shows the status of all the checkouts, looking for issues, like a prison security guard. This work is inactive and unengaging, requiring constant attention - something humans aren't good at. When little they do interact with others, it is in situations where that are upset. We didn't automate anything here, we just changed who does what. We made customers into the people doing checkouts and we made more level staff into managers of them, plus being tech support. This is what companies are trying to do with AI. They want to have fewer employees whose job it is to manage the AIs, directing them to produce. The human is left assigning tasks and checking the results - managers of thankless and soulless machines. The credit for the creation goes to the machines while the employees are seen as low skilled and replaceable. And we end up back at the start: trying to find high skilled people to perform low skilled work based on experience that they only would have had if they had being doing high skilled work to begin with. When everyone is just managing an AI, no one will know what it is supposed to do.

ozim 1y ago

My observation over the years as a software dev was that velocity is overrated. Mostly because all kinds of systems are made for humans - even if we as a dev team were able to pump out features we got pushed back. Exactly because users had to be trained, users would have to be migrated all kinds of things would have to be documented and accounted for that were tangential to main goals. So bottleneck is a feature not a bug. I can see how we should optimize away documentation and tangential stuff so it would happen automatically but not the main job where it needs more thought anyway.

charlie0 1y ago
This is my observation as well, especially in startups. So much spaghetti thrown at walls and that pressure falls in devs to have higher velocity when it should fall on product, sales, exec to actually make better decisions.
jaimebuelta 1y ago
Good iteration process is really important. Just throwing things faster into the wall doesn't help if you don't pause to check which one sticks, or, even worst, you are not even able to know which one sticks. That's a very human reflective process that requires time.
ebiester 1y ago
I'm really working hard to figure out how to optimize away documentation, but it's seemingly harder than writing the code. It's easier to generate the code from the documentation than the documentation from the code, reference documentation (like OpenAPI docs) aside.

d4rkn0d3z 1y ago

AI increases our ability to produce bullshit but doesn't do much to increase our ability to detect bullshit. One sentence of bullshit takes 1000 sentences of clear reasoning to dispel.

staunton 1y ago

What's going to happen is that LLMs will eventually make fewer mistakes, and then people will just put up with more bugs in almost all situations, leading to everything being noticably worse, and build everything with robustness in mind, not correctness. But it will all be cheaper so there you go.

HPsquared 1y ago

There are so, so many person-years spent studying. And it's not enough? Everyone wanting to work in "knowledge work" does 16 years of schooling, very often more. How inefficient are we that this still apparently isn't enough?

pjmorris 1y ago

Excerpted from Tony Hoare's 1980 Turing Award speech, 'The Emperor's Old Clothes'... "At last, there breezed into my office the most senior manager of all, a general manager of our parent company, Andrew St. Johnston. I was surprised that he had even heard of me. "You know what went wrong?" he shouted--he always shouted-- "You let your programmers do things which you yourself do not understand." I stared in astonishment. He was obviously out of touch with present day realities. How could one person ever understand the whole of a modern software product like the Elliott 503 Mark II software system? I realized later that he was absolutely right; he had diagnosed the true cause of the problem and he had planted the seed of its later solution." My interpretation is that whether shifting from delegation to programmers, or to compilers, or to LLMs, the invariant is that we will always have to understand the consequences of our choices, or suffer the consequences.

hamuraijack 1y ago

Is it just me, or is vibe coding only useful for greenfield projects that have minimal complexity? Seems like they collapse once enough complexity has built up.

fhd2 1y ago
I've tried to vibe code small stuff a few times, but there's not one success story. After about 2-4 hours, I'd hit a wall, and ultimately threw it away, because it wasn't worth the manual programming effort it would have required (which is why I tried vibe coding it in the first place). I think vibe coding might be more successful for people doing things an experienced developer can do in their sleep with a few lines of code in Django or something. Something a non programmer might have previously done with some no code tool.

slippybit 1y ago

[dead]

bccdee 1y ago

> AI is scaling the creation side of knowledge work at an exponential rate Why do people keep saying things like this? "Exponential rate"? That's just not true. So far the benefits are marginal at best and limited to relatively simple tasks. It's a truism at this point, even among fans of AI, that the benefits of AI are much more pronounced at junior-level tasks. For complex work, I'm not convinced that AI has "scaled the creation side of knowledge work" at all. I don't think it's particularly useful for the kind of non-trivial tasks that actually take up our time. Amdahl's Law comes into play. If using AI gives you 200% efficiency on trivial tasks, but trivial tasks only take 10% of your time, then you've realized a whopping 5.3% productivity boost. I do not actually spend much time on boilerplate. I spend time debugging half-baked code, i.e. the stuff that LLMs spit out. I realize I'm complaining about the third sentence of the article, but I refuse to keep letting people make claims like this as if they're obviously true. The whole article is based on false premises.

0xWTF 1y ago

The article may not be consistent with what I'm hearing from doctors using ambient dictation, which admittedly fits a slightly different niche than the author's use case, but points to their final prediction that the paths to adoption will be complicated. A number of the docs I'm working with describe using ambient dictation as a game changer. Using the OODA loop analogy of the author: they are tightening the full OODA loop by deferring documentation to the end of the day. Historically this was a disaster because they'd forget the first patient by the end of the day. Now, the first patient's automatically dictated note is perhaps wrong but rich with details the spark sufficient remembrance. Of course MBAs will use this to further crush physicians with additional workload, but for a time, it may help.

omneity 1y ago

I would like to challenge the fundamental premise of the article. Just because you can generate 50 PRs doesn’t mean you should. In fact the same bottleneck they’re describing is present if you have 50 coders in your team. The problem therefore is not how to scale PR review and rather how to select meaningful work to perform, which brings me to the second point made by TFA: humans making judgment calls on whhich PR should be prioritized being a uniquely defining human feature. I beg to differ here as well. All the problems described in the article are high context decisions, you need to take a lot in consideration (user request, product strategy, market dynamics, cost/benefit, rou..) to decide which feature should be prioritized in the next release. What prevents LLMs from being able to help with that is the sheer amount of information to ingest which is a still a limitation despite the long context windows we see nowadays. tl;dr: this is a problem of prioritization and product strategy and nothing specific to AI. Scaling so-called judgment is a red herring and better focus and scope management should be aimed for instead.