9 ms·

AI and the ironies of automation – Part 2

256 points by BinaryIgor 10mo ago

jennyholzer 10mo ago
[dead]
z_ 10mo ago
This is a thought provoking piece. “But at what cost?” We’ve all accepted calculators into our lives as being faster and correct when utilized correctly (Minus Intel tomfoolery), but we emphasize the need to know how to do the math in educational settings. Any post education adult will confirm when confronted with an irregular math problem (or a skill) that there is a wait time to revive the ability. Programming automation having the potential skill decay AND being critical path is … worth thinking about.
- xorcist 10mo ago
  Comparisons with deterministic tools such as calculators will always lead astray. There is no comparable situation where faced with a new problem the AI will just give up. If there is the need for an expert, the need is always there, because there is no indication external to the process that the process will fail.
  - alex989 10mo ago
    I disagree about how calculators and math are deterministic a in real world scenarios where you use math at work. When you compute a formula in your calculator or in a fancy design software, it will always give you answer but it doesn't mean you asked the right question. If you use the wrong units in your input or if you make a typo, if you used the wrong formula, etc., the calculator/software will blindly give you an answer and only an experienced engineer will spot it a first glance. As soon as there is a human in the loop, things get messy. For exemple, if your calculator tells you that a 15m long W200x31 steel beam can resist 215kN•m in bending moment, I know at first glance its at least 4x too much for that length, but how many people reading my comment could? A civil engineer fresh out of college would not.
- singpolyma3 10mo ago
  Calculators don't do math, they do calculating. Which is to say, they don't think for you. There's not much value in being able to quickly compute some expression in a world with calculators. But there's a huge value in knowing how to know which numbers to feed into the calculation.
  - kurthr 10mo ago
    The biggest problem with calculators (rather than slide rules), was that because calculations with big numbers (large mantissa) were so easy, people got used to doing them that way without consideration. Using a slide rule meant inherently knowing order-of-magnitude, rounding, and precision. Once calculators make it easy they enable both new kinds of solutions and new kinds of errors (that you have to separately teach to avoid). At the same time, I basically agree. Humans are very bad calculators and we've needed tools (abacus) for millennia.
  - bitwize 10mo ago
    I derive tremendous value from being able to calculate taxes, tips, and so forth in my head, or right on the receipt, without having to reach for my phone and launch Droid48. (I know some of y'all are also Droid48 bros.) It's even more profound a convenience than knowing how to drive Emacs with just the keyboard and not having to reach for the goddamn mouse.
    - agumonkey 10mo ago
      we need to form a group of intrisics people who enjoyed knowing and learning in depth, not just apply to sell something
- eastbound 10mo ago
  We already have generational programming decay. At 25 years old, kids fresh out of uni can’t write a string.contains() routine. They all use .stream() in Java. Matter of generation, fashion and skills to learn. And concerning the programming of C drivers, Apple is the last company to write a filesystem and they already can’t find anyone able to do it.
nuancebydefault 10mo ago
The article discusses basically 2 new problems with using agentic AI: - When one of the agents does something wrong, a human operator needs to be able to intervene quickly and needs to provide the agent with expert instructions. However since experts do not execute the bare tasks anymore, they forget parts of their expertise quickly. This means the experts need constant training, hence they will have little time left to oversee the agent's work. - Experts must become managers of agentic systems, a role which they are not familiar with, hence they are not feeling at home in their job. This problem is harder to be determined as a problem by people managers (of the experts) since they don't experience that problem often first hand. Indeed the irony is that AI provides efficiency gains, which as they become more widely adopted, become more problematic because they outfit the necessary human in the loop. I think this all means that automation is not taking away everyone's job, as it makes things more complicated and hence humans can still compete.
- jennyholzer 10mo ago
  [dead]
- DiscourseFan 10mo ago
  That's how it tends to go, automation removes some parts of the work but creates more complexity. Sooner or later that will also be automated away, and so on and so forth. AGI evangelists ought to read Marx's Capital.
  - jennyholzer2 10mo ago
    I seriously doubt that there is even one "AGI evangelist" who has the intellectual capacity to read books written for adult audiences.
    - ctoth 10mo ago
      Hi. I am not an evangelist -- I'm quite certain it's going to kill us all! But I would like to think that I'm about the closest thing to an AI booster you might find here, given that I get so much damn utility out of it. I'm interested in reading, I probably read too much! would you like to suggest a book we can discuss next week? I'd be happy to do this with you.
jennyholzer2 10mo ago
[flagged]
- TheOtherHobbes 10mo ago
  Not necessarily. It depends if the process is deterministic and repeatable. If an AI generates a process more quickly than a human, and the process can be run deterministically, and the outputs are testable, then the process can run without direct human supervision after initial testing - which is how most automated processes work. The testing should happen anyway, so any speed increase in process generation is a productivity gain. Human monitoring only matters if the AI is continually improvising new solutions to dynamic problems and the solutions are significantly wrong/unreliable. Which is a management/analysis problem, and no different in principle to managing a team. The key difference in practice is that you can hire and fire people on a team, you can intervene to change goals and culture, and you can rearrange roles. With an agentic workflow you can change the prompts, use different models, and redesign the flow. But your choices are more constrained.
  - lkjdsklf 10mo ago
    The issue is LLMs are, by design, non-deterministic. That means that, with the current technology, there can never be a deterministic agent. Now obviously, humans aren't deterministic either, but the error bars are a lot closer together than they are with LLMs these days. An easy to point at example is the coding agent that removed someones home directory that was circulating around. I'm not saying a human has never done that, but it's far less likely because it's so far out of the realm of normal operations. So as of today, we need humans in the loop. And this is understood by the people making these products. That's why they have all these permissions and prompts for you to accept/run commands and all of that.
    - loa_in_ 10mo ago
      There's lots of _marketing_ promising unsupervised agents. It's important to remember not to drink the cool-aid.
    - 10mo ago
sublimefire 10mo ago
Good discussion of the paper and the observations and ironies. A thing to note is that we do have software factories already, with a bunch of automation in place and folks being trained to deal with incidents. The pools of agents just elevate what we currently have but the tools are still lacking severely. IMO the tools need to improve for us to move forward as it is difficult to observe the decisions of agents when they fall apart. Also, by and large the current AI tools are not in the critical path yet, well except those drones that lock on targets to eliminate them in case of interference, and even then it is ML. Agents can not be in that path due to predictability challenges yet.
wesammikhail 10mo ago
Our of curiosity, does anyone know of a good writeup / blog post made by someone in the industry that revolves around reducing orchestration error rates? Would love to read some more about the topic and I'm looking for a few good resources.
- dloranc 10mo ago
  What do you mean by orchestration?
everdrive 10mo ago
I can feel the skill atrophy creeping in. My very first instinct is go use the LLM. I think much like forcing yourself to exercise, eat right, and avoid social media / distractions, this will be a new modern skillset; do you have the discipline to avoid becoming useless without an LLM? A small few will be great at this, the middle of the bell curve will do "well enough," and you know the story for the rest.
- andy99 10mo ago
  I’ve been using LLMs to code for some time and I look at it differently. I ask myself if I need to understand the code, and if the answer is yes I don’t use an LLM. It’s not a matter of discipline, it’s a sober view of what the minimal amount of work for me is.
  - layer8 10mo ago
    The only time one doesn’t need to understand the code is when it doesn’t matter if the code is correct, or when it can be tested exhaustively for all possible inputs. Both are pretty rare for me.
    - kaffekaka 10mo ago
      I largely agree. But sometimes the program is not destructive and you only need to test for inputs that may/will actually occur. The LLM wrote a script to do some processing? Just test it, if the processing is fine, done. I have many LLM-written scripts and tools to do some semi simple jobs where I have barely even looked at the code because I could see immediately that the job I wanted to do got done.
- delaminator 10mo ago
  I haven't written any code in 6 months. But I can still remember how to code in 6502 machine code from the 1980s.
  - zeroonetwothree 10mo ago
ripe 10mo ago
I really like this author's summary of the 1983 Bainbridge paper about industrial automation. I have often wondered how to apply those insights to AI agents, but I was never able to summarize it as well as OP. Bainbridge by itself is a tough paper to read because it's so dense. It's just four pages long and worth following along: https://ckrybus.com/static/papers/Bainbridge_1983_Automatica.pdf https://ckrybus.com/static/papers/Bainbridge_1983_Automatica... For example, see this statement in the paper: "the present generation of automated systems, which are monitored by former manual operators, are riding on their skills, which later generations of operators cannot be expected to have." This summarizes the first irony of automation, which is now familiar to everyone on HN: using AI agents effectively requires an expert programmer, but to build the skills to be an expert programmer, you have to program yourself. It's full of insights like that. Highly recommended!
- startupsfail 10mo ago
  The same argument was there about needing to be an expert programmer in assembly language to use C, and then same for C and Python, and then Python and CUDA, and then Theano/Tensorflow/Pytorch. And yet here we are, able to talk to a computer, that writes Pytorch code that orchestrates the complexity below it. And even talks back coherently sometimes.
  - gipp 10mo ago
    Those are completely deterministic systems, of bounded scope. They can be ~completely solved, in the sense that all possible inputs fall within the understood and always correctly handled bounds of the system's specifications. There's no need for ongoing, consistent human verification at runtime. Any problems with the implementation can wait for a skilled human to do whatever research is necessary to develop the specific system understanding needed to fix it. This is really not a valid comparison.
    - startupsfail 10mo ago
      There are enormous microcode, firmware and drivers blobs everywhere on any pathway. Even with very privileged access of someone at Intel or NVIDIA, ability to have a reasonable level of deterministic control of systems that involve CPU/GPU/LAN were long gone, almost for a decade now.
jinwoo68 10mo ago
"Most companies are efficiency-obsessed." But what most of them do is not to be more efficient but to be shown to be more efficient. The main reason they are so obsessed with AI is because they want to send the signal that they are pursuing to be more efficient, whether they succeed or not.
- theologic 10mo ago
  Peter Drucker popularized the phrase "Efficiency is doing things right; effectiveness is doing the right things." Being a credibly efficient at doing the wrong things, turns out to be a massive issue inside of most companies. What's interesting is I do think that AI gives opportunity to be massively more effective because if you have the right LLM, that's trained right, you can explore a variety of scenarios much faster than what you can do by yourself. However, we hear very little about this as a central thrust of how to utilize AI into the work space.
  - jjk166 10mo ago
    In my experience plenty of places are quite inefficient at doing the wrong things as well. You might think this reduces the number of wrong things done, but somehow it doesn't.
    - theologic 10mo ago
      It's almost comical isn't it, but it actually turns out that this is a big foundation behind behavioral economics. In essence you can get trapped in an upper level heuristic and never stop for a moment and thinks things through. Another one of my favorite examples is that there is some research out of Harvard that basically suggested that if people would take and spend 15 minutes a day reviewing what they had done and what was important, they increased their productivity 22%. Now you would think that this is so obvious and so dramatic you would have variety of Fortune 500 companies saying "oh my goodness we want all of our workers to be 22% more productive" and so they would simply send out a memo or an email or some sort of process to force people to do some reflecting. I would also suggest that Microsoft had a unique advantage based out of the idea that people should have their own enclosed workspace to do coding. This was deeply entrenched when Bill was running the company day-to-day. And I'm sure as somebody that was a coding phenomenon, it simply made sense to him. But academically, it also makes sense. Microsoft has reversed this policy, but as far as I can tell, it doesn't have anything to do with the research. It has to do with statements about working together efficiently. or AI productivity. If there's real research then it's great. My problem is it just doesn't appear to be any real research behind it. Yet I'm sure many managers at Microsoft thinks that it's very efficient. Of course, if you do know anybody at Microsoft that codes, they have their own opinion, and rather than me repeating hearsay, it would be fantastic to have somebody anonymously post what's really going on here. I'll betcha a nickel that 90% of them are not reporting that they feel a lot more effective.
jiehong 10mo ago
This irony of automation has been dealt with in the aviation industry for pilot for years: auto pilots can actually land the plane in many cases, and do fly the plane on most of the cruise. Yet, pilots are constantly trained on actual scenarios, and are expected to land airplanes manually monthly (and during take off too). This ensures pilots maintain their skills, while the auto pilot helps most of the time. On top of that, plane commands often are half automatic already, aka they are assisted (but not by LLMs!), so it’s a complex comparison.
- libraryofbabel 10mo ago
  Yes, but (to write the second half of your post for you!) regulation and incentives are very different in the aviation industry, because safety and planning for long-tail risks is paramount. Therefore airlines can afford to have their pilots spend thousands of hours training on manual control in various scenarios. By contrast, I don’t think the average software development org will encourage its engineers to hand-roll a sizable proportion of their code, if (still a big if) there are major productivity costs in doing so. Rushing the Next Big Feature out the door will almost always beat out long-term investment in dev training, unfortunately. Don’t get me wrong - manual practice is in some sense the correct solution, and I plan to try and do it myself in the next decade to make sure my skills stay sharp. But I don’t see the industry broadly encouraging it, still less making it mandatory as aviation does. Addendum: as you probably know, even in aviation, this is hard to get right. (This is sometimes called the “children of the magenta” problem, but it’s really Bainbridge again.) The most famous example is perhaps Air France Flight 447[0], where the pilots put the plane into a stall at 35,000ft when they reacted poorly after the autopilot disconnecting, and did not even realize they had stalled the plane. Of course, that crash itself led to more regulations around training in manual scenarios too. [0] https://admiralcloudberg.medium.com/the-long-way-down-the-crash-of-air-france-flight-447-8a7678c37982 https://admiralcloudberg.medium.com/the-long-way-down-the-cr...
  - justincormack 10mo ago
    In most industry now you can't make the things by hand any more, there is no fallback. Once things get designed for automation there is no way back.
steveBK123 10mo ago
I think for most non-coding tasks we are still in the "convincing liar" stage, and not even at the "its right 99.9% of the time and humans need to quickly detect the 0.1% errors" problem. I think a lot of the HN crowd misses this because they are programmers using it for programming. I work at a firm that has given AI tooling to non-developer data analyst type people who otherwise live & die in excel. Much of their day job involves reading PDFs. I occasionally will use some of the firms AI tooling for PDF summarizing/parsing/interrogation/etc type tasks and remain consistently underwhelmed. Stuff like taking 10 PDFs each with a simple 30 row table per PDF, with the same title in each file, it ends up puking on 3-4 out of 10 with silent failures. Row drops, duplicating data, etc. When you point out its missed rows, it goes back and duplicates rows to get to the correct row count. Using it to interrogate standard company filings PDfs that it has been specially trained on and it gave very convincing answers which were wrong because it has silently truncated its search context to only recent year financial filings. Nowhere did it show this limitation to the user. It only became apparent after researching the 4th or 5th company when it decided to caveat its answer with its knowledge window. This invalidated the previous answers as questions such as "when was the first X" or "have they ever reported Y" were operating on incomplete information. Most users of these tool are not that technical, and are going to be much more naive in taking the answers for fact without considering the context.
- Terr_ 10mo ago
  I'm convinced the best use of these systems will be an explicit two-phase process where they just help people prototype and see and learn how to command regular software. For example, imagine describing what files you want to find, and getting back a command-line string of find/grep piping. It doesn't execute anything without confirmation, it doesn't "summarize" the results, it's just a narrow tutor to help people in a translation step. A tool for learning that, ideally, eventually puts itself out of a job. Returning to your PDF scenario: The LLM could help people weave together regular tools of "find regions with keywords" and "extract table as spreadsheet" and "cross-reference two spreadsheets using column values", etc.
throwaway613745 10mo ago
If your process is shit, you're just automating shit at lightning speed. If you're bad at your job, you're automating it at lightning speed. You need have good business process and be good at your job without AI in order to have any chance in hell of being successful with it. The idea that you can just outsource your thinking to the AI and don't need to actually understand or learn anything new anymore is complete delusion.
demorro 10mo ago
These observations were made 40 years ago. I suspect we have solved many of these problems now and have close to fully automated manufacturing and flight systems, or close enough that the training trade-off is worth it. However, this took 40 years and actual fatalities. We should keep that in mind when we're pushing the AI acceleration pedal down ever harder.
analog8374 10mo ago
I spent years creating automated drawing machines. But I can still draw better than any of them with my hand. Not as quickly tho.
dsjoerg 10mo ago
> Typically, before people are put in a leadership role directing humans, they will get a lot of leadership training teaching them the skills and tools needed to lead successfully. I question this.
didibus 10mo ago
A good read, but it reminds me that people see the programmer as being there to identify when the AI makes an error or a mistake. But in my use of AI agents as a programmer and also for other work. I would say that, while yes, you also have to look for mistakes or errors, most of the time I spend is on programming the AI still. The AI agent has no idea what it must produce, what it's meant to do, when it can alter something existing to enable something new, etc. And this is true for both functional and non-functional requirements. Unlike in traditional manufacturing, you've already built your manufacturing pipeline for a precise output, you've got your CAD designs done, you ran your simulations, you've calibrated everything already for what you want. So most of the work remains that of programming the machine.
Animats 10mo ago
There are a few issues here. It's useful to think about AI-driven coding assistants in terms of the SAE levels of automation for automatic driving. - Level 0 - totally manual - Level 1 - a bit of assistance, such as cruise control - Level 2 - speed and steering control that requires constant supervision by a human driver. This is where most of the commercial systems are now. - Level 3 - Level 2, but reliable enough that the human driver doesn't need to supervise constantly. Able to bring the vehicle to a safe stop by itself. Mercedes -Benz Drive Pilot is supposedly level 3. Handoff between computers and human remains a problem. Human still liable for accidents. - Level 4 - Full automation, but not under all conditions. Waymo is Level 4. Human just selects the destination. - Level 5 - Full automation, at least as capable as human drivers under all conditions. Not yet seen. What we're looking at with the various programming assistance AI systems ls Level 2 or Level 3 competence. These are the most troublesome levels. Who's in charge? Who's to blame? The need for such programming assistance systems may be transient, as it clearly is in automotive. Eventually, everybody in automotive will get to Level 4 or better, or drop out due to competitive pressure.
Animats 10mo ago
Bainbridge [1] is interesting, but dated. A more useful version of that discussion from slightly later is "Children of the Magenta", [2] an airline chief pilot talking to his pilots about cockpit automation and how to use it. Requires a basic notion of aviation jargon. There's been progress since then. Although the details are not widely publicized, enough pilots of the F-22, F-35, or the Gripen have talked about what modern fighter cockpit automation is like. The real job of fighter pilots is to fight and win battles, not drive the airplane. A huge amount of effort has been put into simplifying the airplane driving job so the pilot can focus on killing targets. The general idea today is that the pilot puts the pointy end in the right direction and the control systems take care of the details. An F-22 pilot has been quoted as saying that the F-22 is far less fussy than a Cessna as a flying machine. For the F-35, which has a VTOL configuration (B) and a carrier-landing configuration (C), much effort was put into making VTOL landing and carrier landing easy. Not because pilots can't learn to do it, but because training tended to obsess on those tasks. The hard part of Harrier (the only previous successful VTOL fighter) was learning to land the unstable beast without crashing. There were still a lot of Harrier crashes. The hard part of Naval aviator training is landing on a carrier deck. Neither of these tasks has anything to do with the real job of taking a bite out of the enemy, but they consumed most of the training time. So, for the F-35, both of those tasks have enough computer-added stability to make them much easier. One of the stranger features of the F-35 is that it has two main controls, called "inceptors", which correspond to throttle and stick. In normal flight, they mostly work like throttle and stick. But in low-speed hover, the "throttle" still controls speed while the "stick" controls attitude, even though the "stick" is affecting engine speed and the "throttle" is affecting control surfaces in that mode. So the pilot doesn't have to manage the strange transitions of a VTOL craft directly. This refocuses pilot training on using the sensors and weapons to do something to the enemy. Classic training is mostly about the last few minutes of getting home safely. As AI for programming advances, we should expect to devote more user time to analyzing the tactical problem, rather than driving the bus. [1] https://ckrybus.com/static/papers/Bainbridge_1983_Automatica.pdf https://ckrybus.com/static/papers/Bainbridge_1983_Automatica... [2] https://www.youtube.com/watch?v=5ESJH1NLMLs https://www.youtube.com/watch?v=5ESJH1NLMLs
bdangubic 10mo ago
> “ If it does not work properly, you need better prompts” is the usual response if someone struggles with directing agents successfully so much this!
abrookewood 10mo ago
TROJAN WARNING: My AV is reporting issues with this link: 15/12/2025 2:59:56 PM;HTTP filter;file;https://cdn.jsdeliver.net/npm/mathjax@3.2.2/es5/tex-chtml.js;JS/Redirector.SWD https://cdn.jsdeliver.net/npm/mathjax@3.2.2/es5/tex-chtml.js... trojan;connection terminated;
- wizzwizz4 10mo ago
  Ooh, that should be cdn.jsdelivr.net (ver vs vr). Good catch!
alexgotoi 10mo ago
The automation irony: we build AI to reduce human workload, but end up creating systems that need constant human supervision anyway. Classic. What's interesting is this mirrors every automation wave. We thought assembly lines would eliminate human work - instead they just changed what work meant. AI's doing the same, just at software speed instead of industrial speed. Long-term I'm optimistic - automation creates more than it destroys, always has. Short-term though? Messy transition for anyone whose job is 'being the interface layer. Will include this thread in my next issue of https://hackernewsai.com/ https://hackernewsai.com/

256 points by BinaryIgor 10mo ago

[dead]

z_ 10mo ago

This is a thought provoking piece. “But at what cost?” We’ve all accepted calculators into our lives as being faster and correct when utilized correctly (Minus Intel tomfoolery), but we emphasize the need to know how to do the math in educational settings. Any post education adult will confirm when confronted with an irregular math problem (or a skill) that there is a wait time to revive the ability. Programming automation having the potential skill decay AND being critical path is … worth thinking about.

xorcist 10mo ago
Comparisons with deterministic tools such as calculators will always lead astray. There is no comparable situation where faced with a new problem the AI will just give up. If there is the need for an expert, the need is always there, because there is no indication external to the process that the process will fail.
- alex989 10mo ago
  I disagree about how calculators and math are deterministic a in real world scenarios where you use math at work. When you compute a formula in your calculator or in a fancy design software, it will always give you answer but it doesn't mean you asked the right question. If you use the wrong units in your input or if you make a typo, if you used the wrong formula, etc., the calculator/software will blindly give you an answer and only an experienced engineer will spot it a first glance. As soon as there is a human in the loop, things get messy. For exemple, if your calculator tells you that a 15m long W200x31 steel beam can resist 215kN•m in bending moment, I know at first glance its at least 4x too much for that length, but how many people reading my comment could? A civil engineer fresh out of college would not.
singpolyma3 10mo ago
Calculators don't do math, they do calculating. Which is to say, they don't think for you. There's not much value in being able to quickly compute some expression in a world with calculators. But there's a huge value in knowing how to know which numbers to feed into the calculation.
- kurthr 10mo ago
  The biggest problem with calculators (rather than slide rules), was that because calculations with big numbers (large mantissa) were so easy, people got used to doing them that way without consideration. Using a slide rule meant inherently knowing order-of-magnitude, rounding, and precision. Once calculators make it easy they enable both new kinds of solutions and new kinds of errors (that you have to separately teach to avoid). At the same time, I basically agree. Humans are very bad calculators and we've needed tools (abacus) for millennia.
- bitwize 10mo ago
  I derive tremendous value from being able to calculate taxes, tips, and so forth in my head, or right on the receipt, without having to reach for my phone and launch Droid48. (I know some of y'all are also Droid48 bros.) It's even more profound a convenience than knowing how to drive Emacs with just the keyboard and not having to reach for the goddamn mouse.
  - agumonkey 10mo ago
    we need to form a group of intrisics people who enjoyed knowing and learning in depth, not just apply to sell something
eastbound 10mo ago
We already have generational programming decay. At 25 years old, kids fresh out of uni can’t write a string.contains() routine. They all use .stream() in Java. Matter of generation, fashion and skills to learn. And concerning the programming of C drivers, Apple is the last company to write a filesystem and they already can’t find anyone able to do it.

nuancebydefault 10mo ago

The article discusses basically 2 new problems with using agentic AI: - When one of the agents does something wrong, a human operator needs to be able to intervene quickly and needs to provide the agent with expert instructions. However since experts do not execute the bare tasks anymore, they forget parts of their expertise quickly. This means the experts need constant training, hence they will have little time left to oversee the agent's work. - Experts must become managers of agentic systems, a role which they are not familiar with, hence they are not feeling at home in their job. This problem is harder to be determined as a problem by people managers (of the experts) since they don't experience that problem often first hand. Indeed the irony is that AI provides efficiency gains, which as they become more widely adopted, become more problematic because they outfit the necessary human in the loop. I think this all means that automation is not taking away everyone's job, as it makes things more complicated and hence humans can still compete.

jennyholzer 10mo ago
[dead]
DiscourseFan 10mo ago
That's how it tends to go, automation removes some parts of the work but creates more complexity. Sooner or later that will also be automated away, and so on and so forth. AGI evangelists ought to read Marx's Capital.
- jennyholzer2 10mo ago
  I seriously doubt that there is even one "AGI evangelist" who has the intellectual capacity to read books written for adult audiences.
  - ctoth 10mo ago
    Hi. I am not an evangelist -- I'm quite certain it's going to kill us all! But I would like to think that I'm about the closest thing to an AI booster you might find here, given that I get so much damn utility out of it. I'm interested in reading, I probably read too much! would you like to suggest a book we can discuss next week? I'd be happy to do this with you.

jennyholzer2 10mo ago

[flagged]

TheOtherHobbes 10mo ago
Not necessarily. It depends if the process is deterministic and repeatable. If an AI generates a process more quickly than a human, and the process can be run deterministically, and the outputs are testable, then the process can run without direct human supervision after initial testing - which is how most automated processes work. The testing should happen anyway, so any speed increase in process generation is a productivity gain. Human monitoring only matters if the AI is continually improvising new solutions to dynamic problems and the solutions are significantly wrong/unreliable. Which is a management/analysis problem, and no different in principle to managing a team. The key difference in practice is that you can hire and fire people on a team, you can intervene to change goals and culture, and you can rearrange roles. With an agentic workflow you can change the prompts, use different models, and redesign the flow. But your choices are more constrained.
- lkjdsklf 10mo ago
  The issue is LLMs are, by design, non-deterministic. That means that, with the current technology, there can never be a deterministic agent. Now obviously, humans aren't deterministic either, but the error bars are a lot closer together than they are with LLMs these days. An easy to point at example is the coding agent that removed someones home directory that was circulating around. I'm not saying a human has never done that, but it's far less likely because it's so far out of the realm of normal operations. So as of today, we need humans in the loop. And this is understood by the people making these products. That's why they have all these permissions and prompts for you to accept/run commands and all of that.
  - loa_in_ 10mo ago
    There's lots of _marketing_ promising unsupervised agents. It's important to remember not to drink the cool-aid.
  - 10mo ago

sublimefire 10mo ago

Good discussion of the paper and the observations and ironies. A thing to note is that we do have software factories already, with a bunch of automation in place and folks being trained to deal with incidents. The pools of agents just elevate what we currently have but the tools are still lacking severely. IMO the tools need to improve for us to move forward as it is difficult to observe the decisions of agents when they fall apart. Also, by and large the current AI tools are not in the critical path yet, well except those drones that lock on targets to eliminate them in case of interference, and even then it is ML. Agents can not be in that path due to predictability challenges yet.

wesammikhail 10mo ago

Our of curiosity, does anyone know of a good writeup / blog post made by someone in the industry that revolves around reducing orchestration error rates? Would love to read some more about the topic and I'm looking for a few good resources.

dloranc 10mo ago
What do you mean by orchestration?

everdrive 10mo ago

I can feel the skill atrophy creeping in. My very first instinct is go use the LLM. I think much like forcing yourself to exercise, eat right, and avoid social media / distractions, this will be a new modern skillset; do you have the discipline to avoid becoming useless without an LLM? A small few will be great at this, the middle of the bell curve will do "well enough," and you know the story for the rest.

andy99 10mo ago
I’ve been using LLMs to code for some time and I look at it differently. I ask myself if I need to understand the code, and if the answer is yes I don’t use an LLM. It’s not a matter of discipline, it’s a sober view of what the minimal amount of work for me is.
- layer8 10mo ago
  The only time one doesn’t need to understand the code is when it doesn’t matter if the code is correct, or when it can be tested exhaustively for all possible inputs. Both are pretty rare for me.
  - kaffekaka 10mo ago
    I largely agree. But sometimes the program is not destructive and you only need to test for inputs that may/will actually occur. The LLM wrote a script to do some processing? Just test it, if the processing is fine, done. I have many LLM-written scripts and tools to do some semi simple jobs where I have barely even looked at the code because I could see immediately that the job I wanted to do got done.
delaminator 10mo ago
I haven't written any code in 6 months. But I can still remember how to code in 6502 machine code from the 1980s.
- zeroonetwothree 10mo ago

ripe 10mo ago

I really like this author's summary of the 1983 Bainbridge paper about industrial automation. I have often wondered how to apply those insights to AI agents, but I was never able to summarize it as well as OP. Bainbridge by itself is a tough paper to read because it's so dense. It's just four pages long and worth following along: https://ckrybus.com/static/papers/Bainbridge_1983_Automatica.pdf https://ckrybus.com/static/papers/Bainbridge_1983_Automatica... For example, see this statement in the paper: "the present generation of automated systems, which are monitored by former manual operators, are riding on their skills, which later generations of operators cannot be expected to have." This summarizes the first irony of automation, which is now familiar to everyone on HN: using AI agents effectively requires an expert programmer, but to build the skills to be an expert programmer, you have to program yourself. It's full of insights like that. Highly recommended!

startupsfail 10mo ago
The same argument was there about needing to be an expert programmer in assembly language to use C, and then same for C and Python, and then Python and CUDA, and then Theano/Tensorflow/Pytorch. And yet here we are, able to talk to a computer, that writes Pytorch code that orchestrates the complexity below it. And even talks back coherently sometimes.
- gipp 10mo ago
  Those are completely deterministic systems, of bounded scope. They can be ~completely solved, in the sense that all possible inputs fall within the understood and always correctly handled bounds of the system's specifications. There's no need for ongoing, consistent human verification at runtime. Any problems with the implementation can wait for a skilled human to do whatever research is necessary to develop the specific system understanding needed to fix it. This is really not a valid comparison.
  - startupsfail 10mo ago
    There are enormous microcode, firmware and drivers blobs everywhere on any pathway. Even with very privileged access of someone at Intel or NVIDIA, ability to have a reasonable level of deterministic control of systems that involve CPU/GPU/LAN were long gone, almost for a decade now.

jinwoo68 10mo ago

"Most companies are efficiency-obsessed." But what most of them do is not to be more efficient but to be shown to be more efficient. The main reason they are so obsessed with AI is because they want to send the signal that they are pursuing to be more efficient, whether they succeed or not.

theologic 10mo ago
Peter Drucker popularized the phrase "Efficiency is doing things right; effectiveness is doing the right things." Being a credibly efficient at doing the wrong things, turns out to be a massive issue inside of most companies. What's interesting is I do think that AI gives opportunity to be massively more effective because if you have the right LLM, that's trained right, you can explore a variety of scenarios much faster than what you can do by yourself. However, we hear very little about this as a central thrust of how to utilize AI into the work space.
- jjk166 10mo ago
  In my experience plenty of places are quite inefficient at doing the wrong things as well. You might think this reduces the number of wrong things done, but somehow it doesn't.
  - theologic 10mo ago
    It's almost comical isn't it, but it actually turns out that this is a big foundation behind behavioral economics. In essence you can get trapped in an upper level heuristic and never stop for a moment and thinks things through. Another one of my favorite examples is that there is some research out of Harvard that basically suggested that if people would take and spend 15 minutes a day reviewing what they had done and what was important, they increased their productivity 22%. Now you would think that this is so obvious and so dramatic you would have variety of Fortune 500 companies saying "oh my goodness we want all of our workers to be 22% more productive" and so they would simply send out a memo or an email or some sort of process to force people to do some reflecting. I would also suggest that Microsoft had a unique advantage based out of the idea that people should have their own enclosed workspace to do coding. This was deeply entrenched when Bill was running the company day-to-day. And I'm sure as somebody that was a coding phenomenon, it simply made sense to him. But academically, it also makes sense. Microsoft has reversed this policy, but as far as I can tell, it doesn't have anything to do with the research. It has to do with statements about working together efficiently. or AI productivity. If there's real research then it's great. My problem is it just doesn't appear to be any real research behind it. Yet I'm sure many managers at Microsoft thinks that it's very efficient. Of course, if you do know anybody at Microsoft that codes, they have their own opinion, and rather than me repeating hearsay, it would be fantastic to have somebody anonymously post what's really going on here. I'll betcha a nickel that 90% of them are not reporting that they feel a lot more effective.

jiehong 10mo ago

This irony of automation has been dealt with in the aviation industry for pilot for years: auto pilots can actually land the plane in many cases, and do fly the plane on most of the cruise. Yet, pilots are constantly trained on actual scenarios, and are expected to land airplanes manually monthly (and during take off too). This ensures pilots maintain their skills, while the auto pilot helps most of the time. On top of that, plane commands often are half automatic already, aka they are assisted (but not by LLMs!), so it’s a complex comparison.

libraryofbabel 10mo ago
Yes, but (to write the second half of your post for you!) regulation and incentives are very different in the aviation industry, because safety and planning for long-tail risks is paramount. Therefore airlines can afford to have their pilots spend thousands of hours training on manual control in various scenarios. By contrast, I don’t think the average software development org will encourage its engineers to hand-roll a sizable proportion of their code, if (still a big if) there are major productivity costs in doing so. Rushing the Next Big Feature out the door will almost always beat out long-term investment in dev training, unfortunately. Don’t get me wrong - manual practice is in some sense the correct solution, and I plan to try and do it myself in the next decade to make sure my skills stay sharp. But I don’t see the industry broadly encouraging it, still less making it mandatory as aviation does. Addendum: as you probably know, even in aviation, this is hard to get right. (This is sometimes called the “children of the magenta” problem, but it’s really Bainbridge again.) The most famous example is perhaps Air France Flight 447[0], where the pilots put the plane into a stall at 35,000ft when they reacted poorly after the autopilot disconnecting, and did not even realize they had stalled the plane. Of course, that crash itself led to more regulations around training in manual scenarios too. [0] https://admiralcloudberg.medium.com/the-long-way-down-the-crash-of-air-france-flight-447-8a7678c37982 https://admiralcloudberg.medium.com/the-long-way-down-the-cr...
- justincormack 10mo ago
  In most industry now you can't make the things by hand any more, there is no fallback. Once things get designed for automation there is no way back.

steveBK123 10mo ago

I think for most non-coding tasks we are still in the "convincing liar" stage, and not even at the "its right 99.9% of the time and humans need to quickly detect the 0.1% errors" problem. I think a lot of the HN crowd misses this because they are programmers using it for programming. I work at a firm that has given AI tooling to non-developer data analyst type people who otherwise live & die in excel. Much of their day job involves reading PDFs. I occasionally will use some of the firms AI tooling for PDF summarizing/parsing/interrogation/etc type tasks and remain consistently underwhelmed. Stuff like taking 10 PDFs each with a simple 30 row table per PDF, with the same title in each file, it ends up puking on 3-4 out of 10 with silent failures. Row drops, duplicating data, etc. When you point out its missed rows, it goes back and duplicates rows to get to the correct row count. Using it to interrogate standard company filings PDfs that it has been specially trained on and it gave very convincing answers which were wrong because it has silently truncated its search context to only recent year financial filings. Nowhere did it show this limitation to the user. It only became apparent after researching the 4th or 5th company when it decided to caveat its answer with its knowledge window. This invalidated the previous answers as questions such as "when was the first X" or "have they ever reported Y" were operating on incomplete information. Most users of these tool are not that technical, and are going to be much more naive in taking the answers for fact without considering the context.

Terr_ 10mo ago
I'm convinced the best use of these systems will be an explicit two-phase process where they just help people prototype and see and learn how to command regular software. For example, imagine describing what files you want to find, and getting back a command-line string of find/grep piping. It doesn't execute anything without confirmation, it doesn't "summarize" the results, it's just a narrow tutor to help people in a translation step. A tool for learning that, ideally, eventually puts itself out of a job. Returning to your PDF scenario: The LLM could help people weave together regular tools of "find regions with keywords" and "extract table as spreadsheet" and "cross-reference two spreadsheets using column values", etc.

throwaway613745 10mo ago

If your process is shit, you're just automating shit at lightning speed. If you're bad at your job, you're automating it at lightning speed. You need have good business process and be good at your job without AI in order to have any chance in hell of being successful with it. The idea that you can just outsource your thinking to the AI and don't need to actually understand or learn anything new anymore is complete delusion.

demorro 10mo ago

These observations were made 40 years ago. I suspect we have solved many of these problems now and have close to fully automated manufacturing and flight systems, or close enough that the training trade-off is worth it. However, this took 40 years and actual fatalities. We should keep that in mind when we're pushing the AI acceleration pedal down ever harder.

analog8374 10mo ago

I spent years creating automated drawing machines. But I can still draw better than any of them with my hand. Not as quickly tho.

dsjoerg 10mo ago

> Typically, before people are put in a leadership role directing humans, they will get a lot of leadership training teaching them the skills and tools needed to lead successfully. I question this.

didibus 10mo ago

A good read, but it reminds me that people see the programmer as being there to identify when the AI makes an error or a mistake. But in my use of AI agents as a programmer and also for other work. I would say that, while yes, you also have to look for mistakes or errors, most of the time I spend is on programming the AI still. The AI agent has no idea what it must produce, what it's meant to do, when it can alter something existing to enable something new, etc. And this is true for both functional and non-functional requirements. Unlike in traditional manufacturing, you've already built your manufacturing pipeline for a precise output, you've got your CAD designs done, you ran your simulations, you've calibrated everything already for what you want. So most of the work remains that of programming the machine.

Animats 10mo ago

There are a few issues here. It's useful to think about AI-driven coding assistants in terms of the SAE levels of automation for automatic driving. - Level 0 - totally manual - Level 1 - a bit of assistance, such as cruise control - Level 2 - speed and steering control that requires constant supervision by a human driver. This is where most of the commercial systems are now. - Level 3 - Level 2, but reliable enough that the human driver doesn't need to supervise constantly. Able to bring the vehicle to a safe stop by itself. Mercedes -Benz Drive Pilot is supposedly level 3. Handoff between computers and human remains a problem. Human still liable for accidents. - Level 4 - Full automation, but not under all conditions. Waymo is Level 4. Human just selects the destination. - Level 5 - Full automation, at least as capable as human drivers under all conditions. Not yet seen. What we're looking at with the various programming assistance AI systems ls Level 2 or Level 3 competence. These are the most troublesome levels. Who's in charge? Who's to blame? The need for such programming assistance systems may be transient, as it clearly is in automotive. Eventually, everybody in automotive will get to Level 4 or better, or drop out due to competitive pressure.

Animats 10mo ago

Bainbridge [1] is interesting, but dated. A more useful version of that discussion from slightly later is "Children of the Magenta", [2] an airline chief pilot talking to his pilots about cockpit automation and how to use it. Requires a basic notion of aviation jargon. There's been progress since then. Although the details are not widely publicized, enough pilots of the F-22, F-35, or the Gripen have talked about what modern fighter cockpit automation is like. The real job of fighter pilots is to fight and win battles, not drive the airplane. A huge amount of effort has been put into simplifying the airplane driving job so the pilot can focus on killing targets. The general idea today is that the pilot puts the pointy end in the right direction and the control systems take care of the details. An F-22 pilot has been quoted as saying that the F-22 is far less fussy than a Cessna as a flying machine. For the F-35, which has a VTOL configuration (B) and a carrier-landing configuration (C), much effort was put into making VTOL landing and carrier landing easy. Not because pilots can't learn to do it, but because training tended to obsess on those tasks. The hard part of Harrier (the only previous successful VTOL fighter) was learning to land the unstable beast without crashing. There were still a lot of Harrier crashes. The hard part of Naval aviator training is landing on a carrier deck. Neither of these tasks has anything to do with the real job of taking a bite out of the enemy, but they consumed most of the training time. So, for the F-35, both of those tasks have enough computer-added stability to make them much easier. One of the stranger features of the F-35 is that it has two main controls, called "inceptors", which correspond to throttle and stick. In normal flight, they mostly work like throttle and stick. But in low-speed hover, the "throttle" still controls speed while the "stick" controls attitude, even though the "stick" is affecting engine speed and the "throttle" is affecting control surfaces in that mode. So the pilot doesn't have to manage the strange transitions of a VTOL craft directly. This refocuses pilot training on using the sensors and weapons to do something to the enemy. Classic training is mostly about the last few minutes of getting home safely. As AI for programming advances, we should expect to devote more user time to analyzing the tactical problem, rather than driving the bus. [1] https://ckrybus.com/static/papers/Bainbridge_1983_Automatica.pdf https://ckrybus.com/static/papers/Bainbridge_1983_Automatica... [2] https://www.youtube.com/watch?v=5ESJH1NLMLs https://www.youtube.com/watch?v=5ESJH1NLMLs

bdangubic 10mo ago

> “ If it does not work properly, you need better prompts” is the usual response if someone struggles with directing agents successfully so much this!

abrookewood 10mo ago

TROJAN WARNING: My AV is reporting issues with this link: 15/12/2025 2:59:56 PM;HTTP filter;file;https://cdn.jsdeliver.net/npm/mathjax@3.2.2/es5/tex-chtml.js;JS/Redirector.SWD https://cdn.jsdeliver.net/npm/mathjax@3.2.2/es5/tex-chtml.js... trojan;connection terminated;

wizzwizz4 10mo ago
Ooh, that should be cdn.jsdelivr.net (ver vs vr). Good catch!

alexgotoi 10mo ago

The automation irony: we build AI to reduce human workload, but end up creating systems that need constant human supervision anyway. Classic. What's interesting is this mirrors every automation wave. We thought assembly lines would eliminate human work - instead they just changed what work meant. AI's doing the same, just at software speed instead of industrial speed. Long-term I'm optimistic - automation creates more than it destroys, always has. Short-term though? Messy transition for anyone whose job is 'being the interface layer. Will include this thread in my next issue of https://hackernewsai.com/ https://hackernewsai.com/