9 ms·

Looming Liability Machines (LLMs)

158 points by zdw 2y ago

noduerme 2y ago
Forget RCA, we should think bigger! Putting LLMs in charge of nuclear weapons could completely eliminate the root causes of accidents worldwide!
- hislaziness 2y ago
  I know you mean this in jest, but we are much closer to this than we would imagine, the use of LLMs to process communication / translation is becoming ubiquitous. We are 1 bad translation away from a disaster.
  - handfuloflight 2y ago
    Could you illustrate a likely scenario that's in your mind?
    - DaiPlusPlus 2y ago
      > Could you illustrate a likely scenario that's in your mind? The year is 2034; after a surprisingly cutthroat economic trade-war fought between China and the US left the world in the throes of another great recession, a growing wave of anti-China sentiment captures the attention of domestic political leadership which cultivates the movement despite (or more likely: because of) the growing interest from xenophobic reactionaries and other populist movements looking to scapegoat their way out of a dip in GDP - eventually those same poltiical-actors win the presidency and use their democratic mandate to instigate a new McCarthy-era of anti-China paranoia leading to utterly deranged domestic policy, namely as the executive ordering, by-decree, that the State Department terminate the employment of anyone who even speaks Mandarin[1] - a few weeks later in the South China Sea another Filpino/Sino boat-ramming incident escalates into something serious - the US Navy urges the US civilian government to communicate with China over the D.C.-to-Beijing "red telephone" deescalation e-mail system, but no-one knows how to communicate to the Chinese in their own language, so the overworked federal employee manning the red-Outlook-inbox sees nothing wrong with simply having that Microosft Office 365 CoPilot translate it for him - the same AI bot that's somehow always on his screen with that distracting sidebar (despite the best efforts of the US Federal Gov's Active Directory Group Policy) - it wasn't long before the first warheads exploded over North America that the President learned the AI translated the polite request to China for them to "please stop ramming the fishing boats" was received by them as "I'll ram my fish into your Junk...boats". If there's any upside to this story, the collective mass of AI were wiped out first by the high-altitude EMP bursts, leaving us humans with the last-laugh before we were all incinerated moments later - while those not fortunate enough to die instantly instead suffered months of prolonged fatal radiaiton sickness while what little left of civilization collapsed around them[2]. [1]If you think that's too ridiculous to be realistic, consider the Japanese internment-camp policy or Trump's declared Muslim ban. Elsewhere, in the late-1970s (in the age of CT Scanners and VHS tapes), Pol Pot targeted people for wearing glasses. [2]Blame James Burke's editorial slant in his documentary series' for turning me into a nhilist.
      - noduerme 2y ago
        That scenario has me less worried, just because by 2035 all popular politicians in America and China will be AI deepfakes run off the same cloud servers. [edit] I herewith introduce a new shitcoin called NukeCoin. Everyone in China and America gets a NukeCoin that will go up in value every day you hold it that no one nukes anyone.
    - walterbell 2y ago
      Supply chain contract negotiations?
- grugagag 2y ago
  Don’t stop at that, take it further handing over all decision making to LLMs, what could go wrong? In fact, replacing all jobs with LLMs should do in the minds of Silcone Valey venture capitalists. Again, what could go wrong LLMs can’t be used to fix with just one succesive prompt?
- ooterness 2y ago
  "In three years, Cyberdyne will become the largest supplier of military computer systems. All stealth bombers are upgraded with Cyberdyne computers, becoming fully unmanned. Afterwards, they fly with a perfect operational record. The Skynet Funding Bill is passed. The system goes online August 4th, 1997. Human decisions are removed from strategic defense. Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug." -Terminator 2
deleted 2y ago
[deleted]
efitz 2y ago
Why should I think that LLMs would be good at the task of analyzing a cloud incident and determining root cause? LLMs are good at predicting the next word in written language. They are generative; they make new text given a prompt. LLMs do not have base sets of facts about how complex systems work, and do not attempt to reason over a corpus of evidence and facts. as a result, I would expect that an LLM might concoct an interesting story about why such a failure occurred, and it might even be a convincing story if it happened to weave bits of context, accurately into the storyline. It might even, purely randomly, generate a story that actually correctly diagnosed the root cause of the failure, but that would be coincidental based on the similarity of the prompt to text of similar postmortem discussions that were part of its training set. If you had an extremely detailed postmortem document, then I would expect LLM‘s to do a very good job of summarizing such document. But I don’t see why an LLM is an appropriate tool for analyzing failures in complex systems; just as I don’t see a hammer being a very effective tool for tightening bolts. Right now, I am concerned that the relative ease that modern frameworks provide to author LLM based applications, is leading many people to optimistically include LLM technology in attempts to solve problems that it doesn’t seem particularly well suited to solve.
- from-nibly 2y ago
  The main problem is. Most people cant tell the difference between an expert opinion and random words. So if an LLM shoots a bunch of jargon on the screen then, well so does my senior platform engineer so whats the difference? The LLM is cheaper and is always on call.
  - RodgerTheGreat 2y ago
    I see deeply ominous connections between generations of American children who have been educated in a manner which produces alarming rates of functional illiteracy[0] and the widespread popularity of machine learning models that produce nonsense which resembles plausible text if you skim or don't actually read it. [0] https://www.apmreports.org/episode/2019/08/22/whats-wrong-how-schools-teach-reading https://www.apmreports.org/episode/2019/08/22/whats-wrong-ho...
sickblastoise 2y ago
Here’s a simple rule, based on the fact no one has shown that an llm or a compound llm system can produce an output that doesn’t need to be verified for correctness by a human across any input: The rate at which llm/llm compound systems can produce output > the rate at which humans can verify the output I think it follows that we should not use llms for anything critical. The gunghoe adoption and hamfisting of llms into critical processes, like an AWs migration to Java 17, or root cause analysis is plainly premature, naive, and dangerous.
- latentnumber 2y ago
  Why not automate verification itself then? While not possible now, and I would probably never advocate for using LLMs in critical settings, it might be possible to build field-specific verification systems for LLMs with robustness guarantees as well.
  - RodgerTheGreat 2y ago
    If the verification systems for LLMs are built out of LLMs, you haven't addressed the problem at all, just hand-waved a homunculus that itself requires verification. If the verification systems for LLMs are not built out of LLMs and they're somehow more robust than LLMs at human-language problem solving and analysis, then you should be using the technology the verification system uses instead of LLMs in the first place!
    - wbogusz 2y ago
      > If the verification systems for LLMs are not built out of LLMs and they're somehow more robust than LLMs at human-language problem solving and analysis, then you should be using the technology the verification system uses instead of LLMs in the first place! The issue is not in the verification system, but in putting quantifiable bounds on your answer set. If I ask an LLM to multiply large numbers together I can also very easily verify the generated answer by topping it with a deterministic function. I.e. rather than hoping that an LLM can accurately multiply two 10 digit numbers, I have a much easier (and verified) solution by instead asking it to perform this calculation using python and reading me the output
m1keil 2y ago
Folks let's be real. While the tech industry borrows terms and procedures from mature and inherently riskier industries like Aviation, 99% of the tech companies don't share the same risk profile. This means that in most cases, these RCAs are the output of a long and over engineered incident review process that was designed to impress the higher echelon. The problem is, that in a decently sized corporation, you have tens to hundreds of daily fuck ups (also known as "incidents") that completely suck out the free time out of engineers that have to navigate the long game of post incident management process. The utilisation of LLMs on these cases are just engineered solution to the problem of organisational bureaucracy.
- aragilar 2y ago
  Is not aviation, space-flight, etc. part of "tech"? While you can always over-do processes, you can also under-do them as well (I don't think I need to list examples here, just look at the news...). Having the processes also be able to adapt to different needs and requirements is part of having good processes.
  - m1keil 2y ago
    I don't think symantics really matter here. I largely think of "tech" as companies that produce mainly virtual goods in the form of software. Most of these software is largely skins over databases. In most cases, nobody will get hurt if their product will malfunction. This is the vast majority of tech companies today. Exceptions exists. I would take under process any day of the week. From my experience, adding a process is far easier than removing it.
    - aragilar 2y ago
      If you do "tech" == "software" maybe I'd agree with you no one will get hurt (though the cynical part of me says "did you measure this, or do you want this to be true"), but S(cience)T(echnology)E(ngineering)M(athematics) is a thing, and it's definitely not just software.
- droopyEyelids 2y ago
charleslmunger 2y ago
The statement about "4,500 developer-years of work" is insane to me. Java is one of the most backward compatible languages period - other than hashmap iteration order a while back, it's hard to think of what could require that astronomical quantity of engineering effort to upgrade. Do they actually budget over a billion dollars to upgrade Java versions, or is this like "this amazing tool, sed, saved us infinity developer years by replacing strings at one quadrillionth the cost of a $500k human editing each text file by hand"
- deleted 2y ago
  [deleted]
- efitz 2y ago
  <satire/>In unrelated news, Amazon announces layoff of 4500 developer positions...
- DaiPlusPlus 2y ago
  > "The average time to upgrade an application to Java 17 plummeted from what’s typically 50 developer-days to just a few hours." *blink* ...why are they boasting about migrating to Java 17... in 2024? ...and from what versions were they migrating from? Java 17 didn't introduce any significant breaking-changes (IME) for users on Java 16 or even the next previous LTS version, Java 11 - I don't work at Amazon, but surely Amazon isn't in the habit of running on unsupported JVMs? - so assuming these Java projects were being competently maintained, then the only work actually required to migrate to 17 is changing your `org.gradle.java.home=` path to where JDK 17 followed by running your test suite. If Amazon was using a monorepo then 1 person could do this in 5 minutes with a 1-liner awk/sed command - whereas I expect it would likely take an AI far longer to do this, if it's even able to make sense of an Amazon-sized monorepo - they'd also likely re-prompt it for every separate project for reliability's sake. So after considering all that, the "50 days" number he gives, without any context either, is a nice shorthand to communicate his disconnection from what really goes-on inside his org... or he's lying - and he knows he's lying - but he also knows there won't be any negative consequences for him as a result of his lying, so why not lie if it gives you a good story to tell for LinkedIn? In conclusion: these remarks by leadership unintentionally make the company look bad, not good, once you fill-in-the-blanks to make up for what they dind't say. (What's next...? Big Brother increasing our chocolate ration to 20 grammes per week?) ----- One more thing: the comment-replies to the post on LinkedIn are utterly derranged and I genuinely can't put my sense of unease into words.
deleted 2y ago
[deleted]
ryoshu 2y ago
Why would you ever use a non-deterministic model for a deterministic function?
indus 2y ago
Most LLMs have an accuracy benchmark for controlled questions & answers. Even if this accuracy is 95% then in a complex system the probability of getting to the right answer diminishes with each new step being added. This is also the key tenet of an agentic system. While the analysis in the blog is excellent but an answer needs to be found. A layer on top of LLMs for error control/check. As an analogy, in the analog to digital transmission stack of OSI, an error correction mechanism such as frame check sequence (fcs) detects transmission errors in the data link layer.
rboyd 2y ago
FTA: "If we offload the RCA learning/categorization part to the LLM (whatever that means), we wouldn't be able to make much progress in the enhancing reliability and safety part." But you don't offload it in the sense that you expect the tool to completely take the wheel. You ask it for suggestions to inform a human. If the suggestions turn out to only be a distraction in your environment then you abandon the tool. For plenty of environments the suggestions will be hugely useful and save you valuable time during an ongoing outage.
- layer8 2y ago
  They are specifically talking about automating RCA, not about tool-assisted RCA.
  - rboyd 2y ago
    Go read the papers on automated RCA. The algorithms are designed to suggest top K candidates for RCA. By definition at least k-1 are going to be wrong. https://arxiv.org/abs/2305.10638 https://arxiv.org/abs/2305.10638
zebomon 2y ago
This articles speak directly to what I believe will become the unfortunate and inconvenient reality of LLMs' core limitation: at some point, someone has to 1) benefit from what's been generated and 2) (and more importantly) know they're benefiting from what's been generated. If for example you have an LLM agent that's effectively "solved" every security flaw your software may encounter for the next 50 years, unless it can simultaneously impart 50 years of training to the people who rely on the software, it's done nothing but introduce us to more complex flaws that we would need approximately 49 more years of experience to tackle ourselves.

158 points by zdw 2y ago

Forget RCA, we should think bigger! Putting LLMs in charge of nuclear weapons could completely eliminate the root causes of accidents worldwide!

hislaziness 2y ago
I know you mean this in jest, but we are much closer to this than we would imagine, the use of LLMs to process communication / translation is becoming ubiquitous. We are 1 bad translation away from a disaster.
- handfuloflight 2y ago
  Could you illustrate a likely scenario that's in your mind?
  - DaiPlusPlus 2y ago
    > Could you illustrate a likely scenario that's in your mind? The year is 2034; after a surprisingly cutthroat economic trade-war fought between China and the US left the world in the throes of another great recession, a growing wave of anti-China sentiment captures the attention of domestic political leadership which cultivates the movement despite (or more likely: because of) the growing interest from xenophobic reactionaries and other populist movements looking to scapegoat their way out of a dip in GDP - eventually those same poltiical-actors win the presidency and use their democratic mandate to instigate a new McCarthy-era of anti-China paranoia leading to utterly deranged domestic policy, namely as the executive ordering, by-decree, that the State Department terminate the employment of anyone who even speaks Mandarin[1] - a few weeks later in the South China Sea another Filpino/Sino boat-ramming incident escalates into something serious - the US Navy urges the US civilian government to communicate with China over the D.C.-to-Beijing "red telephone" deescalation e-mail system, but no-one knows how to communicate to the Chinese in their own language, so the overworked federal employee manning the red-Outlook-inbox sees nothing wrong with simply having that Microosft Office 365 CoPilot translate it for him - the same AI bot that's somehow always on his screen with that distracting sidebar (despite the best efforts of the US Federal Gov's Active Directory Group Policy) - it wasn't long before the first warheads exploded over North America that the President learned the AI translated the polite request to China for them to "please stop ramming the fishing boats" was received by them as "I'll ram my fish into your Junk...boats". If there's any upside to this story, the collective mass of AI were wiped out first by the high-altitude EMP bursts, leaving us humans with the last-laugh before we were all incinerated moments later - while those not fortunate enough to die instantly instead suffered months of prolonged fatal radiaiton sickness while what little left of civilization collapsed around them[2]. [1]If you think that's too ridiculous to be realistic, consider the Japanese internment-camp policy or Trump's declared Muslim ban. Elsewhere, in the late-1970s (in the age of CT Scanners and VHS tapes), Pol Pot targeted people for wearing glasses. [2]Blame James Burke's editorial slant in his documentary series' for turning me into a nhilist.
    - noduerme 2y ago
      That scenario has me less worried, just because by 2035 all popular politicians in America and China will be AI deepfakes run off the same cloud servers. [edit] I herewith introduce a new shitcoin called NukeCoin. Everyone in China and America gets a NukeCoin that will go up in value every day you hold it that no one nukes anyone.
  - walterbell 2y ago
    Supply chain contract negotiations?
grugagag 2y ago
Don’t stop at that, take it further handing over all decision making to LLMs, what could go wrong? In fact, replacing all jobs with LLMs should do in the minds of Silcone Valey venture capitalists. Again, what could go wrong LLMs can’t be used to fix with just one succesive prompt?
ooterness 2y ago
"In three years, Cyberdyne will become the largest supplier of military computer systems. All stealth bombers are upgraded with Cyberdyne computers, becoming fully unmanned. Afterwards, they fly with a perfect operational record. The Skynet Funding Bill is passed. The system goes online August 4th, 1997. Human decisions are removed from strategic defense. Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug." -Terminator 2

deleted 2y ago

[deleted]

efitz 2y ago

Why should I think that LLMs would be good at the task of analyzing a cloud incident and determining root cause? LLMs are good at predicting the next word in written language. They are generative; they make new text given a prompt. LLMs do not have base sets of facts about how complex systems work, and do not attempt to reason over a corpus of evidence and facts. as a result, I would expect that an LLM might concoct an interesting story about why such a failure occurred, and it might even be a convincing story if it happened to weave bits of context, accurately into the storyline. It might even, purely randomly, generate a story that actually correctly diagnosed the root cause of the failure, but that would be coincidental based on the similarity of the prompt to text of similar postmortem discussions that were part of its training set. If you had an extremely detailed postmortem document, then I would expect LLM‘s to do a very good job of summarizing such document. But I don’t see why an LLM is an appropriate tool for analyzing failures in complex systems; just as I don’t see a hammer being a very effective tool for tightening bolts. Right now, I am concerned that the relative ease that modern frameworks provide to author LLM based applications, is leading many people to optimistically include LLM technology in attempts to solve problems that it doesn’t seem particularly well suited to solve.

from-nibly 2y ago
The main problem is. Most people cant tell the difference between an expert opinion and random words. So if an LLM shoots a bunch of jargon on the screen then, well so does my senior platform engineer so whats the difference? The LLM is cheaper and is always on call.
- RodgerTheGreat 2y ago
  I see deeply ominous connections between generations of American children who have been educated in a manner which produces alarming rates of functional illiteracy[0] and the widespread popularity of machine learning models that produce nonsense which resembles plausible text if you skim or don't actually read it. [0] https://www.apmreports.org/episode/2019/08/22/whats-wrong-how-schools-teach-reading https://www.apmreports.org/episode/2019/08/22/whats-wrong-ho...

sickblastoise 2y ago

Here’s a simple rule, based on the fact no one has shown that an llm or a compound llm system can produce an output that doesn’t need to be verified for correctness by a human across any input: The rate at which llm/llm compound systems can produce output > the rate at which humans can verify the output I think it follows that we should not use llms for anything critical. The gunghoe adoption and hamfisting of llms into critical processes, like an AWs migration to Java 17, or root cause analysis is plainly premature, naive, and dangerous.

latentnumber 2y ago
Why not automate verification itself then? While not possible now, and I would probably never advocate for using LLMs in critical settings, it might be possible to build field-specific verification systems for LLMs with robustness guarantees as well.
- RodgerTheGreat 2y ago
  If the verification systems for LLMs are built out of LLMs, you haven't addressed the problem at all, just hand-waved a homunculus that itself requires verification. If the verification systems for LLMs are not built out of LLMs and they're somehow more robust than LLMs at human-language problem solving and analysis, then you should be using the technology the verification system uses instead of LLMs in the first place!
  - wbogusz 2y ago
    > If the verification systems for LLMs are not built out of LLMs and they're somehow more robust than LLMs at human-language problem solving and analysis, then you should be using the technology the verification system uses instead of LLMs in the first place! The issue is not in the verification system, but in putting quantifiable bounds on your answer set. If I ask an LLM to multiply large numbers together I can also very easily verify the generated answer by topping it with a deterministic function. I.e. rather than hoping that an LLM can accurately multiply two 10 digit numbers, I have a much easier (and verified) solution by instead asking it to perform this calculation using python and reading me the output

m1keil 2y ago

Folks let's be real. While the tech industry borrows terms and procedures from mature and inherently riskier industries like Aviation, 99% of the tech companies don't share the same risk profile. This means that in most cases, these RCAs are the output of a long and over engineered incident review process that was designed to impress the higher echelon. The problem is, that in a decently sized corporation, you have tens to hundreds of daily fuck ups (also known as "incidents") that completely suck out the free time out of engineers that have to navigate the long game of post incident management process. The utilisation of LLMs on these cases are just engineered solution to the problem of organisational bureaucracy.

aragilar 2y ago
Is not aviation, space-flight, etc. part of "tech"? While you can always over-do processes, you can also under-do them as well (I don't think I need to list examples here, just look at the news...). Having the processes also be able to adapt to different needs and requirements is part of having good processes.
- m1keil 2y ago
  I don't think symantics really matter here. I largely think of "tech" as companies that produce mainly virtual goods in the form of software. Most of these software is largely skins over databases. In most cases, nobody will get hurt if their product will malfunction. This is the vast majority of tech companies today. Exceptions exists. I would take under process any day of the week. From my experience, adding a process is far easier than removing it.
  - aragilar 2y ago
    If you do "tech" == "software" maybe I'd agree with you no one will get hurt (though the cynical part of me says "did you measure this, or do you want this to be true"), but S(cience)T(echnology)E(ngineering)M(athematics) is a thing, and it's definitely not just software.
droopyEyelids 2y ago

charleslmunger 2y ago

The statement about "4,500 developer-years of work" is insane to me. Java is one of the most backward compatible languages period - other than hashmap iteration order a while back, it's hard to think of what could require that astronomical quantity of engineering effort to upgrade. Do they actually budget over a billion dollars to upgrade Java versions, or is this like "this amazing tool, sed, saved us infinity developer years by replacing strings at one quadrillionth the cost of a $500k human editing each text file by hand"

deleted 2y ago
[deleted]
efitz 2y ago
<satire/>In unrelated news, Amazon announces layoff of 4500 developer positions...
DaiPlusPlus 2y ago
> "The average time to upgrade an application to Java 17 plummeted from what’s typically 50 developer-days to just a few hours." *blink* ...why are they boasting about migrating to Java 17... in 2024? ...and from what versions were they migrating from? Java 17 didn't introduce any significant breaking-changes (IME) for users on Java 16 or even the next previous LTS version, Java 11 - I don't work at Amazon, but surely Amazon isn't in the habit of running on unsupported JVMs? - so assuming these Java projects were being competently maintained, then the only work actually required to migrate to 17 is changing your `org.gradle.java.home=` path to where JDK 17 followed by running your test suite. If Amazon was using a monorepo then 1 person could do this in 5 minutes with a 1-liner awk/sed command - whereas I expect it would likely take an AI far longer to do this, if it's even able to make sense of an Amazon-sized monorepo - they'd also likely re-prompt it for every separate project for reliability's sake. So after considering all that, the "50 days" number he gives, without any context either, is a nice shorthand to communicate his disconnection from what really goes-on inside his org... or he's lying - and he knows he's lying - but he also knows there won't be any negative consequences for him as a result of his lying, so why not lie if it gives you a good story to tell for LinkedIn? In conclusion: these remarks by leadership unintentionally make the company look bad, not good, once you fill-in-the-blanks to make up for what they dind't say. (What's next...? Big Brother increasing our chocolate ration to 20 grammes per week?) ----- One more thing: the comment-replies to the post on LinkedIn are utterly derranged and I genuinely can't put my sense of unease into words.

deleted 2y ago

[deleted]

ryoshu 2y ago

Why would you ever use a non-deterministic model for a deterministic function?

indus 2y ago

Most LLMs have an accuracy benchmark for controlled questions & answers. Even if this accuracy is 95% then in a complex system the probability of getting to the right answer diminishes with each new step being added. This is also the key tenet of an agentic system. While the analysis in the blog is excellent but an answer needs to be found. A layer on top of LLMs for error control/check. As an analogy, in the analog to digital transmission stack of OSI, an error correction mechanism such as frame check sequence (fcs) detects transmission errors in the data link layer.

rboyd 2y ago

FTA: "If we offload the RCA learning/categorization part to the LLM (whatever that means), we wouldn't be able to make much progress in the enhancing reliability and safety part." But you don't offload it in the sense that you expect the tool to completely take the wheel. You ask it for suggestions to inform a human. If the suggestions turn out to only be a distraction in your environment then you abandon the tool. For plenty of environments the suggestions will be hugely useful and save you valuable time during an ongoing outage.

layer8 2y ago
They are specifically talking about automating RCA, not about tool-assisted RCA.
- rboyd 2y ago
  Go read the papers on automated RCA. The algorithms are designed to suggest top K candidates for RCA. By definition at least k-1 are going to be wrong. https://arxiv.org/abs/2305.10638 https://arxiv.org/abs/2305.10638

zebomon 2y ago

This articles speak directly to what I believe will become the unfortunate and inconvenient reality of LLMs' core limitation: at some point, someone has to 1) benefit from what's been generated and 2) (and more importantly) know they're benefiting from what's been generated. If for example you have an LLM agent that's effectively "solved" every security flaw your software may encounter for the next 50 years, unless it can simultaneously impart 50 years of training to the people who rely on the software, it's done nothing but introduce us to more complex flaws that we would need approximately 49 more years of experience to tackle ourselves.