4 ms·
A transmission error has a strictly contained, predictable blast radius. If a packet drops, the system knows exactly how to handle it: it throws a timeout, drop
by distalx 5mo ago
A transmission error has a strictly contained, predictable blast radius. If a packet drops, the system knows exactly how to handle it: it throws a timeout, drops a connection, or asks for a retry. The worst-case scenario is known.
A reasoning error has an infinite, unpredictable blast radius. When an LLM hallucinates, it doesn't fail safely but it writes perfectly compiling code that does the wrong thing. That "wrong thing" might just render a button incorrectly, or it might silently delete your production database, or open a security backdoor.
You can build reliable abstractions over failures that are predictable and contained. You cannot abstract away unpredictable destruction.
- yunwal 5mo ago> A reasoning error has an infinite, unpredictable blast radius. Says who? It’s quite easy to limit the blast radius of a reasoning error.
- amazingamazing 5mo agoHow so? Suppose you had: Math() Add() Subtract() Program() Math(“calculate rate”) This is intentionally written vaguely. How do you limit that these implementations ensure Program() runs and does the right thing when there is no guarantee Math() or its components are correct? Normally you could use a typed programming language, unit tests, etc, but if LLM is the ultimate abstraction programs will be written line above. At some point traditional software engineering principles will need to apply.
- distalx 5mo agoIn 2024, a Chevy dealership deployed an AI chatbot that confidently agreed to sell a customer a 2024 Chevy Tahoe for $1. It executed a catastrophic business failure simply because it didn't know the logic was wrong. Sure, you can patch that specific case with guardrails, but how many unpredictable edge cases are you going to cover? It only takes a user with a bit of ingenuity to circumvent them. There are already several examples of AI agents getting stuck in infinite loops, burning through massive API bills while achieving absolutely nothing. You can contain a system failure, but you cannot contain a logic failure if the system doesn't know the logic is wrong.
- pear01 5mo agoThis would be more convincing if a single car had been exchanged for $1. It didn't happen. Seems the bug was "contained". Sort of undermines your point re "catastrophic business failure" don't you think?
- yunwal 5mo ago> but how many unpredictable edge cases are you going to cover? This is the wrong question. The correct question is what specific subsets of cases do you allow, similar to any security question
- yunwal 5mo agoYou literally can contain a logic failure. If I execute logic on my computer that’s not connected to the internet it can’t get out of the box. Done. Contained
- td2 5mo agoI mean if your talking about packets, your already one abstraction over the real data Transmission, in wich is noisy. So bits can randomly flip, noise could be interpreted as bits, and bits could get lost. A much larger blast radius
- deleted 5mo ago[deleted]
- harrall 5mo agoA transmission error does not have a strictly contained blast radius. A bad packet could tell a flying probe to fire all thrusters on and deplete its fuel in 15 minutes. What makes a transmission error controlled is all the protection mechanisms on top of it. An LLM cannot delete a production database unless you give it access to do it. My hot take is that many people are naturally more comfortable with deterministic systems that have clearly analyzable outcomes. Software engineering has historically primarily been oriented around deterministic systems and it has attracted that type of thinker. But many of us, myself included, prefer chaotic systems where you can’t fully nail down every cause and effect. The challenge of building a prediction model on top of chaos is exhilarating. I really don’t find many people like me in SWE as in, say, the graphics design department. To me, that’s the underlying threat here — LLMs are rewriting a field that has previously self selected a certain type of person and this, quite understandably, rubs them the wrong way.
- aeon_ai 5mo agoInsightful. Feels like this maps to the J/P of Myers Briggs
- c-linkage 5mo agoI don't need to be able to write proofs about my maths using logic and determinism. If the answer comes out in a way that I like then it has to be correct!
- dpark 5mo agoThis is vapid condescension. The comment you replied to made no statements about math or proofs. They made a statement about working in systems of non determinism effectively. Your statement seems to imply that this is dumb, as if working in a world of full determinism is an option.
- deleted 5mo ago[deleted]
- panarky 5mo ago
- DiscourseFan 5mo agoVery few people are even beginning to understand the constraints of these systems, and none of them have yet been elevated to high enough positions of prominence to rise above the noise of all the hype. Give us some time man, jeez