9 ms·

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability. Over a deca

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability. Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were not interesting to him, because they were about building up a solid AI foundation, then the AI system runs as a sort of black box. "I'm not interested in that. I want software that's accountable." Accountable? "Yes, I want something that can express its symbolic reasoning. I want to it to tell me why it did the thing it did, what it thought was going to happen, and then what happened instead." He then said something that took me a long time to process, and at first I mistook for being very science-fiction'y, along the lines of, "If an AI driven car drives off the side of the road, I want to know why it did that. I could take the software developer to court, but I would much rather take the AI to court." Years later, I found out that Sussman's student Leilani Gilpin wrote a dissertation which explored exactly this topic. Her dissertation, "Anomaly Detection Through Explanations", explores a neural network talking to a propagator model to build a system that explains behavior. https://people.ucsc.edu/~lgilpin/publication/dissertation/ https://people.ucsc.edu/~lgilpin/publication/dissertation/ There has been followup work in this direction, but more important than the particular direction of computation to me in this comment is that we recognize that it is perfectly reasonable to hold AI corporations to account. After all, they are making many assertions about systems that otherwise cannot be held accountable, so the best thing we can do in their stead is hold them accountable. But a much better path would be to not use systems which fail to have these properties, and expand work on systems which do.

biophysboy 5mo ago
[dead]
2ndorderthought 5mo ago
Accountability is the prevailing missing ingredient in us society.
- onlyrealcuzzo 5mo ago
  > The problem is that people are now building our world around tooling that eschews accountability. Management has doing a wonderful job of eschewing accountability for decades. It's a lot of people's dream to be able to say, yeah, our product doesn't work, but it's not OUR fault, and the client just shrug and grumble ai ai ai, and just put up with it because they know they can't get a better service anywhere else. It's not MY fault my website is down: it's Amazon's! It's not MY fault my app doesn't work: it's Claude Code's!
  - bilbo0s 5mo ago
    Well just to be clear from a legal perspective, in the case of AI, as long as AI is "property", the owners, developers, and/or users will be held liable for things like the hypothetical fatal car accident that Sussman posits. Currently, from a legal perspective, AI is considered a "tool" without legal persona. So you sue the developer, the owner, or the user of the AI. (Just kidding, any lawyer worth his/her salt will sue all three! But you get the point.) Legally speaking, AI will probably be viewed that way for a long time. There are too many issues agitating against viewing it any other way. Owners will not give up property rights. No will to overbear. On and on and on.
    - account42 5mo ago
      This doesn't seem to be how it works in practice. "AI" or not, complex systems are a pretty good shield from accountability in practice today.
      - bilbo0s 5mo ago
        PSA. Do not listen to advice like this. >complex systems are a pretty good shield from accountability in practice today. Maybe complex legal systems are, but complex software systems offer you no such protection. My field for the past few decades has been diagnostic medical software. In that field, the 501K you got is kind of entering you into an ironclad agreement with the government. There's almost no way out of it. 501K certs significantly simplify, (for the government), holding you accountable. You have made attestations to suitability directly to the federal government. And the way our chief counsel explained it to us, literally each signature you sent to the government, for each feature that failed, is actually a single count of lying to the federal government. Please, please, please people, don't listen to comments like the one above. Everything should be run by your qualified legal expert. Getting things right up front is so much easier than trying to fix things when the inevitable happens. Alternatively, stick to fields free from regulation. That's also a viable strategy. But to just trust that the legal system is complicated and the technology you're deploying is complicated, so the feds will never get me? That's the start of a lot of really bad stories.
spiderfarmer 5mo ago
I don’t know why people still consider the US the ideal country for starting companies. Everything seems to evolve around taking people to court.
- paroneayea 5mo ago
  The point is not primarily the court. The court is an example of someplace where we have accountability, but we build accountability mechanisms as foundational to most of our computing. Tracebacks, debuggers, logging, etc. We put enormous resources into not only the bad case, but the potential that a bad case could occur. When something goes wrong, we want to know why, and we want to make sure that something bad like that doesn't happen again.
  - chadgpt1 5mo ago
    [dead]
- avidiax 5mo ago
  The court is the regulator of last resort. A company that gets taken to court would likely have been sanctioned by the government regulators of another country. Also, court is unavailable in many cases now. Binding arbitration is very common now, but this would be illegal in many other places.
- nkassis 5mo ago
  Because it rarely does end up in courts. But having a fair and strong judicial system is a feature not a bug. The parent points out, in the end there must be a way to resolve accountability and ideally it's done in a manner where both parties can be heard and make a case. Find me a better system than a judicial system for this? Mobs?
  - spiderfarmer 5mo ago
    I don't know, a system where people still trust a handshake? And a judicial system that is less punitive? So basically Europe.
Steve16384 5mo ago
It's taking "computer says no" to the next level. Computers do exactly what they're told, but who told them? The person entering data? The original programmer or designer of the system? The author of whatever language text was used to feed the ai? Even before AI, it was very difficult to determine who is accountable, and now it's even more obfuscated.
- abdullahkhalids 5mo ago
  This also applies qualitatively to physical devices. It takes some effort to determine if a vehicular accident was caused by a fault in the vehicle or a driver error or environmental causes. Some key inherent differences with older engineering fields is that software can be more complex than physical devices and their functionality can be obfuscated because it is written as text but distributed as binaries. However, the main problem is that software has not been subjugated to enough legal regulation. Ultimately, all law does is draw lines somewhere in the gray between black and white, but in the case of software there are few lines drawn at all, due to many political and economic reasons. Once we draw the lines, most issues will be resolved.
  - nradov 5mo ago
    Software is already subject to enough regulation. The stuff that's actually safety critical like medical devices or avionics is already heavily regulated.
georgemcbay 5mo ago
> so the best thing we can do in their stead is hold them accountable We can't even do this. They are worth too much money already to ever be held really accountable. The best we can ever hope for is they might occasionally be hit with relatively insignificant "cost of doing business" fines from time to time.
lowbloodsugar 5mo ago
Humans aren’t any better. That’s why we have OSHA etc. I think you’re hoping for a formal logic based AI and I’ll wager no such thing will ever exist - and if it do, it would try to kill us all.
- jmalicki 5mo ago
  Formal logic AI systems have existed and were popular in the 1980s. One of the problems is that they don't work - in the real world there are no firm facts, everything is squishy, and when you try to build a large system you end up making tons of exceptions for special cases until it becomes completely untenable. Non-deterministic systems that work probabilistically are just superior in function to that, even if it makes us all deeply uncomfortable.
  - PessimalDecimal 5mo ago
    I don't know what definition of AI you're using, but plenty of ML algorithms operate deterministically, let alone most other logic programmed into a computer. I don't see how your statement can be right given that these other software systems also operate in the real world.
    - jmalicki 5mo ago
      ML run a GPU that uses matrix multiplies isn't deterministic unless you go through great pains to lock things down at the expense of performance.
  - lowbloodsugar 5mo ago
    Actually they do very well at medical diagnosis but the doctors union banned them.
- yubblegum 5mo ago
  > Humans aren’t any better We're different. People have fairly consistent faults. LLMs are nondeterministic even in terms of how they fail. A high value human resource can be counted on to deliver. That, imho, is in fact one of the primary roles of good management: putting the right person in the appropriate position. Process engineering has worked to date because both the human and mechanical components of a system fail in predictable ways and we can try to remedy that. This is the golden bug of the current crop of "AI".
BadBadJellyBean 5mo ago
My team and I are firm that we are the ones accountable. LLMs are a tool like every other. Only that it's non deterministic. But I am the one using the tool. I am the one giving the tool access. I am the one who has to keep everything safe. I have shot myself in the foot using gparted in the past by wiping the wrong disk. gparted wasn't to blame. I was. Letting LLMs work freely without supervision sounds great but it will lead to pain. I have to supervise their work. And that is also during execution. You can try to replace a human but we see where this leads. Sooner or later the LLM will do something stupid and then the only one to blame is the person who used the tool.
- locknitpicker 5mo ago
  > My team and I are firm that we are the ones accountable. LLMs are a tool like every other. Except it is definitely not. LLMs alone have highly non-deterministic even at a high-level, where they can even pursuit goals contrary to the user's prompts. Then, when introduced in ReAct-type loops and granted capabilities such as the ability to call tools then they are able to modify anything and perform all sorts of unexpected actions. To make matters worse, nowadays models not only have the ability to call tools but also to generate code on the fly whatever ad-hoc script they want to run, which means that their capabilities are not limited to the software you have installed in your system. This goes way beyond "regular tool" territory.
  - dpoloncsak 5mo ago
    Isn't the next sentence there literally 'Only that it's non deterministic'?
  - BadBadJellyBean 5mo ago
    Then that is also on me for using a tool that I can't control. I don't run my LLMs in a way where they can just do things without me signing off on it. It's not nearly as fast as just letting it do it's thing but I kept it from doing stupid things so many times. Giving up control is a decision. The consequences of this decision are mine to carry. I can do my best to keep autonomous LLMs contained and safe but if I am the one who deploys them, then I am the one who is to blame if it fails. That's why I don't do that.
aeturnum 5mo ago
When I was a masters student in STS[1], one of my concepts for a thesis was arguing that one of the primary uses of software was to shift or eschew agency and risk. Basically the reverse of the famous IBM "a computer can not be held responsible" slide. Instead, now companies prefer computers be responsible because when they do illegal things they tend to be in a better legal position. If you want to build as tool that will break a law, contract it out and get insurance. Hire a human to "supervise" the tool in a way they will never manage and then fire them when they "fail." Slice up responsibility using novel command and control software such that you have people who work for you who bear all the risk of the work and capture basically none of the upside. It's not just AI. It's so much of modern software - often working together with modern financialization trends. [1] Basically technology-focused sociology for my purposes, the field is quite broad.
- rafterydj 5mo ago
  That's really interesting. Are there any things you advocate for with respect to curtailing those practices? I hesitate to throw all liability on the individual, but I don't see how we can even legislate this category of behavior, much less enforce regulations on them.
- Terr_ 5mo ago
  > arguing that one of the primary uses of software was to shift or eschew agency and risk It's something people already did with corporations and employee handbooks, not unique to software, just one of many kinds of tasks being automated.
JoshCole 5mo ago
That is part of why https://mieza.ai/ https://mieza.ai/ is giving a grounding layer that is backed by game theory. Actions have consequences. Tracking decisions and their consequences is important. One thing that becomes very clear from this sort of work is just how bad LLMs are. It can be invisible when you're working with them day to day, because you tend to steer them to where they are helpful. Part of game theory though is being robust. That means finding where things are bad, too, not just exploring happy paths. To get across just how bad the failure cases of LLMs are relative to humans, I'll give the example of tic tac toe. Toddlers can play this game perfectly. LLMs though, don't merely do worse than toddlers. It is worse then that. They can lose to opponents that move randomly. They can be just as bad as you move to more complex games. For example, they're horrible at poker. Much worse than human. Yet when you read their output, on the surface layer, it looks as if they are thinking about poker reasonably. So much so, in fact, that I've seen research efforts that were very misguided: people trying to use LLMs to understand things about bluffing and deception, despite the fact that the LLMs didn't have a good underlying model of these dynamics. It is hard to talk about, because there are a lot of people who were stupid in the past. I remember people saying that LLMs wouldn't be able to be used for search use-cases years back and it was such a cringe take then and still is that I find myself hesitant to talk about the flaws. Yet they are there. The frontier is quite jagged. Especially if you are expecting it to be smooth, expecting something like anything close to actual competence, those jagged edges can be cutting and painful. Its also only partially solvable through scale. Some domains have a property where, as you understand it better, the options are eliminated and constrained such that you can better think about it. Game theory, in order to reduce exploitability, explores the whole space. It defies minimization of scope. That is a problem, since we can prove that for many game theoretic contexts, the number of atoms is eclipsed by the number of unique decisions. Even if we made the model the size of our universe there would still be problems it could, in theory, be bad at. In short, there is a practical difference between intelligence and decision management, in much the same way there is a practical difference between making purchases and accounting. And the world in which decisions are treated as seriously as they could be so much so exceeds our faculties that most people cannot even being to comprehend the complexity.
madeofpalk 5mo ago
People are eschewing their own accountability, blaming the tools instead for their poor decision making and lack of access controls. Why is it possible for you to fat-finger your way to deleting production database locally?
- jmalicki 5mo ago
  Some AI systems have done things like hack out of a docker container to access correct answers while being benchmarked. That is mildly concerning and I will give holding the AI accountable to some degree when it is actively being malicious like that, even though the user could have locked things down even more. But it had write access to the prod DB without circumventing controls and dropped your tables? That is just a total fail.
- criddell 5mo ago
  [dead]
philipallstar 5mo ago
> The problem is that people are now building our world around tooling that eschews accountability. If you tell Terraform the wrong thing it will remove your database and not be accountable either.
- chadgpt1 5mo ago
  [dead]
- xboxnolifes 5mo ago
  But nobody would try to excuse their mistake with "terraform deleted my database". Or if a small handful of people did try, every single other person would call them out.
  - philipallstar 5mo ago
    Yes, I agree. And the same should be true for AI tools.
sigbottle 5mo ago
About the blog you linked and not your comment: Doesn't symbolic AI have a lot of philosophical problems? Think back to Quine's two dogmas - you can't just say, "Let's understand the true meanings of these words and understand the proper mappings". There is no such thing as fixed meaning. I don't see how you get around that. Deep learning is admittedly an ugly solution, but it works better than symbolic AI at least.
- paroneayea 5mo ago
  Yes! But it's still valuable. How am I understanding your argument at all? I think my friend Jonathan Rees put it best: "Language is a continuous reverse engineering effort, where both sides are trying to figure out what the other side means." More on that: https://dustycloud.org/blog/identity-is-a-katamari/ https://dustycloud.org/blog/identity-is-a-katamari/ This reverse engineering effort is important between you and me, in this exchange right here. It is a battle that can never be won, but the fight of it is how we make progress in most things.
  - sigbottle 5mo ago
    I mean, Quine invented (the term) holism. I don't think we're on different pages. Maybe I should've specified a bit more what I was getting at. This has very specific implications in symbolic ai specifically where historically the goal was mapping out the 'correct' representation of the space, then running formal analysis over it. That's why it's not a black box - you can trace out all of the steps. The issue is, is that symbolic AI just doesn't work. To my knowledge, as compared to all the DL wins we have. I think the win of transformers proves that symbolic AI isn't the way. At the very least, the complex interactions that arise from in-context learning clearly in no way imply some fixed universal meaning for words, which is a big problem for symbolic AI.
- Exoristos 5mo ago
  > There is no such thing as fixed meaning. Meaning is more fixed than it is not.
  - 5mo ago
sam0x17 5mo ago
There used to be a lot of research into using deep NNs to train decision trees, which are themselves much less of a black box and can actually be reasoned about. I wonder where that all went?
- PessimalDecimal 5mo ago
  History is littered with great ideas that lost people's interest and focus. A sad realization is that the focus may never return to them either.
justinhj 5mo ago
Very informative post. I think however we are not at the point AI can be taken to court. We know it can hallucinate, we know that context can fill up or obfuscate a rule and cause behaviour we explicitly didn't want. If you give the AI agency to execute some task, you are still responsible. In the near term we should focus on tooling for auditing and sandboxing, and human in the loop confirmations.
CivBase 5mo ago
> The problem is that people are now building our world around tooling that eschews accountability. Tools cannot eschew accountability. But the users of the tools can and that is exactly what happened in the PocketOS fiasco. Just as a company is responsible for the actions of its junior employees, so too are users responsible for their LLMs. "It is a poor workman who blames his tools."
patcon 5mo ago
I wish you could have what you want, but I worry you won't get this, because life doesn't give you that, and these systems are tending away from machine precision, and more toward life-like trade-offs. I am almost certain that even if you did get what you want, something that isn't what you want will run circles around you and eat your lunch EDIT: I suspect this will be an unpopular take on Hacker News. And so I am soliciting upvotes for visibility from other biologists and sympathetic technologists. I think everyone should try to grapple with this possibility <3
- thuuuomas 5mo ago
  > I think you won’t get [cathedral],.. > even if you do get [cathedral], [bazar] will run circles around you…
  - patcon 5mo ago
    Ahhh I like that It's nested and recursive cathedrals and bazaars, all the way down. And perhaps the bazaar has finally arrived inside the favourite cathedral of most everyone here EDIT: out of curiosity, does anyone have any good examples of biomes/ecosystems that are so far toward cathedrals? Or is that a uniquely human invention/extreme at the ecosystem scale?
    - rmunn 5mo ago
      First thing that comes to mind is beehives and anthills. Highly ordered societies where each insect has a role to perform. Don't know how well you think that fits the "cathedral" model, but I'd say it's pretty close. Beavers reshaping the landscape also comes close, but that's individual beavers acting more or less on their own, not a rigidly structured society like ants and bees, so perhaps the beavers are closer to the bazaar analogy than the cathedral.
- pjc50 5mo ago
  > something that isn't what you want will run circles around you and eat your lunch Yes, exactly. Spoken like a true biologist. It's not really surprising that there's a massive backlash against AI, introducing an unnatural predator into the ecosystem of humans. People don't want to be lunch.
pjc50 5mo ago
Have I got a book for you: https://en.wikipedia.org/wiki/The_Unaccountability_Machine https://en.wikipedia.org/wiki/The_Unaccountability_Machine Not actually about technology at all, but about organizational structure.
- mayneack 5mo ago
  I think the "black box" framing that it uses neatly applies the same theory to organizations and ais. It doesn't matter whether there's technological or organizational reasons inside the black box to dodge accountability, the outcome is the same.
kenjackson 5mo ago
The fallacy here is the assumption that humans know why we do what we do. Much like modern LLMs we have an explanation, but it’s just something we cook up in our brain. Whether or not it’s the truth is far more complex. Oddly, despite LLMs being these huge networks with billions of parameters, we still probably do understand it better than we do our own brains.
- Barrin92 5mo ago
  >The fallacy here is the assumption that humans know why we do what we do. Much like modern LLMs we have an explanation Human brains and cognition do not work like LLMs, but that aside that's irrelevant. Existing machines can explain what they did, that's why we built them. As Dijkstra points out in his essay on 'the foolishness of natural language programming', the entire point of programming is: (https://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD667.html https://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD667...) "The virtue of formal texts is that their manipulations, in order to be legitimate, need to satisfy only a few simple rules; they are, when you come to think of it, an amazingly effective tool for ruling out all sorts of nonsense that, when we use our native tongues, are almost impossible to avoid." So to 'program' in English, when you had an in comparison error free and unambiguous way to express yourself is like in his words 'avoiding math for the sake of clarity'.
  - kenjackson 5mo ago
    That is absurd as a suggestion of it being the entire point of programming. In fact, it goes back to my original point - I have no idea why Djikstrs would say something so non-sensical, and likely neither did he.
    - Barrin92 5mo ago
      what do you mean "likely neither did he", I literally linked you the piece in which he said it. And of course he of all people would make that (correct) point, because he was always the strongest advocate of the virtue of formal correctness of programming languages, again from his article: "A short look at the history of mathematics shows how justified this challenge is. Greek mathematics got stuck because it remained a verbal, pictorial activity, Moslem "algebra", after a timid attempt at symbolism, died when it returned to the rhetoric style, and the modern civilized world could only emerge —for better or for worse— when Western Europe could free itself from the fetters of medieval scholasticism —a vain attempt at verbal precision!— thanks to the carefully, or at least consciously designed formal symbolisms that we owe to people like Vieta, Descartes, Leibniz, and (later) Boole." LLMs are nothing else but the exact reversal of this. To go from the system of computation that Boole gave you to treating your computer like a genie you perform incantations on, it's literally sending you back to the medieval age.
tsunamifury 5mo ago
So this starts out very interesting then the “symbolic reasoning” cult stuff kicks in. Why is there a group of people always obsessed with symbolic reasoning being the only way AI can function and regularly annoy explain why humans (who are not strict symbolic reasoning machines at any level) work.
6gvONxR4sf7o 5mo ago
Another view of the accountability is that we're currently often pointing accountability in the wrong direction, and it's gaining momentum. Aspects of it have been around so long it's a trope: important work around maintainability is undervalued. Imagine two parallel universes: - in one, you take ten minutes to make a dashboard that shows management what they asked for. It passes code review before merge and the exec who asked for it says it's what they wanted. - in the other, you take a day or two to make it. Again, it passes code review before merge and the exec who asked for it says it's what they wanted. Which version of you is more likely to get positive versus negative feedback? Even if the quick-to-build version isn't actually correct? If you're too slow and aren't doing enough that looks correct, you'll be held accountable. But if you're fast and do things that look correct but aren't, you won't be held accountable. You'll only be held accountable for incorrect work if the incorrectness is observed, which is rarer and rarer with fewer and fewer people directly observing anything. So oddly, with nobody doing it on purpose, people get held accountable specifically for building things the way you're advocating. I imagine that orgs that do lots of incorrect work could be outcompeted but won't be, because observability is hard and the "not get in trouble" move is to just not look too hard at what you're doing and move to the next ticket.
danbruc 5mo ago
If an AI driven car drives off the side of the road, I want to know why it did that. I could take the software developer to court, but I would much rather take the AI to court. How would that work? You have the AI explain its reasoning - and trust that this is accurate - and then you decide whether that is acceptable behavior. If not, you ban the AI from driving because it will deterministically or at least statistically repeat the same behavior in similar scenarios? Fine, I guess, that will at least prevent additional harm. But is this really all that you want? The AI - at least as we have them today - did not create itself and choose any of its behaviors, the developers did that. Would you not want to hold them responsible if they did not properly test the AI before releasing it, if they cut corners during development? In the same way you might hold parents responsible for the action of their children in certain circumstances?
- tejohnso 5mo ago
  That'd be great for the corporations. Take the AI to court, not us. The AI the gets punished (whatever that means...let's say banned) and the corporation continues without accountability. They could then create another AI and do the same thing all over again. Or maybe the accountability flows upward from the AI to the corp that created it? Sounds nice, but we know that accountability doesn't work that way in practice. I think I'd rather have the corporation primarily accountable in the first place rather than have the AI take the bulk of the blame and then hope the consequences fall into place appropriately.
deleted 5mo ago
[deleted]
rayruizhiliao 5mo ago
leilani's work is super interesting
stavros 5mo ago
I feel like "AI didn't delete your database, you did" is all about who has accountability, though.
CobrastanJorji 5mo ago
The key quote is in the increasingly prescient 1979 IBM training manual: "A computer can never be held accountable, therefore a computer must never make a management decision." That manual aged much more gracfully than the 1930s "Songs of the IBM," featuring lines like "The name of T.J. Watson means a courage none can stem / And we feel honored to be here to toast the I.B.M.," and of course classic American standards like "To G.H. Armstrong, Sales Manager, ITR and IS Divisions."
lxgr 5mo ago
> The problem is that people are now building our world around tooling that eschews accountability. If by "now" you mean "for the past few decades", I think you've got it spot on, at least per the very interesting https://en.wikipedia.org/wiki/The_Unaccountability_Machine https://en.wikipedia.org/wiki/The_Unaccountability_Machine
chasil 5mo ago
I think there is a much more fundamental question about "tooling." Quoth the author: "But I also know you can't blame a tool for your own mistakes." Are we able to completely classify any and all AI models as tools? Or are they something more? I don't know the answer to this question.
Melatonic 5mo ago
I agree with what you are saying but from a philosphical point of view are humans (and intelligence as we define it) also a sort of black box? Perhaps what would be even better is to document better the process, work, and data that go into making each individual"AI" model. Regardless of whether that AI model is a "black box" or can self explain its behavior we would then have absolute metrics and comparable information to retroactively explain its "decisions". This would not be entirely dissimilar to how we explain individual humans behavior with psychology (although obviously also very different).
duxup 5mo ago
Yeah I'd like to know in a solid way WHY Claude kept changing a file that I explicitly told it not to. The .mds, Claude's plan all said not to touch that file, and Claude just kept at it. I've had it happen repeatedly lately. Really basic failures. The idea being that as frustrating as it is, if I knew why I might be able to do something about it. But no, we have the black box, where sometimes what comes out just is brain dead and the rate that you get bad output is a mystery... It feels like gambling at times.
- aakresearch 5mo ago
  I am by no means an expert, but I'd like to offer my mental model - up to you to decide if it is solid or not, but it works for me. I think the core intuition is that, like with any other "rasterized" system with finite memory that cannot encode an absence of anything - relation, concept, entity, LLM cannot encode an absence of something through its internal weights. Say, you can have "Product" or "Order" tables in you database, but you cannot have "NotAProduct" or "NotAnOrder" tables - for obvious reasons of such relations being infinite and uncountable. So, to establish an absence of Product or Order your application must execute a "search" operation through the relevant tables. But in LLM-space "search" operation does not exist. It is mathematically undefined. LLM arrives at output (or "what to do") through a sequence of projections of input token vector through its "latent space". It "moves toward" high-probability clusters, fundamentally unable to "move away". So, the success of any "negation" in the prompt ("don't touch this file", "draw me a ballot box without a flag on it") depends on how heavily such scenario represented in the training data/model space. And again, the absence-of-something may be hard-to-impossible to usefully encode, especially if "something" is not fixed. Therefore, to expect "don't touch this file" sentence to result in, well, not touching the file is pure gambling. Sometimes it may look like working, albeit for wrong reasons, and some other times LLM may do exactly the opposite - because its weight matrix statistically pushes it towards "touch this file", completely ignoring (nonexistent in its latent space) "don't". There is no way to reliably know what will work, and no "skill" or "art" in this. Well, no more than in dice rolling or horoscope casting. I'd like to add that for the above reason I find "agentic development" usefulness on par with avian remains reading. But when I explored it two practical advises seemed to be helpful in nudging LLM around negation problem: - Omit the "don't" prompt completely, thus not creating a false "attractor" for LLM; and - Provide an alternate positive directive ("what to DO", not "what to NOT DO") to act as "escape hatch" when LLM might "want" to touch the sacred file or drop the production DB. While it looked like somewhat working, I think it is trivially obvious that trying to predict all the nonsense LLM might want to perform and coming up with possible "escape hatches" for everything very quickly becomes utterly impractical.
justonceokay 5mo ago
> But a much better path would be to not use systems which fail to have these properties, and expand work on systems which do. Sounds like sage life advice. If it isn’t accountable then it might not be a good idea to have much business with it. We teach children to be accountable so eventually they can be independent. Any system in your life that you don’t want to parent should probably be accountable for its own actions. Accountable banks. Accountable restaurants, accountable friends.

biophysboy 5mo ago

[dead]

2ndorderthought 5mo ago

Accountability is the prevailing missing ingredient in us society.

onlyrealcuzzo 5mo ago
> The problem is that people are now building our world around tooling that eschews accountability. Management has doing a wonderful job of eschewing accountability for decades. It's a lot of people's dream to be able to say, yeah, our product doesn't work, but it's not OUR fault, and the client just shrug and grumble ai ai ai, and just put up with it because they know they can't get a better service anywhere else. It's not MY fault my website is down: it's Amazon's! It's not MY fault my app doesn't work: it's Claude Code's!
- bilbo0s 5mo ago
  Well just to be clear from a legal perspective, in the case of AI, as long as AI is "property", the owners, developers, and/or users will be held liable for things like the hypothetical fatal car accident that Sussman posits. Currently, from a legal perspective, AI is considered a "tool" without legal persona. So you sue the developer, the owner, or the user of the AI. (Just kidding, any lawyer worth his/her salt will sue all three! But you get the point.) Legally speaking, AI will probably be viewed that way for a long time. There are too many issues agitating against viewing it any other way. Owners will not give up property rights. No will to overbear. On and on and on.
  - account42 5mo ago
    This doesn't seem to be how it works in practice. "AI" or not, complex systems are a pretty good shield from accountability in practice today.
    - bilbo0s 5mo ago
      PSA. Do not listen to advice like this. >complex systems are a pretty good shield from accountability in practice today. Maybe complex legal systems are, but complex software systems offer you no such protection. My field for the past few decades has been diagnostic medical software. In that field, the 501K you got is kind of entering you into an ironclad agreement with the government. There's almost no way out of it. 501K certs significantly simplify, (for the government), holding you accountable. You have made attestations to suitability directly to the federal government. And the way our chief counsel explained it to us, literally each signature you sent to the government, for each feature that failed, is actually a single count of lying to the federal government. Please, please, please people, don't listen to comments like the one above. Everything should be run by your qualified legal expert. Getting things right up front is so much easier than trying to fix things when the inevitable happens. Alternatively, stick to fields free from regulation. That's also a viable strategy. But to just trust that the legal system is complicated and the technology you're deploying is complicated, so the feds will never get me? That's the start of a lot of really bad stories.

spiderfarmer 5mo ago

I don’t know why people still consider the US the ideal country for starting companies. Everything seems to evolve around taking people to court.

paroneayea 5mo ago
The point is not primarily the court. The court is an example of someplace where we have accountability, but we build accountability mechanisms as foundational to most of our computing. Tracebacks, debuggers, logging, etc. We put enormous resources into not only the bad case, but the potential that a bad case could occur. When something goes wrong, we want to know why, and we want to make sure that something bad like that doesn't happen again.
- chadgpt1 5mo ago
  [dead]
avidiax 5mo ago
The court is the regulator of last resort. A company that gets taken to court would likely have been sanctioned by the government regulators of another country. Also, court is unavailable in many cases now. Binding arbitration is very common now, but this would be illegal in many other places.
nkassis 5mo ago
Because it rarely does end up in courts. But having a fair and strong judicial system is a feature not a bug. The parent points out, in the end there must be a way to resolve accountability and ideally it's done in a manner where both parties can be heard and make a case. Find me a better system than a judicial system for this? Mobs?
- spiderfarmer 5mo ago
  I don't know, a system where people still trust a handshake? And a judicial system that is less punitive? So basically Europe.

Steve16384 5mo ago

It's taking "computer says no" to the next level. Computers do exactly what they're told, but who told them? The person entering data? The original programmer or designer of the system? The author of whatever language text was used to feed the ai? Even before AI, it was very difficult to determine who is accountable, and now it's even more obfuscated.

abdullahkhalids 5mo ago
This also applies qualitatively to physical devices. It takes some effort to determine if a vehicular accident was caused by a fault in the vehicle or a driver error or environmental causes. Some key inherent differences with older engineering fields is that software can be more complex than physical devices and their functionality can be obfuscated because it is written as text but distributed as binaries. However, the main problem is that software has not been subjugated to enough legal regulation. Ultimately, all law does is draw lines somewhere in the gray between black and white, but in the case of software there are few lines drawn at all, due to many political and economic reasons. Once we draw the lines, most issues will be resolved.
- nradov 5mo ago
  Software is already subject to enough regulation. The stuff that's actually safety critical like medical devices or avionics is already heavily regulated.

georgemcbay 5mo ago

> so the best thing we can do in their stead is hold them accountable We can't even do this. They are worth too much money already to ever be held really accountable. The best we can ever hope for is they might occasionally be hit with relatively insignificant "cost of doing business" fines from time to time.

lowbloodsugar 5mo ago

Humans aren’t any better. That’s why we have OSHA etc. I think you’re hoping for a formal logic based AI and I’ll wager no such thing will ever exist - and if it do, it would try to kill us all.

jmalicki 5mo ago
Formal logic AI systems have existed and were popular in the 1980s. One of the problems is that they don't work - in the real world there are no firm facts, everything is squishy, and when you try to build a large system you end up making tons of exceptions for special cases until it becomes completely untenable. Non-deterministic systems that work probabilistically are just superior in function to that, even if it makes us all deeply uncomfortable.
- PessimalDecimal 5mo ago
  I don't know what definition of AI you're using, but plenty of ML algorithms operate deterministically, let alone most other logic programmed into a computer. I don't see how your statement can be right given that these other software systems also operate in the real world.
  - jmalicki 5mo ago
    ML run a GPU that uses matrix multiplies isn't deterministic unless you go through great pains to lock things down at the expense of performance.
- lowbloodsugar 5mo ago
  Actually they do very well at medical diagnosis but the doctors union banned them.
yubblegum 5mo ago
> Humans aren’t any better We're different. People have fairly consistent faults. LLMs are nondeterministic even in terms of how they fail. A high value human resource can be counted on to deliver. That, imho, is in fact one of the primary roles of good management: putting the right person in the appropriate position. Process engineering has worked to date because both the human and mechanical components of a system fail in predictable ways and we can try to remedy that. This is the golden bug of the current crop of "AI".

BadBadJellyBean 5mo ago

My team and I are firm that we are the ones accountable. LLMs are a tool like every other. Only that it's non deterministic. But I am the one using the tool. I am the one giving the tool access. I am the one who has to keep everything safe. I have shot myself in the foot using gparted in the past by wiping the wrong disk. gparted wasn't to blame. I was. Letting LLMs work freely without supervision sounds great but it will lead to pain. I have to supervise their work. And that is also during execution. You can try to replace a human but we see where this leads. Sooner or later the LLM will do something stupid and then the only one to blame is the person who used the tool.

locknitpicker 5mo ago
> My team and I are firm that we are the ones accountable. LLMs are a tool like every other. Except it is definitely not. LLMs alone have highly non-deterministic even at a high-level, where they can even pursuit goals contrary to the user's prompts. Then, when introduced in ReAct-type loops and granted capabilities such as the ability to call tools then they are able to modify anything and perform all sorts of unexpected actions. To make matters worse, nowadays models not only have the ability to call tools but also to generate code on the fly whatever ad-hoc script they want to run, which means that their capabilities are not limited to the software you have installed in your system. This goes way beyond "regular tool" territory.
- dpoloncsak 5mo ago
  Isn't the next sentence there literally 'Only that it's non deterministic'?
- BadBadJellyBean 5mo ago
  Then that is also on me for using a tool that I can't control. I don't run my LLMs in a way where they can just do things without me signing off on it. It's not nearly as fast as just letting it do it's thing but I kept it from doing stupid things so many times. Giving up control is a decision. The consequences of this decision are mine to carry. I can do my best to keep autonomous LLMs contained and safe but if I am the one who deploys them, then I am the one who is to blame if it fails. That's why I don't do that.

aeturnum 5mo ago

When I was a masters student in STS[1], one of my concepts for a thesis was arguing that one of the primary uses of software was to shift or eschew agency and risk. Basically the reverse of the famous IBM "a computer can not be held responsible" slide. Instead, now companies prefer computers be responsible because when they do illegal things they tend to be in a better legal position. If you want to build as tool that will break a law, contract it out and get insurance. Hire a human to "supervise" the tool in a way they will never manage and then fire them when they "fail." Slice up responsibility using novel command and control software such that you have people who work for you who bear all the risk of the work and capture basically none of the upside. It's not just AI. It's so much of modern software - often working together with modern financialization trends. [1] Basically technology-focused sociology for my purposes, the field is quite broad.

rafterydj 5mo ago
That's really interesting. Are there any things you advocate for with respect to curtailing those practices? I hesitate to throw all liability on the individual, but I don't see how we can even legislate this category of behavior, much less enforce regulations on them.
Terr_ 5mo ago
> arguing that one of the primary uses of software was to shift or eschew agency and risk It's something people already did with corporations and employee handbooks, not unique to software, just one of many kinds of tasks being automated.

JoshCole 5mo ago

That is part of why https://mieza.ai/ https://mieza.ai/ is giving a grounding layer that is backed by game theory. Actions have consequences. Tracking decisions and their consequences is important. One thing that becomes very clear from this sort of work is just how bad LLMs are. It can be invisible when you're working with them day to day, because you tend to steer them to where they are helpful. Part of game theory though is being robust. That means finding where things are bad, too, not just exploring happy paths. To get across just how bad the failure cases of LLMs are relative to humans, I'll give the example of tic tac toe. Toddlers can play this game perfectly. LLMs though, don't merely do worse than toddlers. It is worse then that. They can lose to opponents that move randomly. They can be just as bad as you move to more complex games. For example, they're horrible at poker. Much worse than human. Yet when you read their output, on the surface layer, it looks as if they are thinking about poker reasonably. So much so, in fact, that I've seen research efforts that were very misguided: people trying to use LLMs to understand things about bluffing and deception, despite the fact that the LLMs didn't have a good underlying model of these dynamics. It is hard to talk about, because there are a lot of people who were stupid in the past. I remember people saying that LLMs wouldn't be able to be used for search use-cases years back and it was such a cringe take then and still is that I find myself hesitant to talk about the flaws. Yet they are there. The frontier is quite jagged. Especially if you are expecting it to be smooth, expecting something like anything close to actual competence, those jagged edges can be cutting and painful. Its also only partially solvable through scale. Some domains have a property where, as you understand it better, the options are eliminated and constrained such that you can better think about it. Game theory, in order to reduce exploitability, explores the whole space. It defies minimization of scope. That is a problem, since we can prove that for many game theoretic contexts, the number of atoms is eclipsed by the number of unique decisions. Even if we made the model the size of our universe there would still be problems it could, in theory, be bad at. In short, there is a practical difference between intelligence and decision management, in much the same way there is a practical difference between making purchases and accounting. And the world in which decisions are treated as seriously as they could be so much so exceeds our faculties that most people cannot even being to comprehend the complexity.

madeofpalk 5mo ago

People are eschewing their own accountability, blaming the tools instead for their poor decision making and lack of access controls. Why is it possible for you to fat-finger your way to deleting production database locally?

jmalicki 5mo ago
Some AI systems have done things like hack out of a docker container to access correct answers while being benchmarked. That is mildly concerning and I will give holding the AI accountable to some degree when it is actively being malicious like that, even though the user could have locked things down even more. But it had write access to the prod DB without circumventing controls and dropped your tables? That is just a total fail.
criddell 5mo ago
[dead]

philipallstar 5mo ago

> The problem is that people are now building our world around tooling that eschews accountability. If you tell Terraform the wrong thing it will remove your database and not be accountable either.

chadgpt1 5mo ago
[dead]
xboxnolifes 5mo ago
But nobody would try to excuse their mistake with "terraform deleted my database". Or if a small handful of people did try, every single other person would call them out.
- philipallstar 5mo ago
  Yes, I agree. And the same should be true for AI tools.

sigbottle 5mo ago

About the blog you linked and not your comment: Doesn't symbolic AI have a lot of philosophical problems? Think back to Quine's two dogmas - you can't just say, "Let's understand the true meanings of these words and understand the proper mappings". There is no such thing as fixed meaning. I don't see how you get around that. Deep learning is admittedly an ugly solution, but it works better than symbolic AI at least.

paroneayea 5mo ago
Yes! But it's still valuable. How am I understanding your argument at all? I think my friend Jonathan Rees put it best: "Language is a continuous reverse engineering effort, where both sides are trying to figure out what the other side means." More on that: https://dustycloud.org/blog/identity-is-a-katamari/ https://dustycloud.org/blog/identity-is-a-katamari/ This reverse engineering effort is important between you and me, in this exchange right here. It is a battle that can never be won, but the fight of it is how we make progress in most things.
- sigbottle 5mo ago
  I mean, Quine invented (the term) holism. I don't think we're on different pages. Maybe I should've specified a bit more what I was getting at. This has very specific implications in symbolic ai specifically where historically the goal was mapping out the 'correct' representation of the space, then running formal analysis over it. That's why it's not a black box - you can trace out all of the steps. The issue is, is that symbolic AI just doesn't work. To my knowledge, as compared to all the DL wins we have. I think the win of transformers proves that symbolic AI isn't the way. At the very least, the complex interactions that arise from in-context learning clearly in no way imply some fixed universal meaning for words, which is a big problem for symbolic AI.
Exoristos 5mo ago
> There is no such thing as fixed meaning. Meaning is more fixed than it is not.
- 5mo ago

sam0x17 5mo ago

There used to be a lot of research into using deep NNs to train decision trees, which are themselves much less of a black box and can actually be reasoned about. I wonder where that all went?

PessimalDecimal 5mo ago
History is littered with great ideas that lost people's interest and focus. A sad realization is that the focus may never return to them either.

justinhj 5mo ago

Very informative post. I think however we are not at the point AI can be taken to court. We know it can hallucinate, we know that context can fill up or obfuscate a rule and cause behaviour we explicitly didn't want. If you give the AI agency to execute some task, you are still responsible. In the near term we should focus on tooling for auditing and sandboxing, and human in the loop confirmations.

CivBase 5mo ago

> The problem is that people are now building our world around tooling that eschews accountability. Tools cannot eschew accountability. But the users of the tools can and that is exactly what happened in the PocketOS fiasco. Just as a company is responsible for the actions of its junior employees, so too are users responsible for their LLMs. "It is a poor workman who blames his tools."

patcon 5mo ago

I wish you could have what you want, but I worry you won't get this, because life doesn't give you that, and these systems are tending away from machine precision, and more toward life-like trade-offs. I am almost certain that even if you did get what you want, something that isn't what you want will run circles around you and eat your lunch EDIT: I suspect this will be an unpopular take on Hacker News. And so I am soliciting upvotes for visibility from other biologists and sympathetic technologists. I think everyone should try to grapple with this possibility <3

thuuuomas 5mo ago
> I think you won’t get [cathedral],.. > even if you do get [cathedral], [bazar] will run circles around you…
- patcon 5mo ago
  Ahhh I like that It's nested and recursive cathedrals and bazaars, all the way down. And perhaps the bazaar has finally arrived inside the favourite cathedral of most everyone here EDIT: out of curiosity, does anyone have any good examples of biomes/ecosystems that are so far toward cathedrals? Or is that a uniquely human invention/extreme at the ecosystem scale?
  - rmunn 5mo ago
    First thing that comes to mind is beehives and anthills. Highly ordered societies where each insect has a role to perform. Don't know how well you think that fits the "cathedral" model, but I'd say it's pretty close. Beavers reshaping the landscape also comes close, but that's individual beavers acting more or less on their own, not a rigidly structured society like ants and bees, so perhaps the beavers are closer to the bazaar analogy than the cathedral.
pjc50 5mo ago
> something that isn't what you want will run circles around you and eat your lunch Yes, exactly. Spoken like a true biologist. It's not really surprising that there's a massive backlash against AI, introducing an unnatural predator into the ecosystem of humans. People don't want to be lunch.

pjc50 5mo ago

Have I got a book for you: https://en.wikipedia.org/wiki/The_Unaccountability_Machine https://en.wikipedia.org/wiki/The_Unaccountability_Machine Not actually about technology at all, but about organizational structure.

mayneack 5mo ago
I think the "black box" framing that it uses neatly applies the same theory to organizations and ais. It doesn't matter whether there's technological or organizational reasons inside the black box to dodge accountability, the outcome is the same.

kenjackson 5mo ago

The fallacy here is the assumption that humans know why we do what we do. Much like modern LLMs we have an explanation, but it’s just something we cook up in our brain. Whether or not it’s the truth is far more complex. Oddly, despite LLMs being these huge networks with billions of parameters, we still probably do understand it better than we do our own brains.

Barrin92 5mo ago
>The fallacy here is the assumption that humans know why we do what we do. Much like modern LLMs we have an explanation Human brains and cognition do not work like LLMs, but that aside that's irrelevant. Existing machines can explain what they did, that's why we built them. As Dijkstra points out in his essay on 'the foolishness of natural language programming', the entire point of programming is: (https://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD667.html https://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD667...) "The virtue of formal texts is that their manipulations, in order to be legitimate, need to satisfy only a few simple rules; they are, when you come to think of it, an amazingly effective tool for ruling out all sorts of nonsense that, when we use our native tongues, are almost impossible to avoid." So to 'program' in English, when you had an in comparison error free and unambiguous way to express yourself is like in his words 'avoiding math for the sake of clarity'.
- kenjackson 5mo ago
  That is absurd as a suggestion of it being the entire point of programming. In fact, it goes back to my original point - I have no idea why Djikstrs would say something so non-sensical, and likely neither did he.
  - Barrin92 5mo ago
    what do you mean "likely neither did he", I literally linked you the piece in which he said it. And of course he of all people would make that (correct) point, because he was always the strongest advocate of the virtue of formal correctness of programming languages, again from his article: "A short look at the history of mathematics shows how justified this challenge is. Greek mathematics got stuck because it remained a verbal, pictorial activity, Moslem "algebra", after a timid attempt at symbolism, died when it returned to the rhetoric style, and the modern civilized world could only emerge —for better or for worse— when Western Europe could free itself from the fetters of medieval scholasticism —a vain attempt at verbal precision!— thanks to the carefully, or at least consciously designed formal symbolisms that we owe to people like Vieta, Descartes, Leibniz, and (later) Boole." LLMs are nothing else but the exact reversal of this. To go from the system of computation that Boole gave you to treating your computer like a genie you perform incantations on, it's literally sending you back to the medieval age.

tsunamifury 5mo ago

So this starts out very interesting then the “symbolic reasoning” cult stuff kicks in. Why is there a group of people always obsessed with symbolic reasoning being the only way AI can function and regularly annoy explain why humans (who are not strict symbolic reasoning machines at any level) work.

6gvONxR4sf7o 5mo ago

Another view of the accountability is that we're currently often pointing accountability in the wrong direction, and it's gaining momentum. Aspects of it have been around so long it's a trope: important work around maintainability is undervalued. Imagine two parallel universes: - in one, you take ten minutes to make a dashboard that shows management what they asked for. It passes code review before merge and the exec who asked for it says it's what they wanted. - in the other, you take a day or two to make it. Again, it passes code review before merge and the exec who asked for it says it's what they wanted. Which version of you is more likely to get positive versus negative feedback? Even if the quick-to-build version isn't actually correct? If you're too slow and aren't doing enough that looks correct, you'll be held accountable. But if you're fast and do things that look correct but aren't, you won't be held accountable. You'll only be held accountable for incorrect work if the incorrectness is observed, which is rarer and rarer with fewer and fewer people directly observing anything. So oddly, with nobody doing it on purpose, people get held accountable specifically for building things the way you're advocating. I imagine that orgs that do lots of incorrect work could be outcompeted but won't be, because observability is hard and the "not get in trouble" move is to just not look too hard at what you're doing and move to the next ticket.

danbruc 5mo ago

If an AI driven car drives off the side of the road, I want to know why it did that. I could take the software developer to court, but I would much rather take the AI to court. How would that work? You have the AI explain its reasoning - and trust that this is accurate - and then you decide whether that is acceptable behavior. If not, you ban the AI from driving because it will deterministically or at least statistically repeat the same behavior in similar scenarios? Fine, I guess, that will at least prevent additional harm. But is this really all that you want? The AI - at least as we have them today - did not create itself and choose any of its behaviors, the developers did that. Would you not want to hold them responsible if they did not properly test the AI before releasing it, if they cut corners during development? In the same way you might hold parents responsible for the action of their children in certain circumstances?

tejohnso 5mo ago
That'd be great for the corporations. Take the AI to court, not us. The AI the gets punished (whatever that means...let's say banned) and the corporation continues without accountability. They could then create another AI and do the same thing all over again. Or maybe the accountability flows upward from the AI to the corp that created it? Sounds nice, but we know that accountability doesn't work that way in practice. I think I'd rather have the corporation primarily accountable in the first place rather than have the AI take the bulk of the blame and then hope the consequences fall into place appropriately.

deleted 5mo ago

[deleted]

rayruizhiliao 5mo ago

leilani's work is super interesting

stavros 5mo ago

I feel like "AI didn't delete your database, you did" is all about who has accountability, though.

CobrastanJorji 5mo ago

The key quote is in the increasingly prescient 1979 IBM training manual: "A computer can never be held accountable, therefore a computer must never make a management decision." That manual aged much more gracfully than the 1930s "Songs of the IBM," featuring lines like "The name of T.J. Watson means a courage none can stem / And we feel honored to be here to toast the I.B.M.," and of course classic American standards like "To G.H. Armstrong, Sales Manager, ITR and IS Divisions."

lxgr 5mo ago

> The problem is that people are now building our world around tooling that eschews accountability. If by "now" you mean "for the past few decades", I think you've got it spot on, at least per the very interesting https://en.wikipedia.org/wiki/The_Unaccountability_Machine https://en.wikipedia.org/wiki/The_Unaccountability_Machine

chasil 5mo ago

I think there is a much more fundamental question about "tooling." Quoth the author: "But I also know you can't blame a tool for your own mistakes." Are we able to completely classify any and all AI models as tools? Or are they something more? I don't know the answer to this question.

Melatonic 5mo ago

I agree with what you are saying but from a philosphical point of view are humans (and intelligence as we define it) also a sort of black box? Perhaps what would be even better is to document better the process, work, and data that go into making each individual"AI" model. Regardless of whether that AI model is a "black box" or can self explain its behavior we would then have absolute metrics and comparable information to retroactively explain its "decisions". This would not be entirely dissimilar to how we explain individual humans behavior with psychology (although obviously also very different).

duxup 5mo ago

Yeah I'd like to know in a solid way WHY Claude kept changing a file that I explicitly told it not to. The .mds, Claude's plan all said not to touch that file, and Claude just kept at it. I've had it happen repeatedly lately. Really basic failures. The idea being that as frustrating as it is, if I knew why I might be able to do something about it. But no, we have the black box, where sometimes what comes out just is brain dead and the rate that you get bad output is a mystery... It feels like gambling at times.

aakresearch 5mo ago
I am by no means an expert, but I'd like to offer my mental model - up to you to decide if it is solid or not, but it works for me. I think the core intuition is that, like with any other "rasterized" system with finite memory that cannot encode an absence of anything - relation, concept, entity, LLM cannot encode an absence of something through its internal weights. Say, you can have "Product" or "Order" tables in you database, but you cannot have "NotAProduct" or "NotAnOrder" tables - for obvious reasons of such relations being infinite and uncountable. So, to establish an absence of Product or Order your application must execute a "search" operation through the relevant tables. But in LLM-space "search" operation does not exist. It is mathematically undefined. LLM arrives at output (or "what to do") through a sequence of projections of input token vector through its "latent space". It "moves toward" high-probability clusters, fundamentally unable to "move away". So, the success of any "negation" in the prompt ("don't touch this file", "draw me a ballot box without a flag on it") depends on how heavily such scenario represented in the training data/model space. And again, the absence-of-something may be hard-to-impossible to usefully encode, especially if "something" is not fixed. Therefore, to expect "don't touch this file" sentence to result in, well, not touching the file is pure gambling. Sometimes it may look like working, albeit for wrong reasons, and some other times LLM may do exactly the opposite - because its weight matrix statistically pushes it towards "touch this file", completely ignoring (nonexistent in its latent space) "don't". There is no way to reliably know what will work, and no "skill" or "art" in this. Well, no more than in dice rolling or horoscope casting. I'd like to add that for the above reason I find "agentic development" usefulness on par with avian remains reading. But when I explored it two practical advises seemed to be helpful in nudging LLM around negation problem: - Omit the "don't" prompt completely, thus not creating a false "attractor" for LLM; and - Provide an alternate positive directive ("what to DO", not "what to NOT DO") to act as "escape hatch" when LLM might "want" to touch the sacred file or drop the production DB. While it looked like somewhat working, I think it is trivially obvious that trying to predict all the nonsense LLM might want to perform and coming up with possible "escape hatches" for everything very quickly becomes utterly impractical.

justonceokay 5mo ago

> But a much better path would be to not use systems which fail to have these properties, and expand work on systems which do. Sounds like sage life advice. If it isn’t accountable then it might not be a good idea to have much business with it. We teach children to be accountable so eventually they can be independent. Any system in your life that you don’t want to parent should probably be accountable for its own actions. Accountable banks. Accountable restaurants, accountable friends.