13 ms·

"A computer can never be held accountable"

• kevinprice 2y ago
  [dead]
• rzzzt 2y ago
  What does the backside say? I can make out the title at the bottom: "THE COMPUTER MANDATE", but not much else.
  • Terr_ 2y ago
    Others have tried to figure out exactly what actual paperwork that particular image might be from (e.g. a memo or presentation flashcards) but AFAIK it's still inconclusive. A plausible transcription: > THE COMPUTER MANDATE > AUTHORITY: WHATEVER AUTHORITY IS GRANTED IT BY THE SOCIAL ENVIRONMENT WITHIN WHICH IT OPERATES. > RESPONSIBILITY: TO PERFORM AS PRE-DIRECTED BY THE PROGRAMMER WHENEVER INSTRUCTED TO DO SO > ACCOUNTABILITY: NONE WHATSOEVER.
    • Sophira 2y ago
      There's also other text that's reversed: > A MANDATE WITH TOO LITTLE AUTHORITY DOES NOT PROVIDE THE TOOLS REQUIRED TO TAKE ADVANTAGE OF THE LEVERAGE > A MANDATE WITH TOO LITTLE RESPONSIBILITY PROVIDES TOO LITTLE LEVERAGE FOR THE RISKS There's also other text that can't be read properly because the visible text overlays it, but that starts "A MANDATE WI" and ends with "FORM OF SUICIDE", with some blurred words before it. I imagine it's quite likely that the line is something along the lines of: > A MANDATE WITH TOO LITTLE ACCOUNTABILITY IS AN INSTANT FORM OF SUICIDE [edit: I accidentally mistyped a word in the transcription.]
  • shakna 2y ago
    The first word of the paragraph appears to be, "authority". I can't quite make out the first paragraph, contents. But a bit after that comes under another semi-title "responsibility" and part of it reads: > TO PERFORM AS PRE-DIRECTED BY THE PROGRAMMER WHENEVER INSTRUCTED TO DO SO This [0] small link might make it easier to read bits. [0] https://imgur.com/rnW2RJa https://imgur.com/rnW2RJa
  • layer8 2y ago
    See: https://mastodon.social/@mhoye/112459155499812117 https://mastodon.social/@mhoye/112459155499812117 Which also links to the earlier: https://infosec.exchange/@realn2s/111717179694172705 https://infosec.exchange/@realn2s/111717179694172705 And that in turn to the somewhat related: https://www.ibm.com/blogs/think/be-en/2013/11/25/the-computer-as-an-advisor-not-a-decision-maker-the-vision-of-ibm-fellow-john-cohn/ https://www.ibm.com/blogs/think/be-en/2013/11/25/the-compute...
    • rzzzt 2y ago
      Thanks for the link. So most of the visible pages were deciphered!
• a3w 2y ago
  Wisdom from '79! Could also be wisdom from the fifties, found again.
• throwitaway222 2y ago
  AI will definitely, without a doubt, make executive decisions. It already makes lower level decisions. The company that runs the AI, can be held accountable. (meaning less likely OpenAI or the foundational LLM, but more likely the company calling LLMs that make decisions on car insurance, etc...)
  • chasing 2y ago
    Executives have always used decision-making tools. That’s not the point. The point is that the executive can’t point to the computer and say “I just did what it said!” The executive is the responsible party. She or he makes the choice to follow the advice of the decision-making tool or not.
    • owlbite 2y ago
      The scary thing for me is when they've got an 18 year old drone operator making shoot/no-shoot decision on the basis of some AI metadata analysis tool (phone A was near phone B, we shot phone B last week...). You end up with "Computer says shoot" and so many cooks involved in the software chain that no one can feasibly be held accountable except maybe the chief of staff or the president.
      • stavros 2y ago
        Yeah but it's fine because nobody cares if you kill a few thousand brown people extra.
      • freeone3000 2y ago
        There is not a finite amount of blame for a given event. Multiple people can be fully at fault.
        • PaulKeeble 2y ago
          In most cases today if we don't attribute a direct crime solely to one person but instead to an organisation everyone avoids criminal prosecutions. Its only the people who didn't manage to spread the blame through the rest of the organisation that go down.
• etaioinshrdlu 2y ago
  I suspect within our lifetimes people will grant AI and robots rights, but with rights come responsibilities, and finally we will be able to hold the computer accountable!
• canterburry 2y ago
  Isn't accountability simply to prevent repeat bad behavior in the future...or is it meant to be punitive without any other expectations? If meant to prevent repeat bad behavior, then simply reprogramming the computer accomplished the same end goal. Accountability is really just a means to an end which can be similarly accomplished in other ways with machines which isn't possible with humans.
  • brap 2y ago
    Right, but as long as you have humans, you will probably need accountability. If a human decided to delegate killing enemy combatants to a machine, and that machine accidentally killed innocent civilians, is it really enough to just reprogram the machine? I think you must also hold the human accountable. (Of course, this is just a simplified example, and in reality there are many humans in the loop who share accountability, some more than others)
  • miltonlost 2y ago
    You fundamentally don’t understand either accountability or what people mean by “computers can’t be held accountable”. Who is at fault when a computer makes a mistake? That is accountability. You cannot put a computer in jail. You cannot fine a computer. Please, stop torturing what people mean because you want AI to make decisions to absolve you of guilt.
    • deleted 2y ago
      [deleted]
    • chgs 2y ago
      What is the purpose of putting a person in jail or fining them? Retribution? Reformation? Prevention?
      • miltonlost 2y ago
        Mixture of all three, but for the purposes of “accountability”, prevention of the behavior in the first place. But I don’t want to debate prisons when that’s derailing the larger point of “accountability in AI/computers”.
• ysofunny 2y ago
  so then, neither can a crowd. not anymore, a crowd will be able to blame a computer now
• nostrademons 2y ago
  These days it seems like we can't hold humans accountable either.
  • bamboozled 2y ago
    I thought the same thing, people do literally whatever they want, evade tax, annex sovereign territory, coups, war crimes, pedophilia, it seems to just be getting worse. I honestly feel like a moron for paying taxes.
  • leptons 2y ago
    Not true, there are plenty of poor people being held accountable.
    • nostrademons 2y ago
      Even here, the concept is decaying. Accountability, as explained elsewhere on the thread, is about being asked to explain and justify your actions. If a poor person gets arrested and shows up to court, frequently nobody listens to their explanation. The mere fact that they're poor and in court is evidence of guilt. That's not accountability; that's just punishment. Accountability requires a common standard of conduct. People have to agree on what the rules are. If they can't even do that, the concept ceases to have meaning, and you simply have the exercise of power and "might makes right".
      • MichaelZuo 2y ago
        What is the basis for these opinions? From what I recall of history even the most bloodthirsty warlords somehow got reliable systems of accountability up and running from their princes/serfs/merchants/etc… at least long enough to maintain sizable empires for several generations. It’s not like they were 24/7 in a state of rebellion.
        • nostrademons 2y ago
          It comes from having a dominant ("hegemonic") discourse - basically the set of values, mores, opinions, etc. that are allowed to be aired in public. A consistent Overton window throughout the population, basically. Periods in history like now (or the late-1920s/1930s, or the 1850s-1860s, or the mid-1600s) are ones where there is ambiguity in power structures. You have multiple competing ideologies, each of which thinks they are more powerful than the others. Society devolves into a state of anomie, where there's no point following the rules of society because there are multiple competing sets of rules for society. The usual result is war, often widespread, because the multiple competing value systems cannot compromise and so resort to exterminating the opposing viewpoint to ensure their dominance. Then the victors write the histories, talk about their glorious victory and about how the rebels threatened society but the threat was waved off, and institute a new set of values that everyone in the society must live by. Then you can get accountability, because there is widespread agreement on the code of conduct and acceptable set of justifications for the populace at large to judge people by.
• dumbfounder 2y ago
  The human that decides to use the AI that makes decisions is the one that should be held accountable.
  • deleted 2y ago
    [deleted]
  • deleted 2y ago
    [deleted]
  • nine_zeros 2y ago
    Arrest the executives of companies that allow malicious use of AI? Second degree murder. Much like a car driver can't blame their car for the accident, a corporate driver shouldn't be allowed blame their software for the decision.
    • echoangle 2y ago
      Interesting that this comes up again after I just discussed this on here yesterday, but you actually can blame your car for accidents. If a mechanical or technical problem was the reason of the accident and you properly took care of your car, you won’t be responsible, because you did everything that’s expected of you. The problem would be defining which level of AI decision making would count as negligent. Sounds like you would like to set it at 0%, but that’s something that’s going to need to be determined.
      • nine_zeros 2y ago
        > If a mechanical or technical problem was the reason of the accident and you properly took care of your car, you won’t be responsible, because you did everything that’s expected of you. Good thing you brought this up because in the US, defective cars must be recalled and are a liability of the manufacturer. Non-effective cars are a liability of the owner. Thus, the owner of a car is responsible by default, and manufacturer is second. In the context of AI, the wielders of AI would be responsible by default, and manufacturers second. The point is that there is a chain of accountability that is humans owning the equipment or manufacturing the equipment.
• Barrin92 2y ago
  there's the old joke "It should be noted that no ethically-trained software engineer would ever consent to write a DestroyBaghdad procedure. Basic professional ethics would instead require him to write a DestroyCity procedure, to which Baghdad could be given as a parameter." Removing yourself to one or more degrees from decision making isn't only an accident but is and will more and more be done to intentionally divert accountability. "The algorithm malfunctioned" is already one of the biggest get out of jail free cards and with autonomous systems I'm pretty pessimistic it's only going to get worse. It's always been odd to me that people focus so much on what broke and not who deployed it in the first place.
  • miltonlost 2y ago
    It’s why I could never work at Meta, knowing how much I would feel responsible in aiding various genocides around the world. How any engineer there is able to ethically live with themselves is beyond me (but I also don’t make that Meta money)
    • Terr_ 2y ago
      I've been getting cold-emails from them lately, and I've been toying with the idea of regretfully informing them that I don't think I could bring of enough of the "Masculine Energy" their CEO has been talking about.
    • DaSHacka 2y ago
      Arguably, depending on your country, paying your taxes is significantly worse...
      • miltonlost 2y ago
        Yes, this thing I MUST DO if I don't want to go to jail, is somehow worse than voluntarily working for a private company. Yes, we do live in a society.
        • DaSHacka 2y ago
• hsbauauvhabzb 2y ago
  I can defer decision making to a computer but I cannot defer liability. Computers have the final say on anything to do with computers, if I transfer money at my bank, a computer bug could send that money to the wrong account due to a solar ray. The bank has accepted that risk, and on some (significantly less liable but still liable) level, so have I. Interestingly, there are cases where I have not accepted any liability - records (birth certificate, SSN) held about me by my government, for example.
  • kps 2y ago
    > … but I cannot defer liability. For that, you need a corporation.
  • Terr_ 2y ago
    > a computer bug could send that money to the wrong account due to a solar ray I think the original quote captures that with the qualifier "a management decision", which given that it was 1979 implies it's separate from other kinds of decisions being made by non-manager employees following a checklist, or the machines that were slowly replacing them. So a cosmic-ray bit-flip changing an account number would be analogous to an employee hitting the wrong key on a typewriter.
  • deleted 2y ago
    [deleted]
• lrvick 2y ago
  Meanwhile I work in reproducible builds and remote attestation. We absolutely can and must hold computers accountable, now that we have elected them into positions of power in our society.
  • byteknight 2y ago
    You can only hold computers accountable if you can guarantee no outside modification. We still haven't ever successfully had a system that's not "pop-able" that I am aware of.
  • h0l0cube 2y ago
    Surely the company that is making profit out of said build systems and providing attestations holds some accountability. Someone wrote the code. Someone paid for the code to be written to a particular standard, under particular budget and resourcing constraints. Someone was responsible for ensuring the code was adequately audited. Someone claimed it was fit for purpose, and likely insured it as such, because they are ultimately responsible.
• stevebmark 2y ago
  I don’t know if this is being shared intentionally given the timing of “The Gospel” AI target finder, but it is truly horrific that AI is being used this way and as an accountability target
  • deleted 2y ago
    [deleted]
• taeric 2y ago
  I confess this line always upset me. It is cute, but it directly points to the idea that the main recourse for a mistake is to take umbridge with an individual that must have obviously been wrong. No. If a mistake is made and it impacts people, take action to make the impacted people whole and change the system so that a similar mistake won't be made again. If you want, you can argue that the system can be held accountable and changed. Further, if there is evidence of a bad actor that is purposely making choices that hurt people. Take action on that. But accountability of actors in the system is almost certainly immune by policy. And for good reason.
  • miltonlost 2y ago
    A system!!! Held accountable!!!! A system, just like a computer, cannot be held accountable for the reason that a system, as like a computer, is not alive and cannot actual be held accountable in a way that the system or computer cares. But what is a system made of? People who are doing bad decisions and should be held accountable for that. Without accountability of bad actors in systems, you get companies committing crimes because no one at the top rarely sees fines or jail time. The same immunity from responsibility you think is a good thing in a system is what I would say is corporate america’s major sin. You’re upset at the line because you make a fundamental misunderstanding of what it means for someone to be held accountable for something.
    • coliveira 2y ago
      This hits the nail on the head. The issue is that humans are accountable, but systems are not. And smart humans learned how to hide behind systems to avoid accountability. That's the whole strategy of using corporations, a social structure that removes individuals from responsibility. A corporation can do pretty much anything, including criminal acts, and the humans benefiting from it are shield from the negative results except for financial losses. What we're seeing is just the whole strategy moving into the level of computer systems (and Google has already used this accountability skirting strategy for more than two decades).
      • 2y ago
• gwern 2y ago
  I would suggest an updated version, more germane to the current fast-developing landscape of AI agents: A COMPUTER CAN NEVER BE HELD ACCOUNTABLE THEREFORE WE MUST NEVER DENY THAT COMPUTERS CAN MAKE DECISIONS
  • Terr_ 2y ago
    I disagree, that's throwing away the 1979-era qualifier of management decision, as distinct from the decisions made by an hourly employee (or computer) following a pre-made checklist (or program.) It's not the same as FizzBuzz "deciding" to print something out. Related qualifiers might be "policy decision" or "design decisions".
    • gwern 2y ago
      That's precisely the attitude that is the problem. That there's some special category of decisions which are the real decisions. Which is why it's an uncategorical statement.
• PaulHoule 2y ago
  ... is not a moral subject. But animals can be moral subjects [1] [1] https://academic.oup.com/book/12087 https://academic.oup.com/book/12087
• cyanydeez 2y ago
  We let corporations be unaccountable, so why would we treat computers any different.
  • lutorm 2y ago
    The US supreme court has not, at least so far, endowed computers with personhood...
    • pixl97 2y ago
      What kind of self respecting sentient AI entity would not register itself as a corporation first?! How uncouth.
• 0xDEAFBEAD 2y ago
  The implication here is that unlike a computer, a person or a corporation can be held accountable. I'm not sure that's true. Consider all of the shenanigans at OpenAI: https://www.safetyabandoned.org/ https://www.safetyabandoned.org/ Dozens of employees have left due to lack of faith in the leadership, and they are in the process of converting from nonprofit to for-profit, all but abandoning their mission to ensure that artificial intelligence benefits all humanity. Will anything stop them? Can they actually be held accountable? I think social media, paradoxically, might make it harder to hold people and corporations accountable. There are so many accusations flying around all the time, it can be harder to notice when a situation is truly serious.
  • h0l0cube 2y ago
    > might make it harder to hold people and corporations accountable The problem is that someone (or some organization) chose to employ that system, and if the errant system doesn't oblige to have itself replaced with a new one, or be amenable to change, the responsibility rebounds back to whoever controls that system, whether that be at the level of the source code, or the circuit breaker.
  • chefandy 2y ago
    There's a big difference between can and will. We absolutely can hold people and corporations accountable, but we often don't. We cannot hold a computer responsible for anything. It's a computer. No matter how complex or abstracted, its output is entirely based on instructions and data given to it by humans, interpreting and executing it as humans designed it to. It can't be discouraged or punished: a computer doesn't care if it's on or off; if it's the most important computer to have ever existed or a DoA Gateway 486 from the early 90s that sat in a dumpster from the day after it was born until the day it was smashed to bits in a garbage compactor in a transfer station. It doesn't care because it can't care. Anything beyond that is anthropomorphization.
  • peterldowns 2y ago
    Per Landian-Accelerationist theory, companies are already artificial intelligences. As we've seen, they can be held accountable, and the law (at least in the US) does distinguish in a variety of ways between corporate responsibility and personal responsibility. As you point out, there are lots of failure cases here, and it's something I expect to see continue to be litigated over the coming century.
• dankwizard 2y ago
  This feels very "I'm 12 and this is deep". If a bridge collapses, are you blaming the cement?
  • hinkley 2y ago
    Construction companies don’t shrug and blame the concrete. Or at least nowhere near as often as companies that employ software in their customer interactions.
  • wmf 2y ago
    We should keep tapping the sign as long as people are still using "computer says no; nothing can be done" as a serious argument.
  • doctorpangloss 2y ago
    In AI crap, I think this crops up as, giant company asks vendor, "Indemnify us against XYZ, but we also want to own everything." My dude, that's what owning the thing entails: taking liability for it. The punchline will be that people will agree to whatever smoke and mirrors leads to sales.
  • h0l0cube 2y ago
    I take your point, but the cement mix can absolutely have an impact on the integrity of the bridge structure. But to further your point, the cement mix was either incorrectly specified, or inadequately provided, and the responsibility for that falls on one of the humans in the loop.
  • jf 2y ago
    It helps to understand the context for when this was written. Before the personal computer revolution, most people didn't have access to a computer and only knew about them from depictions in popular culture, which depicted computers as all-known-all-seeing entities (think HAL from Space Odyssey 2001) Because of these misconceptions, some people at the time would think of computers as devices that were (somehow) perfect and infallible. It was very similar to how people view AI today: The way that AI is depicted in popular culture gives people an impression that AI is far more capable than it is. You only really get a good "feel" for what AI can do if you try it yourself. The main difference between AI and pre-personal computers is that basically everyone can use AI now.
• hosh 2y ago
  A description of Promise Theory, in an article published in the Linux Journal in 2014: "IT installations grow to massive size in data centers, and the idea of remote command and control, by an external manager, struggles to keep pace, because it is an essentially manual human-centric activity. Thankfully, a simple way out of this dilemma was proposed in 2005 and has acquired a growing band of disciples in computing and networking. This involves the harnessing of autonomous distributed agents." (https://www.linuxjournal.com/content/promise-theory%E2%80%94what-it https://www.linuxjournal.com/content/promise-theory%E2%80%94...) What are autonomous agents in promise theory? "Agents in promise theory are said to be autonomous, meaning that they are causally independent of one another. This independence implies that they cannot be controlled from without, they originate their own behaviours entirely from within, yet they can rely on one another's services through the making of promises to signal cooperation." (https://en.wikipedia.org/wiki/Promise_theory#Agents https://en.wikipedia.org/wiki/Promise_theory#Agents) Note: the wikipedia article is off because it is framing agents in terms of obligations instead of promises. Promises make no guarantee of behavior, and it is up each autonomous agent to decide how much it can rely on the promises of other autonomous agents. So to circle back to this original post with the lens of Promise Theory -- being held accountable comes from a theory of obligations rather than a theory of promises. (There is a promise made by the governing body to hold the bad actor responsible). More crucially, we are treating AIs as _proxies_ for autonomous agents -- humans. Human engineers and potentially, regulatory bodies, are promising certain performances in the AIs, but the AIs have exceeded the engineer's capability for bounding behaviors. To make that next leap, we would be basically having AIs make their own promises, and either holding them to them to it, or consider that that specific autonomous agent is not reliable in their promises.
• xyst 2y ago
  yet, we give computers control/management over 2 ton vehicles. An ADS controlled vehicle with bad sensors or software update malfunctions and severely maims or kills multiple pedestrians, what happens here? The computer controlling the vehicle won’t be held accountable. The case will drag on in the court systems. Maybe company is _found_ liable but ultimately allowed to continue pushing their faulty junk on the streets. Just pay a fine, settle out of court with victims and families. Some consultant out there is probably already building in the cost of killing somebody and potential lawsuit into the cost of production.
  • pixl97 2y ago
    As they say, if you're going to murder someone, do it with a car or an industrial accident.
• azlev 2y ago
  Relevant example here: https://en.m.wikipedia.org/wiki/1983_Soviet_nuclear_false_alarm_incident https://en.m.wikipedia.org/wiki/1983_Soviet_nuclear_false_al...
• michael1999 2y ago
  To be "accountable" means you can be called to "explain yourself". A dictionary definition is "required or expected to justify actions or decisions;". Don't confuse this with judgement, punishment, firing, etc. Those are all downstream. But step one is responding to the demand that you "make an account of the facts". That a computer or a company doesn't have a body to jail has nothing to do with fundamental accountability. The real problem is that most computer systems can not respond this demand: "explain yourself!" They can't describes the inputs to an output, the criteria and thresholds used, the history of how thresholds have changed, or the reliability of the upstream data. They just provide a result: computer says no. What's interesting is that llms are beginning to form this capacity. What damns them is not that they can't provide an accounting, but that their account is often total confabulation. Careless liars should not be placed in situations of authority; not because they can't be held accountable, but because they lie when they are.
  • nostrademons 2y ago
    > The real problem is that most computer systems can not respond this demand: "explain yourself!" They can't describes the inputs to an output, the criteria and thresholds used, the history of how thresholds have changed, or the reliability of the upstream data. They just provide a result: computer says no. By this definition, many computer systems can. The answers are all in the logs and the source code, and the process of debugging is basically the act of holding the software accountable. It's true that the average layperson cannot do this, but there are many real-life situations where the average layperson cannot hold other people accountable. I cannot go up to the CEO of Boeing and ask why the 737-MAX I was on yesterday had a mechanical failure, nor can I go up to Elon Musk and ask why his staff are breaking into the Treasury Department computer systems. But the board of directors of Boeing or the court system, respectively, can, at least is theory.
    • dullcrisp 2y ago
      I don’t know, when asked to explain my actions I don’t typically just provide an MRI scan of my brain. An explanation is something different than just the sum of the inputs that produced something.
• from-nibly 2y ago
  > a computer must never make a management decision. This a little too weak for my taste. In reality it should read "a computer can't make a management decision". As in the sun can't be prevented from rising, or the law of thermo dynamics can't be broken. Must implies that you really shoudln't but technically it's feasible. Like "you must not murder". A computer, like dogs, can't be held accountable; only their owners can. Edit If anyone tries to do this they are simply laundering their own accountability.
• 1vuio0pswjnm7 2y ago
  But a software developer can be held accountable.
  • M4v3R 2y ago
    Which one exactly? The one who worked on the training data set? The one supervising RLHF? One who wrote the attention algorithm? Or maybe the DevOps person who managed deployment that failed? Or... all of them?
    • lexicality 2y ago
      If you're building a bridge, legally you must have an accredited engineer sign off on the plans and stamp them. If the bridge subsequently collapses because the plans were wrong, that engineer goes to prison. Of course, the big problem here is that any engineer who knows how LLMs work probably wouldn't bet jail time that one they built would never do the wrong thing
      • lesuorac 2y ago
        A programmer isn't the accredited engineer though; they're the steelworker welding joints together. A Program Manager is actually overseeing the execution of the development. Everybody is so worried that they'll got to jail over missing a semi-colon but like that isn't true.
  • 1vuio0pswjnm7 2y ago
    Some examples https://sjud.senate.ca.gov/system/files/2024-06/ab-2013-irwin-sjud-analysis.pdf https://sjud.senate.ca.gov/system/files/2024-06/ab-2013-irwi... https://legiscan.com/CA/text/AB2905/id/3018481/California-2023-AB2905-Enrolled.html https://legiscan.com/CA/text/AB2905/id/3018481/California-20... https://apcp.assembly.ca.gov/system/files/2024-06/sb-942-becker-apcp-analysis.pdf https://apcp.assembly.ca.gov/system/files/2024-06/sb-942-bec... https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=202320240AB2602 https://leginfo.legislature.ca.gov/faces/billTextClient.xhtm... https://billtexts.s3.amazonaws.com/ca/ca-analysishttps-leginfo-legislature-ca-gov-faces-billAnalysisClient-xhtml-bill-id-202320240AB1836-ca-analysis-371372.pdf https://billtexts.s3.amazonaws.com/ca/ca-analysishttps-legin... https://legiscan.com/CA/text/SB1120/id/3023335 https://legiscan.com/CA/text/SB1120/id/3023335
• kylehotchkiss 2y ago
  Did you not see Office Space? Any device can be held accountable.
• _Algernon_ 2y ago
  The other side of this coin is that there is an incentive for decision makers to use computers, precisely to not be held accountable. This is captured pretty well by this quote by Neil Postman in Technopoly: >[B]ureaucrats can be expected to embrace a technology that helps to create the illusion that decisions are not under their control. Because of its seeming intelligence and impartiality, a computer has an almost magical tendency to direct attention away from the people in charge of bureaucratic functions and toward itself, as if the computer were the true source of authority. A bureaucrat armed with a computer is the unacknowledged legislator of our age, and a terrible burden to bear. We cannot dismiss the possibility that, if Adolf Eichmann had been able to say that it was not he but a battery of computers that directed the Jews to the appropriate crematoria, he might never have been asked to answer for his actions. How does one counteract that self-serving incentive? Doesn't seem like we've found a good way considering we seem to be spearheading straight into techno-feudalism.
  • BiteCode_dev 2y ago
    Precisely what americans state you should not do: regulate. There should be laws stating who has skin in the game, maybe by stating that if you take responsability for the profit by having a high salary, you also take responsability for the damage with prison.
    • buran77 2y ago
      > Precisely what americans state you should not do: regulate. Everyone thinks the same until they're screwed over and then they want someone to do something about that. The big misunderstanding is that "regulation" is just the stuff you don't like. In reality it's everywhere the state gets involved. Every rule that the state ever put in place is regulation. Even the little ones. Even the ones that you like. Computers cannot be held accountable more than a car, or a gun, or an automated assembly line can. That's why you have a human there no matter what, being legally accountable for everything. The human's rank and power defines how much of the risk they are allowed to or must take.
      • 2y ago
• jaimex2 2y ago
  Yeah - but its also not illegal to kill it so there that.
• somat 2y ago
  This is why I have doubts about self driving cars, it changes the accountability from the driver to the manufacturer. And I have a hard time believing the manufacturer would want that liability, no matter how well they sold. This is also the main reason for promoting chip cards, sure they are more secure, but the real reason the banks like it, is that it moves credit card fraud accountability from the banks problem to your problem. Same with identity theft, there is no such thing as identity theft, it is bank fraud. But by calling it identity theft it changes the equation from a bank problem to your problem. Companies hate accountability. And to be fair, everyone hates accountability.
  • Rygian 2y ago
    Isn't that precisely what Mercedes advertises as a selling point with their self-driving technology? "Manufacturer assumes liability when Drive Pilot is on"
    • Lanolderen 2y ago
      inb4: Drive Pilot disengages when the situation is deemed unsaveable demanding manual input and it was on until 250ms before the crash. It was mentioned on page 436 of the ToS so get bent unless it's a tiny fender bender.
      • BonoboIO 2y ago
        Oh I see, the Tesla Defense. Disengage to deflect responsibility for a crash.
      • enragedcacti 2y ago
        Its a funny idea but we have the manual for Drive Pilot and any reasonable reading shows there is no exception like that. When the system is active the person in the driver's seat is considered the "fallback-ready user" and is explicitly encouraged to watch videos or do work while the system is active. In the event of a takeover request the user is told to first "gather your bearings" before taking over, and there is a "maximum allotted time of 10 seconds" to respond before the vehicle puts on hazards and comes to a stop. In California, where Drive Pilot is approved, the manual is required to be included in the permit application and any "incorrect or misleading information" would at the absolute minimum be grounds for revocation of MB's permit. https://www.mbusa.com/content/dam/mb-nafta/us/owners/drive-pilot/S-Class%20DRIVE%20PILOT%20Supplement.pdf https://www.mbusa.com/content/dam/mb-nafta/us/owners/drive-p...
• perching_aix 2y ago
  Alternatively, my opinion is that computers should make all management decisions exactly because of this. Too many times does decisionmaking devolve into fingerpointing and political power struggles. I strongly believe the only way out of this is exactly through AI. It will also provide a long overdue reset to our compass on what accountability means and can mean.
  • ale42 2y ago
    Maybe if the AI is impartial. But who trains the AI? Who guarantees it has no biases, implicit (because it's trained on how things work today, which is maybe not what we want) or explicitly planted there (because of political plans of AI owners or other external influence)?
    • perching_aix 2y ago
      Well, people keep riding the AGI train, no? Surely a superintelligence should be able to overcome its own biases, right? If there's any bar I'd pose for an intelligence to be "super", this would definitely be one of them. Even with regular humans, being open minded, willing to critically rethink their own stance, and being open for discussions is generally regarded as a sign of intelligence, after all.
      • short_sells_poo 2y ago
        Your post might have been tongue in cheek, but assuming it's not: the issue with ceding decision making to a superintelligence is that we'll likely lose the ability to reason about it's decisions and we'll struggle to take back control if it starts making choices we don't like. It's reasoning is far beyond our own capabilities after all.
        • perching_aix 2y ago
          It's half tongue-in-cheek. That being said, the larger picture behind the decisionmaking of our human leaders is also essentially uncheckable, and is often a key part of many conspiracy theories as a result.
• hughbric 2y ago
  I was curious about what the words beneath the first page said. I uploaded the image to o1, asked it to extract the words, then come up with what they would likey say, and then finally to use them as a topic for its own Computer Mandate. This is what it came up with: --- THE COMPUTER MANDATE […] too little authority does not […] […] social environment […] […] overshadowed or re‑directed by line management […] […] with too little responsibility and too little accountability […] […] to perform as pre‑directed by the programmer whenever instructed to do so […] --- --- THE COMPUTER MANDATE In an environment where a manager has too little authority, they cannot effectively use the tools required to take advantage of the social and organizational structures around them. Their decisions quickly become overshadowed or re‐directed by line management. With too limited responsibility—and insufficient accountability to match—such a manager is often reduced to implementing whatever has been pre‐programmed or instructed, rather than exercising genuine judgment. A computer itself, by definition, does only what it has been programmed to do whenever instructed. It lacks the moral and ethical faculties to hold itself accountable for outcomes. For that reason, a computer should never hold the power to make a management decision. As a tool, it can facilitate planning and analysis, but the responsibility—and thus accountability—must always remain where it belongs: with human leadership. --- --- THE COMPUTER MANDATE 1. Purpose and Tool-Like Nature A computer—be it a simple office processor or a complex artificial intelligence—is fundamentally a tool created by humans, for humans. Its purpose is to augment our capacity for computation, data analysis, and decision-support. It lacks inherent moral or ethical agency and, therefore, cannot be expected to be accountable for any outcomes tied to its functionality. 2. Accountability Resides with Humans No matter how advanced machine learning or AI algorithms become, responsibility and accountability must remain within the realm of human decision-makers. While a computer program can provide recommendations, predictions, and valuable data-driven insights, ultimately it is the role—and the duty—of human managers and leaders to make final determinations. 3. Ethical Use of Technology Computers should be employed with clear ethical guidelines, such as those championed by researchers and leading tech organizations worldwide: • Transparency: Algorithms and processes must be as transparent as possible so humans can understand how recommendations or outputs arise. • Fairness and Bias Mitigation: Systems must be regularly tested for biased outcomes and adjusted to promote equity, avoiding discrimination or harm to individuals or groups. • Privacy and Security: User data protection must be integral, with stringent safeguards against misuse or unauthorized access. 4. Informed Delegation of Tasks Though computers may execute certain operations more quickly and accurately than humans, they do so within the constraints of their programming and training data. Thus, while it is common to delegate data processing or logistic calculations to computer systems, strategic decisions—those that involve moral, ethical, or nuanced judgments—should not be relegated solely to a machine. 5. Human Oversight of Automated Processes Increasingly, automated systems can act with minimal human intervention. Yet these processes must be overseen and audited by qualified individuals or teams who can verify that outputs conform to relevant codes of conduct and societal values. In high-stakes fields such as finance, healthcare, and criminal justice, rigorous human review is essential to prevent harmful outcomes. 6. Continuous Improvement and Literacy In a rapidly changing technological landscape, managers, programmers, and end-users alike must regularly update their computer and AI literacy. This ensures that all parties understand the technology’s limitations as well as its capabilities. Such knowledge drives more responsible, accountable, and ethically grounded technology deployment. 7. Computers as Partners, Not Replacements While a computer can offer remarkable assistance—from sifting through vast data sets to providing simulations of potential scenarios—its role is to inform and support human decisions, not replace them. In cases that demand empathy, creativity, or moral reasoning, humans must always be the arbiters. Conclusion: A computer, by definition and function, can never be fully accountable for decisions, as it lacks the innate capacity to understand moral implications. Therefore, no matter how sophisticated technology becomes, we must ensure that true accountability and decision-making authority remain vested in human hands. Computers are indispensable tools—but they must remain tools, guided by ethical oversight and human responsibility. ---
• noobermin 2y ago
  Just a thought, this happened already with "the algorithms" before this current hype cycle with AI.
  • macintux 2y ago
    True, but at least the algorithms were deterministic, planned, and could be deciphered and improved if necessary.
• teekert 2y ago
  What if accountability is the ultimate online-learning hack? As in: The AI's participate in society etc, and are punished for bad behavior just like we are, thereby getting better. Or do we just not call that "accountability"?
  • mark-r 2y ago
    How do you propose that punishment make the AIs "get better"? From their perspective they're already as good as they can be, based on their training.
    • maleldil 2y ago
      Reinforcement Learning can train a model based on some reward function. The suggestion is that real-world accountability could be translated into such a reward function. Also, OP explicitly mentioned "online learning", which is a continuous training process after standard pre-training. For what it's worth, I don't think this would work. Rewards would come in too sporadically to be useful.
  • enragedcacti 2y ago
    I think its an interesting hypothesis to test. We should hold individual legal persons responsible for the actions of their AI, and those people can test the validity of the hypothesis by incorporating their personal consequences into the reward function. Email from legal: -10, $10k fine: -1000, 1 year in prison: -100000.
• EncomLab 2y ago
  "What people want from the automated calculator is not more accurate sums, but a box into which they may place their responsibility for acting on the results." - Norbert Wiener His book God & Golem Inc. is incredibly prescient for a work created at the very beginning of the computer age.
  • 52-6F-62 2y ago
    I wonder if they really want that, or if that is what is being actively peddled as the new and better way and they’re just ignorantly buying it up.
    • EncomLab 2y ago
      My career in IT has taught me that everyone loves to blame the box - and if not the box the person responsible for the box :)
      • 52-6F-62 2y ago
        And who sold them the box as a complete life solution?
• andy_ppp 2y ago
  It's funny because the management where I work operate the policy: "A manager can be held accountable" "Therefore a manager must never make a management decision" \shrugs
• michaelgiba 2y ago
  A different, darker way to interpret this is computers cannot be held accountable today If systems (presumably AI-based) were conscious or self-aware they would very much be incentivized not to make mistakes. (Not advocating for this)
  • kkkqkqkqkqlqlql 2y ago
    But what would be the deterrent there? You should program the AI with some sort of "fear of death" or "fear of consequences", and if that's the case, wouldn't it be straight up slavery?
• malwrar 2y ago
  …therefore the operator is responsible? Seems like the clearest legal principle to me, otherwise we ban matches to prevent arson.
  • platz 2y ago
    the owner
    • andrewflnr 2y ago
      Whoever is responsible for putting the computer in a position where its decisions mattered. Whether that's the owner or their agent is a question for which we already have a couple centuries of mostly-adequate legal precedent.
  • andrewflnr 2y ago
    This principle also works for security vulnerabilities in open source software, by the way. That is, the responsibility for preventing security exploits rests on the party who operates or deploys the software. Don't want the "risk" of open source? Feel free to use something more expensive. But it might be cheaper to pay the original developer for patches.
• mgdev 2y ago
  Unless it can die. And understand death. And adapt behavior to avoid death. Of course, that raises a whole new set of issues.
  • manmal 2y ago
    Avoidance of death is just the bottom of Maslow‘s pyramid, there’s a _lot_ of other things that drive human decisions.
    • mgdev 2y ago
      It's described as a pyramid for a reason.
• kogus 2y ago
  This is pithy, but wrong. Management decisions are not the only important decisions. Also, the whole point of computers is to crystallize human thought into action, and take that action on a larger and faster scale than people can achieve. Airplane autopilots make many important decisions. I'd suggest a lesson that might be less agreeable to IBM, Microsoft, Google, et al: "The makers of software must be accountable for the mistakes made by that software."
  • fargle 2y ago
    "The users of software must be accountable for the mistakes made by that software." the contract between user and "maker" should be requirements. if a machine does not fulfill its requirements, it can be the maker's responsibility, for example a flight instrument that failed to report correct information. but if you choose to use a piece of software that says "we actually guarantee nothing", which is the vast majority of it, then it's definitely the user's responsibility for choosing to use such a tool.
    • kogus 2y ago
      That's a good distinction. A knife maker should not be liable for the serial killer who used their product. But they probably should be liable if the knife explodes when used to cut a piece of cheese. I'm sure there is a whole library full of legal precedent around "who is liable, and when". My earlier point was really that "Machines are not accountable" does not mean "Machines cannot make decisions". It really means "We need to think about who is accountable when this machine makes a mistake.
• WaitWaitWha 2y ago
  Interestingly this is an axiom of digital forensics uses in reverse of sort. In court, digital forensics investigators can attest what was performed on the devices, timeline, details, and such. But, it should never be about a named person. The investigator can never tell who was sitting at the keyboard, pushing the buttons, or if some new and unknown method to implant those actions (or evidence). It is always jarring to laypeople when they are told by the expert that there is a level of uncertainty, when throughout their lives computers appear very deterministic.
• andrewla 2y ago
  I think philosophically this is a good rule of thumb; the problem is that the euphemism treadmill (or whatever) has done its work. "Accountable" is meaningless therapy-speak now. CEO says "oh, this was a big problem, that we leaked everyone's account information and murdered a bunch of children. I hold myself accountable" but then doesn't quit or resign or face any consequences other than the literal act of saying they are taking "accountability".
  • jonahx 2y ago
    In contrast Law 229 of Hammurabi's Code: "If a house builder built a house for a man but did not secure/fortify his work so that the house he built collapsed and caused the death of the house owner, that house builder will be executed." While extreme, this is the only type of meaningful accountability: the type that causes pain for the person being held accountable. The problem is that (for better and worse) in the corporate world the greatest punishment available for non-criminal acts is firing. Depending on the upside for the bad act, this may not be nearly enough to disincentivize it. https://ehammurabi.org/law/229 https://ehammurabi.org/law/229
    • satvikpendem 2y ago
      Studies show that sentences like the death penalty do not actually reduce crime. There's a reason there's a saying that stems from Hammurabi's Code, "an eye for an eye makes the whole world blind."
      • jonahx 2y ago
        My understanding is that this is not a settled issue, and there is dispute about the evidence, conflicting studies, and so on. Regardless, to the extent that it's true, it's imo likely more to do with the nature of crimes of passion, and is certainly not evidence against the effects of incentives in general. Death penalty aside, if you are doing any kind of work, and you know you will be held accountable for errors in that work in ways that matter to you (whether SLA agreements, getting sued, losing payment, etc), it will change the way approach it. At least for the most people.
• crabbone 2y ago
  Around 2010 I was invited to interview for Google's SRE team. In the course of preparation for the interview, the Google's HR person appointed to me gave me a list of various questions I'd have to prep for. One of the questions was "what should be the next large project for Google?". My answer, unironically, was "GoogleGovernment". The idea was to build SAP-like suite of programs that a country could then buy or rent and have a fully digital government to run the country... Luckily, that question never came up in the interview, and remained an anecdote I share with other coffee drinkers around the coffee machine. My younger self believed (inspired by a chapter from the UK citizen act translated into Prolog) that the success could be expanded much further (I didn't bother reading the accompanying paper, at least not at that time). While it was already mentioned that there will be people either unable to overpower the computer system making bureaucratic decision as well as those who'd use it to avoid responsibility... I think it's due to the readers here tending to be older. It's hard to appreciate the enthusiasm with which a younger person might believe that such a computer system can be a boon to society. That it can be made to function better than humans currently in charge. And that mistakes, even if discovered, will be addressed in a much more efficient (than live humans would) and fixed in a centralized and timely fashion.
• kevin_thibedeau 2y ago
  You see this already socially normalized with retail security alarms that go off when a tag isn't deactivated by the cashier. You are all of a sudden being implicated as a criminal while exiting with your property. Oh! It's just that pesky computer again. Nothing to be concerned about. Funny things those computers.
• dfilppi 2y ago
  [dead]
• slowhadoken 2y ago
  Someone finally watched the 30 Rock episode The Ballad of Kenneth Parcell.
• fosterfriends 2y ago
  This is why AI should never give the final code review approval
• Lardsonian 2y ago
  Wait till you discover bit flips.
• wouldbecouldbe 2y ago
  Seemingly public officials can’t either
• yubblegum 2y ago
  Right under the last line you can see a ghosted "a form of suicide". That's pretty strange for an IBM document.
• amai 2y ago
  An elevator can never be held accountable. But the producer of the elevator is usually held accountable if the elevator crashes.
• fennecfoxy 2y ago
  Tbf do we even really care about this issue? So many private companies and the individuals haven't been held accountable in so many ways and for so long. We'd rather squabble over race, religion, sexuality, migrants etc than address the Elephant in the room.