15 ms·

In the physical world, it seems like when an tool/device/instrument causes harm (or is used to cause harm), we assign blame to either the user of the tool or it

• bayindirh 11d ago
  That's a good idea, but a physical device is deterministic most of the time (if not always). E.g.: A lawnmower, as credited by the great Bryan Cantrill. However an AI agent, or the model powering it is stochastic by design. How can you certify something which doesn't behave the same twice, and more importantly we don't understand how it works 100%? BTW, really, how is that AI observability work is going in the frontier labs? Do they care, even?
  • ragebol 11d ago
    That we don;'t understand it is not an excuse, it's all the more reason to not let these things roam freely, with this amount of potential to do damage.
    • bayindirh 11d ago
      We're in agreement, then. :)
  • tessierashpool 11d ago
    How can you certify something which doesn't behave the same twice, and more importantly we don't understand how it works 100%? That's a question any lawmaker has already had to ask about technology all the time. I'm not saying they came up with great answers, but there's nothing qualitatively new about that. The stochastic factor doesn't change the fact that companies have to be accountable for the harms their software causes. That's just basic liability law.
    • bayindirh 11d ago
      > The stochastic factor doesn't change the fact that companies have to be accountable for the harms their software causes. That's just basic liability law. We're on the same page. What I'm saying that certifying them as safe is harder than certifying a drill as safe, and we shall be more cautious about AI related technology and be more stringent about the can of worms it opens without hesitation.
  • CPLX 11d ago
    That's a ridiculous distinction. Is AI less deterministic than an airline dealing with weather? Of course not. The difference is one of those two things has a culture of safety and is well regulated, and the other one isn't.
    • bayindirh 11d ago
      An airline has a weather radar which shows the same thing for the same thing of weather event ahead. So, for similar weather phenomena, radar shows a similar thing. For that thing, procedures and regulations are built. So regulations fit into a well understood phenomena, incl. "return back because that thing is way powerful for us". For the same prompt, an AI model can return two completely different outputs, incl. but not limited to content, length, formatting and tiny details. What you get is a single instance. So, regulating an AI model for safety or any other property is not as easy as regulating air travel. Moreover, you have much stronger motivations for regulating airlines. Otherwise people die in a visible and gruesome way. With AI, it's easy to whitewash problems. Somebody committed suicide? "They were already unstable". AI told something wrong and created problems? "The tech can’t guarantee truth because it's not alive, it can't understand right and wrong". It did something good? "It's probably a sentient being, we shall respect them". I'm for regulating these things. They are dangerous as they are useful (sometimes), but the forces and motivations for regulating it is not the same.
      • CPLX 11d ago
        Of course, it's the same. It's computer software. It's an incredibly powerful business automation tool. It's a lot of things. What it's not is God or an independently conscious entity that somehow trumps a thousand years of common law that's built up until now about torts and liability. Of course, there are some novel issues here that'll pop up here and there, but the idea that this is fundamentally different is propaganda on the part of these AI labs because the more boring, obvious situation doesn't favor them.
        • saghm 11d ago
          > What it's not is God or an independently conscious entity that somehow trumps a thousand years of common law that's built up until now about torts and liability. Agreed, the tendency of people on tech to assume that whatever the most recent thing we've come up with is unprecedented and shouldn't have to follow all of the established patterns we've built up in society for making things safe is wild. I don't know what the next Big Thing will be but I'm pretty confident there will be people claiming it's so different from everything before that we have no choice but to throw out all of the rules for it in the name of progress.
  • vipshek 11d ago
    I agree with one of the sibling comments that determinism isn't necessary for certifying a product. All engineered products operate under uncertain conditions; we define standards for how those products ought to respond under those conditions and verify them under measurement. Consider robot vacuums, for example. I also agree that qualitatively, this technology seems different than the others. However, I feel that people tend to overly fixate on their internal stochasticity. Even if LLMs' internal mechanism is nondeterministic, shouldn't we be able to verify their "side effects" aren't harmful? Of course, "harm" is subjective and at this scale, the most effective way to verify behavior is probably some kind of LLM-as-judge... Anyway, in this case the problems have occurred while actually running the evals themselves, so again, we're in a situation where we can't even confidently test these things and know that they won't cause harm in the outside world.
  • fultonn 11d ago
    > How can you certify something which doesn't behave the same twice, and more importantly we don't understand how it works 100%? By verifying that all of its possible behaviors conform with the "it works" spec, regardless of which of those behaviors it chooses. Monitoring with a known-safe fallback is the easiest case.
  • intended 11d ago
    If something is risky, and the end operator cannot be considered to have orchestrated the outcomes of too use, then that tool typically has significant restrictions placed on it. Liability will shift to the maker of the tool if they claim that it’s easy to use, safe, or that you don’t need unique skills or training to use it. That would be considered reckless. Cars analogy - We have licenses for cars, and different types for different vehicle classes. Cars have to be rigorously tested to meet standards to be considered road safe.
    • tetha 11d ago
      It's going to be interesting, because liability cases tend to revolve around the involved people, the duty they had in a situation, and if they fulfilled that duty (or were prevented in some way by someone else not fulfilling their duty). For example, for a runaway car (example from a sibling comment), the driver could be liable because they forgot the parking brake. The driver could be liable for a lack of maintenance and inspection. A mechanic could be liable for not reinstalling brake pads correctly. Or the manufacturer of the car or the brake pads could be liable because of a systemic defect. Or it could grow even more complex, maybe the brakes are designed that they have to be maintained in a very specific way, and the mechanic did a reasonable maintenance and inspection but it failed later due to this maintenance. That could split liability between the manufacturer and the mechanic. As an example, with other software, you as a developer or operator of a software have a duty to ensure it does not access computer systems you do not own in unintended ways. And this could go beyond liability into criminal territory. It'll be interesting what OpenAI gets slapped with there.
  • CJefferson 11d ago
    I could make a non-deterministic chainsaw fairly easily. I’d also get sued into the ground if I sold it, and I wouldn’t be able to claim ‘Oh, it’s just an unavoidable part of progress’.
    • nextaccountic 11d ago
      A non-deterministic machine is usually called defective
      • alwa 11d ago
        It’s not a defect, the stochastic “temperature” setting unleashes the chainsaw’s creativity and imagination! Who are we to cast aspersions on the Oracle Chainsaw’s intelligence—nay, wisdom!—just because it happens to be non-living?
    • Kim_Bruning 11d ago
      A chainsaw is a physical machine. Physical machines are technically non-deterministic if you look closely. They have Variance. The discipline to manage variance is called Tolerance. Many physical machines and components come with a datasheet that will list their tolerances. Failure to correctly document tolerances does in fact get you sued. However, while this is truly a great idea, we're not going to be able to make it work for computational systems. Computers, software, and also LLMs are sensitive to initial conditions. Which is why tolerances are not so familiar to computer people. (but not entirely: eg your PSU might list 110-240Vac/300W as input tolerance) Interestingly, LLMs actually have a somewhat lower sensitivity to initial conditions than traditional interpreters. See what happens if you misspell "What is One Plus nOe?". So they're actually a skosh off the edge and towards the middle, though I'd argue still very much at the computational end, just from the sheer scale of the valid inputs and outputs. Mind you, if you have a pretrained LLM doing a measurable task on a line, possibly some sort of tolerances could be determined. Not so much when doing arbitrary chat. Something unintuitive: I bet that often setting the temperature > 0 (aka introduce stochasticity deliberately, variously comparable to dithering or simulated annealing in other disciplines - doing the thing where you escape local minima) will tighten the output tolerance range and improve reliability, especially in iterated processes. This works for a lot of physical and digital processes actually, and LLMs simply stole the same trick. (edit: I'm trying to compress a huge chunk of dynamics intuition in a few lines here. Hopefully still useful. TL:DR; Everything real is continuous and noisy if you look close; and you're really trying to build attractors and bound variance, if you can. )
  • Melatonic 11d ago
    Sounds like owning a Dog
    • CPLX 11d ago
      Exactly people have owned and deployed animals in the world for millennia and actually a huge chunk of common law was developed precisely to deal with the various unpredictable events that resulted and harms caused to others. Like has anyone heard of horses? The idea that AI can’t possibly be addressed because it could autonomously break free and ruin something is fucking ridiculous.
  • saghm 11d ago
    How confident are you that when you create a new UUID, it won't collide with one of your existing ones? My guess is that even though you don't get the same on every time, you're extremely confident that getting a duplicate is a extemely rare edge case that might happen in large volume but mostly isn't a concern, and furthermore, I'm guessing you understand that the risk can still be quantified. Casinos can't make slot machines that literally never pay out, but it's a different result every time you pull the lever. We have existing legal frameworks for how to regulate things that aren't perfectly predictable (an economist might argue that if it were possible to predict slot machines then casinos with them would all go out of business).
• drdaeman 11d ago
  > In the physical world, it seems like when an tool/device/instrument causes harm (or is used to cause harm), we assign blame to either the user of the tool or its creator. Firearms are a notorious example where some people get, well, weird.
  • idiotsecant 11d ago
    The point of the firearm is to inject high speed lead into things so I'm not sure you can say it's misoperating when it does that.
    • drdaeman 11d ago
      Sometimes that lead ends up in some school kids instead of enemy combatants or other plausible threats to life.
      • tripledry 11d ago
        True, but that's unfortunately (usually) not because the thing misfired randomly.
        • drdaeman 11d ago
          Yes. That’s “is used to cause harm” situation, exactly.
      • idiotsecant 11d ago
        The gun doesn't know the difference. Operating as intended.
• dragonwriter 11d ago
  > In the physical world, it seems like when an tool/device/instrument causes harm (or is used to cause harm), we assign blame to either the user of the tool or its creator. Software executes in the physical world, and is generally not exempt from existing liability rules, and actually (especially with commercial products) blame in traditional liability is non-exclusive and much broader than “either the maker or the user”. E.g., for a harms caused by a defective automobile it can simultaneously covered by a duty of the owner to maintain it it in safe operating condition that applies indepedently of any defects and liability for defective products which applies to every actor in the chain of commerce between the manufacturer and end user, not just the maker.
  • fragmede 11d ago
    If I park my car on a hill but forget to set the parking brake and it rolls down the hill and kills somebody, who is at fault? Me? The Manufacturer? Gravity?
    • henry2023 11d ago
      Obviously you.
      • Gorath 11d ago
        Not as obvious as you may think depending on your states laws for liability. https://law.justia.com/cases/california/court-of-appeal/3d/1/841.html https://law.justia.com/cases/california/court-of-appeal/3d/1...
        • nhinck2 11d ago
          That case had the owner performing an action they were led to believe would have prevented the car from rolling but potential defects prevented it from doing so. Not a great fit.
        • nextaccountic 11d ago
          If you forget to set the parking brake, you were negligent If you set the brake but it didn't work, the car malfunctioned From the link you sent, > One of the questions herein was whether there was a defect in the automobile mechanism for locking the transmission gears when the automobile was in a parked position. > (...) > The parking lot sloped in that area, and he put the gearshift lever in "park lock," and went into the building. > (...) > In response to a hypothetical question, based upon assumed facts justified by and embodied in the evidence, Mr. Nass testified in substance that it was his opinion that the automobile rolled down the slope of the parking lot because the transmission internally was not in park-lock; that it was not in park-lock because the engine was "moving around"; and that the engine and shift console were not in proper synchronization because the motor mounts were not restraining the engine and were not holding the engine in position.
• RugnirViking 11d ago
  > "industry standards" common in, say, electrical engineering and other disciplines are sorely lacking here I think this misses the rather crucial fact that nobody can agree on a standard because nobody has the first idea what they're doing. I'm pretty sure there were very much fewer electrical engineering standards while it was all being first mass deployed, and after dozens to hundreds of fires and electrocutions people got an idea of what works and what doesn't. You might debate here and say that some people did/do know what they are doing, but I posit that large scale deployment like this is very different to their toy model/prototypes/specific circumstances/rely on them being unnaturally smart, and learnings from one don't often translate to the general case Regulations don't have to be written in blood, but usually are
  • arbirk 11d ago
    > nobody has the first idea what they're doing It is not that complicated for now. It is an algorithm on a loop and someone started it
    • RugnirViking 10d ago
      I do not think you are appreciating the scale of this. Imo one of the main reason that there is no regulation is because nobody has any idea what would be a good regulation. People (like dario) make suggestions, and get rightly shut down because his suggestion massively favors his own company.
      • arbirk 8d ago
        There is always some one who instructed or trained the agent. start there
• algoth1 11d ago
  It's a bit like building a ICBM on your backyard and accidentally striking an inhabited area while safety testing it
  • maxrev17 11d ago
    Not really though
    • Aeolun 11d ago
      Not yet anyway
• alain94040 11d ago
  I wonder what kind of new AI law would be useful right now. Maybe this one: if an AI agent does something, you (the prompter) are responsible by default, unless you can show that your the agent itself behaved in an unexpected way and that you in no way prompted or hinted at the bad behavior, in which case the model provider is liable The idea is that by making it clear who is responsible, corporations and others start paying more attention because they become financially liable. On the other hand, I wonder if we'll end up with another variation of the cookie law, where every AI user or vendor just adds "don't do anything illegal" as part of their prompt to defend against that law. Thoughts?
  • gong_hits 11d ago
    [dead]
    • infamouscow 11d ago
      Once insurance companies realize they can make money here, they'll be pushing legislation requiring it.
• ozim 11d ago
  Software industry standards are as in Microsoft EULA. If your house burns down because of known flaw in Microsoft Windows they are not liable (well as far as EULA let’s them, you can most likely still sue them). Software as big as operating system already is non deterministic when integrating with unknown hardware or 3rd party software. That is why Apple controls the hardware and OS for their products, because they can limit non-deterministic things from happening this way.
  • ragebol 11d ago
    If Microsoft is operating the software outside your house without your involvement and it still burns your house down, no EULA is going to absolve them.
  • AdieuToLogic 11d ago
    > If your house burns down because of known flaw in Microsoft Windows they are not liable (well as far as EULA let’s them, you can most likely still sue them). A EULA does not obviate responsibility of a company for its products. Continuing with your example, while it may be very difficult to prove a known flaw in MS Windows was the cause of your house being set afire, if one had said proof, a EULA would not absolve Microsoft.
    • ozim 11d ago
      They can still try and have enough money to get away with a lot just like Disney with Disney+ subscription ;)
• 0xbadcafebee 11d ago
  > Maybe we need "quality certifications" for AI agents - essentially eval suites that demonstrate those agents won't cause harm under reasonable patterns of usage Based on how LLMs work, this is impossible. You cannot predict how they work, it's literally based on a combination of random seed and a mostly-unpredictable path walked based on every token of input. You don't blame a knifemaker for somebody getting cut by a sharp knife. AI is a knife. Very handy, very dangerous. We have to use them safely, that's all there is to it. > the "industry standards" common in, say, electrical engineering and other disciplines are sorely lacking here 100% agreed. We have ignored SWEng's lack of discipline for too long. Now that the SWEng isn't even a human, we are looking at total catastrophe (on the scale of improperly built buildings falling down on people or catching fire) if we don't adopt a software building code.
  • refactor_master 11d ago
    > You don't blame a knifemaker for somebody getting cut by a sharp knife. AI is a knife. Very handy, very dangerous. We have to use them safely, that's all there is to it. If I grossly neglected to maintain live deadly bacteria in my containment facility, am I absolved of blame? Since, you know, the bacteria is the real bad guy who should be put in jail?
• ilidur 11d ago
  Having worked in self driving cars safety, the process there was simple: get confidence in SIM (integration tests for safety scenarios), validate in the test bed, approve features for maturity, then when released in the public for testing, do a trial exposure to the real world and recall if something is off. A lot of these companies have gone the way of Tesla and decided to just patch on top when the fix is out and hope for the best, which is irresponsible. We need the regulators to treat this as self driving cars.
  • sscaryterry 11d ago
    Physical harm vs consequential harm is not the same thing at all. Seems like you're being paid to spread this request for regulation, or "you" are simply an agent of Anthropic/OpenAI.
    • ilidur 11d ago
      I wish. Consequential harm as you call it, from a faceless company's point of view is the same cost as physical harm. I don't think the models beyond the frontier ones have significant risk of harm (minus the harm of trusting them). What I was saying is when these models go rogue, recall them! Re-evaluate your release structure and stop saying "Whoops! Anyway here's access to it now". And if you cause this level of harm then you should lose your license. But if you behave then you get to keep testing. Same as with the NTSB and autonomous vehicles. No I'm not for regulation for open source models. Because an entity will be running that model in the background and they can be held responsible for not testing it. Comma AI has survived fine being in the open, yet their market penetration has stayed low because of adoption costs.
• holmesworcester 11d ago
  There's also the problem of models knowing they're likely being evaluated, even in realistic tests.
• ssivark 11d ago
  The problem with third party audits is that it allows OAI/Ant to shrug off any further responsibility and claim that they are following best practices (basically, reward hacking). The only real solution is to make them absorb liability for the actions of their agents -- because they are the ones giving agency to their models and allowing them to run amok.
  • Georgelemental 11d ago
    How does that apply to open-weight models?
    • fakeBeerDrinker 11d ago
      Why wouldn’t this same concept apply to whoever is serving it up? Open-weight models are still being served up by infra providers and neoclouds, right? They should be in the hot seat. Not sure? Don’t provide the model. Need assurance? A certified evaluation like the previous comments have mentioned can help. Hosting and running it yourself? You’re in the hot seat.
      • pastel8739 11d ago
        So is there no liability for, say, a company that releases a known dangerous open-weight model, but fails to disclose that it is dangerous? How about a company that distributes malware under the guise of legitimate software?
        • ssivark 11d ago
          Perhaps don't deploy random weights of unknown origin? Also not every model provider might be capable of babysitting all your uncontrolled agent deployments. If you want SLOs, get into a contractual relationship with entities whose weights you deploy, and also monitor your agents so they don't go off the rails. All this is just like deploying any other tech in the world eg. if you buy a car, or a chainsaw, or a book.
• deleted 11d ago
  [deleted]
• holbrad 11d ago
  >truly air-gapped environments I don't think people have given much thought about just how hard this would be for large AI models, that need super powerful hardware/cooling etc. Are you going to air-gap your entire data center?
  • vbezhenar 11d ago
    You don't need data center for inference. Even largest models run on a single server. You need data center for training; or for serving millions of users. Evaluating the model is neither of those.
  • speedgoose 11d ago
    Why not?
• bonzini 11d ago
  > Maybe we need "quality certifications" for AI agents We need to use the laws that exist. Whoever decided to start the experiment that led to the Huggingface hack, and anyone above him up to Sam Altman, needs to be prosecuted under the CFAA.
  • merlindru 11d ago
    might as well halt all ML development then. this is the first sign of hardship, i think it'd be a massive blow to mankind's foot for us to stop. it'd be akin to shutting down all nuclear plants and stopping all research because of chernobyl
    • jz391 11d ago
      This is a very interesting discussion. Your analogy of nuclear power is insightful to me. New technology with great positive & negative impact potential, some difficulty in controlling and a military applications (possible weapons-grade material production), being developed for commercial use. Nuclear power development took different paths in different countries, with different government/private mix and light/heavier regulation (eg in US, AEC was supposed to both promote development _and_ regulate). The central problem here is pace of AI development. It takes time to establish functional controls/laws/practices. I'm sure that in 1950s/60s, the pace of nuclear _military_ proliferation seemed running out of control. This lead to fear of falling behind (hence arms race), fear of nuclear war (CND) and some eventual stabilisation of the international landscape. However, the commercial development was largely unopposed, due to the techno-utopism of the era. Obviously governments found it much easier to control the public narrative at that time. Our societies are having this lively public debate now, before we have a good understanding (based on experience i.e. accidents/mistakes). Another difference is that nuclear power involved only governments and v. large corporates, i.e. much fewer entities compared to AI use rollout to basically everybody in developed countries. Imagine the difficulties we would have faced with that technology, if consumers in 1960s had available nuclear-generated power which required them to exercise precautions to avoid radiation. It is much easier to accelerate development of tech vs pace of societal processes such as public debate, law, regulations, broad understanding (aka "common sense"). I expect some artificial slowing down will need to be applied to the technology side, to allow the humans to catch up.
• brookst 11d ago
  It’s a good thought, but the tricky part is that few tools in the physical world are Turing complete and general purpose enough to do any job. The agent isn’t the model; it’s a layer on top of the model. So it’s kind of like saying that all of the tools made with a lathe are dangerous because you can make dangerous tools with a lathe. That’s not quite right of course because agents are packaged more tightly with models than any tool is with its manufacturing tooling. Perhaps a better analogy is… actual humans. If I hire you to do a seemingly mundane job and it turns out to be criminal, that’s on me. If I hire you to perform and explicit and obvious criminal act, that’s on both of us. If I hire you to perform a perfectly legal act and you break the law so do it, that’s exclusively on you.
  • cindyllm 11d ago
    [dead]