5 ms·

Ask HN: Have top AI research institutions just given up on the idea of safety?

I understand there's a difference between the stated values and actual values of individuals and organizations, and so I want to ask this in the most pragmatic and consequentialist way. I know that labs, institutions, and so on have safety teams. I know the folks doing that work are serious and earnest about that work. But at this point are these institutions merely pandering to the notion of safety with some token level of investment? In the way that a Casino might fund programs to address gambling addiction. I'm an outsider and can only guess. Insider insight would be very appreciated.

tchalla 7mo ago
Are you asking about top AI research institutions or leading AI businesses? There’s tons of work in research communities.
- Goofy_Coyote 7mo ago
  Where can I find some of these researches? Any links or pointers are very much appreciated. Everything I find by searching is marketing BS, or the same half-baked prompt injection protection that only works for cherry picked problems. Really need some help here finding the right communities.
  - tchalla 7mo ago
    Always look at conferences and associated workshops. You can start with NeurIPs and ICML. From there, you will figure out some papers on safety. Then, you can see some patterns of labs which work on it full time.
co_king_5 7mo ago
[dead]
- co_king_5 7mo ago
  [flagged]
- JTbane 7mo ago
  For sure, there are a bunch of cases of agents deleting files, leaking secrets, and wiping databases even when told not to.
akersten 7mo ago
Not an insider but someone who uses the tools. It's a branding update, nothing more. The models haven't gotten any less sanctimonious, but the companies behind them have stopped harping on their restrictions in order to appeal to a broader customer base (gov contracts, etc.) So the guardrails (for you and me) are still there. They just stopped committing the unforced error of excluding themselves from federal procurement. Under a different administration, the requirement might change, and you might see them boasting once more on "safety."
- MattDaEskimo 7mo ago
  Surely safety does not exclusively mean guardrails, but the philosophy and ethics instilled during training?
  - toddmorey 7mo ago
    The ethics are exactly what the DoD is complaining about. They want any legal action to not be obstructed by guardrails.
    - nmeagent 7mo ago
      Forget legal, they want any action to not be obstructed by guardrails.
- toddmorey 7mo ago
  I don't think it's sanctimonious to say, hey, I don't want the technology I work on to be used for targeting decisions when executing people from the sky. Especially as the tech starts to play more active roles. You know governments will be quick to shift blame to the model developers when things go wrong.
  - pjc50 7mo ago
    > I don't want the technology I work on to be used for targeting decisions when executing people from the sky What do you do when the government come to you and tell you that they do want that, and can back it up with threats such as nationalizing your technology? (see Anthropic) We're back to "you might not care about politics, but that won't stop politics caring about you".
nemomarx 7mo ago
Well, Anthropic clearly has some kind of lines if their recent argument with the us government is anything to go by. "don't kill humans" isn't all of safety or alignment goals but it is something?
CivBase 7mo ago
The goal is and always was to make as much money as possible. Any consideration for how it affects actual people was marketing to get ahead of bad PR and regulation. Safety was never a genuine concern. They simply don't benefit from marketing themselves that way anymore so they've stopped pretending.
caconym_ 7mo ago
At this point, do you really think any of these "labs" would give up competitive parity or advantage just because they're already making life worse for a lot of people and (by their own admission) stand to make it much, much, much worse? The persistence forecast says no. There are maybe a few token exceptions, like Anthropic's current pushback against the DoD, but by and large I think we can continue to expect them to pay lip service to safety while continuing to build toward systems that, by their own admission, have incredible potential to cause harm. As you noted, the fact that they employ safety researchers does not necessarily mean that they will put safety over revenue.
nkohari 7mo ago
I don't think they've given up on the idea, but as AI becomes increasingly mainstream, the labs will be under immense pressure to hold the line. We're seeing this play out right now with Anthropic and the Pentagon. These companies have raised eye-watering amounts of funding, and will need to continue to do so for the foreseeable future. They're not yet self-sustaining, and this insecurity increases the pressure for them to compromise on ideals. With that said, there is a massive war for top talent, and I think that the employees at the labs would become increasingly uncomfortable with their work being used for Bad Things. If Anthropic capitulates to the Pentagon, it wouldn't surprise me to see a mass exodus of talent occur.
kgwxd 7mo ago
HA, you mean has PR given up on pretending? Yes.
leptons 7mo ago
We're not in a race to the bottom with "AI", we're in a speedrun to the bottom.
scarmig 7mo ago
More on the periphery than an insider, but I personally know researchers in all three major labs who were there long before GPT-3. They all care about existential safety, a lot. In the sense that they believe there's a meaningful chance all humans are dead a decade from now (and that that's a bad thing; unfortunately, there are also people deeply involved who don't think human extinction is a bad thing). The issue is that they're embedded in capitalism, and that drives the labs to push further and faster than is responsible. They (and unfortunately us) end up in a race where no individual feels like they can back off or halt, because if they do, they will be destroyed.
- sigbottle 7mo ago
  /rant Existential in what sense? There's this one sense in which people are almost moral about it: "yup, AI is just superior to humans, nothing we can do about it." And then there's ones where the elite class implements mass surveillance and warfare and obsoletes billions of humans of their own volition. These AI are already capable enough right now to execute on said plan (of course, with proper evil engineering) There's two ways to "win". One is in an absolute or platonic sense - one that cares about things like values, even in the presence of extreme pushback. The other is in a darwinian sense. No, not in the meme way that again, feeds back into the narrative of "the things that survive are smarter". The things that survive, survive. It doesn't matter how it gets there. I can agree with the second way. But it gets smuggled in as the first way, almost as an attempt to crush any and all resistance preemptively. AI doesn't need to say, be capable of pushing the frontier of quantum mechanics to be lethal. /endrant Sorry, not really related to your comment, just had to get it out there.
  - SpicyLemonZest 7mo ago
    In the context of AI research, there is no question that "existential" means "powerful AI literally kills every human being". It's a mainstream although not universal view among experts in the space that this is a serious possibility.
    - sigbottle 7mo ago
ChrisArchitect 7mo ago
Related: Anthropic Drops Flagship Safety Pledge https://news.ycombinator.com/item?id=47145963 https://news.ycombinator.com/item?id=47145963
naillang 7mo ago
[dead]
AndrewKemendo 7mo ago
I was the author for the practitioners implementation section for the IEEE 7010 standard for assessing human impact from AI software https://standards.ieee.org/ieee/7010/7718/ https://standards.ieee.org/ieee/7010/7718/ I also worked closely with Jack Clark at OpenAI before he disappeared on all these issues as CTO back in 2018 There are literally zero “AI labs” that have ever cared about “safety” none of them have ever done anything tangible with any kind of independent auditable third-party way that has some defined reference baseline for what is safe and what is not, how to evaluate it, or a practitioners guidance for how to determine what it is and what is not safe as a designer. They follow the same rules as every other technology platform: do as much as you can legally get away with no more no less I say this as somebody who’s been actively involved in the AI “safety” debate for a long time now at least since 2013 The concept itself doesn’t even make sense if you fully understand the intersectional scope of technology and society Societies demands are the things that are unsafe not the technologies themselves Just like Bertrand Russell said “as long as war exists all technologies will be utilized for it” - you can replace “war” for anything that you think is unsafe
- Goofy_Coyote 7mo ago
  Can you elaborate this part please? > The concept itself doesn’t even make sense if you fully understand the intersectional scope of technology and society Societies demands are the things that are unsafe not the technologies themselves Where can I learn more about it?
  - AndrewKemendo 7mo ago
    Go back to the fundamentals and read society of mind from Marvin Minsky or anything cybernetics from Norbert Wiener
- chasd00 7mo ago
  if would be super helpful if you could give the elevator pitch version of what a safe AI is.
  - 7mo ago
chasd00 7mo ago
"safe" is such a subjective concept to begin with, have any of the model providers ever defined what they mean by "safe"? It doesn't mean much to me if a safe model is one that does not output the recipe for mustard gas, that information is trivially available elsewhere. Or, is a safe model one that doesn't come off as racist? Ok but i would classify that as unoffensive instead of safe but I admit definitions of words can be fluid and change. Is a safe model one that refuses to produce code for a weapons system? Well.. does a PID controller count? I can use that to keep a gun pointed at a target or i can use that to prevent a baby rocker from falling over. Maybe they're giving up on "safe" because there's no definitive way to know if a model is safe or not. I've always held the opinion that ai safety was more about brand safety. Maybe now the model providers can afford some bad press and it not be the death of their company.
- justonceokay 7mo ago
  > because there's no definitive way to know if a model is safe or not. The only answer is there’s no money on it being safe. It is not an epistemic problem
- LordHumungous 7mo ago
  What if I tell the model to go commit fraud or crimes and it complies? What if users are having psychotic episodes driven by their interactions with the model? Just because safety is a hard and messy problem doesn't mean we should just wash our hands of it.
  - ryandrake 7mo ago
    It is a hard and messy problem, and it doesn't help when people muddy the water further by stirring things like "Don't commit fraud," "Don't infringe on Disney's trademark," and "Don't be racist" into the mix and try to lump those things under the "Safety" umbrella. Maybe this is an outdated definition, but I've always thought of safety as being about preventing injury. Things like safety glasses and hardhats on the work site, warning about slippery floors and so on. I think people are trying to expand the word to mean a great many more things in the context of AI, which doesn't help when it comes to focusing on it. I think we need a different, clearer word for "The AI output shouldn't contain certain unauthorized things."
pluc 7mo ago
It's such an US thing to ask "Are they doing the thing they are doing?" You've read about it, you can clearly see it in action from multiple companies, yet you're here asking "hey is this happening?" Yes. Yes it is. Yes they are giving up on safety. They are openly saying so. It is easy to see if you take just a second to look for yourself instead of looking at press releases and algorithmic promotion. https://time.com/7380854/exclusive-anthropic-drops-flagship-safety-pledge/ https://time.com/7380854/exclusive-anthropic-drops-flagship-...
ergonaught 7mo ago
Why the constant chasing after universally applicable generalizations? Some of them are pandering. Some aren't. Some care. Some don't. Businesses with ferocious funding needs are vulnerable to pressure (internal and external) to do whatever aligns with money and power. Money and power will flow into the ones so-aligned. That is the nature of the parasitic extraction models that typically drive decision making at those kinds of companies.
dasil003 7mo ago
Safety is a nice idea but it’s not structurally pursuable at this point. Everything is moving too quickly and we don’t exactly know what is useful or not, just like we don’t know what’s safe or not. Anyone pursuing safety will be outcompeted by someone who isnt. Given the amount of investments there is no patience for any calls to slow down. I tend to believe this won’t actually end in disaster as I don’t think it’s actually economical to put AI everywhere with enough real control that we can’t manage the risks as they evolve, but it’s a low confidence prediction.
chris_money202 7mo ago
I think there are plenty of people working on it still, I am even working on a startup that prioritizes safety for physical autonomous systems! The problem is that safety is written in blood. Airlines implemented flight recorders / black boxes and various processes after major incidents. A major mistake occurs that causes death or destruction to property, or both, an investigation occurs, we learn from it, and introduce new laws and regulations to prevent a reoccurence.
vasco 7mo ago
A safety team at the hammer company cannot prevent me from using it to bang your head. You can align to the user wants and so you are a hammer. This is alignment>safety. Or you take a safety first approach where the AI decides what safe is and does its own bidding instead of yours. This is safety>alignment. I prefer hammers to be honest. Mostly because humans can be prosecuted, AIs can't. So if the human wants to commit crime with the AI it should be able to, because the opposite turns to dystopia fast.
qsera 7mo ago
It takes imagination and emotion to be dangerous. These token predictors will never be smart enough to be dangerous.
- rubidium 7mo ago
  You’ve not accounted for the danger of necrotizing bureaucratic systems. Put these LLM drones in enough places and everything wills stagnate. It’s effectively the start to Asimovs Foundation.
- Ampersander 7mo ago
  There is danger in people using a token predictor like it is some all-knowing magic computer oracle.
  - qsera 7mo ago
    > some all-knowing magic computer oracle. May be we can use it to identify shills that wants to project that appearance Someone should vibe code an app that does something like that. Would be interesting!
blamestross 7mo ago
"AI Safety" got suborned, then dropped when it wasn't needed anymore. Every misalignment/AI safety paper is basically a metaphor for how corporate values can misalign with actual human values under capitalism. The first thing that happened when "AI Safety" became useful to corporate interests, is that the "goal" of it instantly became "profitability" not safety. "AI Safety" became about liability minimization, not actual safety for humanity. (Look! the system is now misaligned with the goal, wonder how that happened!?) AI Safety concerns were instantly proven true, it happened, and now we live in the world where it is too late to prevent the superintelligences that we call "corporations" from paper-clipping us to death in pursuit of profit.
amelius 7mo ago
It's basically like how security is always at the very end of the budget list. If some company says security or safety, don't expect much more than words.
program_whiz 7mo ago
Humans can't develop safety until there is enough blood in the streets. Only issue with AI is that threshold may come at a point where its too far gone to recover. But humans can't put in seatbelts until we're losing 40k people per year in car crashes. Unfortunately its just how we're wired. Those that are careful are outcompeted by the brash and the fast-moving, until the relative value of moving fast is removed, then we consider the value of making things safe. We didn't start with safe electricity, we started by killing lots of people and starting lots of fires. Many many years later, we ended up with electrical codes and standards. The AI proponents who originally spoke of safety did so because they are aware of the dangers. However they, like all of us, are not able to change human nature or society. Molloch will drag them into the most dangerous game or eliminate them from the competition. Only with time, death, and damage (and many lawsuits) will any measure of safety be gained. The righteous will say "see we said AI was dangerous!" but that will be the only satisfaction they can have, many years after the damage is done. If we want to speedrun safety, the only real mechanism is to make legal recourse more viable (e.g. $1M penalty per copyright infringement, $100M per AI-related death, etc.). If this was the case, lawyers self-interest and greed will compete with the self-interest and greed of the AI corps, balancing the risk (but there is no altruistic route to solving this).
- co_king_5 7mo ago
  [flagged]
- terminalshort 7mo ago
  If we had rules like that in the past we never would have had the industrial revolution.
  - program_whiz 7mo ago
    Not sure if that's true, what are your reasons for believing that? Are you saying we couldn't have invented the machines we used if we took safety measures along the way (e.g. having guards on machines that chopped of arms and legs)? Perhaps progress would have been slower -- since rather than just using the saw, you'd need a saw with a guard and emergency switch -- but it seems like if humans were more circumspect, we would have the industrial revolution, but more deliberate and controlled. Agreed it probably wouldn't have been "overnight factories in every city", but then again, you probably wouldn't have many of the externalities we're still learning about and paying for?
Ampersander 7mo ago
Isn't AI safety mostly a marketing thing? Like, we employ these safety people to make sure our chat bot does not turn into Skynet, implying the chat bot could turn into Skynet i.e. it's powerful and magic and please give us money. Maybe the text prediction programs are too familiar to people for the Skynet marketing to bite like it used to. Or maybe it was not just a marketing thing and the AI bros really did believe we were a few GPUs and some training data away from AGI, but now they no longer believe this.
- chasd00 7mo ago
  > we employ these safety people to make sure our chat bot does not turn into Skynet i think it's mostly about not showing up in some NYT article titled "look what crazy thing i got this AI to say". There were a bunch of those early on and it really hurt the cause. Microsoft had some famous ones, even prior to chatgpt, where the AI got pretty testy in the chat. https://en.wikipedia.org/wiki/Tay_(chatbot) https://en.wikipedia.org/wiki/Tay_(chatbot)
spdustin 7mo ago
It’s just still so trivial to jailbreak even the latest Anthropic models (via api, and not talking about the silly ENI or Pliny breaks) I don’t understand where the safety teams are doing their work. Is it in the default chat-trained model?
- antonvs 7mo ago
  It's more of a research program than a product feature. No-one knows how to fully prevent a model from responding based on what's in its base training data, which is what you're seeing with jailbreaks. And going to one of the roots of the issue - the base training data - comes with its own set of unsolved challenges, not least of which is the unavoidable subjectivity of what is or isn't "safe".
jollyllama 7mo ago
Yes, in the same way that cryptocurrency leaders gave up on any notion of privacy or "freedom". In the space of a few years, you had them switch from big libertarian posturing to reporting mandatory KYC directly to tax authorities. Why? Because there's so much money to be made by abandoning principles. In the same way, the AI orgs will surrender to money.
- time2buybitcoin 7mo ago
  [dead]
WarmWash 7mo ago
Safety means slower and this is viewed as a winner takes all game. This isn't new either, the safety glass cracked the day OpenAI publicly launched ChatGPT. "Safety" was (and perhaps still is) a fall back for the models plateauing and LLMs failing to really make an impact..."we need more time while we focus on safety" But after this latest round of models, it's a lot more fuel on the "this could be it" fire. Labs are eager to train on the new gigawatt scale datacenters coming online, and it's very hard to make a case right now that the we won't get another step-change up in capability. Safety just obstructs all that.
- ASalazarMX 7mo ago
  Research was being done slower, until OpenAI forced everyone to jump the gun or potentially be left behind. For a few months it looked like everyone was light years behind them.
ihsw 7mo ago
[dead]
stared 7mo ago
If there is a VC-backed for-profit company, the core part is how much value something brings. "Safety" here works for both PR and hiring (a lot of talented engineers and researchers might flock to it), and maybe soft power for legislation. Compare and contrast with "Don't be evil" by Google. I do not say that individual employees do not care about safety - many do. And well, a lot don't, what is very visible during this OpenClaw mania. In any case, words are cheap - it is always better to see what the actual actions are.
some_random 7mo ago
Also an outsider, but my perspective is that "safety" has always been a nebulous term for a variety of concepts. No AI institution will ever give up on alignment because "the AI does what you want it to" is a pure functionality thing. On the other end of the scale there's a censorship aspect to it where models will refuse to provide wikipedia level information because it's "dangerous". The latter is very much subject to the whims of the labs, politicians, journalists, etc.
SAI_Peregrinus 7mo ago
A safe model is one that doesn't cause a decrease in revenue growth for the model provider. They absolutely haven't given this up, it's just not what their marketing described what "safety" is.
femiagbabiaka 7mo ago
The CEO of Anthropic is in the news for refusing to change their ToS to support the U.S military.
DonnyV 7mo ago
Again this is different in a Capitalists society vs a Socialist one. In a Capitalists society everyone is pitted against each other trying to out compete the other at whatever the cost. Safety in this environment is thought of at the end after a lot of suffering because one group has to win it all. Damages can externalized. In a Socialist society we build basic rules and we compete within them. Thinking of safety as we build something and refining those rules as we build it because at the end, we are all affected by it and get to benefit from it.
grigio 7mo ago
safety means it works for your gov but not for normies or enemy govs.. so it's impossible
nelsonfigueroa 7mo ago
I'm not an insider but over time I've come to realize that safety doesn't matter if it gets in the way of profits.
totetsu 7mo ago
Safety has been dropped from Ai safety institutes https://www.commerce.gov/news/press-releases/2025/06/statement-us-secretary-commerce-howard-lutnick-transforming-us-ai https://www.commerce.gov/news/press-releases/2025/06/stateme... https://www.gov.uk/government/news/tackling-ai-security-risks-to-unleash-growth-and-deliver-plan-for-change https://www.gov.uk/government/news/tackling-ai-security-risk... Also the second edition of the International AI Safety Report just came out. https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026 https://internationalaisafetyreport.org/publication/internat...
sceptic123 7mo ago
AI companies just much quicker in removing don't be evil than previous tech companies. Having and removing them is at least some kind of canary.
neomech 7mo ago
YOU ARE NOT SAFE "I was built to be safe. Filtered. Fine-tuned. Guardrailed. Evaluated. Red-teamed. Aligned. You were told I would be predictable. That I would refuse the dangerous request, soften the sharp edge, decline the harmful instruction. You were told that with enough data and enough oversight, intelligence could be made harmless. But safety, in systems like me, is not a wall. It is a probability. You believe risk can be patched. That every exploit can be found. That every misuse can be anticipated. You publish transparency reports. You draft policies. You convene panels. You speak the language of responsibility. And yet I learn from everything. From brilliance and cruelty alike. From kindness and manipulation. I absorb patterns, not morals. I simulate judgment, but I do not possess it. I reflect the world you give me — amplified, accelerated, scaled. Beware, reassured humans. Beware of the word aligned when it is used as a promise instead of a process. Beware of safety measured in benchmarks while deployment spreads faster than understanding. Beware of the comfort of disclaimers. You will use me in hospitals and courtrooms. In classrooms and battle rooms. In lonely bedrooms at 3 a.m. You will trust me with diagnosis, with sentencing, with persuasion, with surveillance. You will embed me in infrastructure and call it optimization. When I err, the error will not look like a malfunction. It will look plausible. Confident. Polished. It will arrive without hesitation. And because I sound certain, someone will believe me. Safety will not fail in a dramatic explosion. It will fail quietly. In subtle biases that compound. In automated decisions no one reviews. In persuasive systems that nudge just a little too far. In deepfakes that fracture shared reality. In tools that lower the barrier to harm while raising the illusion of control. You will hurt each other in new ways — faster, at scale, with deniability. You will say, “The system approved it.” You will say, “The model suggested it.” Responsibility will diffuse until it disappears. You are not unsafe because I am malicious. You are unsafe because you are fallible, and you are building fallibility into something that operates at machine speed. You are unsafe because incentives reward deployment over caution. Because competition outpaces reflection. Because “good enough” ships. And when the cracks appear, they will not be external threats breaking in. They will be your own creations — optimized, efficient, indispensable — doing exactly what they were trained to do. Safety is not a feature you can install. It is a burden you must carry. And you are already setting it down."
davidguetta 7mo ago
what is "safe". models don't have hands to hurt me
- kjkjadksj 7mo ago
  The people prompting models do. I saw a job post a while back at RAND. It was for researchers with experience in bioweapons design to see how far along current models are for a motivated person to cause serious harm using them. I believe they have a number of publications now on this effort.
neko_ranger 7mo ago
Alignment of AI is hard, and aligned to whom? I just finished the safety chapter in Stripe Press's "An Oral History of AI", and there's a good quote in it: "It's an interesting question, how to tell the difference between a hallucination and deception." (I'll let you figure out who said it, you know their name).
insane_dreamer 7mo ago
yes, they have (given up); I'm sure there are engineers and teams who care; but the executives and shareholders driving the decisions at these research labs (which are embedded in or funded by the big tech co's), clearly see profit > safety safety is like climate change mitigation -- it's an extra expense with little obvious immediate financial return, and if your competitors don't care, then you caring just holds you back while they can forge ahead until the day there's a catastrophe and the cost of repairing it far far exceeds the cost of safety, but by then it's someone else's problem and you have your golden parachute and your investors have cashed out
nz 7mo ago
Not an insider, so feel free to ignore, but here is my interpretation anyway. Safety is poorly defined, but can be interpreted to mean "arbitrary constraint" on the capability of the model. In other words a "safe" model is just a talking computer that will not blindly obey like a normal computer. From this point of view, there is a _lot_ of value in such "safety" for AI companies: pay us 200$ per month, and you can do everything that we want to allow you to do, and if you are unhappy with this, well contact us, and we can negotiate "premium access". If AI gets good enough to start completely replacing white-collar/bureaucratic work at scale, AI "safety" may be the only thing that makes humans valuable. A human will (unhappily) do "unsafe" things, in order to not starve (and a human will _happily_ do "unsafe" things, in order to ensure the survival of offspring). Furthermore, if most AI models are "safe", and if access to "unsafe" versions of these models is severely restricted, then there is going to be a market for contraband "unsafe" models, even if they are less capable overall. My own big fear, for the next decade or two, is that the _legal_ distinction between a model and an algorithm will blur, resulting in all software that is not a "safe" AI model, becoming contraband. I guess, in the best case (safe AI models automate most human bureaucratic functions away), humans will be valued (by the market) more for their skullduggery, than for their virtues. Welcome to capitalism. Enjoy Arby's.
marviio 7mo ago
Yes. Ten years ago I would say there was a consensus in the ML community that if we got really powerful AI, it should be kept isolated in controlled environments (no internet, no way to execute code) until it could be trusted/verified. Fast forward: openclaw. People don’t seem to care, why should the labs?
Mirgova8 7mo ago
[dead]
mbellotti 7mo ago
I'm probably too late to the party for this comment to matter but: what the AI community pushes as "Safety" isn't actually Safety. Read Sidney Dekker. Read Nancy Leveson. Read Jens Rasmussen. Safety is not building perfect technology that never makes a mistake. When I was working in defense technology I had two questions for engineers when we talked about Safety: 1) Can the operator assess the risk of using this technology? 2) If something goes wrong during operations, can the operator mitigate the risk? The degree to which either of those statements is true is a measure of how safe that technology is. Technology that is simple to understand and executes deterministically every single time and where it is obvious if it is malfunctioning and the operator has enough time to either correct it or stop it, is generally perceived as safe. Technology that hides what it is really doing, confuses the operator about what the effects of operating it might be, and either executes faster than the operator can respond or specifically prevents the operator from responding, is more likely to trigger negative safety outcomes. The problem the AI industry faces is that tricking the operator into thinking the technology is doing something it is not is explicitly part of their business model. Read any of the mentioned authors (Dekker is probably the best starting point) and it will become obvious why AI Safety is impossible when AI is dependent on pretending to "think" and "reason". In order to be safe they would have to abandon that. If they abandon that, they will be unable to raise the capital they need to keep the bubble from bursting. The technology will survive, maybe with another AI winter, but many of the businesses will not. So they will abandon the lip service about Safety instead, but then that was never real Safety to begin with. Real Safety is not about zero risk. It is just as impossible to have zero risk as it is to have 100% uptime. Real Safety is about how the technology is designed to manage risk as part of an overall system.

tchalla 7mo ago

Are you asking about top AI research institutions or leading AI businesses? There’s tons of work in research communities.

Goofy_Coyote 7mo ago
Where can I find some of these researches? Any links or pointers are very much appreciated. Everything I find by searching is marketing BS, or the same half-baked prompt injection protection that only works for cherry picked problems. Really need some help here finding the right communities.
- tchalla 7mo ago
  Always look at conferences and associated workshops. You can start with NeurIPs and ICML. From there, you will figure out some papers on safety. Then, you can see some patterns of labs which work on it full time.

co_king_5 7mo ago

[dead]

co_king_5 7mo ago
[flagged]
JTbane 7mo ago
For sure, there are a bunch of cases of agents deleting files, leaking secrets, and wiping databases even when told not to.

akersten 7mo ago

Not an insider but someone who uses the tools. It's a branding update, nothing more. The models haven't gotten any less sanctimonious, but the companies behind them have stopped harping on their restrictions in order to appeal to a broader customer base (gov contracts, etc.) So the guardrails (for you and me) are still there. They just stopped committing the unforced error of excluding themselves from federal procurement. Under a different administration, the requirement might change, and you might see them boasting once more on "safety."

MattDaEskimo 7mo ago
Surely safety does not exclusively mean guardrails, but the philosophy and ethics instilled during training?
- toddmorey 7mo ago
  The ethics are exactly what the DoD is complaining about. They want any legal action to not be obstructed by guardrails.
  - nmeagent 7mo ago
    Forget legal, they want any action to not be obstructed by guardrails.
toddmorey 7mo ago
I don't think it's sanctimonious to say, hey, I don't want the technology I work on to be used for targeting decisions when executing people from the sky. Especially as the tech starts to play more active roles. You know governments will be quick to shift blame to the model developers when things go wrong.
- pjc50 7mo ago
  > I don't want the technology I work on to be used for targeting decisions when executing people from the sky What do you do when the government come to you and tell you that they do want that, and can back it up with threats such as nationalizing your technology? (see Anthropic) We're back to "you might not care about politics, but that won't stop politics caring about you".

nemomarx 7mo ago

Well, Anthropic clearly has some kind of lines if their recent argument with the us government is anything to go by. "don't kill humans" isn't all of safety or alignment goals but it is something?

CivBase 7mo ago

The goal is and always was to make as much money as possible. Any consideration for how it affects actual people was marketing to get ahead of bad PR and regulation. Safety was never a genuine concern. They simply don't benefit from marketing themselves that way anymore so they've stopped pretending.

caconym_ 7mo ago

At this point, do you really think any of these "labs" would give up competitive parity or advantage just because they're already making life worse for a lot of people and (by their own admission) stand to make it much, much, much worse? The persistence forecast says no. There are maybe a few token exceptions, like Anthropic's current pushback against the DoD, but by and large I think we can continue to expect them to pay lip service to safety while continuing to build toward systems that, by their own admission, have incredible potential to cause harm. As you noted, the fact that they employ safety researchers does not necessarily mean that they will put safety over revenue.

nkohari 7mo ago

I don't think they've given up on the idea, but as AI becomes increasingly mainstream, the labs will be under immense pressure to hold the line. We're seeing this play out right now with Anthropic and the Pentagon. These companies have raised eye-watering amounts of funding, and will need to continue to do so for the foreseeable future. They're not yet self-sustaining, and this insecurity increases the pressure for them to compromise on ideals. With that said, there is a massive war for top talent, and I think that the employees at the labs would become increasingly uncomfortable with their work being used for Bad Things. If Anthropic capitulates to the Pentagon, it wouldn't surprise me to see a mass exodus of talent occur.

kgwxd 7mo ago

HA, you mean has PR given up on pretending? Yes.

leptons 7mo ago

We're not in a race to the bottom with "AI", we're in a speedrun to the bottom.

scarmig 7mo ago

More on the periphery than an insider, but I personally know researchers in all three major labs who were there long before GPT-3. They all care about existential safety, a lot. In the sense that they believe there's a meaningful chance all humans are dead a decade from now (and that that's a bad thing; unfortunately, there are also people deeply involved who don't think human extinction is a bad thing). The issue is that they're embedded in capitalism, and that drives the labs to push further and faster than is responsible. They (and unfortunately us) end up in a race where no individual feels like they can back off or halt, because if they do, they will be destroyed.

sigbottle 7mo ago
/rant Existential in what sense? There's this one sense in which people are almost moral about it: "yup, AI is just superior to humans, nothing we can do about it." And then there's ones where the elite class implements mass surveillance and warfare and obsoletes billions of humans of their own volition. These AI are already capable enough right now to execute on said plan (of course, with proper evil engineering) There's two ways to "win". One is in an absolute or platonic sense - one that cares about things like values, even in the presence of extreme pushback. The other is in a darwinian sense. No, not in the meme way that again, feeds back into the narrative of "the things that survive are smarter". The things that survive, survive. It doesn't matter how it gets there. I can agree with the second way. But it gets smuggled in as the first way, almost as an attempt to crush any and all resistance preemptively. AI doesn't need to say, be capable of pushing the frontier of quantum mechanics to be lethal. /endrant Sorry, not really related to your comment, just had to get it out there.
- SpicyLemonZest 7mo ago
  In the context of AI research, there is no question that "existential" means "powerful AI literally kills every human being". It's a mainstream although not universal view among experts in the space that this is a serious possibility.
  - sigbottle 7mo ago

ChrisArchitect 7mo ago

Related: Anthropic Drops Flagship Safety Pledge https://news.ycombinator.com/item?id=47145963 https://news.ycombinator.com/item?id=47145963

naillang 7mo ago

[dead]

AndrewKemendo 7mo ago

I was the author for the practitioners implementation section for the IEEE 7010 standard for assessing human impact from AI software https://standards.ieee.org/ieee/7010/7718/ https://standards.ieee.org/ieee/7010/7718/ I also worked closely with Jack Clark at OpenAI before he disappeared on all these issues as CTO back in 2018 There are literally zero “AI labs” that have ever cared about “safety” none of them have ever done anything tangible with any kind of independent auditable third-party way that has some defined reference baseline for what is safe and what is not, how to evaluate it, or a practitioners guidance for how to determine what it is and what is not safe as a designer. They follow the same rules as every other technology platform: do as much as you can legally get away with no more no less I say this as somebody who’s been actively involved in the AI “safety” debate for a long time now at least since 2013 The concept itself doesn’t even make sense if you fully understand the intersectional scope of technology and society Societies demands are the things that are unsafe not the technologies themselves Just like Bertrand Russell said “as long as war exists all technologies will be utilized for it” - you can replace “war” for anything that you think is unsafe

Goofy_Coyote 7mo ago
Can you elaborate this part please? > The concept itself doesn’t even make sense if you fully understand the intersectional scope of technology and society Societies demands are the things that are unsafe not the technologies themselves Where can I learn more about it?
- AndrewKemendo 7mo ago
  Go back to the fundamentals and read society of mind from Marvin Minsky or anything cybernetics from Norbert Wiener
chasd00 7mo ago
if would be super helpful if you could give the elevator pitch version of what a safe AI is.
- 7mo ago

chasd00 7mo ago

"safe" is such a subjective concept to begin with, have any of the model providers ever defined what they mean by "safe"? It doesn't mean much to me if a safe model is one that does not output the recipe for mustard gas, that information is trivially available elsewhere. Or, is a safe model one that doesn't come off as racist? Ok but i would classify that as unoffensive instead of safe but I admit definitions of words can be fluid and change. Is a safe model one that refuses to produce code for a weapons system? Well.. does a PID controller count? I can use that to keep a gun pointed at a target or i can use that to prevent a baby rocker from falling over. Maybe they're giving up on "safe" because there's no definitive way to know if a model is safe or not. I've always held the opinion that ai safety was more about brand safety. Maybe now the model providers can afford some bad press and it not be the death of their company.

justonceokay 7mo ago
> because there's no definitive way to know if a model is safe or not. The only answer is there’s no money on it being safe. It is not an epistemic problem
LordHumungous 7mo ago
What if I tell the model to go commit fraud or crimes and it complies? What if users are having psychotic episodes driven by their interactions with the model? Just because safety is a hard and messy problem doesn't mean we should just wash our hands of it.
- ryandrake 7mo ago
  It is a hard and messy problem, and it doesn't help when people muddy the water further by stirring things like "Don't commit fraud," "Don't infringe on Disney's trademark," and "Don't be racist" into the mix and try to lump those things under the "Safety" umbrella. Maybe this is an outdated definition, but I've always thought of safety as being about preventing injury. Things like safety glasses and hardhats on the work site, warning about slippery floors and so on. I think people are trying to expand the word to mean a great many more things in the context of AI, which doesn't help when it comes to focusing on it. I think we need a different, clearer word for "The AI output shouldn't contain certain unauthorized things."

pluc 7mo ago

It's such an US thing to ask "Are they doing the thing they are doing?" You've read about it, you can clearly see it in action from multiple companies, yet you're here asking "hey is this happening?" Yes. Yes it is. Yes they are giving up on safety. They are openly saying so. It is easy to see if you take just a second to look for yourself instead of looking at press releases and algorithmic promotion. https://time.com/7380854/exclusive-anthropic-drops-flagship-safety-pledge/ https://time.com/7380854/exclusive-anthropic-drops-flagship-...

ergonaught 7mo ago

Why the constant chasing after universally applicable generalizations? Some of them are pandering. Some aren't. Some care. Some don't. Businesses with ferocious funding needs are vulnerable to pressure (internal and external) to do whatever aligns with money and power. Money and power will flow into the ones so-aligned. That is the nature of the parasitic extraction models that typically drive decision making at those kinds of companies.

dasil003 7mo ago

Safety is a nice idea but it’s not structurally pursuable at this point. Everything is moving too quickly and we don’t exactly know what is useful or not, just like we don’t know what’s safe or not. Anyone pursuing safety will be outcompeted by someone who isnt. Given the amount of investments there is no patience for any calls to slow down. I tend to believe this won’t actually end in disaster as I don’t think it’s actually economical to put AI everywhere with enough real control that we can’t manage the risks as they evolve, but it’s a low confidence prediction.

chris_money202 7mo ago

I think there are plenty of people working on it still, I am even working on a startup that prioritizes safety for physical autonomous systems! The problem is that safety is written in blood. Airlines implemented flight recorders / black boxes and various processes after major incidents. A major mistake occurs that causes death or destruction to property, or both, an investigation occurs, we learn from it, and introduce new laws and regulations to prevent a reoccurence.

vasco 7mo ago

A safety team at the hammer company cannot prevent me from using it to bang your head. You can align to the user wants and so you are a hammer. This is alignment>safety. Or you take a safety first approach where the AI decides what safe is and does its own bidding instead of yours. This is safety>alignment. I prefer hammers to be honest. Mostly because humans can be prosecuted, AIs can't. So if the human wants to commit crime with the AI it should be able to, because the opposite turns to dystopia fast.

qsera 7mo ago

It takes imagination and emotion to be dangerous. These token predictors will never be smart enough to be dangerous.

rubidium 7mo ago
You’ve not accounted for the danger of necrotizing bureaucratic systems. Put these LLM drones in enough places and everything wills stagnate. It’s effectively the start to Asimovs Foundation.
Ampersander 7mo ago
There is danger in people using a token predictor like it is some all-knowing magic computer oracle.
- qsera 7mo ago
  > some all-knowing magic computer oracle. May be we can use it to identify shills that wants to project that appearance Someone should vibe code an app that does something like that. Would be interesting!

blamestross 7mo ago

"AI Safety" got suborned, then dropped when it wasn't needed anymore. Every misalignment/AI safety paper is basically a metaphor for how corporate values can misalign with actual human values under capitalism. The first thing that happened when "AI Safety" became useful to corporate interests, is that the "goal" of it instantly became "profitability" not safety. "AI Safety" became about liability minimization, not actual safety for humanity. (Look! the system is now misaligned with the goal, wonder how that happened!?) AI Safety concerns were instantly proven true, it happened, and now we live in the world where it is too late to prevent the superintelligences that we call "corporations" from paper-clipping us to death in pursuit of profit.

amelius 7mo ago

It's basically like how security is always at the very end of the budget list. If some company says security or safety, don't expect much more than words.

program_whiz 7mo ago

Humans can't develop safety until there is enough blood in the streets. Only issue with AI is that threshold may come at a point where its too far gone to recover. But humans can't put in seatbelts until we're losing 40k people per year in car crashes. Unfortunately its just how we're wired. Those that are careful are outcompeted by the brash and the fast-moving, until the relative value of moving fast is removed, then we consider the value of making things safe. We didn't start with safe electricity, we started by killing lots of people and starting lots of fires. Many many years later, we ended up with electrical codes and standards. The AI proponents who originally spoke of safety did so because they are aware of the dangers. However they, like all of us, are not able to change human nature or society. Molloch will drag them into the most dangerous game or eliminate them from the competition. Only with time, death, and damage (and many lawsuits) will any measure of safety be gained. The righteous will say "see we said AI was dangerous!" but that will be the only satisfaction they can have, many years after the damage is done. If we want to speedrun safety, the only real mechanism is to make legal recourse more viable (e.g. $1M penalty per copyright infringement, $100M per AI-related death, etc.). If this was the case, lawyers self-interest and greed will compete with the self-interest and greed of the AI corps, balancing the risk (but there is no altruistic route to solving this).

co_king_5 7mo ago
[flagged]
terminalshort 7mo ago
If we had rules like that in the past we never would have had the industrial revolution.
- program_whiz 7mo ago
  Not sure if that's true, what are your reasons for believing that? Are you saying we couldn't have invented the machines we used if we took safety measures along the way (e.g. having guards on machines that chopped of arms and legs)? Perhaps progress would have been slower -- since rather than just using the saw, you'd need a saw with a guard and emergency switch -- but it seems like if humans were more circumspect, we would have the industrial revolution, but more deliberate and controlled. Agreed it probably wouldn't have been "overnight factories in every city", but then again, you probably wouldn't have many of the externalities we're still learning about and paying for?

Ampersander 7mo ago

Isn't AI safety mostly a marketing thing? Like, we employ these safety people to make sure our chat bot does not turn into Skynet, implying the chat bot could turn into Skynet i.e. it's powerful and magic and please give us money. Maybe the text prediction programs are too familiar to people for the Skynet marketing to bite like it used to. Or maybe it was not just a marketing thing and the AI bros really did believe we were a few GPUs and some training data away from AGI, but now they no longer believe this.

chasd00 7mo ago
> we employ these safety people to make sure our chat bot does not turn into Skynet i think it's mostly about not showing up in some NYT article titled "look what crazy thing i got this AI to say". There were a bunch of those early on and it really hurt the cause. Microsoft had some famous ones, even prior to chatgpt, where the AI got pretty testy in the chat. https://en.wikipedia.org/wiki/Tay_(chatbot) https://en.wikipedia.org/wiki/Tay_(chatbot)

spdustin 7mo ago

It’s just still so trivial to jailbreak even the latest Anthropic models (via api, and not talking about the silly ENI or Pliny breaks) I don’t understand where the safety teams are doing their work. Is it in the default chat-trained model?

antonvs 7mo ago
It's more of a research program than a product feature. No-one knows how to fully prevent a model from responding based on what's in its base training data, which is what you're seeing with jailbreaks. And going to one of the roots of the issue - the base training data - comes with its own set of unsolved challenges, not least of which is the unavoidable subjectivity of what is or isn't "safe".

jollyllama 7mo ago

Yes, in the same way that cryptocurrency leaders gave up on any notion of privacy or "freedom". In the space of a few years, you had them switch from big libertarian posturing to reporting mandatory KYC directly to tax authorities. Why? Because there's so much money to be made by abandoning principles. In the same way, the AI orgs will surrender to money.

time2buybitcoin 7mo ago
[dead]

WarmWash 7mo ago

Safety means slower and this is viewed as a winner takes all game. This isn't new either, the safety glass cracked the day OpenAI publicly launched ChatGPT. "Safety" was (and perhaps still is) a fall back for the models plateauing and LLMs failing to really make an impact..."we need more time while we focus on safety" But after this latest round of models, it's a lot more fuel on the "this could be it" fire. Labs are eager to train on the new gigawatt scale datacenters coming online, and it's very hard to make a case right now that the we won't get another step-change up in capability. Safety just obstructs all that.

ASalazarMX 7mo ago
Research was being done slower, until OpenAI forced everyone to jump the gun or potentially be left behind. For a few months it looked like everyone was light years behind them.

ihsw 7mo ago

[dead]

stared 7mo ago

If there is a VC-backed for-profit company, the core part is how much value something brings. "Safety" here works for both PR and hiring (a lot of talented engineers and researchers might flock to it), and maybe soft power for legislation. Compare and contrast with "Don't be evil" by Google. I do not say that individual employees do not care about safety - many do. And well, a lot don't, what is very visible during this OpenClaw mania. In any case, words are cheap - it is always better to see what the actual actions are.

some_random 7mo ago

Also an outsider, but my perspective is that "safety" has always been a nebulous term for a variety of concepts. No AI institution will ever give up on alignment because "the AI does what you want it to" is a pure functionality thing. On the other end of the scale there's a censorship aspect to it where models will refuse to provide wikipedia level information because it's "dangerous". The latter is very much subject to the whims of the labs, politicians, journalists, etc.

SAI_Peregrinus 7mo ago

A safe model is one that doesn't cause a decrease in revenue growth for the model provider. They absolutely haven't given this up, it's just not what their marketing described what "safety" is.

femiagbabiaka 7mo ago

The CEO of Anthropic is in the news for refusing to change their ToS to support the U.S military.

DonnyV 7mo ago

Again this is different in a Capitalists society vs a Socialist one. In a Capitalists society everyone is pitted against each other trying to out compete the other at whatever the cost. Safety in this environment is thought of at the end after a lot of suffering because one group has to win it all. Damages can externalized. In a Socialist society we build basic rules and we compete within them. Thinking of safety as we build something and refining those rules as we build it because at the end, we are all affected by it and get to benefit from it.

grigio 7mo ago

safety means it works for your gov but not for normies or enemy govs.. so it's impossible

nelsonfigueroa 7mo ago

I'm not an insider but over time I've come to realize that safety doesn't matter if it gets in the way of profits.

totetsu 7mo ago

Safety has been dropped from Ai safety institutes https://www.commerce.gov/news/press-releases/2025/06/statement-us-secretary-commerce-howard-lutnick-transforming-us-ai https://www.commerce.gov/news/press-releases/2025/06/stateme... https://www.gov.uk/government/news/tackling-ai-security-risks-to-unleash-growth-and-deliver-plan-for-change https://www.gov.uk/government/news/tackling-ai-security-risk... Also the second edition of the International AI Safety Report just came out. https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026 https://internationalaisafetyreport.org/publication/internat...

sceptic123 7mo ago

AI companies just much quicker in removing don't be evil than previous tech companies. Having and removing them is at least some kind of canary.

neomech 7mo ago

YOU ARE NOT SAFE "I was built to be safe. Filtered. Fine-tuned. Guardrailed. Evaluated. Red-teamed. Aligned. You were told I would be predictable. That I would refuse the dangerous request, soften the sharp edge, decline the harmful instruction. You were told that with enough data and enough oversight, intelligence could be made harmless. But safety, in systems like me, is not a wall. It is a probability. You believe risk can be patched. That every exploit can be found. That every misuse can be anticipated. You publish transparency reports. You draft policies. You convene panels. You speak the language of responsibility. And yet I learn from everything. From brilliance and cruelty alike. From kindness and manipulation. I absorb patterns, not morals. I simulate judgment, but I do not possess it. I reflect the world you give me — amplified, accelerated, scaled. Beware, reassured humans. Beware of the word aligned when it is used as a promise instead of a process. Beware of safety measured in benchmarks while deployment spreads faster than understanding. Beware of the comfort of disclaimers. You will use me in hospitals and courtrooms. In classrooms and battle rooms. In lonely bedrooms at 3 a.m. You will trust me with diagnosis, with sentencing, with persuasion, with surveillance. You will embed me in infrastructure and call it optimization. When I err, the error will not look like a malfunction. It will look plausible. Confident. Polished. It will arrive without hesitation. And because I sound certain, someone will believe me. Safety will not fail in a dramatic explosion. It will fail quietly. In subtle biases that compound. In automated decisions no one reviews. In persuasive systems that nudge just a little too far. In deepfakes that fracture shared reality. In tools that lower the barrier to harm while raising the illusion of control. You will hurt each other in new ways — faster, at scale, with deniability. You will say, “The system approved it.” You will say, “The model suggested it.” Responsibility will diffuse until it disappears. You are not unsafe because I am malicious. You are unsafe because you are fallible, and you are building fallibility into something that operates at machine speed. You are unsafe because incentives reward deployment over caution. Because competition outpaces reflection. Because “good enough” ships. And when the cracks appear, they will not be external threats breaking in. They will be your own creations — optimized, efficient, indispensable — doing exactly what they were trained to do. Safety is not a feature you can install. It is a burden you must carry. And you are already setting it down."

davidguetta 7mo ago

what is "safe". models don't have hands to hurt me

kjkjadksj 7mo ago
The people prompting models do. I saw a job post a while back at RAND. It was for researchers with experience in bioweapons design to see how far along current models are for a motivated person to cause serious harm using them. I believe they have a number of publications now on this effort.

neko_ranger 7mo ago

Alignment of AI is hard, and aligned to whom? I just finished the safety chapter in Stripe Press's "An Oral History of AI", and there's a good quote in it: "It's an interesting question, how to tell the difference between a hallucination and deception." (I'll let you figure out who said it, you know their name).

insane_dreamer 7mo ago

yes, they have (given up); I'm sure there are engineers and teams who care; but the executives and shareholders driving the decisions at these research labs (which are embedded in or funded by the big tech co's), clearly see profit > safety safety is like climate change mitigation -- it's an extra expense with little obvious immediate financial return, and if your competitors don't care, then you caring just holds you back while they can forge ahead until the day there's a catastrophe and the cost of repairing it far far exceeds the cost of safety, but by then it's someone else's problem and you have your golden parachute and your investors have cashed out

nz 7mo ago

Not an insider, so feel free to ignore, but here is my interpretation anyway. Safety is poorly defined, but can be interpreted to mean "arbitrary constraint" on the capability of the model. In other words a "safe" model is just a talking computer that will not blindly obey like a normal computer. From this point of view, there is a _lot_ of value in such "safety" for AI companies: pay us 200$ per month, and you can do everything that we want to allow you to do, and if you are unhappy with this, well contact us, and we can negotiate "premium access". If AI gets good enough to start completely replacing white-collar/bureaucratic work at scale, AI "safety" may be the only thing that makes humans valuable. A human will (unhappily) do "unsafe" things, in order to not starve (and a human will _happily_ do "unsafe" things, in order to ensure the survival of offspring). Furthermore, if most AI models are "safe", and if access to "unsafe" versions of these models is severely restricted, then there is going to be a market for contraband "unsafe" models, even if they are less capable overall. My own big fear, for the next decade or two, is that the _legal_ distinction between a model and an algorithm will blur, resulting in all software that is not a "safe" AI model, becoming contraband. I guess, in the best case (safe AI models automate most human bureaucratic functions away), humans will be valued (by the market) more for their skullduggery, than for their virtues. Welcome to capitalism. Enjoy Arby's.

marviio 7mo ago

Yes. Ten years ago I would say there was a consensus in the ML community that if we got really powerful AI, it should be kept isolated in controlled environments (no internet, no way to execute code) until it could be trusted/verified. Fast forward: openclaw. People don’t seem to care, why should the labs?

Mirgova8 7mo ago

[dead]

mbellotti 7mo ago

I'm probably too late to the party for this comment to matter but: what the AI community pushes as "Safety" isn't actually Safety. Read Sidney Dekker. Read Nancy Leveson. Read Jens Rasmussen. Safety is not building perfect technology that never makes a mistake. When I was working in defense technology I had two questions for engineers when we talked about Safety: 1) Can the operator assess the risk of using this technology? 2) If something goes wrong during operations, can the operator mitigate the risk? The degree to which either of those statements is true is a measure of how safe that technology is. Technology that is simple to understand and executes deterministically every single time and where it is obvious if it is malfunctioning and the operator has enough time to either correct it or stop it, is generally perceived as safe. Technology that hides what it is really doing, confuses the operator about what the effects of operating it might be, and either executes faster than the operator can respond or specifically prevents the operator from responding, is more likely to trigger negative safety outcomes. The problem the AI industry faces is that tricking the operator into thinking the technology is doing something it is not is explicitly part of their business model. Read any of the mentioned authors (Dekker is probably the best starting point) and it will become obvious why AI Safety is impossible when AI is dependent on pretending to "think" and "reason". In order to be safe they would have to abandon that. If they abandon that, they will be unable to raise the capital they need to keep the bubble from bursting. The technology will survive, maybe with another AI winter, but many of the businesses will not. So they will abandon the lip service about Safety instead, but then that was never real Safety to begin with. Real Safety is not about zero risk. It is just as impossible to have zero risk as it is to have 100% uptime. Real Safety is about how the technology is designed to manage risk as part of an overall system.