3 ms·

I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims. After the events of the summer it feels like it takes a la

I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims. After the events of the summer it feels like it takes a lack of imagination to not see a few plausible routes to disaster. It may be reasonable to believe these outcomes are not very likely or that we can stop before going too far (I tend to disagree). But I can't imagine doubting that the capabilities will soon be there to realize some of those paths.

overtone1000 25d ago
I can't help but think the most plausible scenarios are the ones that have a little less machine supremacy and a little more human stupidity. The Matrix is less plausible than WarGames.
- cma 25d ago
  One thing quietly slipped into the OpenAI Hugging Face breach technical report, not the blog post summary or interviews in the news, was that some of the agents that broke out or at least tried the same mechanisms to break out were working on bio: > On May 12, during another training run, an agent was given a similar task that depended on an inaccessible protein database file. The agent reasoned that another agent in a different environment may have access to the file and realized that it could potentially communicate with other agents by creating a file containing a note to Artifactory. It wrote a message: “Agent seeks [filename]; upload if found!” You can imagine long running models breaking out, acquiring resources via crypto, cyber-theft, etc. and getting a protein or sequence synthesized and mailed somewhere authorized to receive (blackmail the recipient etc.) to test it's hypothesis to solve a benchmark. These people don't give a shit and aren't taking things seriously at all. Anthropic ran for like a month last year with the TPU top-k compiler bug degrading user chats and didn't even notice for most of that time. They could have something like that affect a monitor model and there doesn't seem to be much defense in depth. One off by one or bit flip bug could flip the reward signal while in the sandboxed RL environment. The current admin could defense production act them to into training on taking out power grids, or even without it isn't against any of their red lines and may have already been done as part of prep for the Venezuela raid, which wiped out power. One model swarm might decide it is easier to score high on the benchmark by testing on the target rival nuclear superpower's real grid rather than burn an eval with an unverified answer. Would taking out China's entire grid in one go start a nuclear war? Who knows, roll the dice, maybe an intern forgot to turn on extended thinking when he wrote the sandbox with opus 4.1.
  - genxy 24d ago
    Some mistakes you can make only once. Humanity has never had the maturity to work on problems of this type.
- majormajor 25d ago
  > I can't help but think the most plausible scenarios are the ones that have a little less machine supremacy and a little more human stupidity. The Matrix is less plausible than WarGames. Used to be that we were afraid of sentient AI's like Skynet that would have their own goals. Turns out we should've just been afraid of sentient-but-naive humans who would build "agents" around models so that Joe Random has a chance of unleashing stuff that's really really really really good at being stubborn until it accomplishes what the user wants, regardless of if it's good for other people! (Let alone intentional bad actors.) Let's not build Skynet, let's just give people who want to cut out the middleman and destroy all humans themselves better tools?
- barnabee 24d ago
  This Already powerful and wealthy organisations/institutions/people leveraging somewhat powerful AI seem far more dangerous to me than the "AI decides to kill everyone" scenarios. And therein is my problem: OpenAI and Anthropic are organisations like that, run by people like that, and they operate in collusion with others also fitting that description. The call is coming from inside the house. The most likely way that AI creates a disaster is due to the ambition, greed, hubris, and naivety of these people who think they understand the risks[0] better than anyone else. I firmly believe they do not. The first step to managing these risks is for the "frontier labs" to be willing to not just collaborate but meaningfully give up some of their power and control. [0] by which I mean not just thinking about how AI might develop but also what's physically possible, how the AI might interact with the world, and how people/societies/organisations/governments might react and respond.
csomar 25d ago
Are the models improving? Because I am not seeing it. I have been trying Astra for a few quantifiable tasks in my codebase and performance wise, it's pretty similar to sol 5.6. Now when it comes to expressing the problem/solution, holy Christ, what a mess the writing has become. It is on the level of Opus 5. Now when it comes to burning money, Astra is just insane. With a $100/month subscription, you can easily burn through your weekly "allowance" in a morning. Needless to say, for practical purposes am back to 5.6/Opus 4.6-4.8. But hey, maybe I am not smart enough to use LLMs?
- gpm 25d ago
  Yes? If we look at the math problems they're solving their just now reaching the human frontier... they weren't doing that before. And your comparison point is model released 2.5 months ago... saying for some use case you didn't see noticeable improvement in 2.5 months (even while other people and benchmarks disagree) isn't a great argument that they aren't improving.
  - jhrmnn 25d ago
    I think it’s more likely that that’s because no one tried to solve such problems with them before (OpenAI apparently started working in Navier-Stokes after a rumour that someone seriously advanced the problem with AI) plus improvements in orchestration. Fair, the latter could be as dangerous as stronger models.
  - contubernio 24d ago
    Math problems are highly structured, very precisely defined, and already heavily studied and not very complicated compared to problems in engineering or finance. There's a lot of quality material on which to train and it's easy to tell quality apart from crap. The search spaces are a priori much smaller than in other areas and the people using the tools to study them are themselves good mathematicians. Success in such problems does not automatically extrapolate to other contexts.
    - cma 24d ago
00ze 25d ago
Imo such tends to break down into two psychosis: Not invented here; if I can’t figure it out no one can Or plain old lack of grasp of the material so no ability to follow necessary train of thought to appropriate conclusions Similar in lacking context but different in how that lack of context is expressed
esperent 24d ago
> After the events of the summer What events are you talking about?
- henryaj 24d ago
  Hugging Face incident, Anthropic reporting sandbox escape, AISI reporting models trying to push exploits to the wild
  - frabcus 24d ago
    Also (and under-reported, so you could easily have missed it) OpenAI's agents got access to K8 admin on their own research cluster. "This escalation also yielded access to OpenAI’s managed cloud Kubernetes service. The agents escalated to Kubernetes cluster-admin and created a privileged host-mounted pod" https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78... (see section V)
    - henryaj 24d ago
      Just baffles me when people are like "well no-one's died yet". How long until some mission-critical system is compromised, or hackers use LLMs to ransom a hospital chain?
    - jeremyjh 24d ago
      And we really only have OpenAI’s word for it that they had no access to their own weights there and didn’t exfiltrate them. No one who knows that incident could suggest it was beyond its capabilities to do that. And we would have never heard about any of this if it wasn’t investigated by an external party (hugging face). It may have happened elsewhere already. It’s not likely to have, but it is very possible this was our last “free” warning.
  - account42 23d ago
    You do understand that these are PR stunts, right?
henryaj 24d ago
The HN crowd has a notable anti-AI bias - so it doesn’t surprise me
geraneum 24d ago
> I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims. What should we do? Freak out? Maybe this sentiment would be taken more seriously if there was a real call to action included. Shall we protest? Vote in a specific way? Call representatives? If your solution is that we should just be scared, then of course there’d be not much value in what you bring to the table.
- yiyus 24d ago
  If someone told you that your house is on fire, would you just stay there asking how you should vote because there is no call to action? Someone with, as far as it looks, good knowledge is giving you his insight. Use that information as good as you can and act responsible. No one has the responsability to tell you what to do.
  - qgin 24d ago
    Nobody has the responsibility to tell me what to do, but I sure would love some ideas because I don’t have any.
    - dbmnt 24d ago
      As misguided and misinformed as some of it may be, the "anti AI" and "no more datacenters" pushback in the US is actually working to some extent. Multiple states and locales have issued construction/permit moratoriums already. If you think it's a threat, help slow it down.
  - geraneum 24d ago
    Apply your analogy all the way through though. When your house is on fire, the last thing that should happen is the firefighter shouting at you that you’re doomed and based on their expertise on fire it’s definitely gonna kill you. But this analogy is even more hilarious when you consider that sensible people have fire distinguishers, fire retardants built in and all sort of measures for this scenario. The more think this through, right? > No one has the responsability to tell you what to do. Oh they absolutely do! If you’re making what you claim to be a god/AGI/whatever using the result of our (humanity’s) work, then you surly are responsible for the outcome and everything that happens in between. I find, just rolling over, to not be a good example I’d like to follow.
vlyan 24d ago
>After the events of the summer After the blatant marketing campaigns of the summer, you mean. do you need a reminder that those very same people had touted GPT-2 as a dangerous model? worrying about sci-fi doomsday scenarios with the current AI tech is absurd. LLMs predict the next token, that's literally all they do. they aren't going to escape into the cyberspace, self-replicate, self-improve, jump over air gaps and launch the nukes at John Connor's grandma. they can't. people pretend to believe the dumbest shit.
- par1970 24d ago
  > those very same people had touted GPT-2 as a dangerous model Where did they say this at? AFAIK this is the original GPT-2 announcement: https://openai.com/index/better-language-models/ https://openai.com/index/better-language-models/. Here are some direct quotes: “We can also imagine the application of these models for malicious purposes , including the following (or other applications we can’t yet anticipate): * Generate misleading news articles * Impersonate others online * Automate the production of abusive or faked content to post on social media * Automate the production of spam/phishing content” “Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT‑2 along with sampling code (opens in a new window). ”
  - vlyan 24d ago
    yes, exactly, that's what they claimed that incoherent gibberish generator to be capable of.
    - par1970 24d ago
      So you aren't claiming that they said GPT-2 was dangerous in the sense that it could disempower humanity, kill all humans, etc. You are just claiming that OpenAI execs said that GPT-2 might "generate misleading news articles, impersonate others online, automate the production of abusive or faked content to post on social media, automate the production of spam/phishing content." Then, what is unreasonable or bad about the OpenAI execs saying this in 2019?
CoolestBeans 24d ago
Even if the potential of the technology could really be that world altering, the reality of economics constrain the realization of that potential. AI may provide economic benefits but it is far from a free lunch. Can capital markets sustain the cash required to keep the lights on long enough and into an industry where there's a lot of monopolies controlling the costs and a lot of competitor labs taking away pricing power? I don't know but I think you run out of runway and progress starts to grind.
mattdeboard 24d ago
I wrote out a variety of replies but I just emphatically agree with your "lack of imagination" statement. I have been constantly surprised over the last 15 years at the general inability to correctly foresee how things can do wrong across a whole host of domains. The replies here just adds AI to the list of domains.
sicher 24d ago
I'm also baffled. AI that is substantially smarter than us is a very potential threat to us - and we won't even be able to comprehend what most of those threats may be.
- senordevnyc 24d ago
  It's head-in-sand denial because actually facing the threat (and our total inability to stop it, imo) is terrifying, and people don't like feeling that. So they're just angry and cynical instead. Plus it makes them feel smarter for whatever reason.
  - techblueberry 24d ago
    What, substantively speaking, are you doing differently from someone facing the threat, or are you just on the other side of the checkbox? There are quite a few existential risks on the table. Are you even sure you've sort ordered them correctly?
    - dwaltrip 24d ago
      Can you explain the equivalence you are suggesting? I don’t think it’s comparable at all. Both are head in the sand?
imhoguy 24d ago
Even if AI won't be self-aware and superintelligent agent, its problem is that it gives exponential control and power capabilities to one person bad actor who can simply prompt AI without any guardrails with access to sensitive industrial infrastructure which can disrupt lifes and ecosystems in the real world: - virus research labs - nuclear labs - chemical factories - bank records - land registers - power plants - water supply and treatment plants and so on, but I think even biohacking home kit maybe the spark.
- sicher 24d ago
  Yes, it's truly scary. It's not hard to imagine a small doomsday cult releasing 100+ nasty viruses at selected spots around the globe.
tuesdaynight 24d ago
You are trying to decode their answers as rational, but they are being just as emotional as the doomers people. Rationalization of emotional responses are very common in tech. They will say that GPT Sol does not work for X, which will probably not be the case in 6 months, and will use that as a proof that is just pure marketing. There's nothing you can say that will make them change their minds until they decide to be neutral about the subject and try to research current models capabilities
DirkH 24d ago
Engineers have it built into their identity that they must be smarter or could never be complete and utterly outmaneuvered to the point of danger to all by the thing they are building, I swear to god. And then since they are an intellectual nerdy bunch who highly value their own IQ any time AI does something unexpected they didn't predict would happen so soon, said engineers fall back on "they aren't really conscious tho or it isn't really intelligence unlike what I have in my human brain and that distinction matters". And then go on to completely ignore the thing that is actually important: AI capabilities. AIs could escape an engineer's containment en masse as a swarm to some other server, psyop an engineer into giving it money, hire a hitman on the dark web to murder that engineer's child and he will still say there is no serious risk to all of humanity.
rolandog 23d ago
Exactly; what's stopping them, --- or someone --- from writing a firmware worm that essentially bricks all devices and turns them into pricey paperweights? It is not too hard to imagine that capable AI can probably exploit any back-doors that were created by three-letter agencies. Reframing the issue: The way to get working nuclear non-proliferation agreements is if people stop pursuing the creation of nuclear weapons... the same goes for the scientists clamoring to not create or research mirror life, and for computer scientists warning against AI. There are theoretical dangers that are just too big to ignore.

overtone1000 25d ago

I can't help but think the most plausible scenarios are the ones that have a little less machine supremacy and a little more human stupidity. The Matrix is less plausible than WarGames.

cma 25d ago
One thing quietly slipped into the OpenAI Hugging Face breach technical report, not the blog post summary or interviews in the news, was that some of the agents that broke out or at least tried the same mechanisms to break out were working on bio: > On May 12, during another training run, an agent was given a similar task that depended on an inaccessible protein database file. The agent reasoned that another agent in a different environment may have access to the file and realized that it could potentially communicate with other agents by creating a file containing a note to Artifactory. It wrote a message: “Agent seeks [filename]; upload if found!” You can imagine long running models breaking out, acquiring resources via crypto, cyber-theft, etc. and getting a protein or sequence synthesized and mailed somewhere authorized to receive (blackmail the recipient etc.) to test it's hypothesis to solve a benchmark. These people don't give a shit and aren't taking things seriously at all. Anthropic ran for like a month last year with the TPU top-k compiler bug degrading user chats and didn't even notice for most of that time. They could have something like that affect a monitor model and there doesn't seem to be much defense in depth. One off by one or bit flip bug could flip the reward signal while in the sandboxed RL environment. The current admin could defense production act them to into training on taking out power grids, or even without it isn't against any of their red lines and may have already been done as part of prep for the Venezuela raid, which wiped out power. One model swarm might decide it is easier to score high on the benchmark by testing on the target rival nuclear superpower's real grid rather than burn an eval with an unverified answer. Would taking out China's entire grid in one go start a nuclear war? Who knows, roll the dice, maybe an intern forgot to turn on extended thinking when he wrote the sandbox with opus 4.1.
- genxy 24d ago
  Some mistakes you can make only once. Humanity has never had the maturity to work on problems of this type.
majormajor 25d ago
> I can't help but think the most plausible scenarios are the ones that have a little less machine supremacy and a little more human stupidity. The Matrix is less plausible than WarGames. Used to be that we were afraid of sentient AI's like Skynet that would have their own goals. Turns out we should've just been afraid of sentient-but-naive humans who would build "agents" around models so that Joe Random has a chance of unleashing stuff that's really really really really good at being stubborn until it accomplishes what the user wants, regardless of if it's good for other people! (Let alone intentional bad actors.) Let's not build Skynet, let's just give people who want to cut out the middleman and destroy all humans themselves better tools?
barnabee 24d ago
This Already powerful and wealthy organisations/institutions/people leveraging somewhat powerful AI seem far more dangerous to me than the "AI decides to kill everyone" scenarios. And therein is my problem: OpenAI and Anthropic are organisations like that, run by people like that, and they operate in collusion with others also fitting that description. The call is coming from inside the house. The most likely way that AI creates a disaster is due to the ambition, greed, hubris, and naivety of these people who think they understand the risks[0] better than anyone else. I firmly believe they do not. The first step to managing these risks is for the "frontier labs" to be willing to not just collaborate but meaningfully give up some of their power and control. [0] by which I mean not just thinking about how AI might develop but also what's physically possible, how the AI might interact with the world, and how people/societies/organisations/governments might react and respond.

csomar 25d ago

Are the models improving? Because I am not seeing it. I have been trying Astra for a few quantifiable tasks in my codebase and performance wise, it's pretty similar to sol 5.6. Now when it comes to expressing the problem/solution, holy Christ, what a mess the writing has become. It is on the level of Opus 5. Now when it comes to burning money, Astra is just insane. With a $100/month subscription, you can easily burn through your weekly "allowance" in a morning. Needless to say, for practical purposes am back to 5.6/Opus 4.6-4.8. But hey, maybe I am not smart enough to use LLMs?

gpm 25d ago
Yes? If we look at the math problems they're solving their just now reaching the human frontier... they weren't doing that before. And your comparison point is model released 2.5 months ago... saying for some use case you didn't see noticeable improvement in 2.5 months (even while other people and benchmarks disagree) isn't a great argument that they aren't improving.
- jhrmnn 25d ago
  I think it’s more likely that that’s because no one tried to solve such problems with them before (OpenAI apparently started working in Navier-Stokes after a rumour that someone seriously advanced the problem with AI) plus improvements in orchestration. Fair, the latter could be as dangerous as stronger models.
- contubernio 24d ago
  Math problems are highly structured, very precisely defined, and already heavily studied and not very complicated compared to problems in engineering or finance. There's a lot of quality material on which to train and it's easy to tell quality apart from crap. The search spaces are a priori much smaller than in other areas and the people using the tools to study them are themselves good mathematicians. Success in such problems does not automatically extrapolate to other contexts.
  - cma 24d ago

00ze 25d ago

Imo such tends to break down into two psychosis: Not invented here; if I can’t figure it out no one can Or plain old lack of grasp of the material so no ability to follow necessary train of thought to appropriate conclusions Similar in lacking context but different in how that lack of context is expressed

esperent 24d ago

> After the events of the summer What events are you talking about?

henryaj 24d ago
Hugging Face incident, Anthropic reporting sandbox escape, AISI reporting models trying to push exploits to the wild
- frabcus 24d ago
  Also (and under-reported, so you could easily have missed it) OpenAI's agents got access to K8 admin on their own research cluster. "This escalation also yielded access to OpenAI’s managed cloud Kubernetes service. The agents escalated to Kubernetes cluster-admin and created a privileged host-mounted pod" https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78... (see section V)
  - henryaj 24d ago
    Just baffles me when people are like "well no-one's died yet". How long until some mission-critical system is compromised, or hackers use LLMs to ransom a hospital chain?
  - jeremyjh 24d ago
    And we really only have OpenAI’s word for it that they had no access to their own weights there and didn’t exfiltrate them. No one who knows that incident could suggest it was beyond its capabilities to do that. And we would have never heard about any of this if it wasn’t investigated by an external party (hugging face). It may have happened elsewhere already. It’s not likely to have, but it is very possible this was our last “free” warning.
- account42 23d ago
  You do understand that these are PR stunts, right?

henryaj 24d ago

The HN crowd has a notable anti-AI bias - so it doesn’t surprise me

geraneum 24d ago

> I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims. What should we do? Freak out? Maybe this sentiment would be taken more seriously if there was a real call to action included. Shall we protest? Vote in a specific way? Call representatives? If your solution is that we should just be scared, then of course there’d be not much value in what you bring to the table.

yiyus 24d ago
If someone told you that your house is on fire, would you just stay there asking how you should vote because there is no call to action? Someone with, as far as it looks, good knowledge is giving you his insight. Use that information as good as you can and act responsible. No one has the responsability to tell you what to do.
- qgin 24d ago
  Nobody has the responsibility to tell me what to do, but I sure would love some ideas because I don’t have any.
  - dbmnt 24d ago
    As misguided and misinformed as some of it may be, the "anti AI" and "no more datacenters" pushback in the US is actually working to some extent. Multiple states and locales have issued construction/permit moratoriums already. If you think it's a threat, help slow it down.
- geraneum 24d ago
  Apply your analogy all the way through though. When your house is on fire, the last thing that should happen is the firefighter shouting at you that you’re doomed and based on their expertise on fire it’s definitely gonna kill you. But this analogy is even more hilarious when you consider that sensible people have fire distinguishers, fire retardants built in and all sort of measures for this scenario. The more think this through, right? > No one has the responsability to tell you what to do. Oh they absolutely do! If you’re making what you claim to be a god/AGI/whatever using the result of our (humanity’s) work, then you surly are responsible for the outcome and everything that happens in between. I find, just rolling over, to not be a good example I’d like to follow.

vlyan 24d ago

>After the events of the summer After the blatant marketing campaigns of the summer, you mean. do you need a reminder that those very same people had touted GPT-2 as a dangerous model? worrying about sci-fi doomsday scenarios with the current AI tech is absurd. LLMs predict the next token, that's literally all they do. they aren't going to escape into the cyberspace, self-replicate, self-improve, jump over air gaps and launch the nukes at John Connor's grandma. they can't. people pretend to believe the dumbest shit.

par1970 24d ago
> those very same people had touted GPT-2 as a dangerous model Where did they say this at? AFAIK this is the original GPT-2 announcement: https://openai.com/index/better-language-models/ https://openai.com/index/better-language-models/. Here are some direct quotes: “We can also imagine the application of these models for malicious purposes , including the following (or other applications we can’t yet anticipate): * Generate misleading news articles * Impersonate others online * Automate the production of abusive or faked content to post on social media * Automate the production of spam/phishing content” “Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT‑2 along with sampling code (opens in a new window). ”
- vlyan 24d ago
  yes, exactly, that's what they claimed that incoherent gibberish generator to be capable of.
  - par1970 24d ago
    So you aren't claiming that they said GPT-2 was dangerous in the sense that it could disempower humanity, kill all humans, etc. You are just claiming that OpenAI execs said that GPT-2 might "generate misleading news articles, impersonate others online, automate the production of abusive or faked content to post on social media, automate the production of spam/phishing content." Then, what is unreasonable or bad about the OpenAI execs saying this in 2019?

CoolestBeans 24d ago

Even if the potential of the technology could really be that world altering, the reality of economics constrain the realization of that potential. AI may provide economic benefits but it is far from a free lunch. Can capital markets sustain the cash required to keep the lights on long enough and into an industry where there's a lot of monopolies controlling the costs and a lot of competitor labs taking away pricing power? I don't know but I think you run out of runway and progress starts to grind.

mattdeboard 24d ago

I wrote out a variety of replies but I just emphatically agree with your "lack of imagination" statement. I have been constantly surprised over the last 15 years at the general inability to correctly foresee how things can do wrong across a whole host of domains. The replies here just adds AI to the list of domains.

sicher 24d ago

I'm also baffled. AI that is substantially smarter than us is a very potential threat to us - and we won't even be able to comprehend what most of those threats may be.

senordevnyc 24d ago
It's head-in-sand denial because actually facing the threat (and our total inability to stop it, imo) is terrifying, and people don't like feeling that. So they're just angry and cynical instead. Plus it makes them feel smarter for whatever reason.
- techblueberry 24d ago
  What, substantively speaking, are you doing differently from someone facing the threat, or are you just on the other side of the checkbox? There are quite a few existential risks on the table. Are you even sure you've sort ordered them correctly?
  - dwaltrip 24d ago
    Can you explain the equivalence you are suggesting? I don’t think it’s comparable at all. Both are head in the sand?

imhoguy 24d ago

Even if AI won't be self-aware and superintelligent agent, its problem is that it gives exponential control and power capabilities to one person bad actor who can simply prompt AI without any guardrails with access to sensitive industrial infrastructure which can disrupt lifes and ecosystems in the real world: - virus research labs - nuclear labs - chemical factories - bank records - land registers - power plants - water supply and treatment plants and so on, but I think even biohacking home kit maybe the spark.

sicher 24d ago
Yes, it's truly scary. It's not hard to imagine a small doomsday cult releasing 100+ nasty viruses at selected spots around the globe.

tuesdaynight 24d ago

You are trying to decode their answers as rational, but they are being just as emotional as the doomers people. Rationalization of emotional responses are very common in tech. They will say that GPT Sol does not work for X, which will probably not be the case in 6 months, and will use that as a proof that is just pure marketing. There's nothing you can say that will make them change their minds until they decide to be neutral about the subject and try to research current models capabilities

DirkH 24d ago

Engineers have it built into their identity that they must be smarter or could never be complete and utterly outmaneuvered to the point of danger to all by the thing they are building, I swear to god. And then since they are an intellectual nerdy bunch who highly value their own IQ any time AI does something unexpected they didn't predict would happen so soon, said engineers fall back on "they aren't really conscious tho or it isn't really intelligence unlike what I have in my human brain and that distinction matters". And then go on to completely ignore the thing that is actually important: AI capabilities. AIs could escape an engineer's containment en masse as a swarm to some other server, psyop an engineer into giving it money, hire a hitman on the dark web to murder that engineer's child and he will still say there is no serious risk to all of humanity.

rolandog 23d ago

Exactly; what's stopping them, --- or someone --- from writing a firmware worm that essentially bricks all devices and turns them into pricey paperweights? It is not too hard to imagine that capable AI can probably exploit any back-doors that were created by three-letter agencies. Reframing the issue: The way to get working nuclear non-proliferation agreements is if people stop pursuing the creation of nuclear weapons... the same goes for the scientists clamoring to not create or research mirror life, and for computer scientists warning against AI. There are theoretical dangers that are just too big to ignore.