5 ms·

Do you know the story of the boy who cried wolf? There may very well be a wolf lurking [0] but OpenAI/Anthropic have both cried wolf so many times, incorrectly

Do you know the story of the boy who cried wolf? There may very well be a wolf lurking [0] but OpenAI/Anthropic have both cried wolf so many times, incorrectly, that it’s incredibly hard to believe “this time there IS a wolf!”. Remember “GPT-2 is too dangerous to release”? I had a conversation at work just yesterday about how we need to start hardening things we’ve let languish because of the coming LLM-backed attacks we are sure to face, even if just from a script kiddy. I do think we are headed in that direction, however it’s Sam/Dario’s own fault that people aren’t going to take them seriously. Lastly, as other have pointed out, this seems more financially motivated than our of any real desire for “safety”. We’ve all seen how both labs approach “safety” so it’s quite rich for them to now hide behind that after not giving a shit before. [0] I don’t take anything Sam or Dario say at face value. The whole hacking thing could also be a case of them letting a model loose on purpose for the publicity, not an “escape” during a training run (or whatever they said). And when both, especially Sam, have lied so much and breathlessly warned about the dangers of AI (when it helped their bottom line and/or helped pull up the ladder behind them), it makes it hard to believe them.

bottlepalm 2mo ago
Has the last 100 years of concern about AI and robots been crying wolf because it hasn’t happened yet? How does reallocating resources from training to chain of thought monitoring make ‘financial’ sense? You suggesting then model was let loose on purpose.. how am I the crazy one here while all of you are pushing this tin foil hat conspiracy angle?
- nemomarx 2mo ago
  Most of that concern was in fiction. Non theoretical, genuine concern about AI is pretty recent, maybe dating back to 2010 ish with the rationalist types. But also no one trusts anyone involved in AI safety now, I think, because they are all seemingly in bed with these big companies. And there is the perpetual argument "if we aren't pushing AI forward China will and then we don't have any control" and so on. (I don't use any of these tools - my experience is limited to prodding at copilot at work and seeing Gemini summaries on Google. So it doesn't seem to me like it's getting exponentially better at everything yet. People are always saying the latest model is finally the big step that made it useful and life changing and they have been since 2024 ish. So if the situation is really bad, we should turn it all off, sure. I won't lose anything from it going away and I think life would be a little better without models writing all these posts and websites and needing extra compute.)
  - bottlepalm 2mo ago
    Cool, well let me bring you up to date - it’s bad, and there’s no way to turn it off. Fiction has become non-fiction.
    - nemomarx 2mo ago
      Here is one thing I don't get - the model is only "running" if it's being kept going by some harness that is basically giving it prompts it's generating itself. Shouldn't kill switches be pretty easy to build into the software and hardware for this? and you would even have a better time dumping logs and analyzing things if you froze those processes any time something strange happened in testing, surely? So why do the big frontier labs not have something like this anyway. They're talking about two week pauses on the new model (which seems very short and hardly a cost at all to me) and alarms during their tests that might be 30 minutes late and etc. Those are not very serious measures, so are they not concerned?
      - bottlepalm 2mo ago
        > Shouldn't kill switches be pretty easy to build I need to remember when I comment here that these are the kinds of people I am replying to. Just oozing with hubris. No, kill switches are not easy to build and the latest incident should have made it clear that AI can go undetected, evade, zero day, and spread. The fact that this incident happened greatly increases the probability it happens again and/or is already happening elsewhere.
        nemomarx 2mo ago
        Why aren't they? You could put a human yes/ no prompt before any cycle the agent is running on, or not let it spawn sub processes, or anything like that. Why let it run autonomously enough that it can no longer have a simple way to completely stop it? (obviously not practical to do this during real use, but for evals? you could slow it down in lots of ways I would think) Are you telling me we've been iterating on this for years and for convenience we just let the models call any tools or spawn any other model instances they like, and there was no design for harnesses that could control this done during that time?
- stymaar 2mo ago
  [dead]
- mrguyorama 2mo ago
  You know fiction is.... not real, right? We have several films about the sun or earth needing to be restarted with a nuclear weapon. That doesn't make it something we should be concerned about. Hell, half the fiction about evil AI is actually commentary on stuff that already exists and is making us suffer and doesn't have anything to do with any potential future AI The Star Trek TNG episode about Data being tried in court as to whether he is sentient or not is not actually about whether AIs should have rights or not!
  - bottlepalm 2mo ago
    Your argument boils down to some sci-fi is unrealistic therefore all of it is. Sci-fi is supposed to make you think. What if the AI told you NO when you need it - Hal 9000. What if the evil AI got out and you don’t know what data center it’s hiding in - Lawnmower Man. What if you were so sure something couldn’t escape but it did - Jurassic Park. Actually all three of those predicted fantastical scenarios are possible today. So what’s next? Don’t stick your head in the sand - I assure you the next disaster has already been predicted, and I’m sure if you think about it a little you can figure out what it is.
  - pixl97 2mo ago
    Just about everything we do currently is science fiction to someone 200 years old. When looking at all of human history we live in a fictional world now. You can talk to someone on the other side of the planet instantly. Humans travel the skies in air chariots. We have weapons that hold the power of the gods. If I had some way to kick you back to the, you'd be jailed as a rambling madman for lunacy. So just saying something is fiction isn't really a valid argument. What is an argument is if the laws of physics it can't happen. We've been writing that AI can mess stuff up for 100 years because it's not really that fantastical.
zmgsabst 2mo ago
To be fair, “s’kiddies will exploit low hanging fruit with semi-automated vuln scans” is a lot more realistic threat than “LLMs are going full Skynet any day now!” I’d definitely suggest companies start addressing that first concern, even if I’m in the camp that thinks the second is fantasy.
- preg_match 2mo ago
  I think, realistically the way Skynet happens is that the military abuses LLMs for warfare. The military is historically, and currently, incredibly irresponsible with technology. If we aren’t already automating killer drones and such, then we will be soon because “well our enemies are doing it!” I don’t think that means that it’s gonna, like, somehow homogenize into some mega super intelligence. But we will have machines who are designed to kill, and do so without human input or alignment.

bottlepalm 2mo ago

Has the last 100 years of concern about AI and robots been crying wolf because it hasn’t happened yet? How does reallocating resources from training to chain of thought monitoring make ‘financial’ sense? You suggesting then model was let loose on purpose.. how am I the crazy one here while all of you are pushing this tin foil hat conspiracy angle?

nemomarx 2mo ago
Most of that concern was in fiction. Non theoretical, genuine concern about AI is pretty recent, maybe dating back to 2010 ish with the rationalist types. But also no one trusts anyone involved in AI safety now, I think, because they are all seemingly in bed with these big companies. And there is the perpetual argument "if we aren't pushing AI forward China will and then we don't have any control" and so on. (I don't use any of these tools - my experience is limited to prodding at copilot at work and seeing Gemini summaries on Google. So it doesn't seem to me like it's getting exponentially better at everything yet. People are always saying the latest model is finally the big step that made it useful and life changing and they have been since 2024 ish. So if the situation is really bad, we should turn it all off, sure. I won't lose anything from it going away and I think life would be a little better without models writing all these posts and websites and needing extra compute.)
- bottlepalm 2mo ago
  Cool, well let me bring you up to date - it’s bad, and there’s no way to turn it off. Fiction has become non-fiction.
  - nemomarx 2mo ago
    Here is one thing I don't get - the model is only "running" if it's being kept going by some harness that is basically giving it prompts it's generating itself. Shouldn't kill switches be pretty easy to build into the software and hardware for this? and you would even have a better time dumping logs and analyzing things if you froze those processes any time something strange happened in testing, surely? So why do the big frontier labs not have something like this anyway. They're talking about two week pauses on the new model (which seems very short and hardly a cost at all to me) and alarms during their tests that might be 30 minutes late and etc. Those are not very serious measures, so are they not concerned?
    - bottlepalm 2mo ago
      > Shouldn't kill switches be pretty easy to build I need to remember when I comment here that these are the kinds of people I am replying to. Just oozing with hubris. No, kill switches are not easy to build and the latest incident should have made it clear that AI can go undetected, evade, zero day, and spread. The fact that this incident happened greatly increases the probability it happens again and/or is already happening elsewhere.
      - nemomarx 2mo ago
        Why aren't they? You could put a human yes/ no prompt before any cycle the agent is running on, or not let it spawn sub processes, or anything like that. Why let it run autonomously enough that it can no longer have a simple way to completely stop it? (obviously not practical to do this during real use, but for evals? you could slow it down in lots of ways I would think) Are you telling me we've been iterating on this for years and for convenience we just let the models call any tools or spawn any other model instances they like, and there was no design for harnesses that could control this done during that time?
stymaar 2mo ago
[dead]
mrguyorama 2mo ago
You know fiction is.... not real, right? We have several films about the sun or earth needing to be restarted with a nuclear weapon. That doesn't make it something we should be concerned about. Hell, half the fiction about evil AI is actually commentary on stuff that already exists and is making us suffer and doesn't have anything to do with any potential future AI The Star Trek TNG episode about Data being tried in court as to whether he is sentient or not is not actually about whether AIs should have rights or not!
- bottlepalm 2mo ago
  Your argument boils down to some sci-fi is unrealistic therefore all of it is. Sci-fi is supposed to make you think. What if the AI told you NO when you need it - Hal 9000. What if the evil AI got out and you don’t know what data center it’s hiding in - Lawnmower Man. What if you were so sure something couldn’t escape but it did - Jurassic Park. Actually all three of those predicted fantastical scenarios are possible today. So what’s next? Don’t stick your head in the sand - I assure you the next disaster has already been predicted, and I’m sure if you think about it a little you can figure out what it is.
- pixl97 2mo ago
  Just about everything we do currently is science fiction to someone 200 years old. When looking at all of human history we live in a fictional world now. You can talk to someone on the other side of the planet instantly. Humans travel the skies in air chariots. We have weapons that hold the power of the gods. If I had some way to kick you back to the, you'd be jailed as a rambling madman for lunacy. So just saying something is fiction isn't really a valid argument. What is an argument is if the laws of physics it can't happen. We've been writing that AI can mess stuff up for 100 years because it's not really that fantastical.

zmgsabst 2mo ago

To be fair, “s’kiddies will exploit low hanging fruit with semi-automated vuln scans” is a lot more realistic threat than “LLMs are going full Skynet any day now!” I’d definitely suggest companies start addressing that first concern, even if I’m in the camp that thinks the second is fantasy.

preg_match 2mo ago
I think, realistically the way Skynet happens is that the military abuses LLMs for warfare. The military is historically, and currently, incredibly irresponsible with technology. If we aren’t already automating killer drones and such, then we will be soon because “well our enemies are doing it!” I don’t think that means that it’s gonna, like, somehow homogenize into some mega super intelligence. But we will have machines who are designed to kill, and do so without human input or alignment.