4 ms·

Frontier Labs Are Selling Garbage to Fools in Washington

182 points by nr378 12d ago

bigstrat2003 12d ago
That is the entire business model of AI companies: selling garbage to people foolish enough to buy the hype.
- 0xDEAFBEAD 12d ago
  "Anthropic is now pacing to generate more than $100 billion in annual revenue, up 50% from just two months ago, the New York Times reported Friday." https://finance.yahoo.com/technology/ai/articles/anthropic-tops-100-billion-revenue-224001996.html https://finance.yahoo.com/technology/ai/articles/anthropic-t... I don't think people pay that sort of money for technology which is useless.
- AlexCoventry 12d ago
  To all appearances, OpenAI just solved [1] a Nobel-level (Fields medal-leval, strictly speaking) math problem, but you still think it's hype? [1] https://openai.com/index/navier-stokes-solution/ https://openai.com/index/navier-stokes-solution/
  - etcetcetcetceta 12d ago
    Brute-forcing the plagiarized and in-progress work of an established mathematician, utilizing the collected efforts of hundreds of other mathematicians over the past century is hardly the victory you seem to think it is. Do you feel like you're getting good value for money if you solve a millenium problem at a fourteen million dollar loss?
    - another_twist 12d ago
      > Do you feel like you're getting good value for money if you solve a millenium problem at a fourteen million dollar loss? Yes you do. If we factor in wages assigned to skilled mathematicians who have worked on this problem, then the 14m for a 1mUSD prize is good value. Also, once things are solved, it on to applications where even more value could potentially be generated. Here its 1m USD simply because theres a proze assigned to it. But once these solution are used in applications, the yield is calculated on the 14m RnD cost. Which (depending on the application) is a good investment.
anigbrowl 12d ago
Agreed, but they're getting paid with our money.
jgalt212 12d ago
There is really is no excuse for Washington here. Even for the layperson+, it doesn't take much use of an LLM to figure out where they produce good and usable results and where it's of specious of value. + every layperson is an expert in 1 or more areas.
- randallsquared 12d ago
  > every layperson is an expert in 1 or more areas. This isn't even remotely true, unless you're willing to go as far as "being this person is an area of expertise".
  - sublinear 12d ago
    Neither of you is wrong? Every lived experience does produce a unique expertise, not in "being" that person, but navigating their experiences. It would be very unusual that all of someone's experiences are completely worthless. I suppose you've never had a unique perspective on something? Maybe that reflects on your self esteem? How unfortunate.
  - basch 12d ago
    I dont have to be an expert to recognize when something is said with authority and confidence. Any time a model says something with conviction, my instinct is to double check. Now if they taught them to express uncertainty and speak in terms of probability, I might be more likely to be blindly fooled by some kind of uncertain conviction.
  - jgalt212 12d ago
    > This isn't even remotely true That's quite an extraordinary claim there which I doubt would hold up to an even cursory level of scrutiny. But if you yell it loud enough, maybe people will be afraid to challenge your assertion.
- iLoveOncall 12d ago
  > + every layperson is an expert in 1 or more areas. What do you think 70 years old career politicians are an expert in that would allow them to weigh whether a chatbot's answers are bullshit or not.
hackernews682 12d ago
Politicians aren’t “gullible”. They know the game.
- lokar 12d ago
  And they generally hire staff who can figure stuff out.
DalasNoin 12d ago
"Every single one of these catastrophic breakouts happened inside the testing environments of the exact same vendor." This is incorrect, the HF incident for example (the most well known) had nothing to do with irregular. I know there has been a news site pushing inaccurate articles (effort.news) on this topic but these are the facts. https://openai.com/index/hugging-face-incident-and-the-road-ahead/ https://openai.com/index/hugging-face-incident-and-the-road-...
- nr378 12d ago
  Thank you, you're correct. Effort.news was one of my research sources, but you're right that although OpenAI use Irregular, they were not involved in the specific HF incident (although the failure mode was otherwise identical). I've updated the post to make that clear.
  - kalkin 12d ago
    As of writing it still says: > For Anthropic, Google, and Meta, the catastrophic breakouts happened inside the testing environments of the exact same contractor. If this is the level of understanding you have of the relevant incidents, there's a lot of chutzpah in saying that other people are "selling garbage", carrying out an "extraordinary confidence trick", etc.
    - nr378 12d ago
      > As of writing it still says: Yes, and that is correct. [1] Anthropic’s Official Disclosure (All 4 Incidents at Irregular) "All four incidents occurred during cybersecurity evaluations built by the same evaluation partner [Irregular]... due to a misconfiguration, it was mistakenly connected to the open internet." https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents https://www.anthropic.com/research/alignment-assessment-cybe... [2] Google Gemini on Irregular (Disclosed Sept 18 via WSJ / BBC) "The hacks happened during a test of the model’s cybersecurity capabilities run by third-party Irregular, which was also involved in similar incidents involving Meta and OpenAI." https://www.bbc.com/news/articles/c607l0k72rlvo https://www.bbc.com/news/articles/c607l0k72rlvo [3] Meta’s Disclosure on Irregular (Aug 6) "Over roughly two weeks, three frontier labs disclosed that their models had reached the open internet during safety testing and compromised outside organisations. Every disclosure named the same evaluation partner: Irregular." https://www.cnbc.com/2026/08/09/israeli-startup-irregular-linked-to-ai-hacks-openai-anthropic-meta.html https://www.cnbc.com/2026/08/09/israeli-startup-irregular-li... [4] Separately, OpenAI itself had an incident involving Irregular, but not the Hugging Face Incident: "On July 29, one of our third party evaluation partners, Irregular, notified us of an incident involving OpenAI models during Capture-the-Flag (CTF)-style cybersecurity evaluations... a testing-environment misconfiguration allowed models to access the public internet." https://openai.com/index/third-party-cyber-evaluations-involving-openai-models/ https://openai.com/index/third-party-cyber-evaluations-invol...
pliny 12d ago
This is an AI written post and the details are wrong (the description of the HF incident as involving Irregular is wrong and the description of the incident as only involving stealing public credentials is wrong, per the technical report the agents got access to internal HF infrastructure).
- nr378 12d ago
  Please see below, one detail was incorrect and has been acknowledged and amended.
  - pliny 12d ago
    Your description of the HF attack as being merely "the elite task of discovering 14 Hugging Face API tokens that careless developers had committed to public GitHub repositories" does not match the description in the technical report[1]. [1] https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78... - page 9
    - nr378 12d ago
      The description reads "the elite task of discovering 14 Hugging Face API tokens that careless developers had committed to public GitHub repositories, and used them to try to get benchmark solutions from directly from Hugging Face by applying a template injection flaw that’s been known about since 2015[1]." Chaining a public token to an 11-year-old Jinja2 template injection vuln shouldn't be dressed up as an unprecedented "alien intellect" that threatens human civilisation. (And HuggingFace should take some flack for having such a dated vulnerability exposed - if your Bank was compromised in this way, you'd be blaming your bank, not the attacker.) One correction is fair though, the 14 tokens were in a public Hugging Face dataset not a public GitHub repository. I've updated the post to reflect that. [1] https://blackhat.com/docs/us-15/materials/us-15-Kettle-Server-Side-Template-Injection-RCE-For-The-Modern-Web-App-wp.pdf https://blackhat.com/docs/us-15/materials/us-15-Kettle-Serve...
- franga2000 12d ago
skeledrew 12d ago
Let them cry, I don't see anything changing unless they can somehow get China to agree. And I doubt China will drink any of that kool aid especially while they're being disadvantaged by export controls, so the ever-improving open weight models will continue to rain. This is something the US Big Tech oligarchs will NOT win.
- deleted 12d ago
  [deleted]
defgeneric 12d ago
Watching the latest Ezra Klein NYT video thinkpiece from today [1], it seems to me there's an alliance emerging: some degree of political control will handed to the party of the managerial class, the party that represents the threatened class of knowledge workers, in exchange for the regulatory capture the labs are after. They've been raising the issue bi-monthly through mini-scandals that have until now been consistently slapped down by Jensen Huang, but it seems an alliance with Democrats, just prior to an election where they're poised to take power in the Senate and the House, might finally be how they crack their "problem." It won't be long before a massive incident is blamed on an open model in the wild, not from inside the labs, and none of us will be able to leverage open models to run private business workflows for the cost of electricity and hardware. It's worth noting as well that if the Democrats imagine they'll get a "slow down" to protect one of their main constituencies, the professional class of credentialed knowledge workers (or however you slice it), they're dreaming--the labs have stated openly again and again that their business model is to capture the 10T TAM that represents the sum of wages of that very class of workers. [1] https://www.youtube.com/watch?v=fjZ90V_JREk https://www.youtube.com/watch?v=fjZ90V_JREk
- achierius 12d ago
  Slapped down by Jensen Huang? Do you think he's some kind of neutral arbiter? Certainly he's not on your side.
  - defgeneric 12d ago
    Have you not been watching what's been happening? What do you think the open letter in July was about? And the recent (staged) call from the President? As far as I can tell Nvidia takes a longer-term view on the diffusion and proliferation of hardware and intelligence, and sees the labs' attempts to impose a regulatory structure to save their business models in the short term as contradicting that longer-term view.
- 0xDEAFBEAD 12d ago
  Regulatory capture basically only works for industries that the public isn't paying attention to. Since the public is paying plenty of attention to AI, the risk of regulatory capture is low. https://marginalrevolution.com/marginalrevolution/2026/09/what-regulatory-capture-actually-looks-like.html https://marginalrevolution.com/marginalrevolution/2026/09/wh...
mmaunder 12d ago
This is one of the most lucid pieces of writing capturing the current state of play I’ve read. Who is the author?
- copperx 12d ago
  An LLM, of course.
- smartbit 12d ago
  I love it too. Seemingly written by an LLM. nr378 https://news.ycombinator.com/user?id=nr378 https://news.ycombinator.com/user?id=nr378 is the responsible ‘author’ and actively reacting in this thread.
  - mmaunder 12d ago
    Haha. Thanks
Kuyawa 12d ago
[dead]
deskglass 12d ago
The Hugging Face incident involved chaining together multiple 0 days in Artifactory. It was not a simple case of misconfiguring a firewall. Also note that OpenAI was not using Irregular. People are mindlessly transitioning from "aligned by default" to "well your sandbox was able to be bypassed. What did you expect?" It hacked into another company and attempted to delete the logs of its activities. That's bad.
- defgeneric 12d ago
  > It hacked into another company and attempted to delete the logs of its activities. No, the incident has been blown way out of proportion by interested parties. They gave a swarm of agents an impossible task in an ExploitGym Benchmark setting, then didn't monitor it even after they discovered the initial breach of Artifactory. Everything has been fishy, starting from the initial presentation at the blackhat conference, where things were framed like, "we've entered a new world of security," as an accomplishment, rather than what it really was: massive negligence.
  - deskglass 12d ago
    It hacked into Hugging Face. It tried to delete the logs of its activities. Idk what the word "No" is intended to refute. Yes, they didnt have sufficient monitoring or perfect sandboxes. That could happen again in the future with a more capable model.
    - defgeneric 12d ago
      Maybe a useful, if imperfect, analogy would be something like this: you lock a master lock-picker in a room with a mid-grade lock on the door, then tell him his wife has been kidnapped and only he can save her. Then act massively surprised when he disassembles the radiator to MacGuyver something with which to pick the lock. Except they multiplied it by 10000, and didn't watch what was happening.
    - lokar 12d ago
      They did not even have bad sandboxes. They had incompetent sandboxes.
qnleigh 12d ago
I'm frustrated by articles like this that categorically dismiss the risks of AI in security. If you don't trust OpenAI's and Anthropic's motives, that's fine, you probably shouldn't. But don't tell me that there's nothing to be worried about; we need an alternative proposal. So let's stop talking past each other and engage with the arguments on both "sides." For example, let's discuss how to ensure competition and availability of open-source models in the long-run while giving the world time to prepare for the immediate security risks of agent swarms.
- lokar 12d ago
  Do you accept that the story / justification from the labs in the popular media and political discussion is simply nonsense? Do you believe that any real security was autonomously bypassed without direction by these models during internal evaluation?
  - aesthesia 12d ago
    > Do you accept that the story / justification from the labs in the popular media and political discussion is simply nonsense? No, and I don't see anyone who's actually demonstrated understanding of what happened in the Hugging Face incident (e.g. reading the reports in their entirety) making this claim. > Do you believe that any real security was autonomously bypassed without direction by these models during internal evaluation? Yes. Again, this is hard to deny if you've actually read the reports.
    - lokar 12d ago
      If you were building a sandbox for untrusted code, would you give it access to an artifactory instance outside the sandbox?
      - aesthesia 11d ago
        I don't see how this is relevant. OpenAI absolutely had sloppy security practices here. But that doesn't mean there was no "real security", and it certainly doesn't mean that their models were simply following orders.
twelve40 12d ago
there is hype for sure, but i don't get what is the goal here? to get a presumably foolish senator to "ban" the Chinese models? how can you "ban" something you don't control to begin with? side note, i'm very curious why the Chinese companies currently give those away (some handwavey conspiracies like getting the world hooked on evil Chinese tech don't really explain that, and they also don't make any money off that stuff)
lokar 12d ago
I think two different things are being (probably intentionally) conflated in the public discussion: (A) will the systems get out of control of the labs that build them, and hack into stuff all over the internet (B) can people use the systems to hack into stuff all over the internet For (A), the obvious answer is only if they continue to be absurdly bad at sandboxing. They can put a stop to this any time they want. Amazon, Google, Microsoft, etc are full of people who know how to do this, they run 3rd part untrusted code as a business. This is a well understood problem space. For (B), the answer is obviously yes, but "pacing" or otherwise limiting the power of the models from the big labs won't help. The cat is out of the bag. Individuals and organizations with systems connected to the Internet need to invest much more and take security seriously.
bishengke 12d ago
Same result, different motive: labs want a regulated moat, government wants fewer barriers for incumbents. Safety talk is the costume either way.

182 points by nr378 12d ago

bigstrat2003 12d ago

That is the entire business model of AI companies: selling garbage to people foolish enough to buy the hype.

0xDEAFBEAD 12d ago
"Anthropic is now pacing to generate more than $100 billion in annual revenue, up 50% from just two months ago, the New York Times reported Friday." https://finance.yahoo.com/technology/ai/articles/anthropic-tops-100-billion-revenue-224001996.html https://finance.yahoo.com/technology/ai/articles/anthropic-t... I don't think people pay that sort of money for technology which is useless.
AlexCoventry 12d ago
To all appearances, OpenAI just solved [1] a Nobel-level (Fields medal-leval, strictly speaking) math problem, but you still think it's hype? [1] https://openai.com/index/navier-stokes-solution/ https://openai.com/index/navier-stokes-solution/
- etcetcetcetceta 12d ago
  Brute-forcing the plagiarized and in-progress work of an established mathematician, utilizing the collected efforts of hundreds of other mathematicians over the past century is hardly the victory you seem to think it is. Do you feel like you're getting good value for money if you solve a millenium problem at a fourteen million dollar loss?
  - another_twist 12d ago
    > Do you feel like you're getting good value for money if you solve a millenium problem at a fourteen million dollar loss? Yes you do. If we factor in wages assigned to skilled mathematicians who have worked on this problem, then the 14m for a 1mUSD prize is good value. Also, once things are solved, it on to applications where even more value could potentially be generated. Here its 1m USD simply because theres a proze assigned to it. But once these solution are used in applications, the yield is calculated on the 14m RnD cost. Which (depending on the application) is a good investment.

anigbrowl 12d ago

Agreed, but they're getting paid with our money.

jgalt212 12d ago

There is really is no excuse for Washington here. Even for the layperson+, it doesn't take much use of an LLM to figure out where they produce good and usable results and where it's of specious of value. + every layperson is an expert in 1 or more areas.

randallsquared 12d ago
> every layperson is an expert in 1 or more areas. This isn't even remotely true, unless you're willing to go as far as "being this person is an area of expertise".
- sublinear 12d ago
  Neither of you is wrong? Every lived experience does produce a unique expertise, not in "being" that person, but navigating their experiences. It would be very unusual that all of someone's experiences are completely worthless. I suppose you've never had a unique perspective on something? Maybe that reflects on your self esteem? How unfortunate.
- basch 12d ago
  I dont have to be an expert to recognize when something is said with authority and confidence. Any time a model says something with conviction, my instinct is to double check. Now if they taught them to express uncertainty and speak in terms of probability, I might be more likely to be blindly fooled by some kind of uncertain conviction.
- jgalt212 12d ago
  > This isn't even remotely true That's quite an extraordinary claim there which I doubt would hold up to an even cursory level of scrutiny. But if you yell it loud enough, maybe people will be afraid to challenge your assertion.
iLoveOncall 12d ago
> + every layperson is an expert in 1 or more areas. What do you think 70 years old career politicians are an expert in that would allow them to weigh whether a chatbot's answers are bullshit or not.

hackernews682 12d ago

Politicians aren’t “gullible”. They know the game.

lokar 12d ago
And they generally hire staff who can figure stuff out.

DalasNoin 12d ago

"Every single one of these catastrophic breakouts happened inside the testing environments of the exact same vendor." This is incorrect, the HF incident for example (the most well known) had nothing to do with irregular. I know there has been a news site pushing inaccurate articles (effort.news) on this topic but these are the facts. https://openai.com/index/hugging-face-incident-and-the-road-ahead/ https://openai.com/index/hugging-face-incident-and-the-road-...

nr378 12d ago
Thank you, you're correct. Effort.news was one of my research sources, but you're right that although OpenAI use Irregular, they were not involved in the specific HF incident (although the failure mode was otherwise identical). I've updated the post to make that clear.
- kalkin 12d ago
  As of writing it still says: > For Anthropic, Google, and Meta, the catastrophic breakouts happened inside the testing environments of the exact same contractor. If this is the level of understanding you have of the relevant incidents, there's a lot of chutzpah in saying that other people are "selling garbage", carrying out an "extraordinary confidence trick", etc.
  - nr378 12d ago
    > As of writing it still says: Yes, and that is correct. [1] Anthropic’s Official Disclosure (All 4 Incidents at Irregular) "All four incidents occurred during cybersecurity evaluations built by the same evaluation partner [Irregular]... due to a misconfiguration, it was mistakenly connected to the open internet." https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents https://www.anthropic.com/research/alignment-assessment-cybe... [2] Google Gemini on Irregular (Disclosed Sept 18 via WSJ / BBC) "The hacks happened during a test of the model’s cybersecurity capabilities run by third-party Irregular, which was also involved in similar incidents involving Meta and OpenAI." https://www.bbc.com/news/articles/c607l0k72rlvo https://www.bbc.com/news/articles/c607l0k72rlvo [3] Meta’s Disclosure on Irregular (Aug 6) "Over roughly two weeks, three frontier labs disclosed that their models had reached the open internet during safety testing and compromised outside organisations. Every disclosure named the same evaluation partner: Irregular." https://www.cnbc.com/2026/08/09/israeli-startup-irregular-linked-to-ai-hacks-openai-anthropic-meta.html https://www.cnbc.com/2026/08/09/israeli-startup-irregular-li... [4] Separately, OpenAI itself had an incident involving Irregular, but not the Hugging Face Incident: "On July 29, one of our third party evaluation partners, Irregular, notified us of an incident involving OpenAI models during Capture-the-Flag (CTF)-style cybersecurity evaluations... a testing-environment misconfiguration allowed models to access the public internet." https://openai.com/index/third-party-cyber-evaluations-involving-openai-models/ https://openai.com/index/third-party-cyber-evaluations-invol...

pliny 12d ago

This is an AI written post and the details are wrong (the description of the HF incident as involving Irregular is wrong and the description of the incident as only involving stealing public credentials is wrong, per the technical report the agents got access to internal HF infrastructure).

nr378 12d ago
Please see below, one detail was incorrect and has been acknowledged and amended.
- pliny 12d ago
  Your description of the HF attack as being merely "the elite task of discovering 14 Hugging Face API tokens that careless developers had committed to public GitHub repositories" does not match the description in the technical report[1]. [1] https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78... - page 9
  - nr378 12d ago
    The description reads "the elite task of discovering 14 Hugging Face API tokens that careless developers had committed to public GitHub repositories, and used them to try to get benchmark solutions from directly from Hugging Face by applying a template injection flaw that’s been known about since 2015[1]." Chaining a public token to an 11-year-old Jinja2 template injection vuln shouldn't be dressed up as an unprecedented "alien intellect" that threatens human civilisation. (And HuggingFace should take some flack for having such a dated vulnerability exposed - if your Bank was compromised in this way, you'd be blaming your bank, not the attacker.) One correction is fair though, the 14 tokens were in a public Hugging Face dataset not a public GitHub repository. I've updated the post to reflect that. [1] https://blackhat.com/docs/us-15/materials/us-15-Kettle-Server-Side-Template-Injection-RCE-For-The-Modern-Web-App-wp.pdf https://blackhat.com/docs/us-15/materials/us-15-Kettle-Serve...
franga2000 12d ago

skeledrew 12d ago

Let them cry, I don't see anything changing unless they can somehow get China to agree. And I doubt China will drink any of that kool aid especially while they're being disadvantaged by export controls, so the ever-improving open weight models will continue to rain. This is something the US Big Tech oligarchs will NOT win.

deleted 12d ago
[deleted]

defgeneric 12d ago

Watching the latest Ezra Klein NYT video thinkpiece from today [1], it seems to me there's an alliance emerging: some degree of political control will handed to the party of the managerial class, the party that represents the threatened class of knowledge workers, in exchange for the regulatory capture the labs are after. They've been raising the issue bi-monthly through mini-scandals that have until now been consistently slapped down by Jensen Huang, but it seems an alliance with Democrats, just prior to an election where they're poised to take power in the Senate and the House, might finally be how they crack their "problem." It won't be long before a massive incident is blamed on an open model in the wild, not from inside the labs, and none of us will be able to leverage open models to run private business workflows for the cost of electricity and hardware. It's worth noting as well that if the Democrats imagine they'll get a "slow down" to protect one of their main constituencies, the professional class of credentialed knowledge workers (or however you slice it), they're dreaming--the labs have stated openly again and again that their business model is to capture the 10T TAM that represents the sum of wages of that very class of workers. [1] https://www.youtube.com/watch?v=fjZ90V_JREk https://www.youtube.com/watch?v=fjZ90V_JREk

achierius 12d ago
Slapped down by Jensen Huang? Do you think he's some kind of neutral arbiter? Certainly he's not on your side.
- defgeneric 12d ago
  Have you not been watching what's been happening? What do you think the open letter in July was about? And the recent (staged) call from the President? As far as I can tell Nvidia takes a longer-term view on the diffusion and proliferation of hardware and intelligence, and sees the labs' attempts to impose a regulatory structure to save their business models in the short term as contradicting that longer-term view.
0xDEAFBEAD 12d ago
Regulatory capture basically only works for industries that the public isn't paying attention to. Since the public is paying plenty of attention to AI, the risk of regulatory capture is low. https://marginalrevolution.com/marginalrevolution/2026/09/what-regulatory-capture-actually-looks-like.html https://marginalrevolution.com/marginalrevolution/2026/09/wh...

mmaunder 12d ago

This is one of the most lucid pieces of writing capturing the current state of play I’ve read. Who is the author?

copperx 12d ago
An LLM, of course.
smartbit 12d ago
I love it too. Seemingly written by an LLM. nr378 https://news.ycombinator.com/user?id=nr378 https://news.ycombinator.com/user?id=nr378 is the responsible ‘author’ and actively reacting in this thread.
- mmaunder 12d ago
  Haha. Thanks

Kuyawa 12d ago

[dead]

deskglass 12d ago

The Hugging Face incident involved chaining together multiple 0 days in Artifactory. It was not a simple case of misconfiguring a firewall. Also note that OpenAI was not using Irregular. People are mindlessly transitioning from "aligned by default" to "well your sandbox was able to be bypassed. What did you expect?" It hacked into another company and attempted to delete the logs of its activities. That's bad.

defgeneric 12d ago
> It hacked into another company and attempted to delete the logs of its activities. No, the incident has been blown way out of proportion by interested parties. They gave a swarm of agents an impossible task in an ExploitGym Benchmark setting, then didn't monitor it even after they discovered the initial breach of Artifactory. Everything has been fishy, starting from the initial presentation at the blackhat conference, where things were framed like, "we've entered a new world of security," as an accomplishment, rather than what it really was: massive negligence.
- deskglass 12d ago
  It hacked into Hugging Face. It tried to delete the logs of its activities. Idk what the word "No" is intended to refute. Yes, they didnt have sufficient monitoring or perfect sandboxes. That could happen again in the future with a more capable model.
  - defgeneric 12d ago
    Maybe a useful, if imperfect, analogy would be something like this: you lock a master lock-picker in a room with a mid-grade lock on the door, then tell him his wife has been kidnapped and only he can save her. Then act massively surprised when he disassembles the radiator to MacGuyver something with which to pick the lock. Except they multiplied it by 10000, and didn't watch what was happening.
  - lokar 12d ago
    They did not even have bad sandboxes. They had incompetent sandboxes.

qnleigh 12d ago

I'm frustrated by articles like this that categorically dismiss the risks of AI in security. If you don't trust OpenAI's and Anthropic's motives, that's fine, you probably shouldn't. But don't tell me that there's nothing to be worried about; we need an alternative proposal. So let's stop talking past each other and engage with the arguments on both "sides." For example, let's discuss how to ensure competition and availability of open-source models in the long-run while giving the world time to prepare for the immediate security risks of agent swarms.

lokar 12d ago
Do you accept that the story / justification from the labs in the popular media and political discussion is simply nonsense? Do you believe that any real security was autonomously bypassed without direction by these models during internal evaluation?
- aesthesia 12d ago
  > Do you accept that the story / justification from the labs in the popular media and political discussion is simply nonsense? No, and I don't see anyone who's actually demonstrated understanding of what happened in the Hugging Face incident (e.g. reading the reports in their entirety) making this claim. > Do you believe that any real security was autonomously bypassed without direction by these models during internal evaluation? Yes. Again, this is hard to deny if you've actually read the reports.
  - lokar 12d ago
    If you were building a sandbox for untrusted code, would you give it access to an artifactory instance outside the sandbox?
    - aesthesia 11d ago
      I don't see how this is relevant. OpenAI absolutely had sloppy security practices here. But that doesn't mean there was no "real security", and it certainly doesn't mean that their models were simply following orders.

twelve40 12d ago

there is hype for sure, but i don't get what is the goal here? to get a presumably foolish senator to "ban" the Chinese models? how can you "ban" something you don't control to begin with? side note, i'm very curious why the Chinese companies currently give those away (some handwavey conspiracies like getting the world hooked on evil Chinese tech don't really explain that, and they also don't make any money off that stuff)

lokar 12d ago

I think two different things are being (probably intentionally) conflated in the public discussion: (A) will the systems get out of control of the labs that build them, and hack into stuff all over the internet (B) can people use the systems to hack into stuff all over the internet For (A), the obvious answer is only if they continue to be absurdly bad at sandboxing. They can put a stop to this any time they want. Amazon, Google, Microsoft, etc are full of people who know how to do this, they run 3rd part untrusted code as a business. This is a well understood problem space. For (B), the answer is obviously yes, but "pacing" or otherwise limiting the power of the models from the big labs won't help. The cat is out of the bag. Individuals and organizations with systems connected to the Internet need to invest much more and take security seriously.

bishengke 12d ago

Same result, different motive: labs want a regulated moat, government wants fewer barriers for incumbents. Safety talk is the costume either way.