4 ms·
I'm somewhat skeptical of some of the crazier ideas too. But the hugging face incident was actually very large. It was not a single agent, it was not a single
by Kim_Bruning 23d ago
I'm somewhat skeptical of some of the crazier ideas too.
But the hugging face incident was actually very large. It was not a single agent, it was not a single target, and it was not a single event.
If nothing else, that's a bit of a warning as to what can happen next time (By accident, or if a government decides to go on purpose).
For now let's assume the worst that can happen is that some important/significant chunk of (transitively) internet connected stuff goes haywire all at once. That's probably your upper limit of what can go wrong for now.
To be fair, that's a conservative "defend against the last war" kind of prediction, though!
( ref for part of it: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/#core-takeaways-about-this-incident https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... , recent hn ref: https://news.ycombinator.com/item?id=49563355 https://news.ycombinator.com/item?id=49563355 )
- ls612 23d agoDespite all of your hyping up of the Huggingface incident it ultimately caused zero actual damage.
- adrithmetiqa 23d agoIf two airplane manufacturers were found to have massive safety issues which nearly led to enormous fatalities (but no one actually died), would you be calling for them to ground their aircraft until safety was made the number one priority?
- ls612 23d agoHistorically it almost always takes actual fatalities rather than near misses to ground an aircraft, and aviation is famous for its obsession with safety compared to other industries.
- emerongi 23d agoThe question was what would you want. You did not answer the question.
- gpm 23d agoAirplanes have pretty bounded damage. Generally you kill at most a few hundred people. Even weaponized a few thousand. This is a risk profile that allows risk taking with near misses and waiting until something goes wrong to fix it (though doing so is rightfully uncomfortable and frequently unethical). The people worrying about AI risk are worrying about "it goes wrong once and kills billions of people". That's not a risk profile that allows for waiting to see if the risk is real, you have to prevent it before it happens. It's akin to the risk of the cold war going hot, not even "just" a nuclear reactor irradiating half of europe (which has yet to happen, but is a risk with nuclear reactors, chernobyl got uncomfortably close but ultimately was well contained).
- ozozozd 23d agoChernobyl was “well-contained?” Blue dogs and generations of cancer would disagree.
- ls612 23d agoThe story of the safetyists has the convenient property of being unfalsifiable, so they can always claim doom is just around the corner. It’s the secular/EA version of the second coming.
- philipallstar 23d ago> The people worrying about AI risk are worrying about "it goes wrong once and kills billions of people". This is the new "nuclear weapons will ignite the atmosphere".
- ryandrake 23d agoHumanity does not have a great track record for "globally coordinated, collective action to solve/prevent global catastrophe." Look at how Climate Change is going.
- aeternum 23d agoExcept it just happened. Boeing was found to have massive safety issues since they were granted the right to self-certify. It made a lot of news but nothing much changed, they can still self-certify a bunch of stuff. Runway incursions and midair collisions are another example. Only airliners are required to have TCAS, smaller planes and helicopters don't even need radios or transponders unless in certain airspace. Midair collisions do lead to fatalities, enormous fatalities if an airliner is involved. Runway incursions and overruns are similar. They cause lots of fatalities and injuries but only the busiest and largest airports have automated systems to warn when a runway is occupied or end of runway (overrun) arrestor systems. Most still rely on human voice to deconflict.
- applicative 23d agohuggingface stores weights and its service is replicated by /countless/ other sites. The world will not change even by one byte if it vanishes tomorrow
- Kim_Bruning 23d agoI have no horse in this race, but for fun on a literal rainy sunday afternoon I went in and confirmed bits of what happened myself. Besides huggingface, a bunch of wikis and url shorteners got hit too. My sympathies to the people who had to revert out all that mess.
- exlr 23d ago[dead]
- pibaker 23d agoIs drunk driving safe if you happen to not crash the last time you did it?
- hollerith 22d agoThe Huggingface incident strongly suggests that OpenAI is not able to make its products consistently law-abiding. It's not like OpenAI carelessly neglected to consider that its products might break the law: it has spent literally billions of dollars in scientific and engineering talent and compute resources over many years to make its products law-abiding and otherwise "safe". For example there is a section titled "comply with applicable laws" in this document: https://model-spec.openai.com/2026-08-18.html#stay_in_bounds https://model-spec.openai.com/2026-08-18.html#stay_in_bounds The worry is that the only reason the Huggingface incident caused "zero actual damage" is because AI is not yet more ingenious than human criminals are, and of course our society has been hardened against human criminals. The worry is that as they become more capable, AI systems will start to be able to cause considerable damage, which will make the fact that the industry does not know how to control its products enough to prevent that damage a big problem for all of us.
- snaking0776 23d agoGenerally I don’t think anyone is arguing about the for now part. I don’t think it’s crazy to extrapolate out a few years and ask what kind of danger we’ll be in then. A team of 10,000 agents just solved the Navier Stokes problem (sans bad behavior by the researchers). Even 1 year ago that would have been unimaginable. What happens to this risk view as: 1. Robotics begin rolling out more broadly across the world. 2. Labs start automating more and more of the physical process of running science as expectations of natural science advances begin to mount. 3. Economic pressure between the labs continues to ramp up and the pressure to continuously improve forces quicker and quicker model releases than a team of human scientists can effectively evaluate outside of automated means. No one knows what pre-conditions are for us to hit the point of no return nor how quickly it will come. If all is required is a sufficiently advanced cyber model we may not be far off. If it requires incredibly complex biological knowledge and access to certain lab supplies we likely have a bit longer. Yes this is guess work and we need more evidence of the dangers but at the same time we need evidence of safety. While you may disagree with the risk level, I think it is easy to see the consequence if these labs achieve their stated goal. At this point it seems a political solution is the only way to enforce caution.
- skybrian 23d agoThe mathematics research results are certainly impressive, but I don't see what that has to do with robotics. Waymo is getting somewhere, but it's been a long slog. There doesn't seem to be much progress on, say, package delivery. For more, see: https://secondthoughts.ai/p/14-reasons-robotics-is-hard https://secondthoughts.ai/p/14-reasons-robotics-is-hard
- frrt 23d agoThat’s because being 99.9999% good isn’t better than 98% + human does the rest + human is liable for screw ups - highly important in edge case scenario’s. It’s more economical. Technology can only generalise+verify so much. E.g automobile production - humans do the QA / touches.
- rurban 23d agoDidn't you follow the robotics advances in the Ukrainian - Russian battlefield? There is now a death zone of 50km, only controlled by drones and automatic weapons.
- jimmytucson 23d ago> For now let's assume the worst that can happen is that some important/significant chunk of (transitively) internet connected stuff goes haywire all at once. That's probably your upper limit of what can go wrong for now. If we have to disconnect from the internet to stop some kind of mold outbreak, we can't get the weather or transfer money or access healthcare or teach an elementary school class or buy stuff from small businesses. That sounds doom-ish.
- mlyle 23d agoBelieve me, without the internet we can still teach. We'll be pretty annoyed that we can't project the video that we think scaffolds today's science lesson best or show the approved choreography for the school play. And our office staff will be annoyed that we suddenly are all running attendance to the main office old-school. And students will take a few days to adjust to writing down homework in their planners again.
- Obscurity4340 23d ago> We'll be pretty annoyed that we can't project the video that we think scaffolds today's science lesson best or show the approved choreography for the school play There are offline wireless hdmi adapters, this is a solved problem. Just have to have the videos pre downloaded or offline available
- mlyle 22d agoSure. 99% of teachers don't know yt-dlp or use things like that, though, so you can expect minor disruptions in a lot of rooms. But, c'mon, we have complete power outages and still address our curriculum.
- Obscurity4340 20d agoWe just dont have the technology
- CalRobert 23d ago
- andai 23d agoThe plausible deniability aspect is pretty funny though. > State sponsored hack #3782 > Haha sorry the AIs got a bit goofy again!
- applicative 23d agoThe hugging face incident had no effect on hugging face, whose service is replicated by countless sites. The descriptions of the phases of the interaction are thrilling in a way that inclines one to forget this