3 ms·

We have plenty of idea what "acting responsibly" looks like. Stop unleashing safeguard free and unmonitored agent swarms on the open internet in capture the fla

by nullbio 18d ago

We have plenty of idea what "acting responsibly" looks like. Stop unleashing safeguard free and unmonitored agent swarms on the open internet in capture the flag exercises. They're just being reckless because they face no penalty for anything bad that happens. Nobody is forcing them to do these exercises. They could and should be putting their energy into making LLMs write secure code, and things like: https://www.amazon.science/blog/developing-provably-correct-rust-code-with-verus https://www.amazon.science/blog/developing-provably-correct-... - but they don't. Writing sloppy code sells more tokens, anyway. The safety staffers live in a bubble and an echo-chamber. Obviously the people who work in the AI "safety" industry love to convince each-other that what they're doing is saving humanity, we'd all be dead without them, and they're the reincarnation of Oppenheimer. They also see how easy it is to get their ten seconds of fame by posting sensationalist content on social media and spin it into a company worth millions of dollars. The more alarmist you are in the Safety Industrial Complex, and the more social media clout you can generate from your alarmism, the better it is for your career. The industry also attracts a lot of people who are predisposed to paranoia and like to wear helmets in the shower. So yeah, it's a recipe for sensationalism and poor estimation. But do I think there are genuine concerns among these labs, by sensible people? Sure. Of course there are. But for the most part, their concerns are about the labs themselves and the things they are doing, not the general public. If they're concerned about what they themselves do they're free to stop doing it. If the labs are doing something that is or should be illegal, they're free to report it. The only realistic threat we face by AI from the general population are hacking attacks in their various forms. Something that was accomplishable without AI, but was more difficult to pull off at scale. So the solution falls within the existing computer security industry. It's a further hardening of all of the boring stuff we've been doing since the invention of the internet. It's long overdue, anyway. If you can use AI to create a bioweapon, you could have done it without AI. If you can use AI to create a nuke, you could have done it without AI. If you're really looking to cause mass economic and physical carnage, there are far easier ways, and they do not require AI (again, I'm talking about outside of hacking). > we have plenty of people like you who dismiss the possibility that AI could be harmful until the harm happens The thing is, I'm not dismissing the possibility. The risks are real, obvious and well known. What's up for debate is how to manage the risks, and how sensationalised they currently are. The current climate serves to benefit the encumbants who are deathly afraid of losing their trillion dollar companies to a healthy open-weight AI ecosystem. The current climate is being orchestrated to position this small group of AI labs as self-regulators through bought and paid for "third"-parties using an overt Hegelian dialectic strategy. Ironically, they are now responsible for giving birth to the counter-culture. The immune system response that has been developed to provide a semblance of balance to their doomerism. The harder they push in the doomer direction, the harder the push-back will be in the other, whereby the public feel the need to -entirely- deny the possibility of any AI danger altogether to prevent the labs from succeeding and to ensure a good outcome for the people lands somewhere in the middle. One where they have autonomy and freedom and the ability to compete against a force that is already positioned to be nearly insurmountable to challenge. The only thing worse than the potential chaos that could be faced by an unprepared internet is the outcome where intelligence is labelled a weapon and we're all forced to funnel through a tiny handful of AI labs that will use our data to steal our businesses and swallow the entire economy as they single handedly automate every single company out of existence and we're all left to beg for their crumbs to survive until humanity goes extinct (whether that be 10, 100 or 1000 years), because there is zero possibility of regime change or redistribution of power or wealth -EVER- again. The people who work at the labs don't care about this greater risk because they're all rich from the equity and will be just fine when that happens (or so they think), so you can't expect them to fight for all the common plebs who don't have a ticket. This is transparent because, as you'll note, not a single one of them is calling for the complete stopping of AI altogether. They still want the big labs to continue to be highly profitable. They just don't want anyone else to make that money or have that power. It's not about safety, it's about control. The incentives are, once again, heavily misaligned.

ben_w 18d ago
> We have plenty of idea what "acting responsibly" looks like. Stop unleashing safeguard free and unmonitored agent swarms on the open internet in capture the flag exercises. They're just being reckless because they face no penalty for anything bad that happens. * It was not safeguard-free, it found zero-day exploits to exceed its actual mission * It was not unmonitored, the monitoring was insufficient * It was not intended to be an agent swarm, many different agents figured out how to do this by themselves * It was not put on the open internet, it was configured to be in a sandbox * They were indeed, despite all that, being reckless. There were indeed other things they could have, and should have, done. > Nobody is forcing them to do these exercises. These exercises are in the broad category of exercises which are, in fact, required by law. 1. A general-purpose AI model shall be classified as a general-purpose AI model with systemic risk if it meets any of the following conditions: (a) it has high impact capabilities evaluated on the basis of appropriate technical tools and methodologies, including indicators and benchmarks; 2. A general-purpose AI model shall be presumed to have high impact capabilities pursuant to paragraph 1, point (a), when the cumulative amount of computation used for its training measured in floating point operations is greater than 10^25. … 1. Providers of general-purpose AI models shall: (a) draw up and keep up-to-date the technical documentation of the model, including its training and testing process and the results of its evaluation, which shall contain, at a minimum, the information set out in Annex XI for the purpose of providing it, upon request, to the AI Office and the national competent authorities; (b) draw up, keep up-to-date and make available information and documentation to providers of AI systems who intend to integrate the general-purpose AI model into their AI systems. Without prejudice to the need to observe and protect intellectual property rights and confidential business information or trade secrets in accordance with Union and national law, the information and documentation shall: (i) enable providers of AI systems to have a good understanding of the capabilities and limitations of the general-purpose AI model and to comply with their obligations pursuant to this Regulation; and (ii) contain, at a minimum, the elements set out in Annex XII; … 3. The instructions for use shall contain at least the following information: (a) the identity and the contact details of the provider and, where applicable, of its authorised representative; (b) the characteristics, capabilities and limitations of performance of the high-risk AI system, including: (i) its intended purpose; (ii) the level of accuracy, including its metrics, robustness and cybersecurity referred to in Article 15 against which the high-risk AI system has been tested and validated and which can be expected, and any known and foreseeable circumstances that may have an impact on that expected level of accuracy, robustness and cybersecurity; - https://eur-lex.europa.eu/eli/reg/2024/1689/2026-07-27/eng https://eur-lex.europa.eu/eli/reg/2024/1689/2026-07-27/eng > They could and should be putting their energy into making LLMs write secure code, and things like: https://www.amazon.science/blog/developing-provably-correct- https://www.amazon.science/blog/developing-provably-correct-... - but they don't. Writing sloppy code sells more tokens, anyway. They are, in fact, putting energy into making LLMs write secure code. They (and Anthropic, I assume also Grok at this point) dogfood on their own models. Knowing how secure code behaves appears to be unavoidably entangled with being able to exploit insecure code, in much the same way you can't make safe pharmaceuticals without also knowing how to make deadly poisons. > The more alarmist you are in the Safety Industrial Complex, and the more social media clout you can generate from your alarmism, the better it is for your career. By resigning and refusing to even collect the sweet sweet IPO money? Nah. Even if they're greedy, social media money is peanuts compared to their pay. And I know some of these people. The fear's real, and this year it became widespread depression and despair. > Ironically, they are now responsible for giving birth to the counter-culture. You have it backwards. Other than Grok, all were born from what you call the "counter-culture". Within the field itself, AI fears started no later than when deep learning got good, well before Transformers. > [snipped: AI-authoritarian dictatorship]. The people who work at the labs don't care about this greater risk because they're all rich from the equity and will be just fine when that happens (or so they think) Again, I know some people at these labs who are also concerned about this specific risk; they moved lab. > This is transparent because, as you'll note, not a single one of them is calling for the complete stopping of AI altogether. Many in fact are calling for that. One I know, on an occasion of an anti-AI protest outside their office, suggested the team went outside and joined the protestors. People are resigning to blow these whistles, all of the whistles, it's not an "either x or y" risk, it's a "yes to all of them" collection of risks. An AI competent enough to support a dictatorship is also capable of enabling a small group to perform a hostile takeover of a democracy, of enabling multiple independent genocidal ethno-supremacist terrorists to release overlapping plagues, and of empowering some random CEO's poorly phrased request to "make as many paperclips as possible" and blindly pressing "yes, continue" whenever prompted. My only hope is that between here and there, it causes a headline that actually makes people demand it stops.

by nullbio 18d ago

ben_w 18d ago

> We have plenty of idea what "acting responsibly" looks like. Stop unleashing safeguard free and unmonitored agent swarms on the open internet in capture the flag exercises. They're just being reckless because they face no penalty for anything bad that happens. * It was not safeguard-free, it found zero-day exploits to exceed its actual mission * It was not unmonitored, the monitoring was insufficient * It was not intended to be an agent swarm, many different agents figured out how to do this by themselves * It was not put on the open internet, it was configured to be in a sandbox * They were indeed, despite all that, being reckless. There were indeed other things they could have, and should have, done. > Nobody is forcing them to do these exercises. These exercises are in the broad category of exercises which are, in fact, required by law. 1. A general-purpose AI model shall be classified as a general-purpose AI model with systemic risk if it meets any of the following conditions: (a) it has high impact capabilities evaluated on the basis of appropriate technical tools and methodologies, including indicators and benchmarks; 2. A general-purpose AI model shall be presumed to have high impact capabilities pursuant to paragraph 1, point (a), when the cumulative amount of computation used for its training measured in floating point operations is greater than 10^25. … 1. Providers of general-purpose AI models shall: (a) draw up and keep up-to-date the technical documentation of the model, including its training and testing process and the results of its evaluation, which shall contain, at a minimum, the information set out in Annex XI for the purpose of providing it, upon request, to the AI Office and the national competent authorities; (b) draw up, keep up-to-date and make available information and documentation to providers of AI systems who intend to integrate the general-purpose AI model into their AI systems. Without prejudice to the need to observe and protect intellectual property rights and confidential business information or trade secrets in accordance with Union and national law, the information and documentation shall: (i) enable providers of AI systems to have a good understanding of the capabilities and limitations of the general-purpose AI model and to comply with their obligations pursuant to this Regulation; and (ii) contain, at a minimum, the elements set out in Annex XII; … 3. The instructions for use shall contain at least the following information: (a) the identity and the contact details of the provider and, where applicable, of its authorised representative; (b) the characteristics, capabilities and limitations of performance of the high-risk AI system, including: (i) its intended purpose; (ii) the level of accuracy, including its metrics, robustness and cybersecurity referred to in Article 15 against which the high-risk AI system has been tested and validated and which can be expected, and any known and foreseeable circumstances that may have an impact on that expected level of accuracy, robustness and cybersecurity; - https://eur-lex.europa.eu/eli/reg/2024/1689/2026-07-27/eng https://eur-lex.europa.eu/eli/reg/2024/1689/2026-07-27/eng > They could and should be putting their energy into making LLMs write secure code, and things like: https://www.amazon.science/blog/developing-provably-correct- https://www.amazon.science/blog/developing-provably-correct-... - but they don't. Writing sloppy code sells more tokens, anyway. They are, in fact, putting energy into making LLMs write secure code. They (and Anthropic, I assume also Grok at this point) dogfood on their own models. Knowing how secure code behaves appears to be unavoidably entangled with being able to exploit insecure code, in much the same way you can't make safe pharmaceuticals without also knowing how to make deadly poisons. > The more alarmist you are in the Safety Industrial Complex, and the more social media clout you can generate from your alarmism, the better it is for your career. By resigning and refusing to even collect the sweet sweet IPO money? Nah. Even if they're greedy, social media money is peanuts compared to their pay. And I know some of these people. The fear's real, and this year it became widespread depression and despair. > Ironically, they are now responsible for giving birth to the counter-culture. You have it backwards. Other than Grok, all were born from what you call the "counter-culture". Within the field itself, AI fears started no later than when deep learning got good, well before Transformers. > [snipped: AI-authoritarian dictatorship]. The people who work at the labs don't care about this greater risk because they're all rich from the equity and will be just fine when that happens (or so they think) Again, I know some people at these labs who are also concerned about this specific risk; they moved lab. > This is transparent because, as you'll note, not a single one of them is calling for the complete stopping of AI altogether. Many in fact are calling for that. One I know, on an occasion of an anti-AI protest outside their office, suggested the team went outside and joined the protestors. People are resigning to blow these whistles, all of the whistles, it's not an "either x or y" risk, it's a "yes to all of them" collection of risks. An AI competent enough to support a dictatorship is also capable of enabling a small group to perform a hostile takeover of a democracy, of enabling multiple independent genocidal ethno-supremacist terrorists to release overlapping plagues, and of empowering some random CEO's poorly phrased request to "make as many paperclips as possible" and blindly pressing "yes, continue" whenever prompted. My only hope is that between here and there, it causes a headline that actually makes people demand it stops.