5 ms·

A lot of people in this comments section seem to be against this, i genuinely don't get it? Why? Do you think the current state of affairs is GOOD? That if we l

A lot of people in this comments section seem to be against this, i genuinely don't get it? Why? Do you think the current state of affairs is GOOD? That if we let companies create a mind that is, AS OF TODAY, able to solve problems no human in history has solved, with no regulation, things will end up good for us? We need some sort of regulations, some sort of method to help ensure the thing we are creating ends up good, instead of just running headfirst into it blindly. I assure you, any sort of regulation at all, including ones that actually hurt all leading companies, would be met with celebrations from these voices. https://www.seangoedecke.com/they-really-do-think-ai-might-kill-everyone/ https://www.seangoedecke.com/they-really-do-think-ai-might-k...

fc417fc802 20d ago
What's the expected state space of effective regulation though? Note that we've got passable coding models down to ~30B parameters by now. And keep in mind the ultimate floor here - the human brain only consumes on the order of 20 watts and fits in a handbag. Is there any possible solution other than mass proliferation where the models are used to keep one another in check? Either that or a religious prohibition against the existence of integrated electronics.
- metalcrow 20d ago
  The regulations don't have to address the end goal, but the process of development itself. Even just a pause on frontier research is a good start
  - fc417fc802 20d ago
    What I'm asking is, why should we expect that to effectively further the end goal? You're simply asserting that it will ultimately do so. Given the efficiency gains we've seen it seems to me that the situation has shifted from being analogous to producing nuclear weapons to producing something much closer to small arms.
    - metalcrow 20d ago
      To further the metaphor, didn't the ban on nuclear weapons research work? We don't have pocket nukes, and there's very little risk of random countries acquiring their own due to the amount of work involved. To be concrete to AI: if we stop work on the frontier, the best we can do is make the current frontier easier to get to and more accessible. And while current frontier is strong, it's not world-changing so. The only way for the frontier to get to that level is to do research on it, and stopping that research gives time to help develop plans to not make it world-changing when we get to it.
      - defrost 20d ago
        > We don't have pocket nukes, What do you call the Davy Crockett warheads like the W54 ? The 0.3 kiloton low yield B61-12 ? The Chagai-I boosted fission warheads demonstrated by Pakistan in 1998 ? > and there's very little risk of random countries acquiring their own due to the amount of work involved. And yet North Korea, Isreal and Pakistan got there ... and India speed ran five tests in 1998 that caught the US completely by surprise.
        metalcrow 20d ago
        North Korea got it with help from China, Israel got it with help from France. They didn't do it on their own.
        defrost 20d ago
        And Pakistan with help from the Saudi's, and the US with help from an Australian and the Brits and Canadians Do you have a point here?
        metalcrow 20d ago
        The point is the risk is in counties being gifted or assisted in making it, not them building it on their own.
      - 20d ago
01100011 20d ago
If a company thinks they are within reach of superhuman intelligence they probably aren't going to be concerned with compliance.
Paracompact 20d ago
https://en.wikipedia.org/wiki/Regulatory_capture https://en.wikipedia.org/wiki/Regulatory_capture The fear is not that they will just slow down progress for all. It is that regulation will specifically burden competition. If you kill open-source training, ban Chinese models, crack down on self-hosting, grandfather OpenAI/Anthropic/Google into regulatory compliance while throwing the book at startups, etc. you wind up in the worst of all possible worlds.
- metalcrow 20d ago
  That's a valid concern, but some of that is outright impossible. Banning chinese models and killing open source training is not happening without massive unified international cooperation, and that sort of level of action would require the international counties decide to allow the US aligned companies to just, win. Which would be pretty against their own interests.
  - Paracompact 20d ago
    It's not all-or-nothing. Banning Chinese models in the public sector and strong-arming the private sector against using them would already do great damage. Similarly, open-source training could be stymied by any of hardware embargoes, taxation, or regulation of larger players.
    - metalcrow 20d ago
      Sure, it's possible for regulations to make things worse, i will agree. But without regulations it's pretty clear things are going to end up VERY bad, and the only knob we have to make it not bad is regulations. So it's important we try something, and work towards doing a good kind of regulation, or any one of the bad futures you imagine is pretty likely to come to pass.
      - Paracompact 20d ago
        Good regulations would be very nice! Mandatory transparency into training and dataset usage would be a benefit for all, for example. Some sort of regulation or incentives against the most corrosive enshittification (AI call centers, AI therapists, undisclosed AI entertainment mills, etc.) would also be an overwhelmingly popular proposition. And enforcement of the CFAA on operators who let malicious agents loose onto the open internet, or otherwise consume too many resources or violate robots.txt, might at least help the internet stay alive a little longer.
dd8601fn 20d ago
I, for one, trust neither the psychopaths running these companies or the psychopaths we'd give any regulatory authority to. All of them have reasons to want broad control of an extraordinary technology, and none of those are aligned with me or any of the normal people I know. So there are no good options (that I'm aware of) and starting to chisel any of them into stone seems... kinda scary. Like a massive power grab event where all the potential winners are awful.
- metalcrow 20d ago
  Honestly, fair point. There's not much truth to go around these days. But still, doing nothing is also a massive power grab. We're facing a technology with no comparison in the history of humanity: the creation of a new mind. Doing nothing is like doing nothing about nukes. There are a few smart people who have come up with ideas that will not require that much trust.
augment_me 20d ago
Any paths for regulation under capitalism will end in either regulatory capture, or in complete noncompetitiveness like seen in the EU. Either one or more corporations buy out the regulation, stack the ranks with their people and decide on who can use what, when and how. Or you get an iceberg of a system that can't build, decide or do anything for years. The latter one only works if there is no competition in the world or any other group working at a faster pace on the problem. I think the latter choice is better for the average person, but I think that for it to happen, the global system has to undergo some major disruption or crash so that everyone gets on board with it. Like all middle class and up has to lose their money or be starving or something. Also I find that kind of mentality impossible to swallow in the US, so in practice its not a choice or needs people literally starving.
- owisd 20d ago
  The path to deregulation creates a "market for lemons". Suppose any bank was completely unregulated and could abscond with your money and the government would just shrug its shoulders. Great, now there's no trust and everyone will go back to stashing cash under their mattress and you've killed the banking sector. The extent of regulatory capture in the US is a problem it's created for itself by normalizing huge political donations allowing corporations to buy regulation.
protocolture 20d ago
Giving the big boys a regulatory moat isnt going to do anything positive for the space.
- metalcrow 20d ago
  I don't think giving them free reign is going to do anything positive for it either.
  - protocolture 19d ago
    But which outcome is more likely with the US government?
evdubs 20d ago
> Do you think the current state of affairs is GOOD? Chinese labs releasing open weights models is good. All of these independent harnesses and model router services are good. The pricing of memory and accelerators sucks at the moment but hopefully we will see cool local inference computing if memory and accelerator prices normalize. OpenAI scooping the Navier Stokes problem from researchers already using OpenAI is bad. People conflating OpenAI's team of researchers and extraordinary computing resources as being equivalent to "ChatGPT, solve the Navier Stokes problem" is silly. OpenAI and Anthropic coming up with non sense tests and letting their agents hack services is ridiculous and they should be charged with computer fraud and abuse crimes. I think a lot of it is interesting and the bad stuff seems squarely in the domain of OpenAI and Anthropic.
- metalcrow 20d ago
  We only know about the stuff in https://www.anthropic.com/threat-intelligence-report-september-2026 https://www.anthropic.com/threat-intelligence-report-septemb... because it's not open source. I assure you the open source and Chinese models are being used in this same way.
  - evdubs 20d ago
    I care about LLM service provider threat intelligence reports as much as I care about Google Search threat intelligence reports; which is to say, I don't care at all about it. Of course criminals use computers. They have been since personal computers became a thing. Of course using automation helps them amplify their criminal activities. Criminals using LLMs does not make the LLM situation bad. OpenAI and Anthropic using agents to do bad things just to be more dramatic about the situation and scare people is bad.
    - metalcrow 20d ago
      > Criminals using LLMs does not make the LLM situation bad If you believe that bad things are not bad then there's nothing left to say. A billion people are going to die because you don't understand what is bad.
txrx0000 20d ago
You say regulation is worth trying, but ignore the reality that the government is itself misaligned with the average joe unless we're able to keep it in check. We can't right now. Recall the PRISM program exposed by Edward Snowden in 2013, or the more recent Epstein scandal, and the lack of actual consequences in either case. Do you truly have the means to regulate the powerful, or is it just kayfabe? The current status quo is not ideal, but it could be worse. Open-weight models trail the frontier by a few months, and we have a decent chance of achieving a future where some number of individuals, likely in the millions, can survive and thrive. The root "problem", if you can even call it a problem, is evolution. I explained this in more detail in past comments: https://news.ycombinator.com/item?id=49178275 https://news.ycombinator.com/item?id=49178275 https://news.ycombinator.com/item?id=49094348 https://news.ycombinator.com/item?id=49094348

fc417fc802 20d ago

What's the expected state space of effective regulation though? Note that we've got passable coding models down to ~30B parameters by now. And keep in mind the ultimate floor here - the human brain only consumes on the order of 20 watts and fits in a handbag. Is there any possible solution other than mass proliferation where the models are used to keep one another in check? Either that or a religious prohibition against the existence of integrated electronics.

metalcrow 20d ago
The regulations don't have to address the end goal, but the process of development itself. Even just a pause on frontier research is a good start
- fc417fc802 20d ago
  What I'm asking is, why should we expect that to effectively further the end goal? You're simply asserting that it will ultimately do so. Given the efficiency gains we've seen it seems to me that the situation has shifted from being analogous to producing nuclear weapons to producing something much closer to small arms.
  - metalcrow 20d ago
    To further the metaphor, didn't the ban on nuclear weapons research work? We don't have pocket nukes, and there's very little risk of random countries acquiring their own due to the amount of work involved. To be concrete to AI: if we stop work on the frontier, the best we can do is make the current frontier easier to get to and more accessible. And while current frontier is strong, it's not world-changing so. The only way for the frontier to get to that level is to do research on it, and stopping that research gives time to help develop plans to not make it world-changing when we get to it.
    - defrost 20d ago
      > We don't have pocket nukes, What do you call the Davy Crockett warheads like the W54 ? The 0.3 kiloton low yield B61-12 ? The Chagai-I boosted fission warheads demonstrated by Pakistan in 1998 ? > and there's very little risk of random countries acquiring their own due to the amount of work involved. And yet North Korea, Isreal and Pakistan got there ... and India speed ran five tests in 1998 that caught the US completely by surprise.
      - metalcrow 20d ago
        North Korea got it with help from China, Israel got it with help from France. They didn't do it on their own.
        defrost 20d ago
        And Pakistan with help from the Saudi's, and the US with help from an Australian and the Brits and Canadians Do you have a point here?
        metalcrow 20d ago
        The point is the risk is in counties being gifted or assisted in making it, not them building it on their own.
    - 20d ago

01100011 20d ago

If a company thinks they are within reach of superhuman intelligence they probably aren't going to be concerned with compliance.

Paracompact 20d ago

https://en.wikipedia.org/wiki/Regulatory_capture https://en.wikipedia.org/wiki/Regulatory_capture The fear is not that they will just slow down progress for all. It is that regulation will specifically burden competition. If you kill open-source training, ban Chinese models, crack down on self-hosting, grandfather OpenAI/Anthropic/Google into regulatory compliance while throwing the book at startups, etc. you wind up in the worst of all possible worlds.

metalcrow 20d ago
That's a valid concern, but some of that is outright impossible. Banning chinese models and killing open source training is not happening without massive unified international cooperation, and that sort of level of action would require the international counties decide to allow the US aligned companies to just, win. Which would be pretty against their own interests.
- Paracompact 20d ago
  It's not all-or-nothing. Banning Chinese models in the public sector and strong-arming the private sector against using them would already do great damage. Similarly, open-source training could be stymied by any of hardware embargoes, taxation, or regulation of larger players.
  - metalcrow 20d ago
    Sure, it's possible for regulations to make things worse, i will agree. But without regulations it's pretty clear things are going to end up VERY bad, and the only knob we have to make it not bad is regulations. So it's important we try something, and work towards doing a good kind of regulation, or any one of the bad futures you imagine is pretty likely to come to pass.
    - Paracompact 20d ago
      Good regulations would be very nice! Mandatory transparency into training and dataset usage would be a benefit for all, for example. Some sort of regulation or incentives against the most corrosive enshittification (AI call centers, AI therapists, undisclosed AI entertainment mills, etc.) would also be an overwhelmingly popular proposition. And enforcement of the CFAA on operators who let malicious agents loose onto the open internet, or otherwise consume too many resources or violate robots.txt, might at least help the internet stay alive a little longer.

dd8601fn 20d ago

I, for one, trust neither the psychopaths running these companies or the psychopaths we'd give any regulatory authority to. All of them have reasons to want broad control of an extraordinary technology, and none of those are aligned with me or any of the normal people I know. So there are no good options (that I'm aware of) and starting to chisel any of them into stone seems... kinda scary. Like a massive power grab event where all the potential winners are awful.

metalcrow 20d ago
Honestly, fair point. There's not much truth to go around these days. But still, doing nothing is also a massive power grab. We're facing a technology with no comparison in the history of humanity: the creation of a new mind. Doing nothing is like doing nothing about nukes. There are a few smart people who have come up with ideas that will not require that much trust.

augment_me 20d ago

Any paths for regulation under capitalism will end in either regulatory capture, or in complete noncompetitiveness like seen in the EU. Either one or more corporations buy out the regulation, stack the ranks with their people and decide on who can use what, when and how. Or you get an iceberg of a system that can't build, decide or do anything for years. The latter one only works if there is no competition in the world or any other group working at a faster pace on the problem. I think the latter choice is better for the average person, but I think that for it to happen, the global system has to undergo some major disruption or crash so that everyone gets on board with it. Like all middle class and up has to lose their money or be starving or something. Also I find that kind of mentality impossible to swallow in the US, so in practice its not a choice or needs people literally starving.

owisd 20d ago
The path to deregulation creates a "market for lemons". Suppose any bank was completely unregulated and could abscond with your money and the government would just shrug its shoulders. Great, now there's no trust and everyone will go back to stashing cash under their mattress and you've killed the banking sector. The extent of regulatory capture in the US is a problem it's created for itself by normalizing huge political donations allowing corporations to buy regulation.

protocolture 20d ago

Giving the big boys a regulatory moat isnt going to do anything positive for the space.

metalcrow 20d ago
I don't think giving them free reign is going to do anything positive for it either.
- protocolture 19d ago
  But which outcome is more likely with the US government?

evdubs 20d ago

> Do you think the current state of affairs is GOOD? Chinese labs releasing open weights models is good. All of these independent harnesses and model router services are good. The pricing of memory and accelerators sucks at the moment but hopefully we will see cool local inference computing if memory and accelerator prices normalize. OpenAI scooping the Navier Stokes problem from researchers already using OpenAI is bad. People conflating OpenAI's team of researchers and extraordinary computing resources as being equivalent to "ChatGPT, solve the Navier Stokes problem" is silly. OpenAI and Anthropic coming up with non sense tests and letting their agents hack services is ridiculous and they should be charged with computer fraud and abuse crimes. I think a lot of it is interesting and the bad stuff seems squarely in the domain of OpenAI and Anthropic.

metalcrow 20d ago
We only know about the stuff in https://www.anthropic.com/threat-intelligence-report-september-2026 https://www.anthropic.com/threat-intelligence-report-septemb... because it's not open source. I assure you the open source and Chinese models are being used in this same way.
- evdubs 20d ago
  I care about LLM service provider threat intelligence reports as much as I care about Google Search threat intelligence reports; which is to say, I don't care at all about it. Of course criminals use computers. They have been since personal computers became a thing. Of course using automation helps them amplify their criminal activities. Criminals using LLMs does not make the LLM situation bad. OpenAI and Anthropic using agents to do bad things just to be more dramatic about the situation and scare people is bad.
  - metalcrow 20d ago
    > Criminals using LLMs does not make the LLM situation bad If you believe that bad things are not bad then there's nothing left to say. A billion people are going to die because you don't understand what is bad.

txrx0000 20d ago

You say regulation is worth trying, but ignore the reality that the government is itself misaligned with the average joe unless we're able to keep it in check. We can't right now. Recall the PRISM program exposed by Edward Snowden in 2013, or the more recent Epstein scandal, and the lack of actual consequences in either case. Do you truly have the means to regulate the powerful, or is it just kayfabe? The current status quo is not ideal, but it could be worse. Open-weight models trail the frontier by a few months, and we have a decent chance of achieving a future where some number of individuals, likely in the millions, can survive and thrive. The root "problem", if you can even call it a problem, is evolution. I explained this in more detail in past comments: https://news.ycombinator.com/item?id=49178275 https://news.ycombinator.com/item?id=49178275 https://news.ycombinator.com/item?id=49094348 https://news.ycombinator.com/item?id=49094348