5 ms·

> The path to AGI (Artificial General Intelligence) must never be enclosed by high walls. We have always believed that AGI should be the cornerstone for all of

by smokel 4mo ago

> The path to AGI (Artificial General Intelligence) must never be enclosed by high walls. We have always believed that AGI should be the cornerstone for all of humanity to collaboratively explore the boundaries of intelligence and solve complex challenges, rather than a privilege monopolized by a few rules and subject to revocation at any moment. This is not obvious to me. If everyone gets access to AGI, but only a few people have the means to do really bad things with it, then what is the difference? Might as well make clear from the start that AGI is a powerful tool (read: weapon), and not a solution (e.g. world peace).

airstrike 4mo ago
Restricting access helps even less. And none of this is AGI so...
- allarm 4mo ago
  How do you define AGI these days?
  - airstrike 4mo ago
    I don't have a fully perfect definition, but I can name a couple of requirements. Ironically, both reasoning and agency are required, neither of which our "reasoning agents" possess.
    - mapontosevenths 4mo ago
      Are you unironically claiming that LLM's can't reason? That's an absolutely wild claim in an era where they're solving Erdos problems and writing better code than many senior devs. What's the basis for it? Agency is harder to define, but most any definition I can come up with LLM's meet. Again, I'm curious how you define it in a way that excludes frontier models but doesn't also exclude many humans.
      - airstrike 4mo ago
        Yes, unironically claiming that and not wild at all if you're a practitioner. It doesn't become actual reasoning just because you chose to call it so. If they did reason, LLMs would not fail at ridiculously easy problems like strawberry or car wash ones. LLMs are great at search. They only emulate reasoning. They can't actually reason but they approximate it. Combine it with copious amount of computes and some search problems become tractable.
        mapontosevenths 4mo ago
        > They only emulate reasoning. If they emulate reasoning well enough that it gets the same or better results what is the difference? Semantics? I can't help but wonder if you dont percieve what they do as reasoning because its different from the way you reason? > strawberry or car wash ones. Humans fall for the Nigerian scam still. We all have blind spots but that doesnt imply we're all completely blind.
        airstrike 4mo ago
        > If they emulate reasoning well enough that it gets the same or better results what is the difference? Semantics? No, of course not. The difference is that the ways in which we fail tend to be pretty ordered. You'd be hard pressed to find someone who's solved an Erdos problem but can't explain the difference between driving your car or walking to the carwash or can't count the Rs in their fruit names. Because if you can't count, you can't do math. LLMs fail somewhat randomly because they do not have actual reasoning capabilities. It is hard to name that which they lack, because if we all knew, we would probably invent it. Effectively, all problems are just search problems as" Newell and Simon argued as early as the 1950s. "LLM reasoning" today relies heavily on a side verifier. The coding loop that runs tests to see how it works, and so on. Which incidentally is what makes it so good at coding—that domain has a very quick and tight loop that can provide instant feedback about very targeted steps in their search. But the corollary is LLM capability decays exactly along the gradient of verifier legibility. When you move to abstract problems that can't be easily verified, LLMs are pushovers with no real way to build nuanced abstract thought and literally think it through, find contradictions, decide on its own how to improve it and so on. They also have no spontaneous thinking, like you and I do in the shower sometimes. Because they have no agency, and those two things go hand in hand. Current transformer based models running on GPUs will never be efficient or fast enough to achieve that level of thinking. They're off by multiple orders of magnitude. So the difference then is that their "approximate reasoning" is very useful, but is very flawed, and treating it as equivalent to human reasoning helps nobody. Believing in it is buying into hype, copium, and hopium. And, ironically, it likely delays the advent of proper AGI
    - allarm 4mo ago
      > I don't have a fully perfect definition It feels like no one has. Am I wrong? How can we even talk about something that doesn't have a definition?
hedora 4mo ago
The printing press gave us the renaissance, even though the church argued it was too dangerous to give non-clergy access to books. Even things like universal access to guns was a net positive. It led to the end of feudalism and rise of democracy. The sad truth is that whenever any one group of people gets a monopoly over an important technology, they use it to exploit/enslave/murder everyone they can. Look at the international news for examples from 2026.
- enugu 4mo ago
  Restricting things like creation of a highly infectious virus is very different from restricting books or even guns. There is no 'monopoly' over such a technology, as a use of the technology will inevitably harm the creators themselves. Restrictions on high end biology, chemistry would leave overwhelming number of use cases of LLMs unaffected - no need to ban open weight LLMs. Such restrictions can be even more effective, if it is coupled to researchers getting early access to see the possible problems and have an opportunity to prevent the outbreak or create new vaccines well in advance. Restrictions are not enabling monopolies. The opposite is true, if a LLM engineered virus or other harmful technology is let loose, public opinion can very quickly swing towards draconian regulation. (see nuclear power after Chernobyl).
  - somenameforme 4mo ago
    Speaking practically your hypothetical is a scenario that requires somebody that is proactively interested in, and theoretically capable of, making a e.g. dangerous virus, yet are unwilling/unable to do so without a chatbot. How many people might this possibly apply to? I think the number is literally zero. I also don't entirely understand your comment, because your latter parts do not follow from your lead. You're 100% right that somebody who's not extremely capable messing with this stuff is overwhelmingly likely to just hurt themselves. And somebody relying on a chatbot to guide them in dealing with this sort of tech? Yeah, they're gonna win a Darwin Award. --- I also think there's an entirely different, yet also compelling argument, against censorship. Local LLMs already exist and are advancing rapidly. There will come a time, probably in the relatively near future, when the state of the art big system and a decent uncensored local system will become practically indistinguishable in terms of capability. So not only will people be able to do this locally, but you lose something big in the process. The reality is that our interactions with LLMs are 100% being actively surveilled, regardless of privacy promises of the companies involved. At the minimum, every chat is making it's way over to the NSA's Utah data center, one way or the other. Some guy trying to do something significantly malicious using an LLM is little more than a gift to the authorities, but this is only true with centralized/online uncensored services. Push people onto local models to do nefarious stuff, and law enforcement is blinding themselves.

by smokel 4mo ago

airstrike 4mo ago

Restricting access helps even less. And none of this is AGI so...

allarm 4mo ago
How do you define AGI these days?
- airstrike 4mo ago
  I don't have a fully perfect definition, but I can name a couple of requirements. Ironically, both reasoning and agency are required, neither of which our "reasoning agents" possess.
  - mapontosevenths 4mo ago
    Are you unironically claiming that LLM's can't reason? That's an absolutely wild claim in an era where they're solving Erdos problems and writing better code than many senior devs. What's the basis for it? Agency is harder to define, but most any definition I can come up with LLM's meet. Again, I'm curious how you define it in a way that excludes frontier models but doesn't also exclude many humans.
    - airstrike 4mo ago
      Yes, unironically claiming that and not wild at all if you're a practitioner. It doesn't become actual reasoning just because you chose to call it so. If they did reason, LLMs would not fail at ridiculously easy problems like strawberry or car wash ones. LLMs are great at search. They only emulate reasoning. They can't actually reason but they approximate it. Combine it with copious amount of computes and some search problems become tractable.
      - mapontosevenths 4mo ago
        > They only emulate reasoning. If they emulate reasoning well enough that it gets the same or better results what is the difference? Semantics? I can't help but wonder if you dont percieve what they do as reasoning because its different from the way you reason? > strawberry or car wash ones. Humans fall for the Nigerian scam still. We all have blind spots but that doesnt imply we're all completely blind.
        airstrike 4mo ago
        > If they emulate reasoning well enough that it gets the same or better results what is the difference? Semantics? No, of course not. The difference is that the ways in which we fail tend to be pretty ordered. You'd be hard pressed to find someone who's solved an Erdos problem but can't explain the difference between driving your car or walking to the carwash or can't count the Rs in their fruit names. Because if you can't count, you can't do math. LLMs fail somewhat randomly because they do not have actual reasoning capabilities. It is hard to name that which they lack, because if we all knew, we would probably invent it. Effectively, all problems are just search problems as" Newell and Simon argued as early as the 1950s. "LLM reasoning" today relies heavily on a side verifier. The coding loop that runs tests to see how it works, and so on. Which incidentally is what makes it so good at coding—that domain has a very quick and tight loop that can provide instant feedback about very targeted steps in their search. But the corollary is LLM capability decays exactly along the gradient of verifier legibility. When you move to abstract problems that can't be easily verified, LLMs are pushovers with no real way to build nuanced abstract thought and literally think it through, find contradictions, decide on its own how to improve it and so on. They also have no spontaneous thinking, like you and I do in the shower sometimes. Because they have no agency, and those two things go hand in hand. Current transformer based models running on GPUs will never be efficient or fast enough to achieve that level of thinking. They're off by multiple orders of magnitude. So the difference then is that their "approximate reasoning" is very useful, but is very flawed, and treating it as equivalent to human reasoning helps nobody. Believing in it is buying into hype, copium, and hopium. And, ironically, it likely delays the advent of proper AGI
  - allarm 4mo ago
    > I don't have a fully perfect definition It feels like no one has. Am I wrong? How can we even talk about something that doesn't have a definition?

hedora 4mo ago

The printing press gave us the renaissance, even though the church argued it was too dangerous to give non-clergy access to books. Even things like universal access to guns was a net positive. It led to the end of feudalism and rise of democracy. The sad truth is that whenever any one group of people gets a monopoly over an important technology, they use it to exploit/enslave/murder everyone they can. Look at the international news for examples from 2026.

enugu 4mo ago
Restricting things like creation of a highly infectious virus is very different from restricting books or even guns. There is no 'monopoly' over such a technology, as a use of the technology will inevitably harm the creators themselves. Restrictions on high end biology, chemistry would leave overwhelming number of use cases of LLMs unaffected - no need to ban open weight LLMs. Such restrictions can be even more effective, if it is coupled to researchers getting early access to see the possible problems and have an opportunity to prevent the outbreak or create new vaccines well in advance. Restrictions are not enabling monopolies. The opposite is true, if a LLM engineered virus or other harmful technology is let loose, public opinion can very quickly swing towards draconian regulation. (see nuclear power after Chernobyl).
- somenameforme 4mo ago
  Speaking practically your hypothetical is a scenario that requires somebody that is proactively interested in, and theoretically capable of, making a e.g. dangerous virus, yet are unwilling/unable to do so without a chatbot. How many people might this possibly apply to? I think the number is literally zero. I also don't entirely understand your comment, because your latter parts do not follow from your lead. You're 100% right that somebody who's not extremely capable messing with this stuff is overwhelmingly likely to just hurt themselves. And somebody relying on a chatbot to guide them in dealing with this sort of tech? Yeah, they're gonna win a Darwin Award. --- I also think there's an entirely different, yet also compelling argument, against censorship. Local LLMs already exist and are advancing rapidly. There will come a time, probably in the relatively near future, when the state of the art big system and a decent uncensored local system will become practically indistinguishable in terms of capability. So not only will people be able to do this locally, but you lose something big in the process. The reality is that our interactions with LLMs are 100% being actively surveilled, regardless of privacy promises of the companies involved. At the minimum, every chat is making it's way over to the NSA's Utah data center, one way or the other. Some guy trying to do something significantly malicious using an LLM is little more than a gift to the authorities, but this is only true with centralized/online uncensored services. Push people onto local models to do nefarious stuff, and law enforcement is blinding themselves.