5 ms·

I'm sure some people will have issue with my phrasing but, honest question: Are there examples of where we have collective decided not to pursue knowledge? Suc

I'm sure some people will have issue with my phrasing but, honest question: Are there examples of where we have collective decided not to pursue knowledge? Successfully? I guess nuclear weapons might be the best example though research doesn't seem have to actually "stopped" as much as gone underground and we still have country trying to climb that ladder. But I don't know how relevant that is to LLMs/AI. It almost feels like pandora's box is open and our only option is continue to improve them. There is clearly value in what they do and while I can absolutely see the dangers, for example: authoritative governments and surveillance, I'm not convinced to throw the baby out with the bathwater. All of technology back to the printing press (and probably before that) could also be said to make it easier for governments to oppress their citizens. Making laws (and enforcing them!) to prevent governments from doing these things feels like that route forward, not trying to stick our heads in the sand. Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.

Dig1t 3mo ago
Taking the approach with AI that we took with the atomic bomb would be catastrophic. If the only people who are allowed to use this technology are governments, intelligence agencies, and a select few anointed companies, then the risk of authoritarian misuse will skyrocket. I worry that any attempt to limit their use and development will be abused and misdirected. We are already seeing people like Anthropic doing this, they are trying to use anti-AI sentiment to engage in regulatory capture. Go watch Dario’s speeches about how open weight models are dangerous and how they are “not really open”. Everyone can see that much of this “safety” conversation is ultimately just a tactic to shut potential competitors out of the market and establish a monopoly/duopoly.
- joshstrange 3mo ago
  I agree. I guess I should have said something like: "Stopping" LLM research just means it will be in the hands of a few who can abuse it. I'd rather a state of M.A.D. but instead of a handful of countries/governments it's millions/billions of people with access to the models (open ideally). Again, perhaps horribly naive or misguided, I understand that bioterrorism could (is?) a real problem as well as more "mundane" things like building a bomb (nuclear or otherwise). I just feel like limiting access to governments or "blessed" entities is even worse.
rayiner 3mo ago
No, of course not. That would be an insane trust fall. Even relatively small advances in technology give a country world dominating power. Fun fact: India was militarily superior to Britain in the 1600s—a gunpowder empire with a million soldiers—but was taken over by it in the 1700s. Britain’s edge was small: lighter, more maneuverable cannons, standardized ammunition, better military and political organization. Not a first world country versus a third world country—more like the dynamic US versus a sclerotic EU. And that modest edge led to 200 years of colonization. If we slow down on ASI voluntarily we’d be allowing a gap to open up that would make the difference between colonial europe and colonized Asia/Africa look trivial. It would be insane.
- jst1fthsdys 3mo ago
  You overstate the advantages of technology. Mughal India was fragmented and on a sharp decline. The British used politics, finance, and treachery to divide and conquer what was remaining.
- reducesuffering 3mo ago
  > we’d be allowing a gap to open up that would make the difference between colonial europe and colonized Asia/Africa look trivial. An easy choice to make if the alternative is everyone dying instead.
  - rayiner 3mo ago
    The alternative isn’t “everyone dying.” It’s us holding all the cards.
    - hollerith 3mo ago
      Why? Is human extinction not permitted by the laws of physics?
      - rayiner 3mo ago
        That’s a possibility, but not the only one. The two most realistic ones are: we race ahead and maintain our status, or we slow down and open ourselves up to colonization.
kokotajlod 3mo ago
If you read the scenario, you'll see that the regulations are mostly about what people can do with giant compute clusters, and not about the ideas themselves. The ideas themselves are required to be totally transparent to the public. As for historic precedents: Human cloning, human genome editing, and mirror life seem like one precedent; nuclear weapons and nuclear energy another; come to think of it I think drone delivery was strangled by regulations too...? Plan A isn't a proposal to never build superintelligence, it's a proposal to build it more cautiously and transparently.
- ACCount37 3mo ago
  Human cloning and genetic editing isn't stalled because we agreed it's unethical. It's not pursued because at the current level of advancements, it's pretty useless. The things we can do there are niche. It's easy to ban something that's not very useful. If we had a way to make gene edited humans a lot smarter, a lot stronger or live a lot longer? Or a way to quick-grow human bodies to adulthood in a couple years? Capabilities that private actors or countries may want, ethics be damned? That would be closer to what we have with AI right now.
  - fragmede 3mo ago
    It's not useless. We know enough about DNA to be able to make better humans, but people get real squickly real fast when you talk about that. It has stalled because of that. If we go in and just check the DNA for Downs or Parkinson's, we can't have a conversation on aborting the foetus without religious beliefs coming into play. Designer children aren't a thing, despite the ability to edit DNA to do specific things. Hair and eye color are easy enough to go in and edit for. Humanity has decided to opt out of doing that for now.
    - bglazer 3mo ago
      > If we go in and just check the DNA for Downs or Parkinson's You should consider reading the wikipedia page about Parkinson’s disease.
hollerith 3mo ago
Yes, there are examples of where we have collective decided not to pursue a particular technology tree. For one, Japan banned guns for a few centuries. (Its warrior class was politically powerful and judged that guns would disrupt class relations too much.) And there have been successful world-wide bans. For example, following the invention of recombinant DNA technology, scientists convened the Asilomar Conference in 1975. They established a voluntary self-moratorium on certain types of genetic engineering until strict laboratory containment protocols were created. In the 1980s, bioethicists, theologians, and researchers established a hard ethical line between somatic editing (treating an existing patient's non-reproductive cells) and germline editing (altering future generations). No one has performed the latter form of genetic engineering except for Chinese scientist He Jiankui in 2018. (Chinese society used to be more ambivalent about the technology than the West is.) In response, Beijing heavily tightened its laws, classifying heritable gene editing as a high-risk medical technology subject to the penal code, and He Jiankui was sentenced to three years in prison.
- joshstrange 3mo ago
  I completely can see why we'd want to, for ethical reasons, ban germline editing, and I want to be clear that I agree doing so cannot be done ethically, but there is a part of me that is wistful for what could have been. Same with things like CRISPR but it's probably just fun to dream and the reality would be a nightmare.
- ThrowawayR2 3mo ago
  > "For one, Japan banned guns for a few centuries. (Its warrior class was politically powerful and judged that guns would disrupt class relations too much.)" That example works against the argument since that policy was rendered moot when Commodore Perry arrived at Japan in 1853 with a squadron of American warships and demanded opening of trade and diplomatic relations at gunpoint.
  - esafak 3mo ago
    That's what inspections are for.
arethuza 3mo ago
The decision to not go with the development of extremely large thermonuclear weapons might count - the US Sundial Project was supposed to be about 10 gigatons of TNT. Not the most practical weapons but once you get to a certain size delivery arguably stops being a problem - its going to kill everyone anyway so doesn't matter where you let it off! https://en.wikipedia.org/wiki/Sundial_(weapon) https://en.wikipedia.org/wiki/Sundial_(weapon) Edit: Mind you, I wonder if the design for Sundial is stored somewhere...
tim333 3mo ago
The 1972 Biological Weapons Convention? Probably partly successful.
- foobiekr 3mo ago
  The Soviet Union systematically ignored the BWC. There's tons of evidence.
  - xtracto 3mo ago
    Isn't the US famous for not even signing a lot of world treaties like climate accords and others? This is an interesting subject and conversation, but it's moot having it in these culture-centric forums. I wonder if there are Russians discussing plausible scenarios in Vkontakte groups, or Chinese doing the same in whatever Alibaba group sites they use. The problem is that we are all skewed by our media, our ideas and our culture. These type of discussions need the highest kind of political interactions. It's fascinating, specially for someone who lives in a "third world" country, non-aligned to any of these 3 superpowers. Whatever transpires, we are at tge mercy of these (and no, US hasn't treated us "better"). My opinion is that there's no turning back on AGI development. I dont think current governments are capable of getting into an agreement of that size. Specially given the Isolationist stage in the cycle we live in. (In contrast with for example the CFC and Ozone layer issue we had in the 1990s, when the planet was in a globalist kind of stage)
    - foobiekr 3mo ago
      The Russians signed it and then completely ignored the agreement. That is an objective fact and has nothing to do with the rest of the cultural relativism you're engaging in. Whataboutism about the US _not signing_ things is also irrelevant. The whole AI treaty discussion is about a treaty countries will _actually sign_ and the relevance of Russia cheating on the BWC is that the signatures of some countries are absolutely worthless, which _is_ relevant.
4er_transform 3mo ago
While technology has empowered governments, it’s also empowered the individual, and more importantly shifted the material dynamics to better align the incentives of governments with the people. Democracy followed material change, it didn’t precede it. Democracy came about because it was optimal for a power seeking government, not out of the kindness of their heart. A resource extraction based economy sees people as slaves. The true source of power is the resource, people are just a means to an end, so you mistreat the people as much as you can get away with in pursuit of the resource while avoiding revolt. With stable infrastructure, the government makes far more from an educated, rich population that it can tax and use the innovation from. It’s against its own quest for power to interfere too much in the prosperity of its citizens. The incentives are aligned. Solving the AI problem isn’t about stopping the tech or making a bunch of brittle laws. It’s always been about alignment: aligning the large AGI-like entities that are the modern state, the modern economy, representative democracy, or AGI itself, with human prosperity
- tehjoker 3mo ago
  This is a just so story. The main issue today is the lack of democracy in the country and the use of technology to surveil and govern a restive population as the government has less and less legitimacy. The narrative you are telling is the heroic tale of computing and the internet c. 1990-2010. Yasha Levine wrote about how this narrative was preceded by a forgotten one where MIT students protested because the computers were going to be linked to government databases and share data on anti-Vietnam war activists. Despite protestations, activists were correct and this happened, and now it happens at huge scale. http://yashalevine.com/surveillance-valley http://yashalevine.com/surveillance-valley
  - mikestorrent 3mo ago
    Yes, and RMS was correct in his Right to Read and so many other things - we're seeing the slow death of the never-enshrined-in-law right to compute. Luckily open-source is big enough to slow this down; we should all be pretty amazed and appreciative that there even are open-weights models at all, out there, because it is a profoundly democratizing thing.
QuadmasterXLII 3mo ago
If there were examples, their example status would drop the odds that I know about them.
erichocean 3mo ago
> Are there examples of where we have collective decided not to pursue knowledge? Successfully? Studying human bio-diversity since WW2 is the most obvious example, though it hasn't been entirely successful. Genomics is what finally broke the barrier, especially in the last decade or so.
computably 3mo ago
> Are there examples of where we have collective decided not to pursue knowledge? Successfully? Intrinsically, the knowledge humans choose not to pursue will not be much publicized. There's limited value in calling attention to it and it doesn't make for good entertainment. Plenty of examples provided by other comments nonetheless. > Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know. I don't personally think there's intrinsic benefit in disseminating arbitrary knowledge. There's quite some difference between the printing press and nukes.
sometimelurker 3mo ago
we should never make ASI, what I am saying is strongly sane. ASI = not good. AGI, ~human brainpower, can be made safe enough. > Are there examples of where we have collective decided not to pursue knowledge? Successfully? human GMO, some bioweopns, I'm sure theres a long list of awful stuff no one wants to exist.
- voidmain 3mo ago
  If AGI means that AI+robotics can robustly substitute for human labor, and robots are cheaper and faster to build than humans, then (a) anyone ruthless enough can zerg rush and defeat any nations that don't discard humans, (b) no one without a massive robot army will be needed in any way by their rulers. If this isn't a recipe for a horrific outcome, what is?
  - sometimelurker 3mo ago
    > defeat any nations that don't discard humans AGI doesn't do away with nuclear MAD, it just messes with economics and makes many people temporarily jobless. Temporarily because in a literal sense RLVR needs verification to train off of, and a lot of jobs cant be easily checked if theyre done. this includes AI safety people, preschool teachers, psychologists, and probably a lot more, including most of their bosses
    - voidmain 3mo ago
      Nuclear weapons are mostly a threat to cities, and a zerg rush build would just not build any. I think that if all we get is skill cloning for formally verifiable tasks, that is not "AGI" and does not doom us. It probably won't even doom software engineers, because their whole job is also not formally verifiable. I don't trust that the next ten trillion dollars of AI research won't give us data efficient continual learning and agency on par with humans, which is what I would call AGI.
- throwaway27448 3mo ago
  It's not even clear that ASI is a coherent concept. But, I don't trust capital with either.
browski 3mo ago
[flagged]
- cindyllm 3mo ago
  [dead]
alex_young 3mo ago
Perhaps the burning of the library at Alexandria would qualify. How intentional that was is somewhat in question, but the world certainly turned its back on the only collection of written knowledge and let it turn to ash.
- magicalist 3mo ago
  FWIW the burning of the library of Alexandria, and, indeed, its status as "the only collection of written knowledge" are myths.
  - alex_young 3mo ago
    Did it not burn? Was it not the most extensive library of human writing at that time? Please help us all understand. https://en.wikipedia.org/wiki/Library_of_Alexandria https://en.wikipedia.org/wiki/Library_of_Alexandria
    - aesthesia 3mo ago
      From the article you link: > Scholars have interpreted Cassius Dio's wording to indicate that the fire did not actually destroy the entire Library itself, but rather one or more Library warehouses near the docks.[87][81][8][89] Whatever damage Caesar's fire may have caused, evidently the Library was not completely destroyed.[87][81][8][89][3] The geographer Strabo (c. 63 BC – c. 24 AD) mentions visiting the Mouseion, the larger research institution to which the Library was attached, in around 20 BC, several decades after Caesar's fire, indicating that it either survived the fire or was rebuilt soon afterwards.[87][8] Nonetheless, Strabo's manner of talking about the Mouseion shows that it was nowhere near as prestigious as it had been a few centuries prior. It is unknown whether this was due to historical decline or catastrophic destruction.[8] Despite mentioning the Mouseion, Strabo does not mention the Library separately, perhaps indicating that it had been so drastically reduced in stature and significance that Strabo felt it did not warrant separate mention.[8] It is unclear what happened to the Mouseion after Strabo's mention of it.[60] > Further evidence for the Library's survival after 48 BC comes from the fact that the most notable producer of composite commentaries during the late first century BC and early first century AD was a scholar who worked in Alexandria named Didymus Chalcenterus, whose epithet Χαλκέντερος (Chalkénteros) means "bronze guts".[90][87] Didymus is said to have produced somewhere between 3,500 and 4,000 books, making him the most prolific known writer in all of antiquity.[90][82] He was also given the nickname βιβλιολάθης (Biblioláthēs), meaning "book-forgetter" because it was said that even he could not remember all the books he had written.[90][91] Parts of some of Didymus' commentaries have been preserved in the forms of later extracts and these remains are modern scholars' most important sources of information about the critical works of the earlier scholars at the Library of Alexandria.[90] Lionel Casson states that Didymus' prodigious output "would have been impossible without at least a good part of the resources of the library at his disposal".[87]
nharziro 3mo ago
Uhh north Korea?
HaloZero 3mo ago
Human embryo genetic modification has been effectively taboo if not banned until just recently since WWII and the aftermath of the Holocaust. I think some people in the US are proposing doing it now but I don't think anyone has tried it yet without repercussions.
- dash2 3mo ago
  Actually: https://www.economist.com/science-and-technology/2025/11/19/tech-billionaires-want-to-make-gene-edited-babies https://www.economist.com/science-and-technology/2025/11/19/...
aarondong 3mo ago
I like your optimism and I think you will be vindicated. AI is democratic and AI talent is globally distributed. It will just take a while to get online. AI labs do not have a monopoly on human talent, and open source AI only empowers independent science and meritocracy. On a funny note, I think their prompt was: "Hey Fable. Please attribute every piece of scientific and economic progress to AI until 2040. And predict every major geopolitical event. Make no mistakes."
dyauspitr 3mo ago
> Are there examples of where we have collective decided not to pursue knowledge? Successfully? Isn’t that like all of the Middle Ages where we replaced knowledge with an alternate religious reality.
- matkoniecz 3mo ago
  No, that is not a good description of Middle Ages at all. For start, previous era was also deeply religious. So it switched religions, if anything new one was more friendly toward knowledge.
corv 3mo ago
A successful example of reigning in progress: electric bicycles intentionally speed limited for safety
- nradov 3mo ago
  There is no rein on progress. You can buy electric motorcycles without speed governors. And you can drive them on public roads (subject to traffic laws) as long as you're properly licensed, registered, and insured. The distinction between electric "bikes" and motorcycles is mostly artificial.
kortilla 3mo ago
It’s telling terrorists how to make bombs better apparently. Continuing to lower the barrier for that kind of stuff is clearly a negative in the “knowledge deserves to be free” world.
- AussieWog93 3mo ago
  Has there been an uptick in terrorist bombings or is this just a hypothetical at this stage?
  - kortilla 3mo ago
    https://www.nytimes.com/2026/07/10/us/politics/ai-terrorism-boko-haram-nigeria.html https://www.nytimes.com/2026/07/10/us/politics/ai-terrorism-...
RandomLensman 3mo ago
Biological weapons? Yes, there is research on defense, but no big arsenals of weapons etc. My impression from the origin of the bioweapons convention is that collectively people decided that these things are too dangerous in various ways for any advantage that might be derived from them.
greenchair 3mo ago
I can think of many examples that I won't name but you can imagine in biology/medical fields where certain lines of investigation are not performed due to ethical and legal repercussions.
yeasku 3mo ago
[dead]

Dig1t 3mo ago

Taking the approach with AI that we took with the atomic bomb would be catastrophic. If the only people who are allowed to use this technology are governments, intelligence agencies, and a select few anointed companies, then the risk of authoritarian misuse will skyrocket. I worry that any attempt to limit their use and development will be abused and misdirected. We are already seeing people like Anthropic doing this, they are trying to use anti-AI sentiment to engage in regulatory capture. Go watch Dario’s speeches about how open weight models are dangerous and how they are “not really open”. Everyone can see that much of this “safety” conversation is ultimately just a tactic to shut potential competitors out of the market and establish a monopoly/duopoly.

joshstrange 3mo ago
I agree. I guess I should have said something like: "Stopping" LLM research just means it will be in the hands of a few who can abuse it. I'd rather a state of M.A.D. but instead of a handful of countries/governments it's millions/billions of people with access to the models (open ideally). Again, perhaps horribly naive or misguided, I understand that bioterrorism could (is?) a real problem as well as more "mundane" things like building a bomb (nuclear or otherwise). I just feel like limiting access to governments or "blessed" entities is even worse.

rayiner 3mo ago

No, of course not. That would be an insane trust fall. Even relatively small advances in technology give a country world dominating power. Fun fact: India was militarily superior to Britain in the 1600s—a gunpowder empire with a million soldiers—but was taken over by it in the 1700s. Britain’s edge was small: lighter, more maneuverable cannons, standardized ammunition, better military and political organization. Not a first world country versus a third world country—more like the dynamic US versus a sclerotic EU. And that modest edge led to 200 years of colonization. If we slow down on ASI voluntarily we’d be allowing a gap to open up that would make the difference between colonial europe and colonized Asia/Africa look trivial. It would be insane.

jst1fthsdys 3mo ago
You overstate the advantages of technology. Mughal India was fragmented and on a sharp decline. The British used politics, finance, and treachery to divide and conquer what was remaining.
reducesuffering 3mo ago
> we’d be allowing a gap to open up that would make the difference between colonial europe and colonized Asia/Africa look trivial. An easy choice to make if the alternative is everyone dying instead.
- rayiner 3mo ago
  The alternative isn’t “everyone dying.” It’s us holding all the cards.
  - hollerith 3mo ago
    Why? Is human extinction not permitted by the laws of physics?
    - rayiner 3mo ago
      That’s a possibility, but not the only one. The two most realistic ones are: we race ahead and maintain our status, or we slow down and open ourselves up to colonization.

kokotajlod 3mo ago

If you read the scenario, you'll see that the regulations are mostly about what people can do with giant compute clusters, and not about the ideas themselves. The ideas themselves are required to be totally transparent to the public. As for historic precedents: Human cloning, human genome editing, and mirror life seem like one precedent; nuclear weapons and nuclear energy another; come to think of it I think drone delivery was strangled by regulations too...? Plan A isn't a proposal to never build superintelligence, it's a proposal to build it more cautiously and transparently.

ACCount37 3mo ago
Human cloning and genetic editing isn't stalled because we agreed it's unethical. It's not pursued because at the current level of advancements, it's pretty useless. The things we can do there are niche. It's easy to ban something that's not very useful. If we had a way to make gene edited humans a lot smarter, a lot stronger or live a lot longer? Or a way to quick-grow human bodies to adulthood in a couple years? Capabilities that private actors or countries may want, ethics be damned? That would be closer to what we have with AI right now.
- fragmede 3mo ago
  It's not useless. We know enough about DNA to be able to make better humans, but people get real squickly real fast when you talk about that. It has stalled because of that. If we go in and just check the DNA for Downs or Parkinson's, we can't have a conversation on aborting the foetus without religious beliefs coming into play. Designer children aren't a thing, despite the ability to edit DNA to do specific things. Hair and eye color are easy enough to go in and edit for. Humanity has decided to opt out of doing that for now.
  - bglazer 3mo ago
    > If we go in and just check the DNA for Downs or Parkinson's You should consider reading the wikipedia page about Parkinson’s disease.

hollerith 3mo ago

Yes, there are examples of where we have collective decided not to pursue a particular technology tree. For one, Japan banned guns for a few centuries. (Its warrior class was politically powerful and judged that guns would disrupt class relations too much.) And there have been successful world-wide bans. For example, following the invention of recombinant DNA technology, scientists convened the Asilomar Conference in 1975. They established a voluntary self-moratorium on certain types of genetic engineering until strict laboratory containment protocols were created. In the 1980s, bioethicists, theologians, and researchers established a hard ethical line between somatic editing (treating an existing patient's non-reproductive cells) and germline editing (altering future generations). No one has performed the latter form of genetic engineering except for Chinese scientist He Jiankui in 2018. (Chinese society used to be more ambivalent about the technology than the West is.) In response, Beijing heavily tightened its laws, classifying heritable gene editing as a high-risk medical technology subject to the penal code, and He Jiankui was sentenced to three years in prison.

joshstrange 3mo ago
I completely can see why we'd want to, for ethical reasons, ban germline editing, and I want to be clear that I agree doing so cannot be done ethically, but there is a part of me that is wistful for what could have been. Same with things like CRISPR but it's probably just fun to dream and the reality would be a nightmare.
ThrowawayR2 3mo ago
> "For one, Japan banned guns for a few centuries. (Its warrior class was politically powerful and judged that guns would disrupt class relations too much.)" That example works against the argument since that policy was rendered moot when Commodore Perry arrived at Japan in 1853 with a squadron of American warships and demanded opening of trade and diplomatic relations at gunpoint.
- esafak 3mo ago
  That's what inspections are for.

arethuza 3mo ago

The decision to not go with the development of extremely large thermonuclear weapons might count - the US Sundial Project was supposed to be about 10 gigatons of TNT. Not the most practical weapons but once you get to a certain size delivery arguably stops being a problem - its going to kill everyone anyway so doesn't matter where you let it off! https://en.wikipedia.org/wiki/Sundial_(weapon) https://en.wikipedia.org/wiki/Sundial_(weapon) Edit: Mind you, I wonder if the design for Sundial is stored somewhere...

tim333 3mo ago

The 1972 Biological Weapons Convention? Probably partly successful.

foobiekr 3mo ago
The Soviet Union systematically ignored the BWC. There's tons of evidence.
- xtracto 3mo ago
  Isn't the US famous for not even signing a lot of world treaties like climate accords and others? This is an interesting subject and conversation, but it's moot having it in these culture-centric forums. I wonder if there are Russians discussing plausible scenarios in Vkontakte groups, or Chinese doing the same in whatever Alibaba group sites they use. The problem is that we are all skewed by our media, our ideas and our culture. These type of discussions need the highest kind of political interactions. It's fascinating, specially for someone who lives in a "third world" country, non-aligned to any of these 3 superpowers. Whatever transpires, we are at tge mercy of these (and no, US hasn't treated us "better"). My opinion is that there's no turning back on AGI development. I dont think current governments are capable of getting into an agreement of that size. Specially given the Isolationist stage in the cycle we live in. (In contrast with for example the CFC and Ozone layer issue we had in the 1990s, when the planet was in a globalist kind of stage)
  - foobiekr 3mo ago
    The Russians signed it and then completely ignored the agreement. That is an objective fact and has nothing to do with the rest of the cultural relativism you're engaging in. Whataboutism about the US _not signing_ things is also irrelevant. The whole AI treaty discussion is about a treaty countries will _actually sign_ and the relevance of Russia cheating on the BWC is that the signatures of some countries are absolutely worthless, which _is_ relevant.

4er_transform 3mo ago

While technology has empowered governments, it’s also empowered the individual, and more importantly shifted the material dynamics to better align the incentives of governments with the people. Democracy followed material change, it didn’t precede it. Democracy came about because it was optimal for a power seeking government, not out of the kindness of their heart. A resource extraction based economy sees people as slaves. The true source of power is the resource, people are just a means to an end, so you mistreat the people as much as you can get away with in pursuit of the resource while avoiding revolt. With stable infrastructure, the government makes far more from an educated, rich population that it can tax and use the innovation from. It’s against its own quest for power to interfere too much in the prosperity of its citizens. The incentives are aligned. Solving the AI problem isn’t about stopping the tech or making a bunch of brittle laws. It’s always been about alignment: aligning the large AGI-like entities that are the modern state, the modern economy, representative democracy, or AGI itself, with human prosperity

tehjoker 3mo ago
This is a just so story. The main issue today is the lack of democracy in the country and the use of technology to surveil and govern a restive population as the government has less and less legitimacy. The narrative you are telling is the heroic tale of computing and the internet c. 1990-2010. Yasha Levine wrote about how this narrative was preceded by a forgotten one where MIT students protested because the computers were going to be linked to government databases and share data on anti-Vietnam war activists. Despite protestations, activists were correct and this happened, and now it happens at huge scale. http://yashalevine.com/surveillance-valley http://yashalevine.com/surveillance-valley
- mikestorrent 3mo ago
  Yes, and RMS was correct in his Right to Read and so many other things - we're seeing the slow death of the never-enshrined-in-law right to compute. Luckily open-source is big enough to slow this down; we should all be pretty amazed and appreciative that there even are open-weights models at all, out there, because it is a profoundly democratizing thing.

QuadmasterXLII 3mo ago

If there were examples, their example status would drop the odds that I know about them.

erichocean 3mo ago

> Are there examples of where we have collective decided not to pursue knowledge? Successfully? Studying human bio-diversity since WW2 is the most obvious example, though it hasn't been entirely successful. Genomics is what finally broke the barrier, especially in the last decade or so.

computably 3mo ago

> Are there examples of where we have collective decided not to pursue knowledge? Successfully? Intrinsically, the knowledge humans choose not to pursue will not be much publicized. There's limited value in calling attention to it and it doesn't make for good entertainment. Plenty of examples provided by other comments nonetheless. > Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know. I don't personally think there's intrinsic benefit in disseminating arbitrary knowledge. There's quite some difference between the printing press and nukes.

sometimelurker 3mo ago

we should never make ASI, what I am saying is strongly sane. ASI = not good. AGI, ~human brainpower, can be made safe enough. > Are there examples of where we have collective decided not to pursue knowledge? Successfully? human GMO, some bioweopns, I'm sure theres a long list of awful stuff no one wants to exist.

voidmain 3mo ago
If AGI means that AI+robotics can robustly substitute for human labor, and robots are cheaper and faster to build than humans, then (a) anyone ruthless enough can zerg rush and defeat any nations that don't discard humans, (b) no one without a massive robot army will be needed in any way by their rulers. If this isn't a recipe for a horrific outcome, what is?
- sometimelurker 3mo ago
  > defeat any nations that don't discard humans AGI doesn't do away with nuclear MAD, it just messes with economics and makes many people temporarily jobless. Temporarily because in a literal sense RLVR needs verification to train off of, and a lot of jobs cant be easily checked if theyre done. this includes AI safety people, preschool teachers, psychologists, and probably a lot more, including most of their bosses
  - voidmain 3mo ago
    Nuclear weapons are mostly a threat to cities, and a zerg rush build would just not build any. I think that if all we get is skill cloning for formally verifiable tasks, that is not "AGI" and does not doom us. It probably won't even doom software engineers, because their whole job is also not formally verifiable. I don't trust that the next ten trillion dollars of AI research won't give us data efficient continual learning and agency on par with humans, which is what I would call AGI.
throwaway27448 3mo ago
It's not even clear that ASI is a coherent concept. But, I don't trust capital with either.

browski 3mo ago

[flagged]

cindyllm 3mo ago
[dead]

alex_young 3mo ago

Perhaps the burning of the library at Alexandria would qualify. How intentional that was is somewhat in question, but the world certainly turned its back on the only collection of written knowledge and let it turn to ash.

magicalist 3mo ago
FWIW the burning of the library of Alexandria, and, indeed, its status as "the only collection of written knowledge" are myths.
- alex_young 3mo ago
  Did it not burn? Was it not the most extensive library of human writing at that time? Please help us all understand. https://en.wikipedia.org/wiki/Library_of_Alexandria https://en.wikipedia.org/wiki/Library_of_Alexandria
  - aesthesia 3mo ago
    From the article you link: > Scholars have interpreted Cassius Dio's wording to indicate that the fire did not actually destroy the entire Library itself, but rather one or more Library warehouses near the docks.[87][81][8][89] Whatever damage Caesar's fire may have caused, evidently the Library was not completely destroyed.[87][81][8][89][3] The geographer Strabo (c. 63 BC – c. 24 AD) mentions visiting the Mouseion, the larger research institution to which the Library was attached, in around 20 BC, several decades after Caesar's fire, indicating that it either survived the fire or was rebuilt soon afterwards.[87][8] Nonetheless, Strabo's manner of talking about the Mouseion shows that it was nowhere near as prestigious as it had been a few centuries prior. It is unknown whether this was due to historical decline or catastrophic destruction.[8] Despite mentioning the Mouseion, Strabo does not mention the Library separately, perhaps indicating that it had been so drastically reduced in stature and significance that Strabo felt it did not warrant separate mention.[8] It is unclear what happened to the Mouseion after Strabo's mention of it.[60] > Further evidence for the Library's survival after 48 BC comes from the fact that the most notable producer of composite commentaries during the late first century BC and early first century AD was a scholar who worked in Alexandria named Didymus Chalcenterus, whose epithet Χαλκέντερος (Chalkénteros) means "bronze guts".[90][87] Didymus is said to have produced somewhere between 3,500 and 4,000 books, making him the most prolific known writer in all of antiquity.[90][82] He was also given the nickname βιβλιολάθης (Biblioláthēs), meaning "book-forgetter" because it was said that even he could not remember all the books he had written.[90][91] Parts of some of Didymus' commentaries have been preserved in the forms of later extracts and these remains are modern scholars' most important sources of information about the critical works of the earlier scholars at the Library of Alexandria.[90] Lionel Casson states that Didymus' prodigious output "would have been impossible without at least a good part of the resources of the library at his disposal".[87]

nharziro 3mo ago

Uhh north Korea?

HaloZero 3mo ago

Human embryo genetic modification has been effectively taboo if not banned until just recently since WWII and the aftermath of the Holocaust. I think some people in the US are proposing doing it now but I don't think anyone has tried it yet without repercussions.

dash2 3mo ago
Actually: https://www.economist.com/science-and-technology/2025/11/19/tech-billionaires-want-to-make-gene-edited-babies https://www.economist.com/science-and-technology/2025/11/19/...

aarondong 3mo ago

I like your optimism and I think you will be vindicated. AI is democratic and AI talent is globally distributed. It will just take a while to get online. AI labs do not have a monopoly on human talent, and open source AI only empowers independent science and meritocracy. On a funny note, I think their prompt was: "Hey Fable. Please attribute every piece of scientific and economic progress to AI until 2040. And predict every major geopolitical event. Make no mistakes."

dyauspitr 3mo ago

> Are there examples of where we have collective decided not to pursue knowledge? Successfully? Isn’t that like all of the Middle Ages where we replaced knowledge with an alternate religious reality.

matkoniecz 3mo ago
No, that is not a good description of Middle Ages at all. For start, previous era was also deeply religious. So it switched religions, if anything new one was more friendly toward knowledge.

corv 3mo ago

A successful example of reigning in progress: electric bicycles intentionally speed limited for safety

nradov 3mo ago
There is no rein on progress. You can buy electric motorcycles without speed governors. And you can drive them on public roads (subject to traffic laws) as long as you're properly licensed, registered, and insured. The distinction between electric "bikes" and motorcycles is mostly artificial.

kortilla 3mo ago

It’s telling terrorists how to make bombs better apparently. Continuing to lower the barrier for that kind of stuff is clearly a negative in the “knowledge deserves to be free” world.

AussieWog93 3mo ago
Has there been an uptick in terrorist bombings or is this just a hypothetical at this stage?
- kortilla 3mo ago
  https://www.nytimes.com/2026/07/10/us/politics/ai-terrorism-boko-haram-nigeria.html https://www.nytimes.com/2026/07/10/us/politics/ai-terrorism-...

RandomLensman 3mo ago

Biological weapons? Yes, there is research on defense, but no big arsenals of weapons etc. My impression from the origin of the bioweapons convention is that collectively people decided that these things are too dangerous in various ways for any advantage that might be derived from them.

greenchair 3mo ago

I can think of many examples that I won't name but you can imagine in biology/medical fields where certain lines of investigation are not performed due to ethical and legal repercussions.

yeasku 3mo ago

[dead]