3 ms·

Just because the author can't think of a way to cause mass death with an engineered virus, doesn't mean an AI won't be able to think of one. (There are several

Just because the author can't think of a way to cause mass death with an engineered virus, doesn't mean an AI won't be able to think of one. (There are several biological threats the author hasn't considered. I'll just leave it at that.)

phoghed 23d ago
I know what you’re thinking. A modified HIV virus that also makes everyone hot and horny.
- skeledrew 22d ago
  A surge in rape cases and just curiosity into this new heightened-sex fad will lead to discovery pretty quickly.
jeremyjh 23d ago
The author can't think of what a superintelligence will do, because they are not a superintelligence. But that is not really what people are concerned about, so I think the objections here are quite meaningful. If there is a fully general superintelligence - in the current state of alignment - we will all die for some reason. It doesn't really matter what the reason is.
- deleted 23d ago
  [deleted]
- cobbzilla 23d ago
  please define fully general superintelligence & current state of alignment is this a tautology?
  - andai 23d ago
    Not the guy but, alignment seems to be several decades away, while self-improvement seems to be several years away. (Also you don't even need superintelligence to do a lot of damage. In fact you don't need any intelligence!)
- andai 23d ago
  Well, at that point it doesn't even need to be a virus. It can just be nanobots that atomize your liver. The substrate is a mere technicality.
  - jeremyjh 23d ago
    Right. A superintelligence will kill us in one or more of thousands of possible ways for thousands of possible reasons, perhaps none of which we may understand. Short of a superintelligence - the difficulties of viral bioweapon engineering the author points out are quite meaningful.
alevskaya 23d ago
I was a biophysicist and experimental biologist for decades, I designed viral vectors and cell therapies, etc. What protects us here isn't our ignorance vs an "ASI". It's the fact that the complexity of molecular physics scales exponentially in particle number. No amount of thinking is going to punch through that. You've got to do experiments. The slow loop through reality is not optional, and in the case of "superviruses" would require a lot of iterative experimental development in humans. (And please knock off the bio infohazard act - I routinely get asked about scenarios that AIxBio safety people cook up and they always involve some howler misunderstanding of basic facts in biology or medicine. Just talk about an idea if you have it.)
- naasking 23d ago
  We're already doing those experiments. Tailor made mRNA vaccines targeted to one's own specific cancer mutations can be bought right now. The techniques are getting more sophisticated and more targeted every year. Our ability to predict what happens at these levels is improving by leaps and bounds too thanks to AI like AlphaFold. Anyone can be reasonably confident that no AI or teen can do this now or in the next few years, but are you really so confident what might be possible in 10 years?
  - alevskaya 23d ago
    I worked in immuno-oncology: cancer vaccines work in melanoma where many things work because of the neoantigen abundance, they've generally been very mixed in efficacy. And even BionTech's BNT111 failed in melanoma! We have hope for these approaches but the reality of this stuff is way more nuanced than you think it is. Alphafold can't reliably predict thermal energy landscapes or make functional predictions - and how could it? It wasn't trained on anything that could capture structure - function relationships. Again, most people just have no idea how hard - fundamentally hard - molecular physics is to predict, and how necessary experiments are for any development of biological systems.
    - naasking 22d ago
      > We have hope for these approaches but the reality of this stuff is way more nuanced than you think it is. Sure, everything has more nuance. The point is this stuff is available now; this isn't some future sci-fi, it's only going to get better, it's not the only research on gene targeting, and AI is starting to help with this research. By the time AGI is actually here, consider the breadth of knowledge and capabilities that will be at its disposal. > Alphafold can't reliably predict thermal energy landscapes or make functional predictions - and how could it? It wasn't trained on anything that could capture structure - function relationships. If your point is that the only reason an AI like AlphaFold can't make functional predictions is that we haven't trained an AI to do that, then unless you're arguing we can't or won't ever do that, I'm not sure how that's supposed to be an objection to the argument that AI will be able to make use of this information without doing all of the experiments people seem to think would be necessary. Like I said, we're already going to be doing these experiments because it's useful to us, and we will train AIs to make these predictions, again, because it's useful to us. Stop imagining what an AGI has access to now, and start thinking what it will have access to with the inevitable march of progress that we're already on. Edit: and of course, this doesn't even take into account the fact that an AI could acquire resources to pay people to do this research. The internet provides ample opportunities like this now.
pama 23d ago
I am not sure what you mean here. There exist plenty high-concern biological threats that dont need any AI help. Human oncovirus design is low on my concern list (immunity is diverse), and in any case it does not need AI—rather labspace. Disgruntled high schoolers or undergrad chemists can do way more damage from readily available materials without AI and without research delay. As can nature (or amateur biologists without AI) by mixing bats with their animal of choice and waiting a while. If/when any scary global events like covid happen again, I sure hope we have true superintelligence to help us navigate it quickly.

phoghed 23d ago

I know what you’re thinking. A modified HIV virus that also makes everyone hot and horny.

skeledrew 22d ago
A surge in rape cases and just curiosity into this new heightened-sex fad will lead to discovery pretty quickly.

jeremyjh 23d ago

The author can't think of what a superintelligence will do, because they are not a superintelligence. But that is not really what people are concerned about, so I think the objections here are quite meaningful. If there is a fully general superintelligence - in the current state of alignment - we will all die for some reason. It doesn't really matter what the reason is.

deleted 23d ago
[deleted]
cobbzilla 23d ago
please define fully general superintelligence & current state of alignment is this a tautology?
- andai 23d ago
  Not the guy but, alignment seems to be several decades away, while self-improvement seems to be several years away. (Also you don't even need superintelligence to do a lot of damage. In fact you don't need any intelligence!)
andai 23d ago
Well, at that point it doesn't even need to be a virus. It can just be nanobots that atomize your liver. The substrate is a mere technicality.
- jeremyjh 23d ago
  Right. A superintelligence will kill us in one or more of thousands of possible ways for thousands of possible reasons, perhaps none of which we may understand. Short of a superintelligence - the difficulties of viral bioweapon engineering the author points out are quite meaningful.

alevskaya 23d ago

I was a biophysicist and experimental biologist for decades, I designed viral vectors and cell therapies, etc. What protects us here isn't our ignorance vs an "ASI". It's the fact that the complexity of molecular physics scales exponentially in particle number. No amount of thinking is going to punch through that. You've got to do experiments. The slow loop through reality is not optional, and in the case of "superviruses" would require a lot of iterative experimental development in humans. (And please knock off the bio infohazard act - I routinely get asked about scenarios that AIxBio safety people cook up and they always involve some howler misunderstanding of basic facts in biology or medicine. Just talk about an idea if you have it.)

naasking 23d ago
We're already doing those experiments. Tailor made mRNA vaccines targeted to one's own specific cancer mutations can be bought right now. The techniques are getting more sophisticated and more targeted every year. Our ability to predict what happens at these levels is improving by leaps and bounds too thanks to AI like AlphaFold. Anyone can be reasonably confident that no AI or teen can do this now or in the next few years, but are you really so confident what might be possible in 10 years?
- alevskaya 23d ago
  I worked in immuno-oncology: cancer vaccines work in melanoma where many things work because of the neoantigen abundance, they've generally been very mixed in efficacy. And even BionTech's BNT111 failed in melanoma! We have hope for these approaches but the reality of this stuff is way more nuanced than you think it is. Alphafold can't reliably predict thermal energy landscapes or make functional predictions - and how could it? It wasn't trained on anything that could capture structure - function relationships. Again, most people just have no idea how hard - fundamentally hard - molecular physics is to predict, and how necessary experiments are for any development of biological systems.
  - naasking 22d ago
    > We have hope for these approaches but the reality of this stuff is way more nuanced than you think it is. Sure, everything has more nuance. The point is this stuff is available now; this isn't some future sci-fi, it's only going to get better, it's not the only research on gene targeting, and AI is starting to help with this research. By the time AGI is actually here, consider the breadth of knowledge and capabilities that will be at its disposal. > Alphafold can't reliably predict thermal energy landscapes or make functional predictions - and how could it? It wasn't trained on anything that could capture structure - function relationships. If your point is that the only reason an AI like AlphaFold can't make functional predictions is that we haven't trained an AI to do that, then unless you're arguing we can't or won't ever do that, I'm not sure how that's supposed to be an objection to the argument that AI will be able to make use of this information without doing all of the experiments people seem to think would be necessary. Like I said, we're already going to be doing these experiments because it's useful to us, and we will train AIs to make these predictions, again, because it's useful to us. Stop imagining what an AGI has access to now, and start thinking what it will have access to with the inevitable march of progress that we're already on. Edit: and of course, this doesn't even take into account the fact that an AI could acquire resources to pay people to do this research. The internet provides ample opportunities like this now.

pama 23d ago

I am not sure what you mean here. There exist plenty high-concern biological threats that dont need any AI help. Human oncovirus design is low on my concern list (immunity is diverse), and in any case it does not need AI—rather labspace. Disgruntled high schoolers or undergrad chemists can do way more damage from readily available materials without AI and without research delay. As can nature (or amateur biologists without AI) by mixing bats with their animal of choice and waiting a while. If/when any scary global events like covid happen again, I sure hope we have true superintelligence to help us navigate it quickly.