4 ms·

I am concerned about this kind of technology being used to circumvent traditional safeguards and international agreements that prevent the development of biolog

I am concerned about this kind of technology being used to circumvent traditional safeguards and international agreements that prevent the development of biological weapons.

vessenes 1y ago
Well you might be pleased to know that there are large safety teams working at all frontier model companies worried about the same thing! You could even apply if you have related skills.
- alganet 1y ago
  Are these safety teams subject to the oversight of more estabilished international agreements and safeguards?
  - vessenes 1y ago
    I mean.. they work within the legal frameworks of very large corporations with nation state engagement. It's not like they're autonomous anonymous DAOs
    - canyon289 1y ago
      Hi! I work directly on these teams as a model builder and have talked to my colleagues are the other labs well. All our orgs have openings and if you also could consider working for organizations such as the UK AISI team and other independent organizations that are assessing these models. It's a critical field and there is a need for motivated folks.
    - alganet 1y ago
      That does not answer my question.
      - vessenes 1y ago
        It does. It's just your worldview is pretty black and white. All of these teams ultimately answer to executives that report to a board, that board is guided by a Chief Legal Officer that engages with regulators among other groups. Some guidance will come back as direct control. A lot might come back as soft preferences / guidance to executives.
        alganet 1y ago
        My worldview considers the existance of lobbies and human neglect. It seems more nuanced than "believe the management" rethoric.
  - deleted 1y ago
    [deleted]
  - JumpCrisscross 1y ago
    > the oversight of more estabilished international agreements and safeguards? “Unlike the chemical or nuclear weapons regimes, the [Biological Weapons Convention] lacks both a system to verify states' compliance with the treaty and a separate international organization to support the convention's effective implementation” [1]. [1] https://en.wikipedia.org/wiki/Biological_Weapons_Convention https://en.wikipedia.org/wiki/Biological_Weapons_Convention
    - alganet 1y ago
      The UN has a page on Biological Weapons. https://disarmament.unoda.org/en/our-work/weapons-mass-destruction/biological-weapons https://disarmament.unoda.org/en/our-work/weapons-mass-destr... You can read more about the efforts for disarmament there.
      - JumpCrisscross 1y ago
        Where is there oversight? Biological weapons compliance is entirely voluntary. We don’t have international monitors watching America and Russia’s smallpox stockpiles. That’s left to each nation.
        alganet 1y ago
        Your angle is "there is no oversight, why are you asking for it?". It's the same overall angle as the guy who was spreading covid rumours here in this thread. There are efforts at estabilishing it though. And it's hard and expensive for wet labs, but it could be much simpler for things like simulating biological pathways. One could also see your response as "other nations are developing threats, we should race", which I personally think is misguided. Instead of these petty armchair discussions, we should instead focus on being more serious about it.
- bix6 1y ago
  I thought openAI gave up on safety when Anthropic splintered off as well as when they engaged ScaleAI to traumatize people for RLHF? Or Google when they fired Timnit?
  - nl 1y ago
    The GPT5 system card is 59 pages. Pages 5 to 56 address safety in various forms. https://cdn.openai.com/gpt-5-system-card.pdf https://cdn.openai.com/gpt-5-system-card.pdf
jackblemming 1y ago
Seems like no matter how positive the headline about the technology is, there is invariably someone in the comments pointing out a worst case hypothetical. Is there a name for this phenomenon?
- peddling-brink 1y ago
  Rational discourse? Not working for a marketing team? Realism?
- collingreen 1y ago
  Not believing everything you read on the internet? Being jaded from constant fluff and lies? Not having gell-mann amnesia? I get your sentiment of "why you gotta bring down this good thing" but the answer to your actual question is battle scars from the constant barrage of hostile lies and whitewashing we are subject to. It's kind of absurd (and mildly irresponsible) to think "THIS time will be the time things only go well and nobody uses the new thing for something I don't want".
- OJFord 1y ago
  Pessimism?
- jsnell 1y ago
  Performative cynicism?
trhway 1y ago
We’ve just had a virus - specifically engineered to be highly infectious for humans - escaping the lab (which was running very lax safety level - BSL2 instead of required BSL4) and killing millions and shutting down half the globe. So I’m wondering what safeguards and prevention you’re talking about :)
- LeoPanthera 1y ago
  Don't spread misinformation. This myth is widely believed only by Americans. https://en.wikipedia.org/wiki/COVID-19_misinformation#Virus_origin https://en.wikipedia.org/wiki/COVID-19_misinformation#Virus_...
  - trhway 1y ago
    This myth is documented in the EcoHealth Alliance publicly available NIH and DARPA grants documents among others. Wrt your link - Wikipedia unfortunately isn’t subject to the law like those grants.
    - alganet 1y ago
      Covid is irrelevant to the discussion I opened. You're trying to steer the discussion into a place that will lead us nowhere because there's too many artificial polemics around it. The only thing to be said about it that resonates with what I'm concerned with is that anyone that is good in the head wants better international oversight on potential bioweapons development.
  - 4793273022729 1y ago
    He says and quotes Wikipedia.
- alganet 1y ago
  You're trying to deflect the discussion into a polemic tarpit. That's not going to work. I do not endorse the view that covid was engineered. Also, I consider it to be unrelated to what I am concerned about, and I will kindly explain it to you: Traditional labs work with the wet stuff. And there are a lot of safeguards (the levels you mentioned didn't came out of thin air). Of course I am in favor of enforcing the existing safeguards to the most ethical levels possible. However, when I say that I am concerned about AI being used to circumvent international agreements, I am talking about loopholes that could allow progress in the development of bioweapons without the use of wet labs. For example, by carefully weaving around international rules and doing the development using simulations, which can bypass outdated assumptions that didn't foresaw that this could be possible when they were conceived. This is not new. For example, many people were concerned about research on fusion energy related to compressing fuel pellets, which could be seen as a way of weaving around international treatises on the development of precursor components to more powerful nuclear weapons (better triggers, smaller warheads, all kinds of nasty things).
deleted 1y ago
[deleted]

I am concerned about this kind of technology being used to circumvent traditional safeguards and international agreements that prevent the development of biological weapons.

vessenes 1y ago

Well you might be pleased to know that there are large safety teams working at all frontier model companies worried about the same thing! You could even apply if you have related skills.

alganet 1y ago
Are these safety teams subject to the oversight of more estabilished international agreements and safeguards?
- vessenes 1y ago
  I mean.. they work within the legal frameworks of very large corporations with nation state engagement. It's not like they're autonomous anonymous DAOs
  - canyon289 1y ago
    Hi! I work directly on these teams as a model builder and have talked to my colleagues are the other labs well. All our orgs have openings and if you also could consider working for organizations such as the UK AISI team and other independent organizations that are assessing these models. It's a critical field and there is a need for motivated folks.
  - alganet 1y ago
    That does not answer my question.
    - vessenes 1y ago
      It does. It's just your worldview is pretty black and white. All of these teams ultimately answer to executives that report to a board, that board is guided by a Chief Legal Officer that engages with regulators among other groups. Some guidance will come back as direct control. A lot might come back as soft preferences / guidance to executives.
      - alganet 1y ago
        My worldview considers the existance of lobbies and human neglect. It seems more nuanced than "believe the management" rethoric.
- deleted 1y ago
  [deleted]
- JumpCrisscross 1y ago
  > the oversight of more estabilished international agreements and safeguards? “Unlike the chemical or nuclear weapons regimes, the [Biological Weapons Convention] lacks both a system to verify states' compliance with the treaty and a separate international organization to support the convention's effective implementation” [1]. [1] https://en.wikipedia.org/wiki/Biological_Weapons_Convention https://en.wikipedia.org/wiki/Biological_Weapons_Convention
  - alganet 1y ago
    The UN has a page on Biological Weapons. https://disarmament.unoda.org/en/our-work/weapons-mass-destruction/biological-weapons https://disarmament.unoda.org/en/our-work/weapons-mass-destr... You can read more about the efforts for disarmament there.
    - JumpCrisscross 1y ago
      Where is there oversight? Biological weapons compliance is entirely voluntary. We don’t have international monitors watching America and Russia’s smallpox stockpiles. That’s left to each nation.
      - alganet 1y ago
        Your angle is "there is no oversight, why are you asking for it?". It's the same overall angle as the guy who was spreading covid rumours here in this thread. There are efforts at estabilishing it though. And it's hard and expensive for wet labs, but it could be much simpler for things like simulating biological pathways. One could also see your response as "other nations are developing threats, we should race", which I personally think is misguided. Instead of these petty armchair discussions, we should instead focus on being more serious about it.
bix6 1y ago
I thought openAI gave up on safety when Anthropic splintered off as well as when they engaged ScaleAI to traumatize people for RLHF? Or Google when they fired Timnit?
- nl 1y ago
  The GPT5 system card is 59 pages. Pages 5 to 56 address safety in various forms. https://cdn.openai.com/gpt-5-system-card.pdf https://cdn.openai.com/gpt-5-system-card.pdf

jackblemming 1y ago

Seems like no matter how positive the headline about the technology is, there is invariably someone in the comments pointing out a worst case hypothetical. Is there a name for this phenomenon?

peddling-brink 1y ago
Rational discourse? Not working for a marketing team? Realism?
collingreen 1y ago
Not believing everything you read on the internet? Being jaded from constant fluff and lies? Not having gell-mann amnesia? I get your sentiment of "why you gotta bring down this good thing" but the answer to your actual question is battle scars from the constant barrage of hostile lies and whitewashing we are subject to. It's kind of absurd (and mildly irresponsible) to think "THIS time will be the time things only go well and nobody uses the new thing for something I don't want".
OJFord 1y ago
Pessimism?
jsnell 1y ago
Performative cynicism?

trhway 1y ago

We’ve just had a virus - specifically engineered to be highly infectious for humans - escaping the lab (which was running very lax safety level - BSL2 instead of required BSL4) and killing millions and shutting down half the globe. So I’m wondering what safeguards and prevention you’re talking about :)

LeoPanthera 1y ago
Don't spread misinformation. This myth is widely believed only by Americans. https://en.wikipedia.org/wiki/COVID-19_misinformation#Virus_origin https://en.wikipedia.org/wiki/COVID-19_misinformation#Virus_...
- trhway 1y ago
  This myth is documented in the EcoHealth Alliance publicly available NIH and DARPA grants documents among others. Wrt your link - Wikipedia unfortunately isn’t subject to the law like those grants.
  - alganet 1y ago
    Covid is irrelevant to the discussion I opened. You're trying to steer the discussion into a place that will lead us nowhere because there's too many artificial polemics around it. The only thing to be said about it that resonates with what I'm concerned with is that anyone that is good in the head wants better international oversight on potential bioweapons development.
- 4793273022729 1y ago
  He says and quotes Wikipedia.
alganet 1y ago
You're trying to deflect the discussion into a polemic tarpit. That's not going to work. I do not endorse the view that covid was engineered. Also, I consider it to be unrelated to what I am concerned about, and I will kindly explain it to you: Traditional labs work with the wet stuff. And there are a lot of safeguards (the levels you mentioned didn't came out of thin air). Of course I am in favor of enforcing the existing safeguards to the most ethical levels possible. However, when I say that I am concerned about AI being used to circumvent international agreements, I am talking about loopholes that could allow progress in the development of bioweapons without the use of wet labs. For example, by carefully weaving around international rules and doing the development using simulations, which can bypass outdated assumptions that didn't foresaw that this could be possible when they were conceived. This is not new. For example, many people were concerned about research on fusion energy related to compressing fuel pellets, which could be seen as a way of weaving around international treatises on the development of precursor components to more powerful nuclear weapons (better triggers, smaller warheads, all kinds of nasty things).

deleted 1y ago

[deleted]