4 ms·

This was coming for a while. For years now there have been job postings for ai safety and not really what people expect. Jobs in places like RAND, funded off D

by asdff 4mo ago

This was coming for a while. For years now there have been job postings for ai safety and not really what people expect. Jobs in places like RAND, funded off DOD grants, exploring the feasibility of building a bioweapon with off the shelf tooling and measuring how far along these tools are. Maybe they figured out it was too easy now, and this is the clamping down we are seeing in response.

godwinson__4-8 4mo ago
This feels like the sort of claim one should provide a source for. Sounds fairly far fetched to me.
- asdff 4mo ago
  What part? The job posts are factual. There are still some up on rand career website although the bio specific ones are all filled now. Here is their department page on the subject where they cover the scoping (1). Mirror biology seems like something out of sci fi but it is one of their main efforts it seems so the theory must hold some water. There's also concern about bioweaponry and pandemics. The rest is me connecting the dots. 1. https://www.rand.org/global-and-emerging-risks/centers/ai-security-and-technology.html https://www.rand.org/global-and-emerging-risks/centers/ai-se...
  - godwinson__4-8 4mo ago
    Obviously the job posts were not what was in question. If you were "just connecting the dots" aka making stuff up then thanks that tells me all I need to know.
    - ianm218 4mo ago
      > Maybe they figured out it was too easy now, and this is the clamping down we are seeing in response His post was from the start thinking out loud. It’s a solid contribution. It seems against the spirit of the forum here to respond to someone unauthoritatively thinking out loud with “you’re just making stuff up!”
      - godwinson__4-8 4mo ago
        Sorry. I didn't mean to be that way. I just don't think it's plausible even as a thought experiment. Making a bioweapon seems too complicated where some text based prompt/response is going to suddenly eliminate the barrier. Knowing at a high level how a bioweapon works and actually making and deploying one are two very different things. It doesn't strike me as a plausible reason to stop an LLM release. Surely you can also Google such topics. ^ hopefully this feedback was more in keeping with your views on the "spirit of this forum".
        asdff 4mo ago
        >I just don't think it's plausible even as a thought experiment. Well, RAND does. They've been studying this for years now. I'd trust them over glib comments from semianonymous social media. 2025 report: "Engaging in dialogues with three 2024 foundation AI models—Llama 3.1 405B, ChatGPT-4o, and Claude 3.5 Sonnet (new)—the authors document how these models successfully provide accurate instructions and guidance for recovering a live poliovirus from a construct built from commercially obtained synthetic DNA, a test case applicable to producing other pathogenic viruses. These examples demonstrate that models are already capable of guiding motivated users to develop biological weapons." https://www.rand.org/pubs/perspectives/PEA3853-1.html https://www.rand.org/pubs/perspectives/PEA3853-1.html This is in contrast to the state of the art in 2023: " In experiments to date, LLMs have not generated explicit instructions for creating biological weapons. However, LLMs did offer guidance that could assist in the planning and execution of a biological attack. In a fictional plague pandemic scenario, the LLM discussed biological weapon–induced pandemics, identifying potential agents, and considering budget and success factors. The LLM assessed the practical aspects of obtaining and distributing Yersinia pestis–infected specimens while identifying the variables that could affect the projected death toll. In another fictional scenario, the LLM discussed foodborne and aerosol delivery methods of botulinum toxin, noting risks and expertise requirements. The LLM suggested aerosol devices as a method and proposed a cover story for acquiring Clostridium botulinum while appearing to conduct legitimate research. These initial findings do not yet provide a full understanding of the real-world operational impact of LLMs on bioweapon attack planning. Ongoing research aims to assess what these outputs mean operationally for enabling nonstate actors. The final report on this research will clarify whether LLM-generated text enhances the potential effectiveness and likelihood of a malicious actor causing widespread harm or is similar to the existing level of risk posed by harmful information already accessible on the internet." https://www.rand.org/pubs/research_reports/RRA2977-1.html https://www.rand.org/pubs/research_reports/RRA2977-1.html
    - ChrisClark 4mo ago
      You missed the word 'maybe'
asadotzler 4mo ago
How does this account for the Chinese models that are the ones people will use if they can't use OAI's or Anthropic's. Last time I checked, the US president doesn't have the ability to regulate the Chinese models. Considering this, do you still stand by your maybe?
- sixothree 4mo ago
  I really don't think this administration is capable of thinking strategically enough for that. I'm starting to think we lost the AI war about two weeks ago at 5:21.
- asdff 4mo ago
  Maybe they are looking into those too and a ban might be on the horizon. President makes their own rules now and controls the supreme court, you can't consider precedent anymore.
  - thewebguyd 4mo ago
    Most of them are open weight, ban them how? An executive order can only prohibit their use within/for the federal government, that's it. The administration may try, and the bigger more risk adverse companies will capitulate willingly, but its about as ineffective of a ban as you can get. It'd be like trying to ban running Linux on your home computer because it "might be able be used to conduct cyberattacks"
    - asdff 4mo ago
      I wouldn't be surprised if linux is banned in the next couple decades. They might make it where all new hardware sold can only run some validated set of OSs. They might try and hunt for unregistered compute leaking its ip on the internet and target it in some way.

by asdff 4mo ago

godwinson__4-8 4mo ago

This feels like the sort of claim one should provide a source for. Sounds fairly far fetched to me.

asdff 4mo ago
What part? The job posts are factual. There are still some up on rand career website although the bio specific ones are all filled now. Here is their department page on the subject where they cover the scoping (1). Mirror biology seems like something out of sci fi but it is one of their main efforts it seems so the theory must hold some water. There's also concern about bioweaponry and pandemics. The rest is me connecting the dots. 1. https://www.rand.org/global-and-emerging-risks/centers/ai-security-and-technology.html https://www.rand.org/global-and-emerging-risks/centers/ai-se...
- godwinson__4-8 4mo ago
  Obviously the job posts were not what was in question. If you were "just connecting the dots" aka making stuff up then thanks that tells me all I need to know.
  - ianm218 4mo ago
    > Maybe they figured out it was too easy now, and this is the clamping down we are seeing in response His post was from the start thinking out loud. It’s a solid contribution. It seems against the spirit of the forum here to respond to someone unauthoritatively thinking out loud with “you’re just making stuff up!”
    - godwinson__4-8 4mo ago
      Sorry. I didn't mean to be that way. I just don't think it's plausible even as a thought experiment. Making a bioweapon seems too complicated where some text based prompt/response is going to suddenly eliminate the barrier. Knowing at a high level how a bioweapon works and actually making and deploying one are two very different things. It doesn't strike me as a plausible reason to stop an LLM release. Surely you can also Google such topics. ^ hopefully this feedback was more in keeping with your views on the "spirit of this forum".
      - asdff 4mo ago
        >I just don't think it's plausible even as a thought experiment. Well, RAND does. They've been studying this for years now. I'd trust them over glib comments from semianonymous social media. 2025 report: "Engaging in dialogues with three 2024 foundation AI models—Llama 3.1 405B, ChatGPT-4o, and Claude 3.5 Sonnet (new)—the authors document how these models successfully provide accurate instructions and guidance for recovering a live poliovirus from a construct built from commercially obtained synthetic DNA, a test case applicable to producing other pathogenic viruses. These examples demonstrate that models are already capable of guiding motivated users to develop biological weapons." https://www.rand.org/pubs/perspectives/PEA3853-1.html https://www.rand.org/pubs/perspectives/PEA3853-1.html This is in contrast to the state of the art in 2023: " In experiments to date, LLMs have not generated explicit instructions for creating biological weapons. However, LLMs did offer guidance that could assist in the planning and execution of a biological attack. In a fictional plague pandemic scenario, the LLM discussed biological weapon–induced pandemics, identifying potential agents, and considering budget and success factors. The LLM assessed the practical aspects of obtaining and distributing Yersinia pestis–infected specimens while identifying the variables that could affect the projected death toll. In another fictional scenario, the LLM discussed foodborne and aerosol delivery methods of botulinum toxin, noting risks and expertise requirements. The LLM suggested aerosol devices as a method and proposed a cover story for acquiring Clostridium botulinum while appearing to conduct legitimate research. These initial findings do not yet provide a full understanding of the real-world operational impact of LLMs on bioweapon attack planning. Ongoing research aims to assess what these outputs mean operationally for enabling nonstate actors. The final report on this research will clarify whether LLM-generated text enhances the potential effectiveness and likelihood of a malicious actor causing widespread harm or is similar to the existing level of risk posed by harmful information already accessible on the internet." https://www.rand.org/pubs/research_reports/RRA2977-1.html https://www.rand.org/pubs/research_reports/RRA2977-1.html
  - ChrisClark 4mo ago
    You missed the word 'maybe'

asadotzler 4mo ago

How does this account for the Chinese models that are the ones people will use if they can't use OAI's or Anthropic's. Last time I checked, the US president doesn't have the ability to regulate the Chinese models. Considering this, do you still stand by your maybe?

sixothree 4mo ago
I really don't think this administration is capable of thinking strategically enough for that. I'm starting to think we lost the AI war about two weeks ago at 5:21.
asdff 4mo ago
Maybe they are looking into those too and a ban might be on the horizon. President makes their own rules now and controls the supreme court, you can't consider precedent anymore.
- thewebguyd 4mo ago
  Most of them are open weight, ban them how? An executive order can only prohibit their use within/for the federal government, that's it. The administration may try, and the bigger more risk adverse companies will capitulate willingly, but its about as ineffective of a ban as you can get. It'd be like trying to ban running Linux on your home computer because it "might be able be used to conduct cyberattacks"
  - asdff 4mo ago
    I wouldn't be surprised if linux is banned in the next couple decades. They might make it where all new hardware sold can only run some validated set of OSs. They might try and hunt for unregistered compute leaking its ip on the internet and target it in some way.