5 ms·

> The creator of a model can not ensure that a model is never used to do something harmful – any more so that the developer of a web browser, calculator, or wor

by ubj 2y ago

> The creator of a model can not ensure that a model is never used to do something harmful – any more so that the developer of a web browser, calculator, or word processor could. Placing liability on the creators of general purpose tools like these mean that, in practice, such tools can not be created at all, except by big businesses with well funded legal teams. This matches my thoughts on why this is ultimately a bad piece of legislation. It is virtually impossible to ensure that a piece of technology will not be used for "harmful purposes". I agree that such stipulations will be just another roadblock keeping everyone except "big businesses with well funded legal teams" from working on LLMs.

bee_rider 2y ago
Too bad there isn’t an NRA for LLM’s. So far at least nobody has been literally murdered using an LLM, as far as I’ve heard.
sanxiyn 2y ago
As I understand this law does not mandate you to ensure anything. It requires you to follow best practices (to be determined), report safety incidents, etc. You are not even liable for safety incidents, you just need to report them, although it may be embarrassing. Overall, it seems highly reasonable.
- echelon 2y ago
  > requires you to follow best practices (to be determined) Trigger happy regulation for a field that hasn't even come into full swing. It's indicative of an over-active immune system; lawmakers with nothing better to do. Pass laws against improper use and go after the malicious users. Don't ban the technology, the research, or even the applications. (Of which there will be abundant good uses. Many of which we've yet to even see or predict.) Our culture has become obsessed with regulating and limiting freedom on the very principle that it might be harmful. We should be punishing actual measurable, physical and monetary harms. Not imaginary or hypothetical ones. If California passes this, AI companies should leave California behind.
  - realce 2y ago
    > Don't ban the technology, the research, or even the applications. Where is the ban here?
    - echelon 2y ago
      Some highlights from the text of the bill [1], > Other relief as the court deems appropriate, including monetary damages damages, including punitive damages, to persons aggrieved aggrieved, punitive damages, and an order for the full shutdown of a covered model. > A civil penalty in an amount not exceeding 10 percent of the cost, excluding labor cost, to develop the covered model for a first violation and in an amount not exceeding 30 percent of the cost, excluding labor cost, to develop the covered model for any subsequent violation. [1] https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=202320240SB1047 https://leginfo.legislature.ca.gov/faces/billTextClient.xhtm...
  - sanxiyn 2y ago
    That's like saying we should punish when bridge collapses, before that any bridge should be able to be built. You can argue that, but not many will agree.
    - sgift 2y ago
      Because we know how to build bridges so that they don't collapse. The laws of physics that govern bridge-building are well known. The equivalent for AI systems? Not really.
      - sanxiyn 2y ago
        Best practices will say things like "you should test it". While we are ignorant, there are just many reasonable things to do. Human biology is not completely understood, but that does not mean medical checklists are useless.
        Auracle 2y ago
- jahewson 2y ago
  Nothing drives fear into my heart like the phrase “best practices”. Never in my career has it ever meant anything remotely reasonable.
  - derefr 2y ago
    "Best practices" means something sensible in most subfields of capital-E Engineering. (In fact, I think that's where the term originates from, with all other usages being a corruption of the original concept.) In Engineering, "best practices" are the set of "just do X" answers that will let you skip deriving every answer about what material or design to use from first principles for cases where there's a known dominant solution. For example, "for a load-bearing pillar, use steel-reinforced concrete, in a cylindrical shape, with a cross-sectional diameter following formula XYZ given the number of storeys of the building." You can (and eventually must!) still do a load simulation for the building, to see that the pillar can hold things up without cracking — but you don't have to model the building when selecting what material to use; and you don't have to randomly fiddle with the shape or diameter of the pillar until the load holds. You can slap a pillar into the design and be able to predict that it'll hold the load (while not being overly costly in material use!), because "best practices."
drcode 2y ago
Your argument is meaningless if you don't specify what threshold there should be for harm Otherwise you also have to complain about the stifling of open source bioagent research, open source nuclear warheads, open source human cloning protocols Those are also all dual-use technologies that are objectively morally neutral
- titanomachy 2y ago
  > Those are also all dual-use technologies that are objectively morally neutral nuclear warheads?
  - drcode 2y ago
    Part of my open source Mars terraforming plan
    - sgift 2y ago
      Back to the good old plowshare, yes? https://en.wikipedia.org/wiki/Project_Plowshare https://en.wikipedia.org/wiki/Project_Plowshare
  - sanxiyn 2y ago
    Most explosives are used for construction. That's where Nobel Prize came from.
  - nradov 2y ago
    See the history of "peaceful nuclear explosions". The USA and USSR used a few nuclear warheads for civil engineering purposes. It seems crazy now. https://en.wikipedia.org/wiki/Peaceful_nuclear_explosion https://en.wikipedia.org/wiki/Peaceful_nuclear_explosion
- 65a 2y ago
  Laws should be about the outcome, not about processes that may lead to an outcome. It is already illegal in California to produce your own nuclear weapon. Instead of outlawing books, because they allow research into building giant gundam robots, just outlaw giant gundam robots.
adamc 2y ago
Someone who creates very dangerous items needs to take responsibility for them. Or their production needs to be very heavily regulated. That is just a reality. We don't let companies sell grenades on street corners. The running away from responsibility is one of the things I like least about big tech.
- jahewson 2y ago
  This is a strawman argument. LLMs, like books, are not inherently dangerous. Grenades are, and lack any legitimate purpose beyond indiscriminate killing. LLMs are functions of their training data, nothing more. This is evidenced by how we see very different model architectures produce essentially the same result. All of that training data is out there, on the internet, in books; none of that “dangerous” knowledge is banned or regulated, nor should it be.
- deleted 2y ago
  [deleted]
- eightysixfour 2y ago
  All technology has good and bad uses and you can’t hold the maker accountable for all of those. At some point you have to hold users and buyers accountable or just stop developing anything. When a person uses a car to drive into a crowd, do we blame the automobile manufacturer? Do you blame Kali Linux when someone uses it to hack a remote system? What about Apple when an iPhone is used to call in a threat to a school?
- pclmulqdq 2y ago
  After all of the times that I have heard this argument, I now believe that the lesser evil is allowing people to sell grenades on street corners. This logic causes complacency in users of products and removes any responsibility on the part of malicious actors who still find ways to use the "softened" version of these products badly. They will now just blame the people who didn't "soften" them properly. So no thank you, bring back responsibility to end users of products, and allow suppliers to develop the best capabilities they can.
- groby_b 2y ago
pclmulqdq 2y ago
I really hate the (apparently very popular) idea that we should be shifting responsibility away from end users and toward providers and makers of tools. From playgrounds to drugs to software, our society wants to force the suppliers to make things safe by design rather than requiring and educating end users on responsible use.
photochemsyn 2y ago
The definition of harm is buried low in the bill, here's the list: (A) The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties. (B) At least five hundred million dollars ($500,000,000) of damage through cyberattacks on critical infrastructure via a single incident or multiple related incidents. (C) At least five hundred million dollars ($500,000,000) of damage by an artificial intelligence model that autonomously engages in conduct that would violate the Penal Code if undertaken by a human. (D) Other threats to public safety and security that are of comparable severity to the harms described in paragraphs (A) to (C), inclusive. That means AI for drug discovery and materials science development, AI for managing electricity grids and broadband traffic, AI in the financial and health services sectors, etc. Then there's the military-industrial side, which this legislation might not even touch if only federal contracts are involved. Classified military AI development seems reckless, hasn't anyone seen War Games? https://technologymagazine.com/top10/top-10-military-technology-companies-putting-AI-into-action https://technologymagazine.com/top10/top-10-military-technol... At least with open source, the capabilities are more immediately visible.
hendrycks 2y ago
1. The new Frontier Model Division is just receiving information and issuing guidelines. It’s not a licensing regime and isn’t investigating developers. 2. Folks aren’t automatically liable if their highly capable model is used to do bad things, even catastrophic things. The question is whether they took reasonable measures to prevent that. This bill could have used strict liability, where developers would be liable for catastrophic harms regardless of fault, but that's not what the bill does. 3. Overall it seems pretty reasonable that if your model can cause catastrophic harms (which is not true of current models, but maybe true of future models), then you shouldn’t be releasing models in a way that can predictably allow folks to cause those catastrophic harms. If people want a detailed write up of what the bill does, I recommend this thorough writeup by Zvi. In my opinion this is a pretty narrow proposal focused at the most severe risks (much more narrow than, e.g., the EU AI act). https://thezvi.substack.com/p/on-the-proposed-california-sb-1047 https://thezvi.substack.com/p/on-the-proposed-california-sb-...
- sgammon 2y ago
  On point #3, as far as I can tell, the bill criteria defines a "covered model" (a model subject to regulation under this proposal) as any model that can "cause $500,000 of damage" or more if misused. A regular MacBook can cause half a million dollars of damage if misused. Easily. So I think any model of significant size would qualify. Furthermore, the requirement to register and pre-clear models will surely precede open data access, and that means a loss in competitive cover for startups working on new projects. I can easily see disclosure sites being monitored constantly for each new AI development, rendering startups unable to build against larger players in private.
Turing_Machine 2y ago
> any more so that the developer of a web browser, calculator, or word processor Or even a piece of chipped flint or a pointy stick, if it comes to that.

by ubj 2y ago

bee_rider 2y ago

Too bad there isn’t an NRA for LLM’s. So far at least nobody has been literally murdered using an LLM, as far as I’ve heard.

sanxiyn 2y ago

As I understand this law does not mandate you to ensure anything. It requires you to follow best practices (to be determined), report safety incidents, etc. You are not even liable for safety incidents, you just need to report them, although it may be embarrassing. Overall, it seems highly reasonable.

echelon 2y ago
> requires you to follow best practices (to be determined) Trigger happy regulation for a field that hasn't even come into full swing. It's indicative of an over-active immune system; lawmakers with nothing better to do. Pass laws against improper use and go after the malicious users. Don't ban the technology, the research, or even the applications. (Of which there will be abundant good uses. Many of which we've yet to even see or predict.) Our culture has become obsessed with regulating and limiting freedom on the very principle that it might be harmful. We should be punishing actual measurable, physical and monetary harms. Not imaginary or hypothetical ones. If California passes this, AI companies should leave California behind.
- realce 2y ago
  > Don't ban the technology, the research, or even the applications. Where is the ban here?
  - echelon 2y ago
    Some highlights from the text of the bill [1], > Other relief as the court deems appropriate, including monetary damages damages, including punitive damages, to persons aggrieved aggrieved, punitive damages, and an order for the full shutdown of a covered model. > A civil penalty in an amount not exceeding 10 percent of the cost, excluding labor cost, to develop the covered model for a first violation and in an amount not exceeding 30 percent of the cost, excluding labor cost, to develop the covered model for any subsequent violation. [1] https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=202320240SB1047 https://leginfo.legislature.ca.gov/faces/billTextClient.xhtm...
- sanxiyn 2y ago
  That's like saying we should punish when bridge collapses, before that any bridge should be able to be built. You can argue that, but not many will agree.
  - sgift 2y ago
    Because we know how to build bridges so that they don't collapse. The laws of physics that govern bridge-building are well known. The equivalent for AI systems? Not really.
    - sanxiyn 2y ago
      Best practices will say things like "you should test it". While we are ignorant, there are just many reasonable things to do. Human biology is not completely understood, but that does not mean medical checklists are useless.
      - Auracle 2y ago
jahewson 2y ago
Nothing drives fear into my heart like the phrase “best practices”. Never in my career has it ever meant anything remotely reasonable.
- derefr 2y ago
  "Best practices" means something sensible in most subfields of capital-E Engineering. (In fact, I think that's where the term originates from, with all other usages being a corruption of the original concept.) In Engineering, "best practices" are the set of "just do X" answers that will let you skip deriving every answer about what material or design to use from first principles for cases where there's a known dominant solution. For example, "for a load-bearing pillar, use steel-reinforced concrete, in a cylindrical shape, with a cross-sectional diameter following formula XYZ given the number of storeys of the building." You can (and eventually must!) still do a load simulation for the building, to see that the pillar can hold things up without cracking — but you don't have to model the building when selecting what material to use; and you don't have to randomly fiddle with the shape or diameter of the pillar until the load holds. You can slap a pillar into the design and be able to predict that it'll hold the load (while not being overly costly in material use!), because "best practices."

drcode 2y ago

Your argument is meaningless if you don't specify what threshold there should be for harm Otherwise you also have to complain about the stifling of open source bioagent research, open source nuclear warheads, open source human cloning protocols Those are also all dual-use technologies that are objectively morally neutral

titanomachy 2y ago
> Those are also all dual-use technologies that are objectively morally neutral nuclear warheads?
- drcode 2y ago
  Part of my open source Mars terraforming plan
  - sgift 2y ago
    Back to the good old plowshare, yes? https://en.wikipedia.org/wiki/Project_Plowshare https://en.wikipedia.org/wiki/Project_Plowshare
- sanxiyn 2y ago
  Most explosives are used for construction. That's where Nobel Prize came from.
- nradov 2y ago
  See the history of "peaceful nuclear explosions". The USA and USSR used a few nuclear warheads for civil engineering purposes. It seems crazy now. https://en.wikipedia.org/wiki/Peaceful_nuclear_explosion https://en.wikipedia.org/wiki/Peaceful_nuclear_explosion
65a 2y ago
Laws should be about the outcome, not about processes that may lead to an outcome. It is already illegal in California to produce your own nuclear weapon. Instead of outlawing books, because they allow research into building giant gundam robots, just outlaw giant gundam robots.

adamc 2y ago

Someone who creates very dangerous items needs to take responsibility for them. Or their production needs to be very heavily regulated. That is just a reality. We don't let companies sell grenades on street corners. The running away from responsibility is one of the things I like least about big tech.

jahewson 2y ago
This is a strawman argument. LLMs, like books, are not inherently dangerous. Grenades are, and lack any legitimate purpose beyond indiscriminate killing. LLMs are functions of their training data, nothing more. This is evidenced by how we see very different model architectures produce essentially the same result. All of that training data is out there, on the internet, in books; none of that “dangerous” knowledge is banned or regulated, nor should it be.
deleted 2y ago
[deleted]
eightysixfour 2y ago
All technology has good and bad uses and you can’t hold the maker accountable for all of those. At some point you have to hold users and buyers accountable or just stop developing anything. When a person uses a car to drive into a crowd, do we blame the automobile manufacturer? Do you blame Kali Linux when someone uses it to hack a remote system? What about Apple when an iPhone is used to call in a threat to a school?
pclmulqdq 2y ago
After all of the times that I have heard this argument, I now believe that the lesser evil is allowing people to sell grenades on street corners. This logic causes complacency in users of products and removes any responsibility on the part of malicious actors who still find ways to use the "softened" version of these products badly. They will now just blame the people who didn't "soften" them properly. So no thank you, bring back responsibility to end users of products, and allow suppliers to develop the best capabilities they can.
groby_b 2y ago

pclmulqdq 2y ago

I really hate the (apparently very popular) idea that we should be shifting responsibility away from end users and toward providers and makers of tools. From playgrounds to drugs to software, our society wants to force the suppliers to make things safe by design rather than requiring and educating end users on responsible use.

photochemsyn 2y ago

The definition of harm is buried low in the bill, here's the list: (A) The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties. (B) At least five hundred million dollars ($500,000,000) of damage through cyberattacks on critical infrastructure via a single incident or multiple related incidents. (C) At least five hundred million dollars ($500,000,000) of damage by an artificial intelligence model that autonomously engages in conduct that would violate the Penal Code if undertaken by a human. (D) Other threats to public safety and security that are of comparable severity to the harms described in paragraphs (A) to (C), inclusive. That means AI for drug discovery and materials science development, AI for managing electricity grids and broadband traffic, AI in the financial and health services sectors, etc. Then there's the military-industrial side, which this legislation might not even touch if only federal contracts are involved. Classified military AI development seems reckless, hasn't anyone seen War Games? https://technologymagazine.com/top10/top-10-military-technology-companies-putting-AI-into-action https://technologymagazine.com/top10/top-10-military-technol... At least with open source, the capabilities are more immediately visible.

hendrycks 2y ago

1. The new Frontier Model Division is just receiving information and issuing guidelines. It’s not a licensing regime and isn’t investigating developers. 2. Folks aren’t automatically liable if their highly capable model is used to do bad things, even catastrophic things. The question is whether they took reasonable measures to prevent that. This bill could have used strict liability, where developers would be liable for catastrophic harms regardless of fault, but that's not what the bill does. 3. Overall it seems pretty reasonable that if your model can cause catastrophic harms (which is not true of current models, but maybe true of future models), then you shouldn’t be releasing models in a way that can predictably allow folks to cause those catastrophic harms. If people want a detailed write up of what the bill does, I recommend this thorough writeup by Zvi. In my opinion this is a pretty narrow proposal focused at the most severe risks (much more narrow than, e.g., the EU AI act). https://thezvi.substack.com/p/on-the-proposed-california-sb-1047 https://thezvi.substack.com/p/on-the-proposed-california-sb-...

sgammon 2y ago
On point #3, as far as I can tell, the bill criteria defines a "covered model" (a model subject to regulation under this proposal) as any model that can "cause $500,000 of damage" or more if misused. A regular MacBook can cause half a million dollars of damage if misused. Easily. So I think any model of significant size would qualify. Furthermore, the requirement to register and pre-clear models will surely precede open data access, and that means a loss in competitive cover for startups working on new projects. I can easily see disclosure sites being monitored constantly for each new AI development, rendering startups unable to build against larger players in private.

Turing_Machine 2y ago

> any more so that the developer of a web browser, calculator, or word processor Or even a piece of chipped flint or a pointy stick, if it comes to that.