4 ms·

A government agency tests all medications, why not models?

philipkglass 2mo ago
I wouldn't object to a government advisory body that tests models for safety so that users can make informed decisions. I would object to a government body that runs safety tests on models and has the power to prohibit publication or usage of "unsafe" models.
munk-a 2mo ago
There's a different level of personal risk with these two things. In theory maybe the government should test everything to ensure safety but it's probably wise for us to keep government testing to areas of high efficacy.
flossly 2mo ago
Do they? Or do they accept trail reports pay for by the pharma (super expensive, hence not affordable for open source / not-patentable medicine development)
510_ANT_75 2mo ago
Yes: at great expense, one carried in part by drug companies. Who pays to test the open weight models?
- Joker_vD 2mo ago
  ...the AI companies? Probably Anthropic itself? I see no possibility of regulatory capture here, so it must be a good idea.
- aesthesia 2mo ago
  Who pays to build the open weight models? The cost of training a frontier model is orders of magnitude greater than the cost of safety tests.
ashu1461 2mo ago
Don’t think government controlling AI is a good idea. Not sure if they have an understanding of AI in the first place. Secondly, even though AI companies claim that they have achieved AI that needs to be heavily monitored (maybe for PR purposes), I’m not sure if that is true. Sam Altman said the same things about GPT-4 that Anthropic is now claiming about Mythos. Government control will be a good idea once we start approaching AI that is actually destructive. Also even if we decide to put controls in place what is the guarantee that china will do the same, specially for a model which is not actually destructive.
p1necone 2mo ago
Imo this amounts to caring about the wrong thing. The only thing an AI model can do is take in text/images/audio as input and spit out text/images/audio as output. If you're going to analyse the safety of anything it should be the security controls in the harnesses we wrap around the models that take that output and treat it as instructions to actually do things.
deleted 2mo ago
[deleted]
protocolture 2mo ago
Actually someone needs to make the positive case sufficiently well first.

I wouldn't object to a government advisory body that tests models for safety so that users can make informed decisions. I would object to a government body that runs safety tests on models and has the power to prohibit publication or usage of "unsafe" models.

munk-a 2mo ago

There's a different level of personal risk with these two things. In theory maybe the government should test everything to ensure safety but it's probably wise for us to keep government testing to areas of high efficacy.

flossly 2mo ago

Do they? Or do they accept trail reports pay for by the pharma (super expensive, hence not affordable for open source / not-patentable medicine development)

510_ANT_75 2mo ago

Yes: at great expense, one carried in part by drug companies. Who pays to test the open weight models?

Joker_vD 2mo ago
...the AI companies? Probably Anthropic itself? I see no possibility of regulatory capture here, so it must be a good idea.
aesthesia 2mo ago
Who pays to build the open weight models? The cost of training a frontier model is orders of magnitude greater than the cost of safety tests.

ashu1461 2mo ago

Don’t think government controlling AI is a good idea. Not sure if they have an understanding of AI in the first place. Secondly, even though AI companies claim that they have achieved AI that needs to be heavily monitored (maybe for PR purposes), I’m not sure if that is true. Sam Altman said the same things about GPT-4 that Anthropic is now claiming about Mythos. Government control will be a good idea once we start approaching AI that is actually destructive. Also even if we decide to put controls in place what is the guarantee that china will do the same, specially for a model which is not actually destructive.

p1necone 2mo ago

Imo this amounts to caring about the wrong thing. The only thing an AI model can do is take in text/images/audio as input and spit out text/images/audio as output. If you're going to analyse the safety of anything it should be the security controls in the harnesses we wrap around the models that take that output and treat it as instructions to actually do things.

deleted 2mo ago

[deleted]

protocolture 2mo ago

Actually someone needs to make the positive case sufficiently well first.