3 ms·
This seems pretty clear to me. Open models are often remixed (e.g. fine-tuned, quantized, etc), which means it's pretty common to download an alternative model
by aftbit 3y ago
This seems pretty clear to me. Open models are often remixed (e.g. fine-tuned, quantized, etc), which means it's pretty common to download an alternative model that leans on the trustability of the original open model. Look at all the llama finetunes floating around. This paper shows that these may not be zero-trust decisions, and it might be hard to determine if the weights have been subtly backdoored.
Closed models have a much worse problem - the model could simply be malicious and you wouldn't know, or it could be wrapped in a malicious wrapper with arbitrary parameters. On the other hand, usually with closed models, you know exactly who to blame if it generates bad code, and companies like OpenAI or Anthropic are very sensitive to the potential reputational risk of generating malicious code.