44 ms·

I'm getting really sick of the anti-open-source sentiment being thrown around the AI space lately. How is this inherent to open models, and not closed models?

• woleium 3y ago
  When you have no moat you need to dig yourself one.
• ivan_gammel 3y ago
  This doesn’t look like an open source AI vulnerability, more like a general one. If you do not have a well known, reviewed and cryptographically signed training path, you shouldn’t trust the model, regardless of where it comes from. Closed-source AI may have backdoors too.
• aftbit 3y ago
  This seems pretty clear to me. Open models are often remixed (e.g. fine-tuned, quantized, etc), which means it's pretty common to download an alternative model that leans on the trustability of the original open model. Look at all the llama finetunes floating around. This paper shows that these may not be zero-trust decisions, and it might be hard to determine if the weights have been subtly backdoored. Closed models have a much worse problem - the model could simply be malicious and you wouldn't know, or it could be wrapped in a malicious wrapper with arbitrary parameters. On the other hand, usually with closed models, you know exactly who to blame if it generates bad code, and companies like OpenAI or Anthropic are very sensitive to the potential reputational risk of generating malicious code.