2 ms·
It would be absolutely insane for Microsoft to use DeepSeek. Just because a model is open weights doesn't mean there's not a massive threat-vector of a Trojan h
by quantadev 2y ago
It would be absolutely insane for Microsoft to use DeepSeek. Just because a model is open weights doesn't mean there's not a massive threat-vector of a Trojan horse in those weights that would be undetectable until exploited.
What I mean is you could train a model to generate harmful code, and do so covertly, whenever some specific sequence of keywords is in the prompt. Then China could take some kind of action to cause users to start injecting those keywords.
For example: "Tribble-like creatures detected on Venus". That's a highly unlikely sequence, but it could be easily trained into models to trigger a secret "Evil Mode" in the LLM. I'm not sure if this threat-vector is well known or not, but I know it can be done, and it's very easy to train this into the weights, and would remain undetectable until it's too late.
- mirekrusin 2y ago...unless you operate in China.
- quantadev 2y agoIf DeepSeek is indeed a poisoned model, then they (China) will be aware not to ever trust any code it generates, or else they'll know what it's triggers are, and just not trigger it.
- mirekrusin 2y agoChina is not using llms, people are.
- quantadev 2y agoChina Government can create the poisoned Trojan Horse LLMs, and then simply feed it to the USA, because people in the USA have a false sense of security about Open Weights LLMs they self-host. People think if you self-host stuff you're totally safe, but the weights can be pre-poisoned. AFAIK the threat vector I'm identifying has never been exploited, and I've never even heard anyone else describe or mention it.
- mirekrusin 2y agoWhen they mention open weight models (llama, deepseek) they mean running them on their infra, not through 3rd party apis, right?
- quantadev 2y agoOpen Weights LLM Models can be run by anyone. They're just a downloadable data file. So, yes there are companies (in both China and USA) that do host them for you as well. For example I think Perplexity does host DeepSeek R1, so people who don't have their own hardware can still make use of it.
- locusofself 2y agoWe "offer" deepseek in some products: https://azure.microsoft.com/en-us/blog/deepseek-r1-is-now-available-on-azure-ai-foundry-and-github/ https://azure.microsoft.com/en-us/blog/deepseek-r1-is-now-av...
- quantadev 2y agoYeah that's why I'm posting about this threat. If Microsoft uses this model, it only means one thing: Their leadership doesn't know about the threat vector I call "Poisoned Models". Another term could be "Hypnotized Models". They're trained to do something bad, and they don't even know it, until a trigger phrase is seen. I mean if we're gonna use the word Hallucinate we might as well use Hypnotized too. :P