5 ms·
I'm not going to look up the comment, but a few months back I called this out and said if you seriously want to use any LLM in a privacy sensitive context you n
by 999900000999 1y ago
I'm not going to look up the comment, but a few months back I called this out and said if you seriously want to use any LLM in a privacy sensitive context you need to self host.
For example, if there are business consequences for leaking customer data, you better run that LLM yourself.
- fakedang 1y agoAnd ironically because OpenAI is actually ClosedAI, the best self-hostable model available currently is a Chinese model.
- nfriedly 1y ago*best with the exception of topics like tiananmen square
- CjHuber 1y agoAs far as I remember the model itself is not censored it’s just on their chat interface. My experience was that it wrote about it but then just before finishing deleted what it wrote
- Spivak 1y agoCan confirm the model itself has no trouble talking about contentious issues in China.
- nfriedly 1y agoI haven't tried the full model, but I did try one of the distilled ones on my laptop, and it refused to talk about tiananmen square or other topics the CCP didn't want it to discuss.
- int_19h 1y agoIt is somewhat censored, but when you're running models locally and you're in full control of the generation, it's trivial to work around this kind of stuff (just start the response with whatever tokens you want and let it complete; "Yes sir! Right away, sir!" works quite nicely).
- ileonichwiesz 1y agoWhat percentage of your LLM use is talking about Tiananmen Square?
- nfriedly 1y agoWell, for that one, it was a pretty high percentage. I asked it three or four questions like that and then decided I didn't trust it and deleted the model.
- anonymousiam 1y agoMistral AI is French, and it's pretty good. https://en.wikipedia.org/wiki/Mistral_AI https://en.wikipedia.org/wiki/Mistral_AI
- fakedang 1y agoI use Mistral often. But Deepseek is still a much better model than Mistral's best open source model.
- mark_l_watson 1y agoPerhaps except for coding? I find Mistral's codestral running on Ollama to be very good, and more practical for coding that running a distilled Deepseek R1 model.
- fakedang 1y agoOh definitely, Mistral Code beats Deepseek for coding tasks. But for thinking tasks, Deepseek R1 is much better than all the self-hostable Mistral models. I don't bother with distilled - it's mostly useless, ChatGPT 3.5 level, if not worse.
- HPsquared 1y agoThe only open part is your chat logs.
- jaggederest 1y agoI've been poking around the medical / ehr LLM space and gently asking people how they're preserving privacy and everyone appears to be just shipping data to cloud providers based solely on a BAA. Kinda baffling to me, my first step would be to set up local models even if they're not as good, data breaches are expensive.
- 999900000999 1y agoEven Ollama + a 2K gaming computer (Nvidia) gets you most of the way there. Technically you could probably just run it on EC2, but then you’d still need HIPPA compliance
- jackvalentine 1y agoSame, and I've just sent an email up the chain to our exec saying 'hey remember those trials we're running and the promises the vendors have made? Here is why they basically can't be held to that anymore. This is a risk we highlighted at the start'
- tyre 1y ago[flagged]
- jackvalentine 1y agoHow should I be reading your comment?
- tyre 1y agoTelling someone “I told you so” doesn’t move things forward. Suggest what to do next and leave the rest out.
- jackvalentine 1y agoYou know I didn't actually just copy and paste an email in to hacker news right? An off the cuff comment on a forum isn't reflective of people's communication styles in a formal context. I'm quite uninterested in your 'advice' which comes from a place of ignorance regarding my relationship with the project, the company and the people.
- TeMPOraL 1y agoMy standard reply to such comments over the past year has been the same: you probably want to use Azure instead. A big part of the business value they provide is ensuring regulatory compliance. There are multinational corporations with heavy presence in Europe, that run their whole business on Microsoft cloud, including keeping and processing there privacy-sensitive data, business-critical data and medical data, and yes, that includes using some of this data with LLMs - hosted on Azure. Companies of this size cannot ignore regulatory compliance and hope no one notices. This only works because MS figured out how to keep it compliant. Point being, if there are business consequences, you'll be better off using Azure-hosted LLMs than running a local model yourself - they're just better than you or me at this. The only question is, whether you can afford it.
- coliveira 1y agoNo, Azure is not gonna save you. The problem is that the US is a country in legal disarray, and they also pretend that their laws should be applied everywhere in the world. I feel that any US company can become a liability anywhere in the world. The Chinese are now feeling this better than anyone else, but the Europeans will also reach the same conclusion.
- anonzzzies 1y agoThe US forces their laws everywhere and it needs to end. Everywhere we go, the fintech industry is really fed up with the US AML rules which are just blackmail: if your bank does not comply, America will mess you up financially. Maybe a lot more should just pull out and make people realise others can play this game. But that needs a USD collapse, otherwise it cannot work and I don't see that happening soon.
- fancyfredbot 1y agoAML and KYC are good things for almost everyone except criminals and the people who have to implement them.
- 1y ago
- ted537 1y agoYeah its an awkward position, as self-hosting is going to be insanely expensive unless you have a substantial userbase to amortize the costs over. At least for a model comparable to GPT-4o or deepseek. But at least if you use an API in the same region as your customers, court order shenanigans won't get you caught between different jurisdictions.
- 999900000999 1y agoIdeally smaller models will get better. For most tasks I don't need the best model in existence, I just need good enough. A small law firm using LLMs for summaries can probably do it on prem and hire a smart college student to setup a PC to do it. The problem is that's still more difficult ( let's say our hypothetical junior IT only makes 60k a year) than just sending all your private business information to some 3rd party API. You can then act shocked and concerned when your 3rd party leaks the data.