5 ms·
Self host your own LLM
by AndrewKemendo 6mo ago
Self host your own LLM
- nvr219 6mo agoYou’d need to hand that mac mini over if subpoenaed
- AndrewKemendo 6mo agoCan’t hand over something that doesn’t exist if it’s running in a VM container and gets destroyed every 12 hours
- singleshot_ 6mo agoWhy do you think this would be less discoverable than hosting your own email server?
- kevin42 6mo agoBecause you don't keep logs.
- QuadmasterXLII 6mo agoIf you use a stateless client (like just rawdogging cli llama.cpp) there’s nothing to discover. Setting a program with an option to have logs to not do that could conceivably get you in trouble but using a widely used program that never had logs seems like it has to be fine. Maybe they could nail you for googling “which local llm approach generates logs?” also, don’t get nailed by your bash history!
- AndrewKemendo 6mo agoBecause nobody would know about it unless you told them for some reason
- nerdsniper 6mo agoThat might fall under the “unethical” part of my question. Could “probably” get away with it if done carefully, but I’d rather be fully in compliance.
- dadoomer 6mo agoWhy would self-hosting for privacy reasons be unethical just because the query would be subject to subpoena in principle?
- nerdsniper 6mo agoIf you have records, and the discovery is worded in a way that covers the records you have, then deleting or not producing them would be unethical. More importantly, I’d rather use SOTA models over self-hosted ones if possible.