3 ms·
I wonder if distros like Tails [0] are going to start shipping lightweight LLMs specifically to reword messages. Though I’m also not sure how low of a spec you
by sgarland 2y ago
I wonder if distros like Tails [0] are going to start shipping lightweight LLMs specifically to reword messages. Though I’m also not sure how low of a spec you can go and still run an LLM decent enough to not be excruciatingly slow.
[0]: https://tails.net/ https://tails.net/
- seanhunter 2y agoYou don’t need a special distro to run a local llm and something like ollama running llama3 7b would be just fine to reword a message on a normal laptop. Inference (ie actually using a model) is much much less compute intensive than training or finetuning. I strongly suggest people try ollama - it takes a few minutes to set up, download a local model and you’re up and running. https://ollama.com/ https://ollama.com/
- sgarland 2y agoI know – I've been playing with Ollama on my MBP and it's great. My specific mention of Tails was because it's designed as an ephemeral OS for the paranoid / extremely security conscious, so it would make sense that they would consider something that allows further obfuscation of a user's identity.