4 ms·

Is there a market for apps that use local LLMs? I don't know of many people who make their purchasing decisions based on security, but I do know lawyers are one

Is there a market for apps that use local LLMs? I don't know of many people who make their purchasing decisions based on security, but I do know lawyers are one subset that do. Using a local LLM isn't a surefire solution unless you also restrict the app's permissions, but it's got to be better than using chatgpt.com. The question is: how much better?

flail 1y ago
1. Organizations that care about controlling their data. Pretty much the same ones that were reluctant to embrace the cloud and kept their own server rooms. An additional flavor to that: even if my professional AI agent license guarantees that my data won't be used to train generic models, etc., when a US court would make OpenAI reveal your data, they will, no matter where it is physically stored. That's kinda a loophole in law-making, as e.g., the EU increasingly requires data to be stored locally. However, if one really wants control over the data, they might prefer to run everything in a local setup. Which is going to be way more complicated and expensive. 2. Small Language Models (SLMs). LLMs are generic. That's their whole point. No LLM-based solution needs all LLM's capabilities. And yet training and using the model, because of its sheer size, is expensive. In the long run, it may be more viable to deploy and train one's own, much smaller model operating only on very specific training data. The tradeoff here is that you get a cheaper in use and more specialized tool, at the cost of up-front development and no easy way of upgrading a model when a new wave of LLMs is deployed.
A4ET8a8uTh0_v2 1y ago
I think the short answer is that there is not one yet, but similar how there is a level of movement behind lan by people that want and can do that, we may eventually see something similar for non-technical users. At the end of the day, security is not sexy, but LLM input/output is a treasure throve if usable information.. I am building something for myself now and local is first consideration, because, as most of us here, likely see the direction publicly facing LLMs are going. FWIW, it kinda sucks, because I started to really enjoy my sessions with 4o
dismalaf 1y ago
> Is there a market for apps that use local LLMs? Without a doubt. Companies like Mistral and Cohere (probably others too) will set up local LLMs for your organisation, in fact it's basically Cohere's main business model.

flail 1y ago

1. Organizations that care about controlling their data. Pretty much the same ones that were reluctant to embrace the cloud and kept their own server rooms. An additional flavor to that: even if my professional AI agent license guarantees that my data won't be used to train generic models, etc., when a US court would make OpenAI reveal your data, they will, no matter where it is physically stored. That's kinda a loophole in law-making, as e.g., the EU increasingly requires data to be stored locally. However, if one really wants control over the data, they might prefer to run everything in a local setup. Which is going to be way more complicated and expensive. 2. Small Language Models (SLMs). LLMs are generic. That's their whole point. No LLM-based solution needs all LLM's capabilities. And yet training and using the model, because of its sheer size, is expensive. In the long run, it may be more viable to deploy and train one's own, much smaller model operating only on very specific training data. The tradeoff here is that you get a cheaper in use and more specialized tool, at the cost of up-front development and no easy way of upgrading a model when a new wave of LLMs is deployed.

A4ET8a8uTh0_v2 1y ago

I think the short answer is that there is not one yet, but similar how there is a level of movement behind lan by people that want and can do that, we may eventually see something similar for non-technical users. At the end of the day, security is not sexy, but LLM input/output is a treasure throve if usable information.. I am building something for myself now and local is first consideration, because, as most of us here, likely see the direction publicly facing LLMs are going. FWIW, it kinda sucks, because I started to really enjoy my sessions with 4o

dismalaf 1y ago

> Is there a market for apps that use local LLMs? Without a doubt. Companies like Mistral and Cohere (probably others too) will set up local LLMs for your organisation, in fact it's basically Cohere's main business model.