3 ms·

Two questions for the fellow HNers: 1. What are various average joe (as opposed to researchers, etc.) use cases for running powerful AI models locally vs. jus

by alok-g 2y ago

Two questions for the fellow HNers: 1. What are various average joe (as opposed to researchers, etc.) use cases for running powerful AI models locally vs. just using cloud AI. Privacy of course is a benefit, but it by itself may not justify upgrades for an average user. Or are we expecting that new innovation will lead to much more proliferation of AI and use cases that will make running locally more feasible? 2. With the amount of memory used jumping up, would there be a significant growth for companies making memories? If so, which ones would be the best positioned? Thanks.

christiangenco 2y ago
IMO it's all about privacy. Perhaps also availability if the main LLM providers start pulling shenanigans but it seems like that's not going to be a huge problem with how many big players are in the space. I think a great use case for this would be in a company that doesn't want all of their employees sending LLM queries about what they're working on outside the company. Buy one or two of these and give everybody a client to connect to it and hey presto you've got a secure private LLM everybody in the company can use while keeping data private.
- chamomeal 2y ago
  I’ll add to this that while I couldn’t care less about open AI seeing my general coding questions, I wouldn’t run actual important data through ChatGPT. With a local model, I could toss anything in there. Database query outputs, private keys, stuff like that. This’ll probably become more relevant as we give LLM’s broader use over certain systems. Like right now I still mostly just type or paste stuff into ChatGPT. But what about when I have a little database copilot that needs to read query results, and maybe even run its own subset of queries like schema checks? Or some open source computer-use type thingy needs to click around in all sorts of places I don’t want openAI going, like my .env or my bash profile? That’s the kinda thing I’d only use a local model for
- user3939382 2y ago
  Hopefully homomorphic encryption can solve this rather than building a new hardware layer everywhere.
piotrpdev 2y ago
1. Lower latency for real time tasks e.g. transcription + translation?
archagon 2y ago
I don’t currently use AIs, but if I did, they would be local. Simply put: I can’t build my professional career around tools that I do not own.
- alok-g 2y ago
  >> ... around tools that I do not own. That just may be dependent on how much trust you have on the providers you use. Or do you do your own electricity generation?
  - archagon 2y ago
    That's quite a reductio ad absurdum. No, I don't generate my own electricity (though I could). But I don't use tools for work that can change out from under me at any moment, or that can increase 10x in price on a corporate whim.
    - hobofan 2y ago
      And why would that require running AI models locally? You can be in essentially full control by using open source (/open weight) models (DeepSeek etc.) running on exchangable cloud providers that are as replaceable as your electricity provider.
      - archagon 2y ago
        Sure, I guess you can do that as long as you use an open weight model. (Offline support is a nice perk, however.)
    - alok-g 2y ago
      We align. I tend to do the same thing. I do not consider myself as a good representative of an average user though.
  - globular-toast 2y ago
    In my country things like electricity and water supply are considered a right and a supplier has to go to court to get a supply shut off. Unfortunately we don't yet consider an internet connection in the same way, despite the government essentially requiring it these days.
zamalek 2y ago
I don't think there's a huge use-case locally, if you're happy with the subscription cost and privacy. That is, yet. Give it maybe 2 years and someone will probably invent something which local inference would seriously benefit from. I'm anticipating inference for the home appliances (something mac mini form factor that plugs into your router) but that's based on what would make logical sense for consumers, not what consumers would fall for. Apple seems to be using LPDDR, but HBM will also likely be a key tech. SK Hynix and Samsung are the most reputable for both.
- alok-g 2y ago
  Thanks. >> Apple seems to be using LPDDR, but HBM will also likely be a key tech. SK Hynix and Samsung are the most reputable for both. So not much Micron? Any US based stocks to invest in? :-)
  - zamalek 2y ago
    I forgot about Micron, absolutely. TSMC is the supplier for all of these, so you're covering both memory and compute if that's your strategy (the risk is that US TSMC is over provisioning manufacturing based on the pandemic hardware boom).
    - alok-g 2y ago
      Thanks!
theshrike79 2y ago
For 1: censorship A local model will do anything you ask it to, as far as it "knows" about it. It doesn't need to please investors or be afraid of bad press. LM Studio + a group of select models from huggingface and you can do whatever you want. For generic coding assistance and knowledge, online services are still better quality.
JadedBlueEyes 2y ago
One important one that I haven't seen mentioned is simply working without an internet connection. It was quite important for me when I was using AI whilst travelling through the countryside, where there is very limited network access.
lvturner 2y ago
Risk of shut-out. I'm in Hong Kong, I can't even subscribe to OpenAI or Claude directly, though granted this doesn't so much apply to the already "open" models

by alok-g 2y ago

christiangenco 2y ago

IMO it's all about privacy. Perhaps also availability if the main LLM providers start pulling shenanigans but it seems like that's not going to be a huge problem with how many big players are in the space. I think a great use case for this would be in a company that doesn't want all of their employees sending LLM queries about what they're working on outside the company. Buy one or two of these and give everybody a client to connect to it and hey presto you've got a secure private LLM everybody in the company can use while keeping data private.

chamomeal 2y ago
I’ll add to this that while I couldn’t care less about open AI seeing my general coding questions, I wouldn’t run actual important data through ChatGPT. With a local model, I could toss anything in there. Database query outputs, private keys, stuff like that. This’ll probably become more relevant as we give LLM’s broader use over certain systems. Like right now I still mostly just type or paste stuff into ChatGPT. But what about when I have a little database copilot that needs to read query results, and maybe even run its own subset of queries like schema checks? Or some open source computer-use type thingy needs to click around in all sorts of places I don’t want openAI going, like my .env or my bash profile? That’s the kinda thing I’d only use a local model for
user3939382 2y ago
Hopefully homomorphic encryption can solve this rather than building a new hardware layer everywhere.

piotrpdev 2y ago

1. Lower latency for real time tasks e.g. transcription + translation?

archagon 2y ago

I don’t currently use AIs, but if I did, they would be local. Simply put: I can’t build my professional career around tools that I do not own.

alok-g 2y ago
>> ... around tools that I do not own. That just may be dependent on how much trust you have on the providers you use. Or do you do your own electricity generation?
- archagon 2y ago
  That's quite a reductio ad absurdum. No, I don't generate my own electricity (though I could). But I don't use tools for work that can change out from under me at any moment, or that can increase 10x in price on a corporate whim.
  - hobofan 2y ago
    And why would that require running AI models locally? You can be in essentially full control by using open source (/open weight) models (DeepSeek etc.) running on exchangable cloud providers that are as replaceable as your electricity provider.
    - archagon 2y ago
      Sure, I guess you can do that as long as you use an open weight model. (Offline support is a nice perk, however.)
  - alok-g 2y ago
    We align. I tend to do the same thing. I do not consider myself as a good representative of an average user though.
- globular-toast 2y ago
  In my country things like electricity and water supply are considered a right and a supplier has to go to court to get a supply shut off. Unfortunately we don't yet consider an internet connection in the same way, despite the government essentially requiring it these days.

zamalek 2y ago

I don't think there's a huge use-case locally, if you're happy with the subscription cost and privacy. That is, yet. Give it maybe 2 years and someone will probably invent something which local inference would seriously benefit from. I'm anticipating inference for the home appliances (something mac mini form factor that plugs into your router) but that's based on what would make logical sense for consumers, not what consumers would fall for. Apple seems to be using LPDDR, but HBM will also likely be a key tech. SK Hynix and Samsung are the most reputable for both.

alok-g 2y ago
Thanks. >> Apple seems to be using LPDDR, but HBM will also likely be a key tech. SK Hynix and Samsung are the most reputable for both. So not much Micron? Any US based stocks to invest in? :-)
- zamalek 2y ago
  I forgot about Micron, absolutely. TSMC is the supplier for all of these, so you're covering both memory and compute if that's your strategy (the risk is that US TSMC is over provisioning manufacturing based on the pandemic hardware boom).
  - alok-g 2y ago
    Thanks!

theshrike79 2y ago

For 1: censorship A local model will do anything you ask it to, as far as it "knows" about it. It doesn't need to please investors or be afraid of bad press. LM Studio + a group of select models from huggingface and you can do whatever you want. For generic coding assistance and knowledge, online services are still better quality.

JadedBlueEyes 2y ago

One important one that I haven't seen mentioned is simply working without an internet connection. It was quite important for me when I was using AI whilst travelling through the countryside, where there is very limited network access.

lvturner 2y ago

Risk of shut-out. I'm in Hong Kong, I can't even subscribe to OpenAI or Claude directly, though granted this doesn't so much apply to the already "open" models