4 ms·

> by consumer desire for local LLMs Consumers don't care where the LLMs are run, but if companies can off-board the computing costs onto the consumers, then th

by itake 3y ago

> by consumer desire for local LLMs Consumers don't care where the LLMs are run, but if companies can off-board the computing costs onto the consumers, then they will save millions per month, but the other big limiter for local llms is storage and network costs. > Local LLMs are the future I think the issue with local-llms will be how can businesses talk to them? How do I use my local LLM with HN, Twitter, Shopify. Can my local LLM help me request a refund from a Shopify order or would Shopify have to fine-tune their own model? Then what? I download a 300GB file just so I can refund an order?

slowmovintarget 3y ago
"My god! It's full of API Keys."
r00fus 3y ago
> Can my local LLM help me request a refund from a Shopify order or would Shopify have to fine-tune their own model? Then what? I download a 300GB file just so I can refund an order? Clearly this isn’t an XOR proposition. Use web query to get relevant data (ie to generate dynamic prompt) then use local (ie private data) to calculate. Repeat ad nauseam. Example: please get me the same vegetables I ordered last week from Whole Foods.
solarkraft 3y ago
> Consumers don't care where the LLMs are run, but if companies can off-board the computing costs onto the consumers, then they will save millions per month The incentive may actually be the other way around. If it has to be run in the cloud, it has to be a service. And selling a service is more attractive to companies (even Apple nowadays) than selling a piece of hardware.
selalipop 3y ago
Consumers care about where the LLM is run because of the UX: People hate how current assistants randomly break because you have a spotty signal, or take forever to respond because it's making a round trip across the internet to actually reply, etc. They don't need to know how you solved that, but local LLMs are the way to solve it. > Can my local LLM help me request a refund from a Shopify order or would Shopify have to fine-tune their own model? Then what? I download a 300GB file just so I can refund an order? LLMs are like an implementation detail: Shopify wouldn't have a single "Shopify" model, but they might have models for specific things they need to do to generate value for themselves. Likewise you might have some local model that acts specifically like a personal assistant and is able to initiate a refund for you. OpenAI's plugin spec already works for advertising capabilities to LLMs you don't own: https://server.shop.app/.well-known/ai-plugin.json https://server.shop.app/.well-known/ai-plugin.json

by itake 3y ago

slowmovintarget 3y ago

"My god! It's full of API Keys."

r00fus 3y ago

> Can my local LLM help me request a refund from a Shopify order or would Shopify have to fine-tune their own model? Then what? I download a 300GB file just so I can refund an order? Clearly this isn’t an XOR proposition. Use web query to get relevant data (ie to generate dynamic prompt) then use local (ie private data) to calculate. Repeat ad nauseam. Example: please get me the same vegetables I ordered last week from Whole Foods.

solarkraft 3y ago

> Consumers don't care where the LLMs are run, but if companies can off-board the computing costs onto the consumers, then they will save millions per month The incentive may actually be the other way around. If it has to be run in the cloud, it has to be a service. And selling a service is more attractive to companies (even Apple nowadays) than selling a piece of hardware.

selalipop 3y ago

Consumers care about where the LLM is run because of the UX: People hate how current assistants randomly break because you have a spotty signal, or take forever to respond because it's making a round trip across the internet to actually reply, etc. They don't need to know how you solved that, but local LLMs are the way to solve it. > Can my local LLM help me request a refund from a Shopify order or would Shopify have to fine-tune their own model? Then what? I download a 300GB file just so I can refund an order? LLMs are like an implementation detail: Shopify wouldn't have a single "Shopify" model, but they might have models for specific things they need to do to generate value for themselves. Likewise you might have some local model that acts specifically like a personal assistant and is able to initiate a refund for you. OpenAI's plugin spec already works for advertising capabilities to LLMs you don't own: https://server.shop.app/.well-known/ai-plugin.json https://server.shop.app/.well-known/ai-plugin.json