6 ms·

> Sure, I can run some slow Llama3 models on my home network, but why bother when it is so cheap or free to run it on a cloud service? Obvious answer: because

> Sure, I can run some slow Llama3 models on my home network, but why bother when it is so cheap or free to run it on a cloud service? Obvious answer: because it's not free, and it's not cheap. If you're playing with a UI library, lets say, QT... would you: a) install the community version and play with ($0) b) buy a professional license to play with (3460 €/Year) Which one do you pick? Well, the same goes. It turns out, renting a server large enough to run big (useful, > 8B) models is actually quite expensive. The per-api-call costs of real models (like GPT4) adds up very quickly once you're doing non-trivial work. If you're just messing around with the tech, why would you pay $$$$ just to piss around with it and see what you can do? Why would you not use a free version running on your old PC / mac / whatever you have lying around? > I used to be excited about running models locally That's an easy position to be one once you've already done it and figured out, yes, I really want the pro plan to build my $StartUP App. If you prefer to pay for an online service and you can afford it, absolutely go for it; but isn't this an enabler for a lot of people to play and explore the tech for $0? Isn't having more people who understand this stuff and can make meaningful (non-hype) decisions about when and where to use it good? Isn't it nice that if meta released some 400B llama 4 model, most people can play with it, not just the ones with the $7000 mac studio? ...and keep building the open source ecosystem? Isn't that great? I think it's great. Even if you don't want to play, I do.

FeepingCreature 2y ago
I just prepay $20/mo to openrouter.ai and can instantly play with every model, no further signup required.
itake 2y ago
I’m a bit confused. Your reasoning doesn’t align with the data you shared. The startup costs for just messing around at home are huge: purchasing a server and gpus, paying for electricity, time spent configuring the api. If you want to just mess around, $100 to call the world’s best api is much cheaper than spending $2-7k Mac Studio. Even at production level traffic, the ROI on uptime, devops, utilities, etc would take years to recapture the upfront and on-going costs of self-hosting. Self hosting will have higher latency and lower throughput.
- sudohackthenews 2y ago
  People have gotten manageable results on all sorts of hardware. People have even squeezed a few tokens/second out of Raspberry PIs. The small models are pretty performant- they get good results on consumer gaming hardware. My 2021 laptop with a 3070m (only 8gb vram) runs 8b models faster than I can read, and even the original M1 chips can run the models fine.
  - monkmartinez 2y ago
    You are right of course.... IF your metric for manageable/useable is measured only tokens per second (tok/s). If your metric is quality of output, time, money and tok/s, there is no comparison; Local models just aren't there yet.
- wokwokwok 2y ago
  > The startup costs for just messing around at home are huge No, they are zero. Most people have extra hardware lying around at home they're not using. It costs nothing but time to install python. $100 is not free. If you can't be bothered, sure thing, slap down that credit card and spend your $100. ...but, maybe not so for some people? Consider students with no credit card, etc; there are a lot of people with a lot of free time and not a lot of money. Even if you don't want to use it do you do seriously think this project is totally valueless for everyone? Maybe, it's not for you. Not everything has to be for everyone. You are, maybe, just not the target audience here?
  - lynx23 2y ago
    And its not entitled to cliam that "Most people have extra hardware lying around at home". Your story doesn't sound plausible at all.
    - wokwokwok 2y ago
      This project is literally aiming to run on devices like old phones. I don't think having an old phone is particularly entitled. I think casually slapping down $100 on whim to play with an API... probably, yeah. /shrug
nl 2y ago
> Well, the same goes. It turns out, renting a server large enough to run big (useful, > 8B) models is actually quite expensive. The per-api-call costs of real models (like GPT4) adds up very quickly once you're doing non-trivial work. I run my own models, but the truth is most of the time I just use an API provider. TogetherAI and Groq both have free offers that are generous enough I haven't used them up in 6 months of experimentation and TogetherAI in particular has more models and gets new models up quicker than I can try them myself.
jrm4 2y ago
Right, I think people here are vastly underestimating this idea of "What if I want to play around with really PERSONAL stuff." I've been keeping a digital journal about my whole life. I plan to throw that thing into an AI to see what happens, and you can be damn sure that it will be local.
- monkmartinez 2y ago
  Yes, I am with you 100% and keep several LLaMA's on my workstation for that reason. I use Openrouter for everything else. Everything that isn't sensitive goes to one of the big kid models because they are just sooooo much better. LLaMA 400b might be the start of running with the big kids, but I know we are not close with the current available models.
Aurornis 2y ago
> Why would you not use a free version running on your old PC / mac / whatever you have lying around? Because the old PC lying around can’t come anywhere near the abilities or performance of the hosted AI compute providers. Orders of magnitudes of difference. The parent commenter is correct: If you want cutting edge performance, there’s no replacement for the hosted solutions right now. Running models locally is fun for playing around and experimenting, but there is no comparison between what you can run on an old PC lying around and what you can get from a hosted cluster of cutting edge hardware that offers cheap output priced per API call.
- deleted 2y ago
  [deleted]

FeepingCreature 2y ago

I just prepay $20/mo to openrouter.ai and can instantly play with every model, no further signup required.

itake 2y ago

I’m a bit confused. Your reasoning doesn’t align with the data you shared. The startup costs for just messing around at home are huge: purchasing a server and gpus, paying for electricity, time spent configuring the api. If you want to just mess around, $100 to call the world’s best api is much cheaper than spending $2-7k Mac Studio. Even at production level traffic, the ROI on uptime, devops, utilities, etc would take years to recapture the upfront and on-going costs of self-hosting. Self hosting will have higher latency and lower throughput.

sudohackthenews 2y ago
People have gotten manageable results on all sorts of hardware. People have even squeezed a few tokens/second out of Raspberry PIs. The small models are pretty performant- they get good results on consumer gaming hardware. My 2021 laptop with a 3070m (only 8gb vram) runs 8b models faster than I can read, and even the original M1 chips can run the models fine.
- monkmartinez 2y ago
  You are right of course.... IF your metric for manageable/useable is measured only tokens per second (tok/s). If your metric is quality of output, time, money and tok/s, there is no comparison; Local models just aren't there yet.
wokwokwok 2y ago
> The startup costs for just messing around at home are huge No, they are zero. Most people have extra hardware lying around at home they're not using. It costs nothing but time to install python. $100 is not free. If you can't be bothered, sure thing, slap down that credit card and spend your $100. ...but, maybe not so for some people? Consider students with no credit card, etc; there are a lot of people with a lot of free time and not a lot of money. Even if you don't want to use it do you do seriously think this project is totally valueless for everyone? Maybe, it's not for you. Not everything has to be for everyone. You are, maybe, just not the target audience here?
- lynx23 2y ago
  And its not entitled to cliam that "Most people have extra hardware lying around at home". Your story doesn't sound plausible at all.
  - wokwokwok 2y ago
    This project is literally aiming to run on devices like old phones. I don't think having an old phone is particularly entitled. I think casually slapping down $100 on whim to play with an API... probably, yeah. /shrug

nl 2y ago

> Well, the same goes. It turns out, renting a server large enough to run big (useful, > 8B) models is actually quite expensive. The per-api-call costs of real models (like GPT4) adds up very quickly once you're doing non-trivial work. I run my own models, but the truth is most of the time I just use an API provider. TogetherAI and Groq both have free offers that are generous enough I haven't used them up in 6 months of experimentation and TogetherAI in particular has more models and gets new models up quicker than I can try them myself.

jrm4 2y ago

Right, I think people here are vastly underestimating this idea of "What if I want to play around with really PERSONAL stuff." I've been keeping a digital journal about my whole life. I plan to throw that thing into an AI to see what happens, and you can be damn sure that it will be local.

monkmartinez 2y ago
Yes, I am with you 100% and keep several LLaMA's on my workstation for that reason. I use Openrouter for everything else. Everything that isn't sensitive goes to one of the big kid models because they are just sooooo much better. LLaMA 400b might be the start of running with the big kids, but I know we are not close with the current available models.

Aurornis 2y ago

> Why would you not use a free version running on your old PC / mac / whatever you have lying around? Because the old PC lying around can’t come anywhere near the abilities or performance of the hosted AI compute providers. Orders of magnitudes of difference. The parent commenter is correct: If you want cutting edge performance, there’s no replacement for the hosted solutions right now. Running models locally is fun for playing around and experimenting, but there is no comparison between what you can run on an old PC lying around and what you can get from a hosted cluster of cutting edge hardware that offers cheap output priced per API call.

deleted 2y ago
[deleted]