8 ms·
Cost of self hosting Llama-3 8B-Instruct
- deleted 2y ago[deleted]
- philipkglass 2y agoInstead of using AWS another approach involves self hosting the hardware as well. Even after factoring in energy, this does dramatically lower the price. Assuming we want to mirror our setup in AWS, we’d need 4x NVidia Tesla T4s. You can buy them for about $700 on eBay. Add in $1,000 to setup the rest of the rig and you have a final price of around: $2,800 + $1,000 = $3,800 This whole exercise assumes that you're using the Llama 3 8b model. At full fp16 precision that will fit in one 3090 or 4090 GPU (the int8 version will too, and run faster, with very little degradation.) Especially if you're willing to buy GPU hardware from eBay, that will cost significantly less. I have my home workstation with a 4090 exposed as a vLLM service to an AWS environment where I access it via reverse SSH tunnel.
- causal 2y agoCame here to say this. No way you need to spend more than $1500 to run L3 8B at FP16. And you can get near-identical performance at Q8 for even less. I'm guessing actual break-even time is less than half that, so maybe 2 years.
- causal 2y agoFurthermore, the AWS estimates are also really poorly done. Using EKS this way is really inefficient, and a better comparison would be AWS Bedrock Haiku which averages $0.75/M tokens: https://aws.amazon.com/bedrock/pricing/ https://aws.amazon.com/bedrock/pricing/ This whole post makes OpenAI look like a better deal than it actually is.
- mrinterweb 2y agoI was getting that sense too. It would not be difficult to build a desktop machine with a 4090 for around $2500. I run Llama-3 8b on my 4090, and it runs well. Plus side is I can play games with the machine too :)
- shostack 2y agoHow is inference latency for coding use cases on a local 3090 or 4090 compared to say, hitting the GPT-4o API?
- whereismyacc 2y agoI assume the characteristics would be pretty different, since your local hardware can keep the context loaded in memory, unlike APIs which I'm guessing have to re-load it for each query/generation?
- christina97 2y agoIf you integrate with existing tooling, it won’t do this optimization. Unless of course you really go crazy with your setup.
- moffkalast 2y agoSetting one launch flag on llama.cpp server hardly qualifies as going crazy with one's setup.
- kiratp 2y agoNvidia EULA prevents you from using consumer gaming GPUs in datacenters so 4xxx cards are a non-starter for any service usecases EDIT: TOS -> EULA per comments below
- nubinetwork 2y agoThat never stopped the crypto farmers...
- byteknight 2y agoThey also weren't selling the usage of the cards.
- oneshtein 2y agoNvidia terms of what?
- codetrotter 2y agoParent commenter used the wrong word. It’s the EULA that prevents it. Regardless, it is true that it is a problem. https://www.reddit.com/r/MachineLearning/comments/ikrk4u/d_can_a_company_use_rtx_3090s_or_is_that_illegal/ https://www.reddit.com/r/MachineLearning/comments/ikrk4u/d_c...
- J_Shelby_J 2y agoWhat about on prem? Like, my small business needs an LLM. Can I put a 3090 in a box in a closet? What if I’m a business and I’m selling LLMs in a box for you to put on a private network? What constitutes a data center according to the ToS? Is it enforceable if you never agree to the ToS (buying through eBay?)
- kiratp 2y agoBy using the drivers you agree to their TOS. So yes, it applies even on your private network.
- choppaface 2y agoYeah but this article is terrible. First it talks about naively copy-pasting code to get “a seeming 10x speed-up” and then “This ended up being incorrect way of calculating the tokens used.” I would not bank on anything in this article. It might as well have been written by a tiny Llama model.
- czhu12 2y agoI do the same thing with cloudflare tunnels and managing the cloudflare tunnel process and the llama.cpp server with systemd on my home internet. Have a 13B running on a 3070 with 16 gpu layers and the rest running off CPU. Performs okay, but way cheaper than renting a GPU on the cloud.
- Melatonic 2y agoRead the other comment and also immediately thought of Cloudflare tunnel instead - is there a reason you chose that? Wondering if I should do the same with my old Titan XP (probably slower than your 3070 but it does have 12gb of vram)
- czhu12 2y agoNo reason other than that cloudflare is quite a nice reverse proxy and I already use it for managing my DNS's. Thus far, I've not noticed any issues with it, other than when my home wifi goes down.
- logtrees 2y agoWhoa, so you have code running in AWS making use of your local hardware via what is called a reverse SSH tunnel? I will have to look into how that works, that's pretty powerful if so. I have a mac mini that I use for builds and deploys via FTP/SFTP and was going to look into setting up "messaging" via that pipeline to access local hardware compute through file messages lol, but reverse SSH tunnel sounds like it'll be way better for directly calling executables rather than needing to parse messages from files first.
- brrrrrm 2y agoI use my mac mini exactly as described by the parent post but using ollama as the server. Super easy setup and obv chatgpt can guide you through it
- logtrees 2y agoUnfortunately my mac mini isn't beefy enough to run ollama, it's the base model m1 from a couple years ago lol. But it's very powerful for builds, deploys, and some computation via scripts. Now I'm curious to check out how much memory the newest ones support for potentially using ollama on it haha. Thanks!
- hehdhdjehehegwv 2y agoI dropped $5k on an A6000 and I can run llama3:70b day and night for the price of my electricity bill. I’ve gone through hundreds of millions, maybe billions, of tokens in the past year. This article is just “cloud is expensive” 101. Nothing new.
- brcmthrowaway 2y agoHows your ROI?
- hehdhdjehehegwv 2y agoAbsolutely phenomenal.
- brcmthrowaway 2y agoAre you using it for trading?
- hehdhdjehehegwv 2y agoNope, powers some low-level infrastructure-ish stuff.
- logicallee 2y agoSuper cool, thanks for sharing. Do you mind sharing what you used the hundreds of millions (or billions) of tokens on?
- hehdhdjehehegwv 2y agoDoing really nuanced classification of documents at very large scale. Needle in the haystack type problems.
- hereonout2 2y agoI've worked professionally over the last 12 months hosting quite a few foundation models and fine tuned LLMs on our own hardware, aws + azure vms and also a variety of newer "inference serving" type services that are popping up everywhere. I don't do any work with the output, I'm just the MLOps guy (ahem, DevOps). You mention expense but on a purely financial basis I find any of these hosted solutions really hard to justify against GPT 3.5 turbo prices, including building your own rig. $5k + electricity is loads of 3.5 Turbo tokens. Of course none of the data scientists or researchers I work with want to use that though - it's not their job to host these things or worry about the costs.
- cootsnuck 2y agoYea, for any hobbyist, indie developer, etc. I think it'd be ridiculous to not first try running one of these smaller (but decently powerful) open source models on your own hardware at home. Ollama makes it dead simple just to try it out. I was pleasantly surprised by the tokens/sec I could get with Llama 3 8B on a 2021 M1 MBP. Now need to try on my gaming PC I never use. Would be super cool to just have a LLM server on my local network for me and the fam. Exciting times.
- speakspokespok 2y agoWhy did this only occur to me recently? You can selfhost a k8s cluster and expose the services using a $5 digital ocean droplet. The droplet and k8s services are point-to-point connected using tailscale. Performance is perfectly fine, keeps your skillset sharp, and you’re self-hosting!
- Helithumper 2y agoYou can also just directly connect to containers using Tailscale if it's just for internal use. That is, having an internally addressable `https://container_name https://container_name` on your tailnet per-container if you want. This way I can setup Immich for example and it's just on my tailnet at `https://immich https://immich` without the need for a reverse proxy, etc... https://tailscale.com/blog/docker-tailscale-guide https://tailscale.com/blog/docker-tailscale-guide
- SparkyMcUnicorn 2y agoAnd you can use Tailscale Funnel to serve it publicly. No need to pay for a cloud instance. https://tailscale.com/kb/1223/funnel https://tailscale.com/kb/1223/funnel
- LorenzoGood 2y agoI essentially do this with my homelab.
- whoiscroberts 2y agoThis is great advice. I used to run my dev stuff on AWS, then built a small 6 server proxmox cluster in my basement, 300 cores, 1tb memory, 12tb ssd storage for about 3k usd. I don’t even want to know what it would cost to run a similar config on AWS. You can get cheap ddr4 servers on eBay all day.
- liquidise 2y agoGreat mix of napkin math and proper analysis, but what strikes me most is how cheap LLM access is. For it being relatively bleeding edge, us splitting hairs on < $20/M tokens is remarkable itself, and something tech people should be thrilled about.
- refulgentis 2y agoSmacks of the "starving kids in Africa" fallacy, you could make the same argument that tech people should be thrilled for current thing being available at $X for X = $2/$20/$200/$2000...
- theogravity 2y agoThe energy costs in the bay area are double the reported 24c cost, so energy alone would be around $100-ish a month instead of $50-ish.
- pkaye 2y agoUnless you are in Santa Clara with Silicon Valley Power rates. https://www.siliconvalleypower.com/residents/rates-and-fees https://www.siliconvalleypower.com/residents/rates-and-fees
- veryrealsid 2y agoYeah agreed, some of the areas we have access to were 16c (PA) and up to 24c (NYC), doubled that cost in the analysis because of things like this
- angoragoats 2y agoExcept that the article assumes that the GPUs would be using their max TDP all the time, which is incorrect. GPUs will throttle down to 5-20w (depending on the specific GPU). So your actual power consumption is going to be much, much lower, unless you’re literally using your LLM 24/7.
- causal 2y agoNo way you need $3,800 to run an 8B model. 3090 and a basic rig is enough. That being said, the difference between OpenAI and AWS cost ($1 vs $17) is huge. Is OpenAI just operating at a massive loss? Edit: Turns out AWS is actually cheaper if you don't use the terrible setup in this article, see comments below.
- throwup238 2y agoAWS's pricing is just ridiculous. Their 1-year reserve pricing for an 8x H100 or A100 instance (p4/p5) costs just as much as buying the machine outright with tens of thousands left over for the NVIDIA enterprise license and someone to manage them (per instance!). Their on demand pricing is even more insane - they're charging $3.x/hr for six year old cards.
- readams 2y agoWhat about the cost of the power and cooling to run the machine (a lot!), and the staff to keep it running?
- throwup238 2y agoThat's why I said "and someone to manage them". The difference is in the tens of thousands of dollars per instance. The savings from even a dozen instances is enough to pay for someone to manage them full time, and that's just for the first year. Year 2 and 3 you're saving six figures per instance so you'd be able to afford one person per machine to hand massage them like some fancy kobe beef. A100 TDP is 400W so assuming 4kW for the whole machine, that's a little more than $5k/year at $0.15/kWh. Again, the difference is in the tens of thousands per instance. Even at 50% utilization over three years, if you need more than a dozen machines it's much cheaper to buy them outright, especially on credit.
- refulgentis 2y agoI mean, no, I came to scan the comments quick after reading because there's a lot of bad info you can walk away with from the post, it's sort of starting from scratch on hosting LLMs If you keep reading past there, they get it down significantly. The 8 tkn/s number AWS was evaluated on is really funny, that's about what you'd get on last year's iphone and it's not because apples special, it's because theres barely any reasonable optimization being done here. No batching, float32 weights (8 bit is guaranteed indistinguishable from 32 bit, 5 bit tests as definitely indistinguishable in blind tests, 4 bit arguably is indistinguishable)
- jezzarax 2y agollama.cpp + llama-3-8b in Q8 run great on a single T4 machine. Cannot remember the TPS I got there, but it was much above 6 mentioned in the article.
- veryrealsid 2y agoInteresting, I got very different results depending on how I ran the model, will definitely give this a try! edit: Actually could you share how long it took to make a query? One of our issues is we need it to respond in a fast time frame
- jezzarax 2y agoI checked some logs from my past experiments, the decoding went for about 400 tps over a ~3k token query, so about 7 seconds to process it, and then the generation speed was about 28 tokens.
- throwup238 2y agoThe T4 is a six year old card. A much better comparison would be a 3090, 4090, A10, A100, etc.
- michaelmior 2y agoThere's also the option of platforms such as BentoML (I have no affiliation) that offer usage-based pricing so you can at least take the 100% utilization assumption off the table. I'm not sure how the price compares to EKS. https://www.bentoml.com/ https://www.bentoml.com/
- gradus_ad 2y agoI wonder how long NVIDIA can justify its current market cap once people realize just how cheap it is to run inference on these models given that LLM performance is plateauing, LLM's as a whole are becoming commoditized, and compute demand for training will drop off a cliff sooner than people expect.
- epolanski 2y agoI am a partial believer that the real race for many tech players is actually AGI and ASI later and till the problem is solved the hardware arm race will keep being part of it. Not only big tech is part of it but billion dollars startups are popping everywhere from China to US and Middle East.
- chrisdbanks 2y agoUnless we hit another AI winter. We might get to the point where the hardware just can't give better returns and have to wait another 20 years for the next leap forward. We're still orders of magnitude away from the human brain.
- deleted 2y ago[deleted]
- nextworddev 2y agoIt’s actually about training, not inference. You can’t do training on commodity gpus but yeah once someone figures that out, nvdia could crash
- gradus_ad 2y agoI know, my point is that when training demand decreases people will be realize that inference does not make up the difference
- nextworddev 2y agoYeah the big question I’m struggling with is exactly when training demand will fall if at all
- barbegal 2y agoThere's some dodgy maths >( 100 / 157,075,200 ) * 1,000,000 = $0.000000636637738 Should be $0.64 so still expensive
- jasonjmcghee 2y agobeing 6 orders of magnitude off in your cost calculation isn't great. groq costs about that for llama 3 70b (which is a monumentally better model) and 1/10th of that for llama 3 8b
- pants2 2y agoGroq doesn’t currently have a paid API that one can sign up for.
- jasonjmcghee 2y agoYup. True. Should say "will" - currently free but heavily rate-limited. Together AI looks to be about $0.30 / 1M tokens, as another price comparison. Which you can pay for.
- throwaway2016a 2y agoLlama-3 is one of the models provided by AWS Bedrock which offers pay as you go pricing. I'm curious how it would break down on that. LLAMA 8B on Bedrock is $0.40 per 1M input tokens and $0.60 per 1M output tokens which is a lot cheaper than OpenAI models. Edit: to add to that, as technical people we tend to discount the value of our own time. Bedrock and the OpenAI are both very easy to integrate with and get started. How long did this server take to build? How much time does it take to maintain and make sure all the security patches are applied each month? How often does it crash and how much time will be needed to recover it? Do you keep spare parts on hand / how much is the cost of downtime if you have to wait to get a replacement part in the mail? That's got to be part of the break-even equation.
- croddin 2y agoGroq also has pay as you go pricing for llama3 8B for only $0.05/$0.08 that is very fast.
- sergiotapia 2y agoGroq is actually allowing you to pay now and get real service?
- coder543 2y agoThe option to pay is still listed as coming soon, but I also see pricing information in the settings page, so maybe it actually is coming somewhat sooner. I’m seeing $0.05/1M input and $0.10/1M output for llama3 8B, which is not exactly identical to what the previous person quoted. Either way, I wish Groq would offer a real service to people willing to pay.
- croddin 2y agoI found the .05/.08 here: https://wow.groq.com/ https://wow.groq.com/
- refulgentis 2y ago
- AaronFriel 2y agoThese costs don't line up with my own experiments using vLLM on EKS for hosting small to medium sized models. For small (under 10B parameters) models on g5 instances, with prefix caching and an agent style workload with only 1 or a small number of turns per request, I saw on the order of tens of thousands of tokens/second of prefill (due to my common system prompts) and around 900 tokens/second of output. I think this worked out to around $1/million tokens of output and orders of magnitude less for input tokens, and before reserved instances or other providers were considered.
- veryrealsid 2y agoInteresting, I think how the model runs makes a big difference and I plan to re-run this experiment with different models and different ways of running the model.
- winddude 2y agodoes aws not have lower vcpu and memory instances with multiple T4s? because with 192gbs of memory and 24 cores, you're paying for a ton of resources you won't be using if you're only running inference.
- yousif_123123 2y agodeepinfra.com hosts Llama 3 8b for 8 cents per 1m tokens. I'm not sure it's the cheapest but it's pretty cheap. There may be even cheaper options. (Haven't used it in production, thinking to use it for side projects).
- kiratp 2y ago3 year commit pricing with Jetstream + Maxtext on TPU v5e is $0.25 per million tokens. On demand pricing put it at about $0.45 per million tokens. Source: We use TPUs at scale at https://osmos.io https://osmos.io Google Next 2024 session going into detail: https://www.youtube.com/watch?v=5QsM1K9ahtw https://www.youtube.com/watch?v=5QsM1K9ahtw https://github.com/google/JetStream https://github.com/google/JetStream https://github.com/google/maxtext https://github.com/google/maxtext
- qihqi 2y agoFor pytorch users: checkout the sister project: https://github.com/google/jetstream-pytorch/blob/main/benchmarks/summary.md https://github.com/google/jetstream-pytorch/blob/main/benchm...
- xmonkee 2y agoDoes anyone know the impact of the prompt size in terms of throughput? If I'm only generating 10 tokens, does it matter if my initial prompt is 10 tokens or 8000 tokens? How much does it matter?
- vinni2 2y agoGgml Q8 models on ollama can run on much cheaper hardware without losing much performance.
- ilaksh 2y agoKind of a ridiculous approach, especially for this model. Use together.ai, fireworks.ai, RunPod serverless, any serverless. Or use ollama with the default quantization, will work on many home computers, including my gaming laptop which is about 5 years old.
- angoragoats 2y agoAgreed with the sentiments here that this article gets a lot of the facts wrong, and I'll add one: the cost for electricity when self-hosting is dramatically lower than the article says. The math assumes that each of the Tesla T4s will be using their full TDP (70W each) 24 hours a day, 7 days a week. In reality, GPUs throttle down to a low power state when not in use. So unless you're conversing with your LLM literally 24 hours a day, it will be using dramatically less power. Even when actively doing inference, my GPU doesn't quite max out its power usage. Your self-hosted LLM box is going to use maybe 20-30% of the power this article suggests it will. Source: I run LLMs at home on a machine I built myself.
- baobabKoodaa 2y agoIf we care about cost efficiency when running LLMs, the most important things are: 1. Don't use AWS, because it's one of the most expensive cloud providers 2. Use quantized models, because they offer the best output quality per money spent, regardless of the budget This article, on the other hand, focuses exclusively on running an unquantized model on AWS...
- johnklos 2y agoSelf hosting means hosting it yourself, not running it on Amazon. I think the distinction the author intends to make is between running something that can't be hosted elsewhere, like ChatGPT, versus running Llama-3 yourself. Overlooking that, the rest of the article feels a bit strange. Would we really have a use case where we can make use of those 157 million tokens a month? Would we really round $50 of energy cost to $100 a month? (Granted, the author didn't include power for the computer) If we buy our own system to run, why would we need to "scale your own hardware"? I get that this is just to give us an idea of what running something yourself would cost when comparing with services like ChatGPT, but if so, we wouldn't be making most of the choices made here such as getting four NVIDIA Tesla T4 cards. Memory is cheap, so running Llama-3 entirely on CPU is also an option. It's slower, of course, but it's infinitely more flexible. If I really wanted to spend a lot of time tinkering with LLMs, I'd definitely do this to figure out what I want to run before deciding on GPU hardware, then I'd get GPU hardware that best matches that, instead of the other way around.
- williamstein 2y ago> Self hosting means hosting it yourself, not running it on Amazon. No. I googled "self hosting", read the first few definitions, and they agree with the article, not you. E.g., wikipedia -- https://en.wikipedia.org/wiki/Self-hosting_(web_services) https://en.wikipedia.org/wiki/Self-hosting_(web_services)
- johnklos 2y agoThe very first definition from the link you provide is: > Self-hosting is the practice of running and maintaining a website or service using a private web server, instead of using a service outside of someone's own control. Hosting anything on Amazon is not "using a private web server" and is the very definition of using "a service outside of someone's own control". The fact that the rest of the article talks about "enabled users to run their own servers on remote hardware or virtual machines" is just wrong. It's not "their own servers", and we don't have "more control over their data, privacy" when it's literally in the possession of others.
- awesomebubbles 2y ago[dead]
- mark_l_watson 2y agoUntil January this year I mostly used Google Colab for both LLMs and deep learning projects. In January I spent about $1800 getting Apple Silicon M2Pro 32G. When I first got it, I was only so-so happy with the models I could run. Now I am ecstatically happy with the quality of the models I can run on this hardware. I sometimes use Groq Llama3 APIs (so fast!) or OpenAI APIs, but I mostly use my 32G M2 system. The article calculates cost of self-hosting, but I think it is also good taking into account how happy I am self hosting on my own hardware.
- rfw300 2y agoI agree with most of the criticisms here, and will add on one more: while it is generally true that you can’t beat “serverless” inference pricing for LLMs, production deployments often depend on fine-tuned models, for which these providers typically charge much more to host. That’s where the cost (and security, etc.) advantage for running on dedicated hardware comes in.
- cloudking 2y agoWhat do you use it for? What problems does it solve?
- k__ 2y agoHalf-OT: can I shard Llama3 and run it on multiple wasm processes?
- Havoc 2y ago>initial server cost of $3,800 Not following? Llama 8B is like 17ish gigs. You can throw that onto a single 3090 off ebay. 700 for the card and another 500 for some 2nd hand basic gaming rig. Plus you don't need a 4 slot PCIE mobo. Plus it's a gen4 pcie card (vs gen3). Plus skipping the complexity of multi-GPU. And wouldn't be surprised if it ends up faster too (everything in one GPU tends to be much faster in my experience, plus 3090 is just organically faster 1:1) Or if you're feeling extra spicy you can do same on a 7900XTX (inference works fine on those & it's likely that there will be big optimisation gains in next months).
- Sohcahtoa82 2y ago> Llama 8B is like 17ish gigs. You can throw that onto a single 3090 off ebay Someone correct me if I'm wrong, but I've always thought you needed enough VRAM to have at least double the model size so that the GPU has enough VRAM for the calculated values from the model. So that 17 GB model requires 34 GB of RAM. Though you can quantize to fp8/int8 with surprisingly little negative effect and then run that 17 GB model with 17 GB of VRAM.
- jokethrowaway 2y agoNo, you don't need that much Here is a calculator (if you have a GPU you want to use EXL2, otherwise GGUF) https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calcul... Also model quantisation goes a long way with surprisingly little loss in quality.
- wkat4242 2y agoYou do need more if you use larger context sizes though. It can really blow up to multiple times the model size even for 128k context. Edit: Oh I see that calculator you linked shows that too. My information was more trial and error, thanks for the calculator link.
- sgt101 2y agoRunning 13b code llama on my m1 macbook pro as I type this...
- badgersnake 2y agoI’ve used llama3 on my work laptop with ollama. It wrote an amazing pop song about k-nearest neighbours in the style of PJ and Duncan’s ‘Let’s Get Ready to Rhumble’ called ‘Let’s Get Ready to Classify’ For everything else it’s next to useless.
- forrest2 2y agoA single synchronous request is not a good way to understand cost here unless your workload is truly singular tiny requests. Chatgpt handles many requests in parallel and this article's 4 GPU setup certainly can handle more too. It is miraculous that the cost comparison isn't worse given how adversarial this test is. Larger requests, concurrent requests, and request queueing will drastically reduce cost here.
- axegon_ 2y agoUp until not too long ago I assumed that self-hosting an llm would come at an outrageous cost. I have a bunch of problems with LLM's in general. The major one is that all LLMs(even openAI) will produce output which will give anyone a great sense of confidence, only to be later slapped across the face with the harsh reality-for anything involving serious reasoning, chances are the response you got was at large bullshit. The second one is that I do not entirely trust those companies with my data, be it OpenAI, Microsoft or Github or any other. That said, a while ago there was this[1] thread on here which helped me snatch a brand new, unboxed p40 for peanuts. Really, the cost was 2 or 3 jars of good quality peanut butter. Sadly it's still collecting dust since although my workstation can accommodate it, cooling is a bit of an issue - I 3D printed a bunch of hacky vents but I haven't had the time to put it all together. The reason why I went this road was phi-3, which blew me away by how powerful, yet compact it is. Again, I would not trust it with anything big, but I have been using it for sifting through a bunch of raw, unstructured text and extract data from it and it's honestly done wonders. Overall, depending on your budget and your goal, running an llm in your home lab is a very appealing idea. [1] https://news.ycombinator.com/item?id=39477848 https://news.ycombinator.com/item?id=39477848
- deleted 2y ago[deleted]
- yieldcrv 2y agothis is not what I consider self hosting but ok I would like to compare the costs vs hardware on prem, so this helps with one side of the equation
- agcat 2y agoThis is a good way to do math. But honestly, how many products actually have 100% utilisation. I did some math a few months ago but mostly on the basis of active users, on what would be the % difference if you have 1k to 10K users/mo. You can run this as low as $0.3K/mo on Serverless GPUs and $0.7K/mo on EC2. The pricing is outdated now. Here is the piece -https://www.inferless.com/learn/unraveling-gpu-inference-costs-for-llms-openai-aws-and-inferless https://www.inferless.com/learn/unraveling-gpu-inference-cos...
- wesleyyue 2y agoSurprised no comments are pointing out that the analysis is pretty far off simply due to the fact that the author runs with batch size of 1. The cost being 100x - 1000x what API providers are charging should be a hint that something is seriously off, even if you expect some of these APIs to be subsidized.
- segmondy 2y agoI own an 8 GPU cluster that I built for super cheap < $4,000. 180gb vram, 7 24gb + 1 24gb. There are tons of models that I run that's not hosted by any provider. The only way to run it is to host myself. Furthermore, the author has 39 tokens in 6 seconds. For llama3-8b, I get almost 80 tk/s and if parallel, can easily get up to 800 tk/s. Most users at home infer only one at a time because they are doing chat or role play. If you are doing more serious work, you will most likely have multiple inference running at once. When working with smaller models, it's not unusual to have 4-5 models loaded at once with multiple inference going. I have about 2tb of models downloaded, I don't have to shuffle data back and forth to the cloud, etc. To each their own, the author's argument is made today by many on why you should host in the cloud. Yet if you are not flush with cash and a little creative, it's far cheaper to run your own server than in the cloud. To run llama-3 8b. A new $300 3060 12gb will do, it will load fine in Q8 gguf. If you must load in fp16 and cash is a problem a $160 P40 will do. If performance is desired a used 3090 for ~$650 will do.
- kennethwolters 2y agoI am looking into renting Hetzner GEX44 dedicated server to run a couple models on with Ollama. I haven't done the arithmetics yet but I wouldn't be surprised to see a 100x cost-decrease compared to OpenAI APIs (granted the models I'll run on the GEX44 machines will be less powerful)
- ekkyv6 2y agoWhat kind of setup were you able to do for so cheap? I'd love to be able to do more locally. I have access to a single RTX A5000 at work, but it is often not enough for what I'm wanting to do, and I end up renting cloud GPU.
- segmondy 2y agohttps://www.reddit.com/r/LocalLLaMA/comments/1bqv5au/144gb_vram_for_about_3500/ https://www.reddit.com/r/LocalLLaMA/comments/1bqv5au/144gb_v...
- Melatonic 2y ago
- jokethrowaway 2y agoYeah, or you can get a gpu server with 20GB VRAM on hetzner for ~200 EUR per month. Runpod and DigitalOcean are also quite competitive on prices if you need a different GPU. AWS is stupidly expensive.
- hereonout2 2y agoExpensive in general but combine some decent tooling and spot instances and it can be insanely cheap. The latest Nvidia L4 GPUs (24GB) instances are currently less than 15c p/h spot. T4s are around 20c per hour spot, though they are smaller and slower. I've been provisioning hundreds of these at a time to do large batch jobs at a fraction of the price of commercial solutions (i.e. 10-100x cheaper). Any problem that fits in a smaller GPU and can be expressed as a batch job using spot instances can be done very cheaply on AWS.
- waldrews 2y agoHetzner GPU servers at $200/month for an RTX 4000 with 20GB seem competitive. Anyone have experience with what kind of token throughput you could get with that?
- deleted 2y ago[deleted]
- cheptsov 2y agoWith dstack you can either utilize multiple affordable cloud GPU providers at once to get the cheapest GPU offer or also use an own cluster of on-prem servers. Dstack supports both altogether. Disclaimer: I’m a core contributor to dstack
- visarga 2y agoI just bought a $1099 MacBook Air M3, I get about 10 tokens/s for a q5 quant. Doesn't even get hot, and I can take it with me on the plane. It's really easy to install ollama.
- melbourne_mat 2y agoThis is another one of those "I used this for 5 minutes and found this out" naive posts which add nothing useful. Check out the host LLM's at home crowd. One app to look at is llama.cpp. Model compression is one of the first techniques to successfully run models on low capacity hardware.
- guluarte 2y ago? you can run llama 3 8b with a 3060
- sophiadancini 2y ago[dead]
- MansorQuincy 2y ago[dead]