31 ms·
Instead of using AWS another approach involves self hosting the hardware as well. Even after factoring in energy, this does dramatically lower the price. Assum
by philipkglass 2y ago
Instead of using AWS another approach involves self hosting the hardware as well. Even after factoring in energy, this does dramatically lower the price.
Assuming we want to mirror our setup in AWS, we’d need 4x NVidia Tesla T4s. You can buy them for about $700 on eBay.
Add in $1,000 to setup the rest of the rig and you have a final price of around:
$2,800 + $1,000 = $3,800
This whole exercise assumes that you're using the Llama 3 8b model. At full fp16 precision that will fit in one 3090 or 4090 GPU (the int8 version will too, and run faster, with very little degradation.) Especially if you're willing to buy GPU hardware from eBay, that will cost significantly less.
I have my home workstation with a 4090 exposed as a vLLM service to an AWS environment where I access it via reverse SSH tunnel.
- causal 2y agoCame here to say this. No way you need to spend more than $1500 to run L3 8B at FP16. And you can get near-identical performance at Q8 for even less. I'm guessing actual break-even time is less than half that, so maybe 2 years.
- causal 2y agoFurthermore, the AWS estimates are also really poorly done. Using EKS this way is really inefficient, and a better comparison would be AWS Bedrock Haiku which averages $0.75/M tokens: https://aws.amazon.com/bedrock/pricing/ https://aws.amazon.com/bedrock/pricing/ This whole post makes OpenAI look like a better deal than it actually is.
- mrinterweb 2y agoI was getting that sense too. It would not be difficult to build a desktop machine with a 4090 for around $2500. I run Llama-3 8b on my 4090, and it runs well. Plus side is I can play games with the machine too :)
- shostack 2y agoHow is inference latency for coding use cases on a local 3090 or 4090 compared to say, hitting the GPT-4o API?
- whereismyacc 2y agoI assume the characteristics would be pretty different, since your local hardware can keep the context loaded in memory, unlike APIs which I'm guessing have to re-load it for each query/generation?
- christina97 2y agoIf you integrate with existing tooling, it won’t do this optimization. Unless of course you really go crazy with your setup.
- moffkalast 2y agoSetting one launch flag on llama.cpp server hardly qualifies as going crazy with one's setup.
- kiratp 2y agoNvidia EULA prevents you from using consumer gaming GPUs in datacenters so 4xxx cards are a non-starter for any service usecases EDIT: TOS -> EULA per comments below
- nubinetwork 2y agoThat never stopped the crypto farmers...
- byteknight 2y agoThey also weren't selling the usage of the cards.
- oneshtein 2y agoNvidia terms of what?
- codetrotter 2y agoParent commenter used the wrong word. It’s the EULA that prevents it. Regardless, it is true that it is a problem. https://www.reddit.com/r/MachineLearning/comments/ikrk4u/d_can_a_company_use_rtx_3090s_or_is_that_illegal/ https://www.reddit.com/r/MachineLearning/comments/ikrk4u/d_c...
- J_Shelby_J 2y agoWhat about on prem? Like, my small business needs an LLM. Can I put a 3090 in a box in a closet? What if I’m a business and I’m selling LLMs in a box for you to put on a private network? What constitutes a data center according to the ToS? Is it enforceable if you never agree to the ToS (buying through eBay?)
- kiratp 2y agoBy using the drivers you agree to their TOS. So yes, it applies even on your private network.
- choppaface 2y agoYeah but this article is terrible. First it talks about naively copy-pasting code to get “a seeming 10x speed-up” and then “This ended up being incorrect way of calculating the tokens used.” I would not bank on anything in this article. It might as well have been written by a tiny Llama model.
- czhu12 2y agoI do the same thing with cloudflare tunnels and managing the cloudflare tunnel process and the llama.cpp server with systemd on my home internet. Have a 13B running on a 3070 with 16 gpu layers and the rest running off CPU. Performs okay, but way cheaper than renting a GPU on the cloud.
- Melatonic 2y agoRead the other comment and also immediately thought of Cloudflare tunnel instead - is there a reason you chose that? Wondering if I should do the same with my old Titan XP (probably slower than your 3070 but it does have 12gb of vram)
- czhu12 2y agoNo reason other than that cloudflare is quite a nice reverse proxy and I already use it for managing my DNS's. Thus far, I've not noticed any issues with it, other than when my home wifi goes down.
- logtrees 2y agoWhoa, so you have code running in AWS making use of your local hardware via what is called a reverse SSH tunnel? I will have to look into how that works, that's pretty powerful if so. I have a mac mini that I use for builds and deploys via FTP/SFTP and was going to look into setting up "messaging" via that pipeline to access local hardware compute through file messages lol, but reverse SSH tunnel sounds like it'll be way better for directly calling executables rather than needing to parse messages from files first.
- brrrrrm 2y agoI use my mac mini exactly as described by the parent post but using ollama as the server. Super easy setup and obv chatgpt can guide you through it
- logtrees 2y agoUnfortunately my mac mini isn't beefy enough to run ollama, it's the base model m1 from a couple years ago lol. But it's very powerful for builds, deploys, and some computation via scripts. Now I'm curious to check out how much memory the newest ones support for potentially using ollama on it haha. Thanks!
- hehdhdjehehegwv 2y agoI dropped $5k on an A6000 and I can run llama3:70b day and night for the price of my electricity bill. I’ve gone through hundreds of millions, maybe billions, of tokens in the past year. This article is just “cloud is expensive” 101. Nothing new.
- brcmthrowaway 2y agoHows your ROI?
- hehdhdjehehegwv 2y agoAbsolutely phenomenal.
- brcmthrowaway 2y agoAre you using it for trading?
- hehdhdjehehegwv 2y agoNope, powers some low-level infrastructure-ish stuff.
- logicallee 2y agoSuper cool, thanks for sharing. Do you mind sharing what you used the hundreds of millions (or billions) of tokens on?
- hehdhdjehehegwv 2y agoDoing really nuanced classification of documents at very large scale. Needle in the haystack type problems.
- hereonout2 2y agoI've worked professionally over the last 12 months hosting quite a few foundation models and fine tuned LLMs on our own hardware, aws + azure vms and also a variety of newer "inference serving" type services that are popping up everywhere. I don't do any work with the output, I'm just the MLOps guy (ahem, DevOps). You mention expense but on a purely financial basis I find any of these hosted solutions really hard to justify against GPT 3.5 turbo prices, including building your own rig. $5k + electricity is loads of 3.5 Turbo tokens. Of course none of the data scientists or researchers I work with want to use that though - it's not their job to host these things or worry about the costs.
- cootsnuck 2y agoYea, for any hobbyist, indie developer, etc. I think it'd be ridiculous to not first try running one of these smaller (but decently powerful) open source models on your own hardware at home. Ollama makes it dead simple just to try it out. I was pleasantly surprised by the tokens/sec I could get with Llama 3 8B on a 2021 M1 MBP. Now need to try on my gaming PC I never use. Would be super cool to just have a LLM server on my local network for me and the fam. Exciting times.
- speakspokespok 2y agoWhy did this only occur to me recently? You can selfhost a k8s cluster and expose the services using a $5 digital ocean droplet. The droplet and k8s services are point-to-point connected using tailscale. Performance is perfectly fine, keeps your skillset sharp, and you’re self-hosting!
- Helithumper 2y agoYou can also just directly connect to containers using Tailscale if it's just for internal use. That is, having an internally addressable `https://container_name https://container_name` on your tailnet per-container if you want. This way I can setup Immich for example and it's just on my tailnet at `https://immich https://immich` without the need for a reverse proxy, etc... https://tailscale.com/blog/docker-tailscale-guide https://tailscale.com/blog/docker-tailscale-guide
- SparkyMcUnicorn 2y agoAnd you can use Tailscale Funnel to serve it publicly. No need to pay for a cloud instance. https://tailscale.com/kb/1223/funnel https://tailscale.com/kb/1223/funnel
- LorenzoGood 2y agoI essentially do this with my homelab.
- whoiscroberts 2y agoThis is great advice. I used to run my dev stuff on AWS, then built a small 6 server proxmox cluster in my basement, 300 cores, 1tb memory, 12tb ssd storage for about 3k usd. I don’t even want to know what it would cost to run a similar config on AWS. You can get cheap ddr4 servers on eBay all day.