4 ms·
Why this is interesting: To my knowledge, this is the first time you are able to run the largest Llama, at competitive speed, on a consumer GPU (or something li
by Palmik 3y ago
Why this is interesting: To my knowledge, this is the first time you are able to run the largest Llama, at competitive speed, on a consumer GPU (or something like A40).
On many tasks, fine-tuned Llama can outperform GPT-3.5-turbo or even GPT-4, but with a naive approach to serving (like HuggingFace + FastAPI), you will have hard time beating the cost of GPT-3.5-turbo even with the smaller Llama models, especially if your utilization is low.
- politelemon 3y agoCould you explain what you mean by naive approach to serving?
- Palmik 3y agoSure, there are many ways one can be accidentally inefficient, some examples: Batching: Your throughput (and thus cost per token) will be much worse if you don't do batching (meaning running inference for several inputs at once). To do batching, you either need to have a high sustained QPS or adjust your workload to be bursty (but this has utilization issues). Inference engine: The regular HuggingFace, even with built in optimizations, is not competitive against inference engines like vllm, exllama, etc. Utilization: Depending on your scale, it might be hard for you to have a nice flat utilization, or to even utilize one GPU at 100% capacity. This means you need to solve scaling up & down your machines and the issues associated with it (can you live with cold starts? etc.) Hardware: Forget server-less GPUs like Replicate etc, their markups compared to on-demand pricing is usually >10x. On-demand A100 or H100, provided you can even get a quota for them, are also expensive. Spot instances are better. Older GPUs (A10, T4, etc.) are better. Your own GPU cluster is likely the best if you have the scale (and likely, you can resell your cluster within the next 18months without much if any depreciation). For these reasons, I have been toying with the idea of providing a dead simple service for fine-tuning & serving open-source LLMs where the users actually own (and can download) the weights. If anyone is interested in this and would like to chat, let me know.
- TheIronYuppie 3y agoWe’d be interested in helping! aronchick (at) expanso (dot) io
- quickthrower2 3y agoInterested in this. Are you thinking of it being a SaaS type thing? I have some rudimentary experience, i.e. training NanoGPT on both Modal.com and Vast.ai (which is vastly cheaper, but privacy/cost trade off there). It could be a layer on top of $gpu_cloud (so end user signs up for the cloud) and the service makes use of those resources.
- senseiV 3y agoDo you inference NanoGPT with say a Flask app? I made a WebUI for tiny Llama, but it doesn't work for NanoChatGPt
- quickthrower2 3y agoHa ha not sure yet, I have only done it from the command line. Still got a lot to learn. Building something like this will be good for that!
- algo_trader 3y agoThanks for the writeup Our special case is a single model, high throughput and latency unsensitive. Can we use multiple lower-memory GPUs, split the the model "horizontally", and pipeline each batch/inference across the GPUs? Is this a common scenarios handled by the engines you mentioned ?
- read_if_gay_ 3y agorunning 70b on 24gb has been possible for months (so basically forever) it just got like 2x speedup now. the issue that it's a lobotomized ~2bit quant remains.
- redox99 3y agoThat's absolutely false, unless you mean running llamacpp with half the layers on the CPU and half the layers on the GPU. And that was like 2t/s.
- yumraj 3y ago> running llamacpp with half the layers on the CPU and half the layers on the GPU How do you do that? Could you please provide a link if you have it handy.
- checkyoursudo 3y agoQuantisation tradeoff is an interesting question. My experience so far is that larger models with lower bit quantisation seem to perform better than smaller models with higher bit quantisation. So, e.g., 30B-4bit appears to perform better than 13B-8bit. But, then, an 8-bit quantisation seems not really distinguishable from the non-quantised model, so what benefit might you gain from not quantising the model. Of course, speed, accuracy, power consumption, etc are all considerations, sometimes at odds with each other.
- woadwarrior01 3y ago
- BoorishBears 3y agoSaying fine-tuned LLaMA can outperform GPT-3.5-turbo on some tasks would be reasonably generous. Saying it can outperform GPT-3.5-turbo on "many tasks" would be venturing into unreasonable territory. Saying it can outperform GPT-3.5-turbo or even GPT-4 on many tasks is just setting people up for disappointment: There's a reason ARC (common sense reasoning) never gets mentioned when touting OS LLMs vs commercial, and the gap in ARC performance is amplified terribly when you try to apply chain-of-thought
- Palmik 3y agoI am totally with you when it comes to reasoning, or when it comes to generally / universally capable models. But many problems don't require reasoning, and are very concrete / specific.
- SOLAR_FIELDS 3y agoI use my models primarily for code generation and evaluation. Does the latter part fall into “reasoning” territory?
- Palmik 3y agoAdvanced code generation, the kind you would need for something like AutoGPT or other agent-like system, is definitely reasoning and GPT-4 or Claude 2 will dominate. But most of the code generation from GitHub Copilot or other systems (like Replit) is much simpler pattern matching (and indeed, Replit afaik uses very small models). What do you mean by evaluation?
- SOLAR_FIELDS 3y agoMy personal use case is most often supplying files and generating unit tests, as well as supplying a section of code with a prompt about that section to evaluate the approach contextually with some other unseen code. Which perhaps is another form of reasoning.
- benob 3y ago
- jumpCastle 3y agoBut you can fine tune gpt 3.5 turbo, so your comparison is not clear.
- Palmik 3y agoFine tuned gpt-3.5 is 8x the cost of regular gpt-3.5 for inference, other thread where this is discussed: https://news.ycombinator.com/item?id=37494335 https://news.ycombinator.com/item?id=37494335 This may be acceptable for some use cases, bot not for others where even regular gpt-3.5 may not be economical.
- Havoc 3y agoI doubt a 2.55bit quantised 70bn model will beat gpt3.5 on a meaningful percentage of tasks