5 ms·
Not sure who this is aimed at? The avg programmer probably doesn’t have the gear on hand to run this at the required pace Cool nonetheless
by Havoc 3y ago
Not sure who this is aimed at? The avg programmer probably doesn’t have the gear on hand to run this at the required pace
Cool nonetheless
- Spivak 3y agoPeople who want to host the models presumably, AWS bedrock will def include it.
- blackoil 3y agoHow feasible would it be too fine tune using internal code and have an enterprise copilot.
- keriati1 3y agoWe actually run already in-house ollama server prototype for coding assistance with deepseek coder and it is pretty good. Now if we would get a model for this, that is on chatgpt 4 level, I would be super happy.
- eurekin 3y agoDid you finetune a model?
- lozenge 3y agoConsidering a number of Saas offer this service, I'd say it's feasible.
- jejeyyy77 3y agoalready been done
- ttul 3y agoYeah, but if your company wants to rent an H100, you can deploy this for your developers for much less than the cost of a developer…
- connorgutman 3y agoThis is targeted towards GPU rental services like RunPod as well as API providers such as together AI. Together.ai is charging $0.90/1M tokens at 70B parameters. https://www.together.ai/pricing https://www.together.ai/pricing
- moyix 3y agoYou can run it on a Macbook M1/M2 with 64GB of RAM.
- 2OEH8eoCRo0 3y agoHow? It's larger than 64GB.
- coder543 3y agoQuantization is highly effective at reducing memory and storage requirements, and it barely has any impact on quality (unless you take it to the extreme). Approximately no one should ever be running the full fat fp16 models during inference of any of these LLMs. That would be incredibly inefficient. I run 33B parameter models on my RTX 3090 (24GB VRAM) no problem. 70B should easily fit into 64GB of RAM.
- 2OEH8eoCRo0 3y agoI'm aware but is it still LLaMA 70B at that point?
- coder543 3y agoYes. Quantization does not reduce the number of parameters. It does not re-train the model.
- deleted 3y ago[deleted]
- andy99 3y agoIt's a legit question, the model will be worse in some way... I've seen it discussed that all things being equal more parameters is better (meaning it's better to take a big model and quantized it to fit in memory than use a smaller unquantized model that fits), but a quantized model wouldn't be expected to run identically to or as well as the full model.
- svara 3y agoIt's aimed at OpenAI's moat. Making sure they don't accumulate too much of one. No one actually has to use this, it just needs to be clear that LLM as a service won't be super high margin because competition can simply start building on Meta's open source releases.
- deleted 3y ago[deleted]
- FrustratedMonky 3y agoSo. Strange as it seems, is Meta being more 'Open', than OpenAI that was created to be the 'open' option to fight off Meta and Google?
- holoduke 3y agoMeta is becoming the good guy. Its actually a smart move. Some extra reputation points wont hurt Meta.
- chasd00 3y agoIf Meta can turn the money making sauce in GenAI from model+data to just data then it's in a very good position. Meta has tons of data.
- justapassenger 3y ago[dead]
- majani 3y agoSometimes you arrive at your intended solution in a roundabout way
- adventured 3y agoThe moat is all but guaranteed to be the scale of the GPUs required to operate these for a lot of users as they get ever larger, specifically the extreme cost that is going along with that. Anybody have $10 billion sitting around to deploy that gigantic open source set-up for millions of users? There's your moat and only a relatively few companies will be able to do it. One of Google's moats is, has been, and will always be the scale required to just get into the search game and the tens of billions of dollars you need to compete in search effectively (and that's before you get to competing with their brand). Microsoft has spent over a hundred billion dollars trying to compete with Google, and there's little evidence anybody else has done better anywhere (Western Europe hasn't done anything in search, there's Baidu out of China, and Yandex out of Russia). VRAM isn't moving nearly as fast as the models are progressing in size. And it's never going to. The cost will get ever greater to operate these at scale. Unless someone sees a huge paradigm change for cheaper, consumer accessible GPUs in the near future (Intel? AMD? China?). As it is, Nvidia owns the market and they're part of the moat cost problem.
- kungfupawnda 3y agoI got it to build and run the example app on my M3 max with 36 gb ram. Memory pressure was around 32 gb
- dimask 3y agoDid you quantise it? At what level and what was your impression compared to other recent smaller models at that quantisation, if so?
- kungfupawnda 3y agoNo I just ran it out of the box but I had to modify the source code to run for Mac. Instructions here: https://github.com/facebookresearch/llama/pull/947/ https://github.com/facebookresearch/llama/pull/947/
- deleted 3y ago[deleted]
- dimask 3y agoThere are companies like phind that offer copilot-like services using finetuned versions of CodeLlama-34B, which imo are actually good. But I do not know if such a larger model is gonna be used in such a context.
- oceanplexian 3y agoI have a multi-GPU rig designed exactly for this purpose :) Check out r/localllama. There are literally dozens of us!