18 ms·
Meta AI releases Code Llama 70B
- turnsout 3y agoGiven how good some of the smaller code models are (such as Deepseek Coder at 6.7B), I'll be curious to see what this 70B model is capable of!
- jasonjmcghee 3y agoMy personal experience is that Deepseek far exceeds code llama of the same size, but it was released quite a while ago.
- turnsout 3y agoAgreed—I hope Meta studied Deepseek's approach. The idea of a Deepseek Coder at 70B would be exciting.
- whimsicalism 3y agoThe “approach” is likely just training on more tokens.
- imjonse 3y agoIt's the quality of data and the method of training, not just the amount of tokens (per their paper release a few days ago)
- hackerlight 3y agoThere's a deepseek coder around 30-35b and it has almost identical performance to the 7b on benchmarks.
- ignoramous 3y agoAlphaCodium is the newest kid on the block that's SoTA pass@5 on coding tasks (authors claim at least 2x better than GPT4): https://github.com/Codium-ai/AlphaCodium https://github.com/Codium-ai/AlphaCodium As for small models, Microsoft has been making noise with the unreleased WaveCoder-Ultra-6.7b (https://arxiv.org/abs/2312.14187 https://arxiv.org/abs/2312.14187).
- eurekin 3y agoAre weights available?
- moyix 3y agoAlphaCodium is more of a prompt engineering / flow engineering strategy, so it can be used with existing models.
- deleted 3y ago[deleted]
- passion__desire 3y agoAlphaCodium author says he should have used DSPy https://twitter.com/talrid23/status/1751663363216580857 https://twitter.com/talrid23/status/1751663363216580857
- hackerlight 3y agoIs this better than GPT4's Grimoire?
- CrypticShift 3y agoPhind [1] uses the larger 34B Model. Still, I'm also curious what they are gonna do with this one. [1] https://news.ycombinator.com/item?id=38088538 https://news.ycombinator.com/item?id=38088538
- hackerlight 3y agoSeems worse according to this benchmark: https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard https://huggingface.co/spaces/bigcode/bigcode-models-leaderb...
- colesantiago 3y agoLlama is getting better and better, I heard this and Llama 3 will start to be good as GPT-4. Who would have thought that Meta, that has been chucking billions on the metaverse is on the forefront of Open Source AI. Not to mention their stock is up and they are worth $1TN, again. Not sure how I feel about this given the fact of all the scandals that have plagued them and the massive 1BN fine from the EU, Cambridge Analytica, and last of all caused a genocide in Myanmar. Goes to show that nobody cares about all of these scandals and just moves on onto the future, allowing Facebook to still collect all this data for their models. If any other startup or mid sized company had at least two of these large scandals, they would be dead in the water.
- austinpena 3y agoI'm really curious what their goal is
- asylteltine 3y ago[dead]
- Ruq 3y agoZuck just wants another robot to talk to.
- throwup238 3y agoPoor guy just wants a friend that won't sell him out to the FTC or some movie producers.
- hendersoon 3y agoTheir goals are clear, dominance and stockholder value. What I'm curious about is how they plan to monetize it.
- refulgentis 3y agoUse it in products, ex. the chatbots
- ramshanker 3y agoAre these trained on internal Code bases or just the public repositories?
- albertzeyer 3y agoWithout knowing any details, I'm almost sure that they did not train it on internal code, as it might be possible to reveal that code otherwise (given the right prompt, or just looking at the weights).
- make3 3y agopeople are able to extract training data from llms with different methods, so there's no way this was trained on internal code
- deleted 3y ago[deleted]
- eigenvalue 3y agoWould be a really bad idea to train on internal code I would think. Besides, there is no shortage of open source code (even open source created by Meta) out there.
- changoplatanero 3y agoCorrect that it’s a bad idea to train on internal code. However surprisingly there is a shortage of open source code. These models are trained on substantially all the available open source code that these companies can get their hands on.
- andy99 3y agoThe github [0] hasn't been fully updated, but it links to a paper [1] that describes how the smaller code llama models were trained. It would be a good guess that this model is similar. [0] https://github.com/facebookresearch/codellama https://github.com/facebookresearch/codellama [1] https://ai.meta.com/research/publications/code-llama-open-foundation-models-for-code/ https://ai.meta.com/research/publications/code-llama-open-fo...
- Ninjinka 3y agoBenchmarks?
- Havoc 3y agoNot sure who this is aimed at? The avg programmer probably doesn’t have the gear on hand to run this at the required pace Cool nonetheless
- Spivak 3y agoPeople who want to host the models presumably, AWS bedrock will def include it.
- blackoil 3y agoHow feasible would it be too fine tune using internal code and have an enterprise copilot.
- b33j0r 3y agoThere is a bait and switch going on, and sam altman or mark zuckerberg are the first to tell you. “No one can compete with us, but it’s cute to try! Make applications though” —almost direct quote from Sam Altman. I have 64gb and an RTX 3090 and a macbook M3, and I already can’t run a lot of the newest models even in their quantized form. The business model requires this to be a subscription service. At least as of today…
- kuczmama 3y agoRealistically, what hardware would be required to run this? I assumed a RTX 3090 would be enough?
- summarity 3y agoA Studio Mac with an M1 Ultra (about 2800 USD used) is actually a really cost effective way to run in. Its total system power consumption is really low, even spitting out tokens at full tilt (<250W).
- kkzz99 3y agoRTX 3090 has 24GB of memory, a quantized llama70b takes around 60GB of memory. You can offload a few layers on the gpu, but most of them will run on the CPU with terrible speeds.
- bk146 3y agoCan someone explain Meta's strategy with the open source models here? Genuine question, I don't fully undestand. (Please don't say "commoditize your complement" without explaining what exactly they're commoditizing...)
- apples_oranges 3y agoOT: You „don‘t“ or you „don’t fully“ understand? ;) (I try to train myself to say it right ..)
- eurekin 3y agoTotal speculation: Yann LeCun is there and he is really passionate about the technology and openness
- observationist 3y agoThe faux-open models mean the models can't be used in competing products. The open code base means enthusiasts and amateurs and other people hack on Meta projects and contribute improvements. They get free R&D and suppress competition, while looking like they have principles. Yann is clueless about open source principles, or the models would have been Apache or some other comparably open license. It's all ruthless corporate strategy, regardless of the mouth noises coming out of various meta employees.
- importantbrian 3y agoMeta's choice of license doesn't indicate that Yann is clueless about open-source principles. I don't know about Meta specifically, but in most companies choosing the license for open source projects involves working with a lot of different stakeholders. He very easily could have pushed for Apache or MIT and some other interest group within Meta vetoed it.
- sangnoir 3y ago> The faux-open models mean the models can't be used in competing products. Just because certain entities can't profitably use a product or obtain a license doesn't make it not-open. AGPL is open, for an extreme example. This argument is also subjective, and not new - "Which is more open BSD-style licenses or GPL?" has ben a guaranteed flameware starter for decades.
- edweis 3y agoHow come a company as big as Meta still uses bit.ly ?
- nemothekid 3y agoWhat else would they use?
- 3pt14159 3y agoSomething like meta.com/our_own_tech_handles_this
- nemothekid 3y agoNot sure it's preferable to hire people at fb salaries to maintain a link shortener than just to use a reputable free one?
- transcriptase 3y agofb.com seems like a reasonable choice.
- Cthulhu_ 3y agoTheir own?
- huac 3y agotheir own shortener, e.g. fb.me, presumably
- simonw 3y agoHere's the model on Hugging Face: https://huggingface.co/codellama/CodeLlama-70b-hf https://huggingface.co/codellama/CodeLlama-70b-hf
- israrkhan 3y agoI hope someone will soon post a quantized version that I can run on my macbook pro.
- annjose 3y agoOllama has released the quantized version. https://ollama.ai/library/codellama:70b https://ollama.ai/library/codellama:70b https://x.com/ollama/status/1752034686615048367?s=20 https://x.com/ollama/status/1752034686615048367?s=20 Just need to run `ollama run codellama:70b` - pretty fast on macbook.
- kungfupawnda 3y agoI'm trying to understand how this works.. does it actually run the model on the MacBook Pro? Sorry I am totally new to this...
- annjose 3y agoYes, it runs a quantized [1] version of the model locally. This version uses low-precision data types to represent reduced weights and activations (8-bit integer instead of 32-bit). The specific model published by Ollama uses 4-bit quantization [2] and that's why it is able to run on MacBook pro. If you want to try it out, this blog post[3] shows how to do it step by step - pretty straightforward. [1] https://huggingface.co/docs/optimum/concept_guides/quantization https://huggingface.co/docs/optimum/concept_guides/quantizat... [2] https://ollama.ai/library/codellama:70b https://ollama.ai/library/codellama:70b [3] https://annjose.com/post/run-code-llama-70B-locally/ https://annjose.com/post/run-code-llama-70B-locally/
- deleted 3y ago
- theLiminator 3y agoCurious what's the current SOTA local copilot model? Are there any extensions in vscode that give you a similar experience? I'd love something more powerful than copilot for local use (I have a 4090, so I should be able to run a decent number of models).
- Eisenstein 3y agoWhen this 70b model gets quantized you should be able to run it fine on your 4090. Check out 'TheBloke' on huggingface and the llamacpp to run the gguf files.
- coder543 3y agoI think your take is a bit optimistic. I like quantization as much as the next person, but even the 2-bit model won’t fit entirely on a 4090: https://huggingface.co/TheBloke/Llama-2-70B-GGUF https://huggingface.co/TheBloke/Llama-2-70B-GGUF I would be uncomfortable recommending less than 4-bit quantization on a non-MoE model, which is ~40GB on a 70B model.
- nox101 3y agofortunately it will run on my UMA mac. it's made me curious what the trade offs are. Would I be better off with a 4090 or a Mac with 128+gig of uma memory
- coder543 3y agoEven the M3 Max seems to be slower than my 3090 for LLMs that fit onto the 3090, but it’s hard to find comprehensive numbers. The primary advantage is that you can spec out more memory with the M3 Max to fit larger models, but with the exception of CodeLlama-70B today, it really seems like the trend is for models to be getting smaller and better, not bigger. Mixtral runs circles around Llama2-70B and arguably ChatGPT-3.5. Mistral-7B often seems fairly close to Llama2-70B. Microsoft accidentally leaked that ChatGPT-3.5-Turbo is apparently only 20B parameters. 24GB of VRAM is enough to run ~33B parameter models, and enough to run Mixtral (which is a MoE, which makes direct comparisons to “traditional” LLMs a little more confusing.) I don’t think there’s a clear answer of what hardware someone should get. It depends. Should you give up performance on the models most people run locally in hopes of running very large models, or give up the ability to run very large models in favor of prioritizing performance on the models that are popular and proven today?
- ahmednazir 3y agoCan you explain why big tech company make a race to release an open source model? If model is free and open source then how will they earn and how will they compete with others?
- jampekka 3y agoCommoditize your complement?
- stainablesteel 3y agothey want to incentivize dependency
- martingoodson 3y agoBaptiste Roziere gave a great talk about Code Llama at our meetup recently: https://m.youtube.com/watch?v=_mhMi-7ONWQ https://m.youtube.com/watch?v=_mhMi-7ONWQ I highly recommend watching it.
- doctoboggan 3y agoIs there a quantized version available for ollama or is it too early for that?
- coder543 3y agoAlready there, it looks like: https://ollama.ai/library/codellama https://ollama.ai/library/codellama (Look at “tags” to see the different quantizations)
- robin-whg 3y agoThis is a prime example of the positive aspects of capitalism. Meta has its own interests, of course, but as a side effect, this greatly benefits consumers.
- LVB 3y agoI'm not very plugged into how to use these models, but I do love and pay for both ChatGPT and GitHub Copilot. How does one take a model like this (or a smaller version) and leverage it in VS Code? There's a dizzying array of GPT wrapper extensions for VS Code, many of which either seem like kind of junk (10 d/ls, no updates in a year), or just lead to another paid plan, at which point I might as well just keep my GH Copilot. Curious what others are doing here for Copilot-esque code completion without Copilot.
- petercooper 3y agohttps://continue.dev/ https://continue.dev/ is a good place to start.
- sestinj 3y agobeat me to the punch : )
- speedgoose 3y agoContinue doesn’t support tab completion like Copilot yet. A pull/merge request is being worked on: https://github.com/continuedev/continue/pull/758 https://github.com/continuedev/continue/pull/758
- sestinj 3y agoRelease coming later this week!
- just_testing 3y agoThanks! I'm going to recommend it for my students who are interested in tinkering with LLMs.
- sa-code 3y agoThank you for your efforts
- 3y ago
- deleted 3y ago[deleted]
- chrishare 3y agoCredit where credit is due, Meta has had a fantastic commitment towards open source ML. You love to see it.
- joshspankit 3y agoYes but: if the commitment is driven by internal researchers and coders standing firm about making their work open source (a rumour I’ve heard a couple times), most of the credit goes to them.
- mvkel 3y agoWasn't LLaMa originally a leak that they were then forced to spin into an open source contribution? Not to diminish the value of the contribution, but "commitment" is an interesting word choice.
- hnfong 3y agoIt wasn't a leak in the typical sense. They sent the weights to pretty much everyone who asked nicely. When you send something interesting to thousands of people without vetting their credentials, you'd expect the stuff to get "leaked" out eventually (and sooner rather than later). I'd say it's more appropriate to say the weights were "pirated" than "leaked". That said, you're probably correct that the community that quickly formed around the "pirated" weights might have influenced Zuckerberg to decide to make llama2's more freely accessible.
- raxxorraxor 3y agoIf the image synthesis is an example, I believe this to be a winning strategy in the long run to get the most competent model. The LLM space just moves a bit slower since the entry requirements for compute power are that much higher and data preparation is maybe a bit more "dry" than tagging images.
- satvikpendem 3y agoMeta has source available licenses, not open source ones.
- anonymousDan 3y agoCan anyone tell me what kind of hardware setup would be needed to fine-tune something like this? Would you need a cluster of GPUs? What kind of size + GPU spec would you think is reasonable (e.g. wrt VRAM per GPU etc).
- a_wild_dandan 3y agoI just use my M2 MacBook Pro. Works great on big models.
- pandominium 3y agoEveryone is mentioning using 4090 and a smaller model, but I rarely see an analysis where the energy consumption is used. I think Copilot is already highly subsidized by Microsoft. Let's say you use Copilot around 30% of your daily work hours. How much kWh does an opensource 7B or 13B model use then in a month on one 4090? EDIT: I think for a 13B at 30% use per day it comes around 30$/no on energy bill. So probably with a even more smaller but capable model can beat the Copilot monthly subscription.
- Retric 3y agoSubscription models are generally subsidized by people barely using them. So I wouldn’t be surprised if the average is closer to 10%.
- Lapha 3y agoRunning models locally using GPU inference shouldn't be too bad as the biggest impact in terms of performance is ram/vram bandwidth rather than compute. Some rough power figures for a dual AMD GPU setup (24gb vram total) on a 5950x (base power usage of around 100w) using llama.cpp (i.e., a ChatGPT style interface, not Copilot): 46b Mixtral q4 (26.5 gb required) with around 75% in vram: 15 tokens/s - 300w at the wall, nvtop reporting GPU power usage of 70w/30w, 0.37kWh 46b Mixtral q2 (16.1 gb required) with 100% in vram: 30 tokens/s - 350w, nvtop 150w/50w, 0.21kWh. Same test with 0% in vram: 7 tokens/s - 250w, 0.65kWh 7b Mistral q8 (7.2gb required) with 100% in vram: 45 tokens/s - 300w, nvtop 170w, 0.12kWh The kWh figures are an estimate for generating 64k tokens (around 35 minutes at 30 tokens/s), it's not an ideal estimate as it only assumes generation and ignores the overhead of prompt processing or having longer contexts in general. The power usage essentially mirrors token generation speed, which shouldn't be too surprising. The more of the model you can load into fast vram the faster tokens will generate and the less power you'll use for the same amount of tokens generated. Also note that I'm using mid and low tier AMD cards, with the mid tier card being used for the 7b test. If you have an Nvidia card with fast memory bandwidth (i.e., a 3090/4090), or an Apple ARM Ultra, you're going to see in the region of 60 tokens/s for the 7b model. With a mid range Nvidia card (any of the 4070s), or an Apple ARM Max, you can probably expect similar performance on 7b models (45 t/s or so). Apple ARM probably wins purely on total power usage, but you're also going to be paying an arm and a leg for a 64gb model which is the minimum you'd want to run medium/large sized models with reasonable quants (46b Mixtral at q6/8, or 70b at q6), but with the rate models are advancing you may be able to get away with 32gb (Mixtral at q4/6, 34b at q6, 70b at q3). I'm not sure how many tokens a Copilot style interface is going to churn though but it's probably in the same ballpark. A reasonable figure for either interface at the high end is probably a kWh a day, and even in expensive regions like Europe it's probably no more than $15/mo. The actual cost comparison then becomes a little complicated, spending $1500 on 2 3090s for 48gb of fast vram isn't going to make sense for most people, similarly making do with whatever cards you can get your hands on so long as they have a reasonable amount of vram probably isn't going to pay off in the long run. It also depends on the size of the model you want to use and what amount of quantisation you're willing to put up with, current 34b models or Mixtral at reasonable quants (q4 at least) should be comparable to ChatGPT 3.5, future local models may end up getting better performance (either in terms of generation speed or how smart they are) but ChatGPT 5 may blow everything we have now out of the water. It seems far too early to make purchasing decisions based on what may happen, but most people should be able to run 7b/13b and maybe up to 34/46b models with what they have and not break the bank when it comes time to pay the power bill.
- fullspectrumdev 3y agoThis looks potentially interesting if it can be ran locally on say, an M2 Max or similar - and if there’s an IDE plugin to do the Copilot thing. Anything that saves me time writing “boilerplate” or figuring out the boring problems on projects is welcome - so I can expend the organic compute cycles on solving the more difficult software engineering tasks :)
- siilats 3y agoWe made a Jetbrains plugin called CodeGPT to run this locally https://plugins.jetbrains.com/plugin/21056-codegpt https://plugins.jetbrains.com/plugin/21056-codegpt
- bredren 3y agoAre seamless conversations still handled, using the truncation method described in #68? I was curious if some kind of summary or compression of old exchanges flagged as such might allow the app to remember stuff that had been discussed but fallen outside the token limit. But possibly request key details lost during summary to bring them back into the new context. I had thought chatgpt was doing something like this but haven’t read about it.
- dhess 3y agoCan anyone recommend an Emacs mode for this model?
- cashweaver 3y agoI recommend gptel.
- sidcool 3y agoI am trying this on perplexity labs. How does it work so fast?
- d_sc 3y agoHope the M3 Ultra Studio comes along soon. Would be great to get a cpu/gpu of that level + the ram capacity it would likely be able to handle.
- d_sc 3y agoAny good resources or suggestions for system/pre-prompt for general coding or when targeting a specific language? ie, using the CodeLlama and working on typescript, ruby, rust, elixir etc.. is there a universal prompt that gives good results or would you want to adjust the prompt depending on the language you're targeting?