13 ms·
Codestral: Mistral's Code Model
- bloopernova 2y agoDoes anyone know of a link to a codegen comparison page? In other words, you write your request, and it's submitted to multiple codegen engines, so you can compare the output.
- rohansood15 2y agoNot the same, but we evaluated how good LLMs are at fixing code and just posted it on HN: https://news.ycombinator.com/item?id=40511689 https://news.ycombinator.com/item?id=40511689
- asadm 2y agoHow do people do infilling these days? In olden times models used to provide a way to provide suffix separately.
- mousetree 2y agoHow does this compare to Github Copilot? It's not shown in their comparison
- nkozyra 2y agoNot sure how much current Copilot varies from the original Codex, but another set of benchmarks here: https://paperswithcode.com/sota/code-generation-on-humaneval https://paperswithcode.com/sota/code-generation-on-humaneval
- ramon156 2y agoKnowing the training data GH has I doubt it's comparable, then again I don't have the benchmarks
- ramon156 2y agoAfter typing this I tried the live chat out and it honestly seems a lot more promising than current GH Copilot, very nice!
- ssgodderidge 2y agoAre you saying GH has more than Codestral and therefore GH has a better model? Or that Codestral would be better because Codestral is not littered with "bad" code?
- nkozyra 2y agoBad code is obviously very subjective, but I would wager that GH places a much higher value on feedback mechanisms like stars, issues, PRs, velocity, etc. Their ubiquity likely allows them to automatically cherry-pick less "bad code."
- nicce 2y agoNothing prevents Mistral do the same if they want to. Issues and and PRs are public information, exposed by APIs, and not that much rate limited.
- rohansood15 2y agoCopilot primarily uses GPT-3.5, which is outclassed by Llama3-70B. And this model claims to be slightly better than Llama3-70B. Edit: For those who don't believe me, https://github.com/microsoft/vscode-copilot-release/issues/664 https://github.com/microsoft/vscode-copilot-release/issues/6.... Gpt-4 for chat, 3.5 for code.
- jasonjmcghee 2y agoGitHub Copilot uses GPT-3.5? I was under the impression it was a custom codex model with a surrogate local model as per https://github.blog/2023-02-14-github-copilot-now-has-a-better-ai-model-and-new-capabilities/ https://github.blog/2023-02-14-github-copilot-now-has-a-bett... When did this change?
- Rastonbury 2y agoWhen it first launched it, I too didn't know they had changed the model from the original codex which came similar time as gpt-3.5
- jasonjmcghee 2y ago> Gpt-4 for chat, 3.5 for code That thread is comparing sidebar chat to inline chat. Doesn't discuss code completions afaict.
- localfirst 2y agoIt's miles better. In fact I stopped using expensive GPT-4 Codestral just works, its quick, output is accurate its kinda scary.
- esafak 2y agoIt is fast all right, but the quality is not there. I asked it to implement OAuth with Stytch and Ktor and it made everything up. I pointed out the correct name for the package and asked if it really knew the SDK, and it apologized and repeated the same made up code after merely changing the name of the package.
- cco 2y agoThis is actually why we (Stytch) haven't rolled out any of these "chatbots for code". We have a big list of example questions we get from devs trying us out and we've tested several home grown and third party providers and thus far haven't seen anything good enough that we'd put into production. Thanks for testing this out for us! I'll cross it off our list :)
- Zambyte 2y agoLink to the huggingface page: https://huggingface.co/mistralai/Codestral-22B-v0.1 https://huggingface.co/mistralai/Codestral-22B-v0.1
- andruby 2y agoThis is an open weights 22B model. The download on Huggingface is 44GB. Is there a rule-of-thumb estimate for how much RAM this would need to be used locally? Is the RAM requirement the same for a GPU and "unified" RAM like Apple silicon?
- fnbr 2y agoThe rule of thumb is roughly 44gb, as most models are trained in bf16, and require 16 bits per parameter, so 2 bytes. You need a bit more for activations, so maybe 50GB? you need enough RAM and HBM (GPU RAM) so it’s a constraint on both.
- sharbloop 2y agoWhich GPU card can I buy to run this model? Can it run on commercial RTX3090 or does it need a custom GPU?
- Havoc 2y ago3090 or 4090 will be able to run quantized 22B models. Though realistically for code completion smaller models will be better due to speed
- TechDebtDevin 2y agoEasy..
- Novosell 2y agoMost GPUs still use GDDR I'm pretty sure, not HBM. Do you mean VRAM?
- mauricio 2y ago22B params * 2 bytes (FP16) = 44GB just for the weights. Doesn't include KV cache and other things. When the model gets quantized to say 4bit ints, it'll be 22B params * 0.5 bytes = 11GB for example.
- tosh 2y ago
- sashank_1509 2y agoSeems nice but some preliminary testing against GPT-4o shows it’s lacking a bit. It does a pretty good job for easy questions though
- jasonjmcghee 2y agoGPT-4o is really oddly hit or miss for code. Sometimes it outperforms GPT-4 in quality by a fair amount, and other times it starts repeating itself. Duplicating function definitions, even misremembering what things are named. It seems to have to do with length. If the output exceeds a few thousand tokens, it seems to experience some pretty bad failure modes.
- afro88 2y ago4o can only output 4k tokens. So the training to complete an answer within 4k tokens is probably kicking in and nerfing the quality
- localfirst 2y agopersonally this has performed consistently and just as good if not better than GPT-4 what strikes me is the consistency and lack of hallucination you got in GPT4o making in unusuable for any reliable code gen
- swyx 2y agoi've been noticing that there's a divergence in philosophy between Llama style LLMs (Mistral are Meta alums so I'm counting them in tehre) and OpenAI/GPT style LLMs when it comes to code. GPT3.5+ prioritized code very heavily - there's no CodeGPT, its just GPT4, and every version is better than the last. Whereas the Llama/Mistral models are now shipping the general language model first, then adding CodeLlama/Codestral with additional pretraining (it seems like we don't know how much more tokens are on this one, but CodeLLama was 500B-1T extra tokens of code). Zuck has mentioned recently that he doesnt see coding ability as important for his usecases, whereas obviously OpenAI is betting heavily on code as a way to improve LLM reasoning for AGI.
- memothon 2y ago>Zuck has mentioned recently That's a really surprising thing to hear, where did you see that? The only quote I've seen is this one: >“One hypothesis was that coding isn’t that important because it’s not like a lot of people are going to ask coding questions in WhatsApp,” he says. “It turns out that coding is actually really important structurally for having the LLMs be able to understand the rigor and hierarchical structure of knowledge, and just generally have more of an intuitive sense of logic.” https://www.theverge.com/2024/1/18/24042354/mark-zuckerberg-meta-agi-reorg-interview https://www.theverge.com/2024/1/18/24042354/mark-zuckerberg-...
- imachine1980_ 2y agoMake Sense, they want better interaction whit users for Whatsapp, Instagram and Facebook marketers, content creation and moderation,and their glasses(ai /ar) I don't see in that context why the should push more effort into llm coding, is sad anyways
- whoami_nr 2y agoHe mentioned it on the Dwarkesh podcast: https://www.youtube.com/watch?v=bc6uFV9CJGg https://www.youtube.com/watch?v=bc6uFV9CJGg
- Davidzheng 2y ago
- sebzim4500 2y agoVery impressed with it based on a short live chat, feels insanely fast considering its capability. chat.mistral.ai
- kergonath 2y agoWe'll see how fast it is on consumer hardware once decent quantisations are available.
- colesantiago 2y agoI'm so happy now LLMs are democratising access to programming, especially open models like what Meta with Llama and Mistral is doing with Codestral are doing. The abundance of programming is going to allow almost everyone to become a great programmer. This is so exciting to see and each day programming is becoming a solved problem so we can focus on other things.
- skydhash 2y agoShadow libraries did more to democratize anything than LLMs. And following a book like Elixir in Action (Manning) will get you there faster than chatting with LLMs or copilot generating code for you.
- smokel 2y agoIn my experience these tools amplify the quality of a programmer. I have seen good programmers dramatically increase their productivity, but I've also seen others copy-pasting for loops inside other for loops where one loop would definitely suffice. We're not quite there yet.
- croes 2y agoI'm curious for the long-term effect. I observe a certain laziness in myself when it comes to certain problems. It's easier to ask a LLM and debug provided code, but I ask myself if I'm losing some problem solving capabilities in the long run because of this. Similar to the loss of speed in doing mental arithmetic because of calculators on the smartphone.
- bubbleRefuge 2y agoAbsolutely it amplifies. Complex and esoteric configuration of frameworks, for example, entails so much reading and Googling and can be very time consuming without AI. AI can help to bring custom software to the markets that could not otherwise afford to pay for it.
- icedchai 2y agoI'm skeptical. I've run into people who used LLMs to code, then can't debug it without someone else's help. It may get you 80% there though.
- jhonatan08 2y agoDo we have a list of the 80+ languages it was trained on? I couldn't find it
- ddavis 2y agoMy favorite thing to ask the models designed for programming is: "Using Python write a pure ASGI middleware that intercepts the request body, response headers, and response body, stores that information in a dict, and then JSON encodes it to be sent to an external program using a function called transmit." None of them ever get it right :)
- bongodongobob 2y agoCool, you've identified that your prompt is inadequate for the task. 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?'
- TechDebtDevin 2y agoDamn, show us your brilliant prompt then. LLMs cannot do this, not even in python, of which there are libraries like Blacksheep that honestly make it a trivial task.
- bongodongobob 2y agoMy point is that you shouldn't expect to one shot everything. Have it start by writing a spec, then outline classes and methods, then write the code, and feed it debug stuff.
- bottom999mottob 2y agoExactly, expecting one shot 100% working code with one prompt is ridiculous at this point. It's why libraries like Aider are so useful, because you can iteratively diff generated code until it's useable.
- TechDebtDevin 2y agoSure it's impossible at this point, but the point of a benchmark isn't to complete the task it's to test it's efficacy overall and to see progress. None of them are 100% at even the simplistic python benchmarks, doesn't mean we shouldn't measure that capability. But sure, I get it. That's not how they are intended to be used but that's also not the point the commenter was laying out.
- IMTDb 2y agoIs there a way to use this within VSCode like copilot , meaning having the "shadow code" appear while you code instead of having to tho back-and-forth between the editor and a chat-like interface ? For me, a significant component of the quality of these tools resides on the "client" side; being able to engineer a prompt that will yield to accurate code being generated by the model. The prompt needs to find and embed the right chunks from the user current workspace, or even from his entire org repos. The model is "just" one piece of the puzzle.
- meiraleal 2y agoI created a simple CLI app that does this in my workspace, which is under source control so after the LLM execution all the changes are highlighted by diff and the LLM also creates a COMMIT_EDITMSG file describing what it changed. Now I don't use chatgpt anymore, only this cli tool. I never saw something like this integrated directly on VSCode tho (and isn't my preferred workflow anyway, command line works better).
- pyepye 2y agoNot using Codestral (yet) but check out Continue.dev[1] with Ollama[2] running llama3:latest and starcoder2:3b. It gives you a locally running chat and edit via llama3 and autocomplete via starcoder2. It's not perfect but it's getting better and better. [1] https://www.continue.dev/ https://www.continue.dev/ [2] https://ollama.com/ https://ollama.com/
- jmorgan 2y agoCodestral was just published here as well: https://ollama.com/library/codestral https://ollama.com/library/codestral
- sa-code 2y agoThis doesn't give the "shadow text" that the user specifically mentioned
- mijoharas 2y agoWow... That site (continue.dev) managed to consistently crash my mobile google chrome. I've had the odd crash now and again, but I can't think of many sites that will reliably make it hard crash. It's almost impressive.
- resource_waste 2y ago[flagged]
- analyte123 2y agoThe license for this [1] prohibits use of the model and its outputs for any commercial activity, or even any "live" (whatever that means) conditions, commercial or not. There seems to be an exclusion for using the code outputs as part of "development". But wait! It also prohibits "any internal usage by employees in the context of the company's business activities". However you interpret these clauses, this puts their claims and comparisons on completely unequal ground. They only compare to other open-weight models, not GPT-4 or Opus, but a normal company or individual can do whatever they want with the Llama weights and outputs. LangChain? "Your favourite coding and building environment"? Who cares? It seems you're not allowed to integrate this with anything else and show it to anyone, even as an art project. [1] https://mistral.ai/licenses/MNPL-0.1.md https://mistral.ai/licenses/MNPL-0.1.md
- croes 2y agoIt's more like a demo version you can evaluate before you need to buy a commercial license. On whose code is Mistral trained?
- rvnx 2y agoYour code, my code, etc. But there is a common case with law; copyright do not apply when you have billions. Examples: recurring infringement from Microsoft on open-source projects, Google scraping content to build their own database, etc...
- foobiekr 2y agoThere's some irony in the fact that people will ignore this license in exactly the same way Mistral and all the other LLM guys ignore the copyright and licensing on the works they ingest.
- deleted 2y ago[deleted]
- nicce 2y agoIn many countries you even can't claim copyright for the output of the AI to use license like this.
- croes 2y ago>Usage Limitation - You shall only use the Mistral Models and Derivatives (whether or not created by Mistral AI) for testing, research, Personal, or evaluation purposes in Non-Production Environments; - Subject to the foregoing, You shall not supply the Mistral Models, Derivatives, or Outputs in the course of a commercial activity, whether in return for payment or free of charge, in any medium or form, including but not limited to through a hosted or managed service (e.g. SaaS, cloud instances, etc.), or behind a software layer
- jstummbillig 2y agoHow I interact with new model reports at this point: Open the page, ctrl + f, "gpt-4" and skip the rest.
- isoprophlex 2y agoDoes it do SQL, and if so, which dialects? I am having a hard time figuring out what it is actually trained on
- sebzim4500 2y agoThey claim good results in a SQL benchmark but they don't specify what dialects it knows.
- esafak 2y agoAre there any IDE plugins that index your entire code base in order to provide contextual responses AND let you pick between the latest models? If not, consider it a product idea ;)
- pmmucsd 2y agoThere are plugins for various IDEs that operate like copilot but let you select model you want to use, just supply your key. CodeGPT for JetBrains/Android Studio is pretty good. I think you can even use a model running locally.
- saturatedfat 2y agoSupermaven, but you don’t get model choice.
- elmariachi 2y agoCody by Sourcegraph allows you to do this. It doesn't have Codestral yet but probably will soon.
- jdorfman 2y agoWe are working on it.
- artninja1988 2y agoThis is a business model I can get behind. The model is under a non-commercial license, but it's open weights and they have their official API for it
- YetAnotherNick 2y agoWhat's the business model for semi open source models like these? Is it just because they can't be fully closed as they have to then compare with OpenAI. Who would pay for these model if better is available for cheaper from Anthropic or Google.
- mirekrusin 2y agoFifty shades of "open".
- isaacrolandson 2y agoWill this run on an M3 48GB?
- piskov 2y agoYou’ll need 44GB just for the weights By default only 75% of unified memory is available to GPU if you have >36GB. So with 48 total only 36 is available for GPU with is lower than 44. tldr; without quantization you will not be able to run it.
- stalked_why 2y agohttps://medium.com/@tedisaacs/codestral-for-ai-coding-models-size-matters-3efcf9f4f856 https://medium.com/@tedisaacs/codestral-for-ai-coding-models... This article says 88GB without quantization. Though it then goes on to make a ridiculous claim that if you had 128GB of RAM, then using up 88GB of 128GB would make everything else really slow because I guess the think the remaining 40GB of RAM somehow isn't enough for your OS and desktop apps. So it’s probably not a very authoritative source.
- Sytten 2y agoIs there a vscode extension that could plug any model out there and have a similar experience to copilot. I always want to try them but I cant be bothered to do a whole setup each time.
- lioeters 2y agoHaven't used it personally, but someone up-thread mentioned: > Llama Coder is a better and self-hosted Github Copilot replacement for VS Code https://github.com/ex3ndr/llama-coder https://github.com/ex3ndr/llama-coder And: > Open-source VS Code and JetBrains extensions that enable you to easily create your own modular AI software development system https://github.com/continuedev/continue https://github.com/continuedev/continue
- gsuuon 2y agoHow does the Mistral non-production license work for small/hobby/indie projects? Has anyone tried to get approval for that kind of use?
- James_K 2y ago> Democratising code Did yall see what happened when they democratised art? I don't want to have a billion and one AI garbage libraries to sift through before I can find something reliable and human-made. At least the potential for creating horrific political software is slightly lower than with simple images.
- gavin_gee 2y agowhat the heck is this for, if you can't use it for commercial work?
- ein0p 2y agoIf I can’t use the output of this in practical code completion use cases, it’s meaningless, because GH Copilot exists. Idk what they’re thinking or what business model they’re envisioning - Copilot is far and away the best model of this kind anyway
- hallman76 2y agodumb question - why not create smaller language-specific models?