8 ms·
Codestral Mamba
- Kinrany 2y agoIs there a good explanation of the Mamba architecture?
- altilunium 2y agohttps://x.com/bycloudai/status/1813311769047138568 https://x.com/bycloudai/status/1813311769047138568
- simonw 2y agoThere's a paper: https://arxiv.org/abs/2312.00752 https://arxiv.org/abs/2312.00752 I haven't seen any good non-paper explainers yet.
- alecco 2y agohttps://thegradient.pub/mamba-explained/ https://thegradient.pub/mamba-explained/ https://jackcook.com/2024/02/23/mamba.html https://jackcook.com/2024/02/23/mamba.html https://www.kolaayonrinde.com/blog/2024/02/11/mamba.html https://www.kolaayonrinde.com/blog/2024/02/11/mamba.html
- localfirst 2y agoany sort of evals on how it compares to closed models like chat gpt 4 or open ones like WizardLLM ?
- sa-code 2y agoIt's great to see a high-profile model using Mamba2!
- culopatin 2y agoDoes anyone have a video or written article that would get one up to speed with a bit of the history/progression and current products that are out there for one to try locally? This is coming from someone that understands the general concepts of how LLMs work but only used the general publicly available tools like ChatGPT, Claude, etc. I want to see if I have any hardware I can stress and run something locally, but don’t know where to start or even what are the available options.
- Kydlaw 2y agoIf I understand correctly what you are looking for, Ollama might be a solution (https://ollama.com/ https://ollama.com/)?. I have no affiliation, but I lazily use this solution when I want to run a quick model locally.
- TechDebtDevin 2y agoBetter yet install Open Web GUI and ollama at the same time via docker. Most people will want a familiar GUI rather than the terminal. https://github.com/open-webui/open-webui https://github.com/open-webui/open-webui This will install ollama and open web GUI: For GPU support run: docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama Use for CPU only support: docker run -d -p 3000:8080 -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama
- Der_Einzige 2y agoWhy do people recommend this instead of the much better oobabooga text-gen-webui? https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui It's like you hate settings, features, and access to many backends!
- TechDebtDevin 2y agoTo each their own, how are you using these extra features? I personally am not looking to spend a bunch on API credits and don't have the hardware to run models larger than 7-8b parameters. I use local llms almost exclusively for formatting notes and as a reading assistant/summarizer and therefor don't need these features.
- sva_ 2y agoIf you mean LLM in general, maybe try llamafile first https://github.com/Mozilla-Ocho/llamafile https://github.com/Mozilla-Ocho/llamafile
- bhouston 2y agoWhat are the steps required to get this running in VS Code? If they had linked to the instructions in their post (or better yet a link to a one click install of a VS Code Extension), it would help a lot with adoption. (BTW I consider it malpractice that they are at the top of hacker news with a model that is of great interest to a large portion of the users where and they do not have a monetizable call to action on the page featured.)
- leourbina 2y agoIf you can run this using ollama, then you should be able to use https://www.continue.dev/ https://www.continue.dev/ with both IntelliJ and VSCode. Haven’t tried this model yet - but overall this plugin works well.
- scosman 2y agoThey say no llama.cpp support yet, so no ollama yet (which uses llama.cpp)
- deleted 2y ago[deleted]
- sadeshmukh 2y agoOllama is supported: https://docs.continue.dev/setup/select-provider https://docs.continue.dev/setup/select-provider
- trsohmers 2y agoThey meant that there is no support for Codestral Mamba for llama.cpp yet.
- deleted 2y ago[deleted]
- HanClinto 2y agoCorrect. The only back-end that Ollama uses is llama.cpp, and llama.cpp does not yet have Mamba2 support. The issues to track Mamba2 and Codestral Mamba support are here: https://github.com/ggerganov/llama.cpp/issues/8519 https://github.com/ggerganov/llama.cpp/issues/8519 https://github.com/ggerganov/llama.cpp/issues/7727 https://github.com/ggerganov/llama.cpp/issues/7727 Mamba support was added in March of this year: https://github.com/ggerganov/llama.cpp/pull/5328 https://github.com/ggerganov/llama.cpp/pull/5328 I have not yet seen a PR to address Mamba2.
- alexandre_ganso 2y ago[dead]
- monkeydust 2y agoAny recommended product primers to Mamba vs Transformers - pros/cons etc?
- ertgbnm 2y agohttps://www.youtube.com/watch?v=X5F2X4tF9iM https://www.youtube.com/watch?v=X5F2X4tF9iM This is what introduced me to them. May be a bit outdated at this point.
- bhouston 2y agoThis video is good: https://www.youtube.com/watch?v=N6Piou4oYx8 https://www.youtube.com/watch?v=N6Piou4oYx8. As are the other videos on the same YouTube account.
- red2awn 2y agoA very good primer to state-space models (from which Mamba is based on) is The Annotated S4 [1]. If you want to dive into the code I wrote a minimal single-file implementation of Mamba-2 here [2]. [1]: https://srush.github.io/annotated-s4/ https://srush.github.io/annotated-s4/ [2]: https://github.com/tommyip/mamba2-minimal https://github.com/tommyip/mamba2-minimal
- flakiness 2y agoFor those who are text oriented: https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mamba-and-state https://newsletter.maartengrootendorst.com/p/a-visual-guide-... The paper author has a blog series but I don't think it's for general public https://tridao.me/blog/2024/mamba2-part1-model/ https://tridao.me/blog/2024/mamba2-part1-model/
- croemer 2y agoThe first sentence is wrong. The website says: > As a tribute to Cleopatra, whose glorious destiny ended in tragic snake circumstances but according to Wikipedia this is not true: > When Cleopatra learned that Octavian planned to bring her to his Roman triumphal procession, she killed herself by poisoning, contrary to the popular belief that she was bitten by an asp.
- dghlsakjg 2y agoMaybe Octavian was the snake?
- deleted 2y ago[deleted]
- rjurney 2y agoI believe this is in dispute among sources.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- skybrian 2y agoYes, that seems to be a myth, but exact circumstances seem rather uncertain according to the Wikipedia article [1]: > [A]ccording to the Roman-era writers Strabo, Plutarch, and Cassius Dio, Cleopatra poisoned herself using either a toxic ointment or by introducing the poison with a sharp implement such as a hairpin. Modern scholars debate the validity of ancient reports involving snakebites as the cause of death and whether she was murdered. Some academics hypothesize that her Roman political rival Octavian forced her to kill herself in a manner of her choosing. The location of Cleopatra's tomb is unknown. It was recorded that Octavian allowed for her and her husband, the Roman politician and general Mark Antony, who stabbed himself with a sword, to be buried together properly. I think this rounds to “nobody really knows.” The “glorious destiny” seems kind of shaky, too. It’s just a throwaway line anyway. [1] https://en.m.wikipedia.org/wiki/Death_of_Cleopatra https://en.m.wikipedia.org/wiki/Death_of_Cleopatra
- rjurney 2y agoBut I JUST switched from GPT4o to Claude! :( Kidding, but it isn't clear how to use this thing, as others have pointed out.
- ukuina 2y agoWhat made you switch?
- pelagicAustral 2y agoI'm using both, been doing that for months now. I can confidently assert that while Claude is getting better and better, GPT 4 and 4o seem the be getting dumbed down for some unexplained reason. Claude is now my go-to for anything code. (I do Ruby and C#, btw, other might have a different experience)
- marcyb5st 2y agoI guess they are distilling the models so that they can save $$$ on serving.
- ldjkfkdsjnv 2y agoGPT4o is way behind sonnet 3.5
- mountainriver 2y agoHuh I guess all the benchmarks are wrong then
- magnio 2y agoThey announce the model is on HuggingFace but don't link to it. Here it is: https://huggingface.co/mistralai/mamba-codestral-7B-v0.1 https://huggingface.co/mistralai/mamba-codestral-7B-v0.1
- dvfjsdhgfv 2y agoThe link is already there in the text, they probably just fixed it.
- imjonse 2y agoThe MBPP column should bold DeepSeek as it has a better score than Codestral.
- smith7018 2y agoWhich means Codestral Mamba and DeepSeek both lead four benchmarks. Kinda takes the air out the announcement a bit.
- ed 2y agoThey're in roughly the same class but totally different architectures Deepseek uses a 4k sliding window compared to Codestral Mamba's 256k+ tokens
- causal 2y agoIt should be corrected but the interesting aspect of this release is the architecture. To stay competitive while only needing linear inference time and supporting 256k context is pretty neat.
- mbowcut2 2y agoTHIS. People don't realize the importance of Mamba competing on par with transformers.
- imtringued 2y agoLinear attention is terrible for chatbot style request-response applications, but if you're giving the model the prompt and then let it scan the codebase and fill in the middle, then linear attention should work pretty decently. The performance benefit should also have a much bigger impact, since you're reprocessing the same code over and over again.
- attentive 2y agocodegeex4-all-9b beats them "on paper" so that's why it's not in the benchmarks.
- deleted 2y ago[deleted]
- pzo 2y agoweird they compare to deepseek-coder v1.5 when we already have v2.0. Any advantage to use codestral mamba apart from that it's lighter in weights?
- sam_goldman_ 2y agoYou can try this model out using OpenAI's API format with this TypeScript SDK: https://github.com/token-js/token.js https://github.com/token-js/token.js You just need a Mistral API key: https://console.mistral.ai/api-keys/ https://console.mistral.ai/api-keys/
- deleted 2y ago[deleted]
- thot_experiment 2y agoDoes anyone have a favorite FIM capable model? I've been using codellama-13b through ollama w/ a vim extension i wrote and it's okay but not amazing, I definitely get better code most of the time out of Gemma-27b but no FIM (and for some reason codellama-34b has broken inference for me)
- xoranth 2y agoIs the extension you wrote public?
- thot_experiment 2y agono but it's super super janky and simple hodgepode of stack overflow and gemma:27b generated code, i'll just put it in the comment here, you just need CURL on your path and vim that's compiled with some specific flag function! GetSurroundingLines(n) let l:current_line = line('.') let l:start_line = max([1, l:current_line - a:n]) let l:end_line = min([line('$'), l:current_line + a:n]) let l:lines_before = getline(l:start_line, l:current_line - 1) let l:lines_after = getline(l:current_line + 1, l:end_line) return [l:lines_before, l:lines_after] endfunction function! AIComplete() let l:n = 256 let [l:lines_before, l:lines_after] = GetSurroundingLines(l:n) let l:prompt = '<PRE>' . join(l:lines_before, "\n") . ' <SUF>' . join(l:lines_after, "\n") . ' <MID>' let l:json_data = json_encode({ \ 'model': 'codellama:13b-code-q6_K', \ 'keep_alive': '30m', \ 'stream': v:false, \ 'prompt': l:prompt \ }) let l:response = system('curl -s -X POST -H "Content-Type: application/json" -d ' . shellescape(l:json_data) . ' http://localhost:11434/api/generate') let l:completion = json_decode(l:response)['response'] let l:paste_mode = &paste set paste execute "normal! a" . l:completion let &paste = l:paste_mode endfunction nnoremap <leader>c :call AIComplete()<CR>
- xoranth 2y agoThanks!
- taf2 2y agoHow does this work in vim?
- kristianp 2y agoSimilarly, is there a way to use it with Kate or Sublime Text?
- tatsuya4 2y agoJust did a quick test in the https://model.box https://model.box playground, and it looks like the completion length is noticeably shorter than other models (e.g., gpt-4o). However, the response speed meets expectations..
- flakiness 2y agoSo Mamba is supposed to be faster and the article claims that. But they don't have any latency numbers. Has anyone tried this? And then, is it fast(er)?
- lolinder 2y agoI know it's just a throwaway line, but the bit about Cleopatra at the top feels in poor taste. It's completely inaccurate in that no one has ever attributed her death to a "mamba", and even the asp that some sources claim has been disputed. But even aside from that, it just feels weird that a human being's death has turned into a random reference you can make in a throwaway joke while advertising a product. They're certainly not the first to use Cleopatra this way nor the most egregious, but there are plenty of other random mamba jokes that could have filled in there and both made more sense and been less crass.
- alluro2 2y agoMaybe it was generated :)
- virgildotcodes 2y agoToo soon!
- quantisan 2y agoRick and Morty ref? https://youtu.be/rYJP2YWtZl8?si=NIvyeVwCJaExq92U https://youtu.be/rYJP2YWtZl8?si=NIvyeVwCJaExq92U
- solarkraft 2y agoI do agree, but there’s worse. Ever heard of Patrice Lumumba? He was a congolese politician involved in its independence and democratization. He was shot with the involvement of western forces. There’s a drink named after him: Hot chocolate with a shot of rum.
- PoignardAzur 2y agoAlso, saying that she had a "glorious destiny" is incredibly inaccurate historically. She was kinda crap at her job.
- zamalek 2y agoIs this the active Codestral model on Le Chat? I got quite some mixed results from it tonight.
- modeless 2y ago> Unlike Transformer models, Mamba models offer the advantage of linear time inference and the theoretical ability to model sequences of infinite length > We have tested Codestral Mamba on in-context retrieval capabilities up to 256k tokens Why only 256k tokens? Gemini's context window is 1 million or more and it's (probably) not even using Mamba.
- rileyphone 2y agoGemini is probably using ring attention. But scaling to that size requires more engineering effort in terms of interlink that goes beyond the purpose of this release from Mistral.
- solarkraft 2y agoI kinda just want something that can keep up with the original version of Copilot. It was so much better than the crap they’re pumping out now (keeps messing up syntax and only completing a few characters at a time).
- heeton 2y agoHave you tried supermaven? It replaced copilot for me a couple of months ago.
- karolist 2y agoI tried it, uses GPT-4o, the $10 sign up credit dissapeared in a few hours of intense coding, I'm not paying $500/mo for a fancy autocomple. Manual instruct style chat about code with Claude-Sonnet-3.5 is the best price/perf I've tried so far, through poe.com I use around 30k credits per day of coding of the 1M monthly allotment, I think it was $200/y. It's not available directly in my country. I've tried a bunch of local models too but Claude is just next level and inference is very cheap.
- PufPufPuf 2y agoOn the pricing page it says $10/month, what you describe sounds like pay-as-you-go, are you sure it was this service you tried?
- karolist 2y agoYes that the same, in the "Your Account" section of the service you have "Chat Credits" field, it was seeded with $10 initially and I got it down to $0.11 in one evening. You are right about the pricing page, it does say it's $10/month, but I believe it's $10/month up to some amount of queries in actuality, because they obviously can't offer infinite inference for $10. Maybe I was "holding it wrong", sending too much complete files as part of prompt and the token use and context window skyrocketed, but I absolutely used my initial $10 in one evening, unlike with Claude...
- 2y ago
- altilunium 2y ago[dead]