4 ms·

What are the steps required to get this running in VS Code? If they had linked to the instructions in their post (or better yet a link to a one click install o

What are the steps required to get this running in VS Code? If they had linked to the instructions in their post (or better yet a link to a one click install of a VS Code Extension), it would help a lot with adoption. (BTW I consider it malpractice that they are at the top of hacker news with a model that is of great interest to a large portion of the users where and they do not have a monetizable call to action on the page featured.)

leourbina 2y ago
If you can run this using ollama, then you should be able to use https://www.continue.dev/ https://www.continue.dev/ with both IntelliJ and VSCode. Haven’t tried this model yet - but overall this plugin works well.
- scosman 2y ago
  They say no llama.cpp support yet, so no ollama yet (which uses llama.cpp)
  - deleted 2y ago
    [deleted]
  - sadeshmukh 2y ago
    Ollama is supported: https://docs.continue.dev/setup/select-provider https://docs.continue.dev/setup/select-provider
    - trsohmers 2y ago
      They meant that there is no support for Codestral Mamba for llama.cpp yet.
  - deleted 2y ago
    [deleted]
  - HanClinto 2y ago
    Correct. The only back-end that Ollama uses is llama.cpp, and llama.cpp does not yet have Mamba2 support. The issues to track Mamba2 and Codestral Mamba support are here: https://github.com/ggerganov/llama.cpp/issues/8519 https://github.com/ggerganov/llama.cpp/issues/8519 https://github.com/ggerganov/llama.cpp/issues/7727 https://github.com/ggerganov/llama.cpp/issues/7727 Mamba support was added in March of this year: https://github.com/ggerganov/llama.cpp/pull/5328 https://github.com/ggerganov/llama.cpp/pull/5328 I have not yet seen a PR to address Mamba2.
- osmano807 2y ago
  Unrelated, all my devices freeze when accessing this page, desktop Firefox and Chrome, mobile Firefox and Brave. Is this the best alternative to access code ai helpers besides the GitHub Copilot and Google Gemini on VSCode?
  - raphaelj 2y ago
    I've been using it for a few months (with Starcoder 2 for code, and GPT-4o for chat). I find the code completion actually better than Github Copilot. My main complain is that the chat sometimes fails to correctly render some GPT-4o output (e.g. LaTeX expressions), but it's mostly fixed with a custom system prompt. It also significantly reduces the battery life of my Macbook M1, but that's expected.
  - oliverulerich 2y ago
    I'm quite happy with Cody from Sourcegraph https://marketplace.visualstudio.com/items?itemName=sourcegraph.cody-ai https://marketplace.visualstudio.com/items?itemName=sourcegr...
sleepytimetea 2y ago
Looking through the Quickstart docs, they have an API that can generate code. However, I don't think they have a way to do "Day 2" code editing. Also, doesn't seem to have a freemium tier...need to start paying even before trying it out ? "Our API is currently available through La Plateforme. You need to activate payments on your account to enable your API keys."
- sv123 2y ago
  I signed up when codestral was first available and put my payment details in. Been using it daily since then with continue.dev but my usage dashboard shows 0 tokens, and so far have not been billed for anything... Definitely not clear anywhere, but it seems to be free for now? Or some sort of free limit that I am not hitting.
  - sunaookami 2y ago
    Through codestral.mistral.ai? It's free until August 1st: https://docs.mistral.ai/capabilities/code_generation/ https://docs.mistral.ai/capabilities/code_generation/ >Monthly subscription based, free until 1st of August
refulgentis 2y ago
"All you need is users" doesn't seem optimal IMHO, Stability.ai providing an object lesson in that. They just released weights, and being a for profit, need to optimize for making money, not eyeballs. It seems wise to guide people to the API offering.
- bhouston 2y ago
  On top of Hacker News (the target demographic for coders) without an effective monetizable call to action? What a missed opportunity. Github Copilot makes +100M/year, if not way way more. Having a VS Code extension for Mistral would be a revenue stream if it was one-click and better or cheaper than Github Copilot. It is malpractice in my mind to not be doing this if you are investing in creating coding models.
  - refulgentis 2y ago
    I see, that makes sense: make an extension and charge for it. I assumed they meant free x local. It doesn't seem rational to make this one paid: its significantly smaller than their better model, and even more so than Copilot's.
  - treyd 2y ago
    How the hell does Copilot make $100M/yr? That seems an order of magnitude higher than I would expect at the high end.
    - ketzo 2y ago
      if we’re talking individual subscriptions that’s ~1M paying subscribers. honestly that number would not totally shock me? plus they’ve got some kinda enterprise/team offering; assuming they charge extra there, I could easily see $100M ARR but that’s pure conjecture, and generous at that; I don’t think we have any hard numbers
    - kcb 2y ago
      I was thinking the opposite...remember there's enterprise subscriptions and multi-million dollar contracts with single companies.
yogeshp 2y ago
Website codegpt.co also has a plugin for both VS Code and Intellij. When model becomes available in Ollama, you can connect plugin in VS code to local ollama instance.
PufPufPuf 2y ago
Currently the best (most user-friendly) way to run models locally is to use Ollama with Continue.dev. This one is not available yet, though: https://github.com/ggerganov/llama.cpp/issues/8519 https://github.com/ggerganov/llama.cpp/issues/8519
DalasNoin 2y ago
I feel like local models could be an amazing coding experience because you could disconnect from the internet. Usually I need to open chatgpt or google every so often to solve some issue or generate some function, but this also introduces so many distractions. imagine being able to turn off internet completely and only have a chat assistant that runs locally. I fear though that it is just going to be a bit to slow at generating tokens on CPU to not be annoying.
- regularfry 2y ago
  I don't have a gut feel for how much difference the Mamba arch makes to inference speed, nor how much quantisation is likely to ruin things, but as a rough comparison Mistral-7B at 4 bits per param is very usable on CPU. The issue with using any local models for code generation comes up with doing so in a professional context: you lose any infrastructure the provider might have for avoiding regurgitation of copyright code, so there's a legal risk there. That might not be a barrier in your context, but in my day-to-day it certainly is.
antifa 2y ago
Maybe not this model, but checkout TabbyML for offline/selfhostws LLMs in vscode.
- sfsylvester 2y ago
  Also looks like an older version of Codestral works well with TabbyML: https://tabby.tabbyml.com/blog/2024/07/09/tabby-codestral/ https://tabby.tabbyml.com/blog/2024/07/09/tabby-codestral/ Thank you for sharing, this is almost exactly what I've been looking for, for ages!

leourbina 2y ago

If you can run this using ollama, then you should be able to use https://www.continue.dev/ https://www.continue.dev/ with both IntelliJ and VSCode. Haven’t tried this model yet - but overall this plugin works well.

scosman 2y ago
They say no llama.cpp support yet, so no ollama yet (which uses llama.cpp)
- deleted 2y ago
  [deleted]
- sadeshmukh 2y ago
  Ollama is supported: https://docs.continue.dev/setup/select-provider https://docs.continue.dev/setup/select-provider
  - trsohmers 2y ago
    They meant that there is no support for Codestral Mamba for llama.cpp yet.
- deleted 2y ago
  [deleted]
- HanClinto 2y ago
  Correct. The only back-end that Ollama uses is llama.cpp, and llama.cpp does not yet have Mamba2 support. The issues to track Mamba2 and Codestral Mamba support are here: https://github.com/ggerganov/llama.cpp/issues/8519 https://github.com/ggerganov/llama.cpp/issues/8519 https://github.com/ggerganov/llama.cpp/issues/7727 https://github.com/ggerganov/llama.cpp/issues/7727 Mamba support was added in March of this year: https://github.com/ggerganov/llama.cpp/pull/5328 https://github.com/ggerganov/llama.cpp/pull/5328 I have not yet seen a PR to address Mamba2.
osmano807 2y ago
Unrelated, all my devices freeze when accessing this page, desktop Firefox and Chrome, mobile Firefox and Brave. Is this the best alternative to access code ai helpers besides the GitHub Copilot and Google Gemini on VSCode?
- raphaelj 2y ago
  I've been using it for a few months (with Starcoder 2 for code, and GPT-4o for chat). I find the code completion actually better than Github Copilot. My main complain is that the chat sometimes fails to correctly render some GPT-4o output (e.g. LaTeX expressions), but it's mostly fixed with a custom system prompt. It also significantly reduces the battery life of my Macbook M1, but that's expected.
- oliverulerich 2y ago
  I'm quite happy with Cody from Sourcegraph https://marketplace.visualstudio.com/items?itemName=sourcegraph.cody-ai https://marketplace.visualstudio.com/items?itemName=sourcegr...

sleepytimetea 2y ago

Looking through the Quickstart docs, they have an API that can generate code. However, I don't think they have a way to do "Day 2" code editing. Also, doesn't seem to have a freemium tier...need to start paying even before trying it out ? "Our API is currently available through La Plateforme. You need to activate payments on your account to enable your API keys."

sv123 2y ago
I signed up when codestral was first available and put my payment details in. Been using it daily since then with continue.dev but my usage dashboard shows 0 tokens, and so far have not been billed for anything... Definitely not clear anywhere, but it seems to be free for now? Or some sort of free limit that I am not hitting.
- sunaookami 2y ago
  Through codestral.mistral.ai? It's free until August 1st: https://docs.mistral.ai/capabilities/code_generation/ https://docs.mistral.ai/capabilities/code_generation/ >Monthly subscription based, free until 1st of August

refulgentis 2y ago

"All you need is users" doesn't seem optimal IMHO, Stability.ai providing an object lesson in that. They just released weights, and being a for profit, need to optimize for making money, not eyeballs. It seems wise to guide people to the API offering.

bhouston 2y ago
On top of Hacker News (the target demographic for coders) without an effective monetizable call to action? What a missed opportunity. Github Copilot makes +100M/year, if not way way more. Having a VS Code extension for Mistral would be a revenue stream if it was one-click and better or cheaper than Github Copilot. It is malpractice in my mind to not be doing this if you are investing in creating coding models.
- refulgentis 2y ago
  I see, that makes sense: make an extension and charge for it. I assumed they meant free x local. It doesn't seem rational to make this one paid: its significantly smaller than their better model, and even more so than Copilot's.
- treyd 2y ago
  How the hell does Copilot make $100M/yr? That seems an order of magnitude higher than I would expect at the high end.
  - ketzo 2y ago
    if we’re talking individual subscriptions that’s ~1M paying subscribers. honestly that number would not totally shock me? plus they’ve got some kinda enterprise/team offering; assuming they charge extra there, I could easily see $100M ARR but that’s pure conjecture, and generous at that; I don’t think we have any hard numbers
  - kcb 2y ago
    I was thinking the opposite...remember there's enterprise subscriptions and multi-million dollar contracts with single companies.

yogeshp 2y ago

Website codegpt.co also has a plugin for both VS Code and Intellij. When model becomes available in Ollama, you can connect plugin in VS code to local ollama instance.

PufPufPuf 2y ago

Currently the best (most user-friendly) way to run models locally is to use Ollama with Continue.dev. This one is not available yet, though: https://github.com/ggerganov/llama.cpp/issues/8519 https://github.com/ggerganov/llama.cpp/issues/8519

DalasNoin 2y ago

I feel like local models could be an amazing coding experience because you could disconnect from the internet. Usually I need to open chatgpt or google every so often to solve some issue or generate some function, but this also introduces so many distractions. imagine being able to turn off internet completely and only have a chat assistant that runs locally. I fear though that it is just going to be a bit to slow at generating tokens on CPU to not be annoying.

regularfry 2y ago
I don't have a gut feel for how much difference the Mamba arch makes to inference speed, nor how much quantisation is likely to ruin things, but as a rough comparison Mistral-7B at 4 bits per param is very usable on CPU. The issue with using any local models for code generation comes up with doing so in a professional context: you lose any infrastructure the provider might have for avoiding regurgitation of copyright code, so there's a legal risk there. That might not be a barrier in your context, but in my day-to-day it certainly is.

antifa 2y ago

Maybe not this model, but checkout TabbyML for offline/selfhostws LLMs in vscode.

sfsylvester 2y ago
Also looks like an older version of Codestral works well with TabbyML: https://tabby.tabbyml.com/blog/2024/07/09/tabby-codestral/ https://tabby.tabbyml.com/blog/2024/07/09/tabby-codestral/ Thank you for sharing, this is almost exactly what I've been looking for, for ages!