11 ms·
Stable Code 3B: Coding on the Edge
- ihaag 3y agoTerrible model
- keyle 3y agoThat is fantastic. I'm building a small macOS SwiftUI client with llama cpp built in, no server-client model, and it's already so useful with models like openhermes chat 7B, and fast. If this opens it to smaller laptops, wow! We truly live in crazy time. The rate of improvement in this field is off the walls.
- turnsout 3y agoThat sounds awesome! Can you share any details about how you're working with llama cpp? Is it just via the Swift <> C bridge? I've toyed with the idea of doing this, and wonder if you have any pointers before I get started.
- joshmarlow 3y agoNot sure if this is where your head is, but I think there's a lot of value in integrating LLMs directly into complex software. Jira, Salesforce, maybe K8s - should all have an integrated LLMs that can walk you through how to perform a nuanced task in the software.
- dpacmittal 3y agoWhy would the LLM walk you through and not just do the nuanced task on its own?
- pennomi 3y agoI assume the human maintains some of the necessary context in their meat memory.
- joshmarlow 3y agoIMO, for many real business use cases, the hallucinations are still a big deal. Once we have models that are more reliable, I think it makes sense to go down that path - the AI is the interface to the software. But until we're there, a system that just provides guidance that the user can validate is a good stepping stone - and one I suspect is immediately feasible!
- debarshri 3y agoWalkthrough is generally performed once or not so frequently. It would be a bad investment if you just use it for just this use case
- alpaca128 3y agoA beginner tutorial is also not used frequently by users, but that doesn't make it a bad investment. I an LLM can help a lot with getting familiar with the tool it could be pretty valuable, especially after a UI rework etc.
- manmal 3y agoImagine good error messages, with hints for mitigation and maybe smart retry w/ mitigations applied.
- emadm 3y ago3b is good for 8gb MacBook Air etc. 7b is slightly too big. Sure these will continue to improve, phi2 is a good base as well
- artninja1988 3y agoGiven the complete failure of the first stable lm, I'm interested to try this one out. Haven't really seen a small language model, except mixtral 7b that's really useful for much. I also hope stability comes out with a competitor to the new midjourney and dalle models! That's what put them on the map in the first place
- brianjking 3y agoAll of the Mistral versions have been excellent, including the OpenHermes versions. I encourage you to check out Phi-2 as well, it's the only 3b model I've found really quite interesting outside of Replit's code model built into Replit Core.
- tarruda 3y agoDeepseek coder 6.7B is very useful for coding and can run in consumer GPUs. I use the 6bit GGUF quantized version on a laptop RTX 3070
- emadm 3y agoWe released a competitor to runway recently that beat it on blind tests, plus way faster image in sdxl turbo We have been working on ComfyUI for the next step and new image models Midjourney and others are pipelines versus models so we have a higher bar to jump but the og stable diffusion team are working hard!
- mchiang 3y agoIt's amazing to see more smaller models being released. This creates opportunities for more developers to run it on their local computers, and makes it easier to fine-tune for specific needs.
- brcmthrowaway 3y agoHas anyone tried starting with a smaller modeling, then RLing until it improves to the bigger model?
- lfkdev 3y agoHow is this compared to the current GitHub Copilot?
- brianjking 3y agoA 3B tiny model is not going to compare to GitHub copilot. However, there are plenty of nice 7B models that are excellent at code and I encourage you to try them out.
- londons_explore 3y agoIf you just want to get stuff done, use the best tools like a Milwaukee Drill - and right now, thats copilot/gpt-4. If you don't want to be tied to a company and like opensource, feel free to connect a toy motor to an AA battery to drill your holes... Or to use Llama/Stable Code 3B.
- yreg 3y agoA self-hosted solution is a common requirement for security reasons.
- mistercheph 3y agoit’s going to be real hard to pry the carburetors out of this guy’s cold dead hands!
- bearjaws 3y agoYou clearly have never used these other tools. Mixtral / Deepseek perform very well on coding challenges. I've used them against local code without issues, sometimes they are a bit optimistic and produce too much, but thats far better than producing too little (like GPT4 does).
- irthomasthomas 3y agoOpenai just invisibly dropped my API requests to a lower model with a 4k context limit. And my commit scripts started failing for being over the context limit. It's buried in the docs somewhere that low tier api users will be served on lower models during peek times. So,I guess they're like a Milwaukee Drill that will sometimes refuse to work unless you buy more drill credits.
- photon_collider 3y agoHow reliable are these benchmarks?
- ilaksh 3y agoI think the trick is that they are just comparing to other tiny models. None of the little models, including this one, are comparable to the performance of the larger models for any significant coding problem. I think what these are useful for is mostly giving people hints inside of a code editor. Occasionally filling in the blank.
- knicholes 3y agoI've got a machine with 4 3090s-- Anyone know which model would perform the best for programming? It's great this can run on a machine w/out a graphics card and is only 3B params, but I have the hardware. Might as well use it.
- SushiHippie 3y agoHere is a leader board of some models https://huggingface.co/spaces/mike-ravkine/can-ai-code-results https://huggingface.co/spaces/mike-ravkine/can-ai-code-resul... Don't know how biased this leaderboard is, but I guess you could just give some of them a try and see for yourself.
- SparkyMcUnicorn 3y agoThis is a much better leaderboard: https://evalplus.github.io/leaderboard.html https://evalplus.github.io/leaderboard.html I've seen the CanAiCode leaderboard several times (and used many of the models listed), but I wouldn't use it to pick a model. It's not a bad list, but the benchmark is too limited. The results are not accurately ranked from best to worst. For example the deepseek 33b model is ranked 5 spots lower than the 6.7b model, but the 33b model is definitely better. WizardCoder 15b is near the top while WizardCoder 33b is ranked 26 spots lower, which is a wildly inaccurate ranking. It's worth noting that those 33b models score in the 70s for HumanEval and HumanEval+ while the 15b model scores in the 50s.
- SushiHippie 3y agoThanks
- tarruda 3y agoAFAIK deepseek coder family are the best open coding models. I haven't tested, but I think deepseek coder 33b can run in a single RTX 3090 when 4-bit quantized. In your case you might be able to run the non quantized version
- Havoc 3y ago
- tarruda 3y agoNote that they don't compare with deepseek coder 6.7b, which is vastly superior to much bigger coding models. Surpassing codellama 7b is not that big of a deal today. The most impressive thing about these results is how good the 1.3B deepseek coder is.
- danielbln 3y agoDeepseek-coder-6.7B really is a quite surprisingly capable model. It's easy to give it a spin with ollama via `ollama run deepseek-coder:6.7b`.
- triyambakam 3y agoThanks for the tip with ollama
- behnamoh 3y agoollama is actually not a great way to run these models as it makes it difficult to change server parameters and doesn't use `mlock` to keep the models in memory.
- discordance 3y agoIf you do: 1. ollama run deepseek-coder:6.7b 2. pip install litellm 3. litellm --model deepseek-coder:6.7b You will have a local OpenAI compatible API for it.
- a_wild_dandan 3y ago
- connorgutman 3y agoFYI: This model is already available on Ollama.
- kleiba 3y agoIt's quite amazing - I often find that I read quite positive comments towards LLM tools for coding. Yet, an "Ask HN" I posted a while ago (and which admittedly didn't gain much traction) seemed to mirror mostly negative/pessimistic responses. https://news.ycombinator.com/item?id=38803836 https://news.ycombinator.com/item?id=38803836 Was it just that my submission didn't find enough / more balanced commenters?
- simonw 3y agoYou only got comments from six people so yeah, definitely not representative.
- yorwba 3y agoYou got two positive and two negative responses. You replied only to the negative responses. Now you think that the responses were mostly negative. I blame salience bias. Anyways, there's also a difference between "are you excited about this new thing becoming available" and "now that you've used it, do you like the experience". The former is more likely to feature rosy expectations and the latter bitter disappointment. (Though it could also be the other way around, with people dismissing it at first and then discovering that it's kind of nice actually.)
- Havoc 3y agoThe precise wording matters. How has it changed your work life leads people down the rabbit hole of will coding jobs be safe. This one is a lot more neutral/technical.
- weebull 3y agoIf somebody can show me a coding task that LLMs have successfully done that isn't an interview question or a documentation snippet, I might start to value it. Spending huge amount of resource to be a bit better at autocompleting code doesn't have value to me. I want it to solve significant problems, and it's looking like it can't do it and scaling it to be able to is totally impractical. > In aggregate, training all 9 Code Llama models required 400K GPU hours of computation on hardware of type A100-80GB (TDP of 350-400W). That is: * 45⅔ GPU years * 160 MWh or... * 45 average UK homes annual electric consumption * 18 average US homes * 64 average drivers annual milage in an EV. ...and that's just the GPUs. Add on all the rest of the system (s).
- akulbe 3y agoI just tried this model with Koboldcpp on my LLM box. I got gibberish back. My prompt - "please show me how to write a web scraper in Python" The response? <blockquote> I've written my first ever python script about 5 months ago and I really don't remember anything except for the fact that I used Selenium in order to scrape websites (in this case, Google). So you can probably just copy/paste all of these lines from your own Python code which contains logic to determine what value should be returned when called by another piece of software or program. </blockquote>
- endofreach 3y agoThis doesn't seem like gibberish though?
- connorgutman 3y agoSame thing with Ollama.
- Tiberium 3y agoBut it's a code completion model, not a chat/instruct one.
- deleted 3y ago[deleted]
- SushiHippie 3y agoIt's very likely a "completion model" and not instruct/chat fine-tuned. So you'd need to prompt it through comments or by starting with a function name, basically the same as one would prompt GitHub copilot. e.g. # the following code implements a webscraper in python class WebScraper: (I didn't try this, and I'm not good at prompting, but something along the lines of this example should yield better results)
- MrNeon 3y agoIt is weird that it is not mentioned in the model card but I'm pretty sure it is a completion model, not tuned as an instruct model. edit: the webpage does call it "Stable Code Completion"
- jjtheblunt 3y agoJargon naivete question: isn't "on the edge" normally implying on a server side with minimal routers hops to the client, not on client side?
- devindotcom 3y agoafaik "edge" nearly always means taking place on the device a user is interacting with. no server involved except perhaps as authentication etc. but there is probably some other situation where "edge" could mean local infra or caching.
- WatchDog 3y agoI think the etymology of “edge computing” is derived from “network edge”, ie the outer shell of some network/autonomous system. The closest point within your control that interfaces with devices outside of your control. Seeing the term get used to describe client devices themselves kinda muddies the terminology.
- jjtheblunt 3y agoAgreed. (autocorrect style typo above: etymology)
- WatchDog 3y agofixed
- ImprobableTruth 3y agoI've definitely seen edge computing used in the context of IoT to refer to compute done on sensing devices. The less narrow meaning at least isn't really "new".
- xer0x 3y ago+1 yes, for a service using network caching like using Cloudflare. I would've referred to their CDN as the Edge of our network.
- herval 3y agoCan anyone explain what’s Stability’s business model (or plan for one)? I get why Meta releases tons of models, but still can’t quite understand what stability is trying to achieve
- seydor 3y agoto be bought by meta
- bogwog 3y agoThis is all an elaborate mating ritual
- sangnoir 3y agoSeems like the standard open-core playbook: > This model is included in our new Stability AI Membership. Visit our Membership page to take advantage of our commercial Core Model offerings, including SDXL Turbo & Stable Video Diffusion. A hypothetical Stable Code 13B/70B could be hosted only, with more languages or specialized use-cases (Stable Code 3B iOS-Swift-Turbo)
- emadm 3y agoMembership with upsell to support, custom models and more Plus licensed variant models like stable audio and on chip installation like arm for specialist models eg Japanese law or Indonesian accounting
- JCM9 3y agoDon’t entirely understand Stability’s business model. They’ve been putting out a lot of models recently and Stable Diffusion was novel at the time, but now their models consistently seem to be somewhat second rate compared to other things out there. For example Midjourney now seems to have far surpassed them in the image generation front. After raising a ton of funding Stability seems to just be throwing a bunch of stuff out there that’s OK but no longer ground breaking. What am I missing? Many other startups in the space will like face similar issues given the rapid commoditization of these models and the underlying tech. It’s very easy to spend a fortune building a model that offers a short lived incremental improvement at best before one can just quickly swap it out for something else someone else paid to train.
- liuliu 3y ago> one can just quickly swap it out for something else someone else paid to train. That doesn't seem to be the case. There are very limited open-source models outside of the small-LLM bubble.
- JCM9 3y agoThe open space on small models is a whole other developing angle, but O was referring to the general commoditization of a lot of these models. With rare exception after launch it seems the lifespan of any of these models is rather limited. From a business standpoint that sort of scenario is generally very unattractive and thus was trying to understand if they have some other angle they’re trying to play here to make a viable business out of this. Or the business model can just be get acquired before that matters and let that be someone else’s problem to figure out.
- liuliu 3y agoThe comoditization of models outside of LLM is delusional. There is no comparable open source image / video models to the private ones.
- smusamashah 3y agoMidjourney has better quality but does not offer any control. Community has done and is still doing a a lot with SD models because they can be played and tinkered with in any way anyone wants to.
- swyx 3y ago> License: Other > Commercial Applications > This model is included in our new Stability AI Membership. Visit our Membership page to take advantage of our commercial Core Model offerings, including SDXL Turbo & Stable Video Diffusion. what exactly is the license lol. can people use this or is this "see dont touch"
- neurostimulant 3y agoIt's free for noncommercial use. If you use it in your company, your company should pay the membership fee. afaik most openai competitors also use similar usage restriction (e.g. free for noncommercial or research use, contact us for commercial license).
- jillesvangurp 3y agoThere are a growing number of open source options out there. I was playing with Simon Willison's excellent llm cli tool this morning and tried out some models from the gpt4all project. One of the better ones come from companies like mistral which release their models under the Apache license. Gpt4all has a UI as well that you can use with models running locally on your laptop.
- blagie 3y agoThis basically means "Get sued." There is no clear legal, definition of "noncommercial," and courts have gone all sorts of different way on what constitutes commercial use. This is where CC NC licenses imploded. A lot of places (hello, MIT!) intentionally use CC NC licenses to make things appear more open than they are.
- nuz 3y agoIf you make money using it, pay them. If you're using it for free, don't worry
- blagie 3y agoThat's not the way legal cases went. Indeed, they went all over the place. That's the reason you see "IANAL" disclaimers all over the internet. Legal advice from non-lawyers can be problematic in many ways. Some jurisdictions, although not where I live, you can even go to jail for giving bad legal advice without being a licensed lawyer.
- outcoldman 3y agoI was able to run this model in http://lmstudio.ai http://lmstudio.ai as well. Just remove Compatibility Guess in Filters, so you can see all the models. LM Studio can load it and run requests against it.
- Valaria 3y ago[flagged]
- Valarit 3y ago[flagged]
- ayibaifie 3y ago[flagged]
- rahimnathwani 3y agoHow are people using codellama and this in their workflows? I found one option: https://github.com/xNul/code-llama-for-vscode https://github.com/xNul/code-llama-for-vscode But I'm guessing there are others, and they might differ in how they provide context to the model.
- weebull 3y agoEllama for Emacs look promising, but I only tried to install it this morning.
- alwinaugustin 3y agoI've been experimenting with code-llama extensively on my laptop, and from my experience, it seems that these models are still in their early stages. I primarily utilize them through a Web UI, where they can successfully refactor code given an existing snippet. However, it's worth noting that they cannot currently analyze entire codebases or packages, refining them based on the most suitable solutions using the most appropriate algorithms. While these models offer assistance to some extent, there is room for improvement in their ability to handle more complex and comprehensive coding scenarios.
- danielmarkbruce 3y agoI think there is a decent chance SourceGraph will figure this all out. The most important thing at this point is figuring what context to feed. They can build up a nice graph of a codebase and I expect from there they can put in the best context and then boom. They might also be able to train a model more intelligently by generating training data from said graphs.
- jdorfman 3y ago> I've been experimenting with code-llama extensively on my laptop You can use/try code-llama with Cody https://sourcegraph.com/blog/cody-vscode-1.1.0-release#:~:text=Autocomplete%20with%20Code%20Llama%20Integration https://sourcegraph.com/blog/cody-vscode-1.1.0-release#:~:te...
- weebull 3y agoI'm honestly failing to see the utility for LLMs, because the context for any given problem is far too small, and we're already at 33B parameter models. They just don't seem to be a technology that scales to an interesting problem size.
- sytelus 3y agoWhy authors miss to compare with Phi-2?
- hcarlens 3y agoAgreed, and not only do they not compare their model to Phi-2 directly, the benchmarks they report don't overlap with the ones in the Phi-2 post[1], making it hard for a third party to compare without running benchmarks themselves. (In turn, in the Phi-2 post they compare Phi-2 to Llama-2 instead of CodeLlama, making it even harder) [1]: https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/ https://www.microsoft.com/en-us/research/blog/phi-2-the-surp...
- alastairr 3y agodoes anyone have recommendations for addins to integrate these 'smaller' llms into an IDE like VSCode? I'm pretty embedded with GH copilot, but curious to explore other options.
- hospitalJail 3y agoSeems like they caught the Apple Marketing bug and are chasing things noonecares about. Great 3B model, everyone is already running 7B models over here. Maybe one day when I need to do offline coding on my cellphone, it will be really useful.
- oliviarobet 3y ago[flagged]
- oliviarobet 3y ago[flagged]