6 ms·
Replit's new AI Model now available on Hugging Face
- SushiHippie 3y ago"Intended use" from their readme: > Replit intends this model be used by anyone as foundational model for application-specific fine-tuning without strict limitations on commercial use. > The model is trained specifically for code completion tasks. Nice, I expected that I would need to give my E-Mail address to them and that it would be ""free"".
- krasin 3y agoNice, Apache 2.0 license. Thank you, Replit! https://huggingface.co/replit/replit-code-v1_5-3b https://huggingface.co/replit/replit-code-v1_5-3b
- IceHegel 3y agoI know a lot depends on architecture and number representation, but do people have a sense for how big a compute cluster is needed to train these classes of models from 1.5B, 3B, 7B, 13B, 70B? Didn’t Meta say they trained on 2k A100s for LLama 2?
- amasad 3y agoWe're on a budget :) trained on 128 H100-80GB GPUs for a week (200B tokens over 5 epochs, ie 1T tokens). Tech talk here with timestamp: https://www.youtube.com/live/veShHxQYPzo?si=UlcU9j2kC-C4oWvj&t=2531 https://www.youtube.com/live/veShHxQYPzo?si=UlcU9j2kC-C4oWvj...
- nojvek 3y agoEach H100 is ~$30,000, so $3.8M in capex cost. Roughly $1/hr/GPU in power cost so looking at 128247 = $21,504. Cheap compared to OpenAI, but not something an indiehacker can do by themselves unless they have millions to burn.
- grey8 3y agoThe Huggingface page of Replit 3Bs says "The model has been trained on the MosaicML platform on 128 H100-80GB GPUs." Source: https://huggingface.co/replit/replit-code-v1_5-3b https://huggingface.co/replit/replit-code-v1_5-3b I'm not an ML engineer, just interested in the space - but as a general ballpark, training these models from scratch needs hundreds to thousands of GPUs.
- dr_dshiv 3y agoAny vibe checks on this model? How does it compare to gpt4 for coding?
- redox99 3y agoThis being a 3B model isn't remotely comparable to GPT4. WizardCoder 34B and Phind 34B are the only models remotely comparable, and they are still slightly worse than GPT 3.5 (let alone GPT4).
- aorth 3y agoHow about Mistral 7B? I saw this article recently: https://wandb.ai/byyoung3/ml-news/reports/Fine-Tuning-Mistral7B-on-Python-Code-With-A-Single-GPU---Vmlldzo1NTg0NzY5 https://wandb.ai/byyoung3/ml-news/reports/Fine-Tuning-Mistra...
- redox99 3y agoMistral 7B is very cool for its size. But unfortunately no open model is close to GPT4 as of right now.
- tarruda 3y agoIf the rumors around GPT4 being a mixture of expert models are true, the this comparison is not fair. What would be interesting is compare GPT4 at a certain task with a small model fine tuned for that task.
- johnthewise 3y agoGPT4 being a mixture of experts is irrelevant imo like we don't care about how many layers there are in a network and how wide those layers are or which type of activation functions are actually used etc. all that matters are we can run it on a specific hardware and the results.
- 3y ago
- evertedsphere 3y ago> When fine-tuned on public Replit user code, the model outperforms models of much larger size such as CodeLlama7B: The table just below this shows the other models doing better on half of the benchmarks; the Replit column being in boldface is misleading.
- itake 3y agoI don't think the boldface is meant to mean "better". I just thought it was meant to draw attention to their numbers.
- IanCal 3y agoThe standard is to bold the best figure per column. If none are significantly different you don't bold any generally but it's standard practice to use this to highlight which approach is best in each task.
- TOMDM 3y agoAgreed this threw me. I think colouring a column is the common approach to drawing attention to your own while still respecting the best is bold custom, which they've sort of done with the header, but personally I'd have gone with the cell background for the column.
- deleted 3y ago[deleted]
- iamsaitam 3y agoAnyone knows how to get this working locally with vscode?
- infruset 3y agoI have the same question, and more generally: Any generic way of doing this for any of the open source or semi open source models, especially Mistral[0]? [0] https://news.ycombinator.com/item?id=37675496 https://news.ycombinator.com/item?id=37675496
- ZeroCool2u 3y agoThis is a good starting place https://marketplace.visualstudio.com/items?itemName=HuggingFace.huggingface-vscode https://marketplace.visualstudio.com/items?itemName=HuggingF...
- coder543 3y agoThat is not what I consider "local", since that uses cloud inference by default (and last I checked, they provided no useful guidance for changing that). I don’t consider cloud inference to count as getting it working “locally” as requested by the comment above yours. Refact worked nicely and worked locally when I tried it a few weeks ago, but the challenge with any new model is making it be supported by the existing software: https://github.com/smallcloudai/refact/ https://github.com/smallcloudai/refact/
- ZeroCool2u 3y ago"Choose your model Requests for code generation are made via an HTTP request. You can use the Hugging Face Inference API or your own HTTP endpoint, provided it adheres to the API specified here[1] or here[2]." It's fairly easy to use your own model locally with the plugin. You can just use the one of the community developed inference servers, which are listed at the bottom of the page, but here's the links[3] to both[4]. [1]: https://huggingface.co/docs/api-inference/detailed_parameters#text-generation-task https://huggingface.co/docs/api-inference/detailed_parameter... [2]: https://huggingface.github.io/text-generation-inference/#/Text%20Generation%20Inference/generate https://huggingface.github.io/text-generation-inference/#/Te... [3]: https://github.com/wangcx18/llm-vscode-inference-server https://github.com/wangcx18/llm-vscode-inference-server [4]: https://github.com/wangcx18/llm-vscode-inference-server https://github.com/wangcx18/llm-vscode-inference-server
- infruset 3y ago> Encompasses Replit's top 30 programming languages with a custom trained 32K vocabulary for high performance and coverage Any idea where the list can be found?
- lioeters 3y ago> The model is trained in bfloat16 on 1T tokens of code (~200B tokens over 5 epochs, including linear cooldown) for 30 programming languages from a subset of permissively licensed code from Bigcode's Stack Dedup V2 dataset and a dev-oriented samples from StackExchange. Following the link to the "Stack Dedup V2" page: https://huggingface.co/datasets/bigcode/the-stack-dedup https://huggingface.co/datasets/bigcode/the-stack-dedup > The Stack contains over 6TB of permissively-licensed source code files covering 358 programming languages. The full list can be found here. https://huggingface.co/datasets/bigcode/the-stack-dedup/blob/main/programming-languages.json https://huggingface.co/datasets/bigcode/the-stack-dedup/blob... It requires login to see the JSON file.
- ms337 3y agojust added the list to the README on Hugging Face!
- amelius 3y agoTitle should include "code generation language model".
- wokwokwok 3y agoI’ve been using mistral and code llama to generate large volumes of code recently, and I have to say… These small models just suck compared to the larger ones. It get it, they’re quick and cheap (ha! Relatively) to make and good for research and fine tuning… …but can anyone here speak authoritatively on fine tuning and getting good results out? I’ve been super disappointed by how bad even the q6 13B code llama model is at generating consistent code (Ie. It even compiles, forget doing what you asked) > about 30 lines in length. These smaller model seem good for a line or two, maybe, but gosh… it’s an effort to do anything useful with them out of the box. Carefully crafted prompt. Tests, hand written. Iterate: prompt, compile, run tests, generate code metrics. Accept code that passes the tests and beats the target threshold. You’re looking at like 7-10 iterations per prompt to get anything for simple (< 30 lines) functions, and maybe no candidates after 30 iterations for longer complex requests. Are people just using this for 5 line code snippets and autocomplete? Or is there a way to get better results by fine tuning?
- a_bonobo 3y agoI feel your experience fits with what's described in the OP article in https://news.ycombinator.com/item?id=37830011 https://news.ycombinator.com/item?id=37830011 (the most upvoted discussion goes somewhere else) I've had the same experience as you. Quantitative, these models are decent. Qualitative, in my daily work, they're not good at all! We'll need better tests.
- tarruda 3y ago> Or is there a way to get better results by fine tuning? It seems that the general wisdom around LLMs is that you can get very good performance on small models if you fine tune for a specific task. In the case of code generation, I think you might get a good performance by fine tuning it on a specific programming language + codebase or architectural pattern. The main problem with fine tuning is getting a good dataset, so a cheap alternative would be to put a few examples of what you want to generate in the prompt. Then you would save these prompts as task specific "fine tunes" that you would select when you need to accomplish something. You might find this discussion helpful: https://news.ycombinator.com/item?id=37813806 https://news.ycombinator.com/item?id=37813806
- kodablah 3y agoSo many code models seem to be used for code generation purposes, but is there a general effort to apply these as static/code analysis tooling? It'd be nice to write my rules in English and have somewhat predictable behavior when analyzing small bits of code. I have had success with GPT4 and a bit less with StarCoderPlus, but I have to build the engine to chop up code, send pieces to remote as needed, cache results for same-hashed snippets, etc. Surely someone is working on general AI powered code analysis tooling?
- michaelmior 3y agoThe first version of the model said that infill was coming. I was hoping to see that in this version, but I guess we have to wait for v2.
- coder543 3y agoAIUI, one of the most notable limitations of the first version of this model was that it couldn't Fill In The Middle (FIM), it could only provide completion.[0] This blog post doesn't mention FIM either, so I guess that's still missing? The demos I've seen of Replit Ghostwriter indicate that it is still possible to get good results without FIM, as long as you have good enough software around the model, but I think FIM could still improve things further. The much smaller Refact-1.6B model supports FIM[1], and Refact-1.6B worked pretty well when I tested it a few weeks ago. People (like the most upvoted comment in this thread) who are expecting any of these small models to write entire programs for them based on a simple prompt seem to misunderstand the purpose of these smaller models, which is to be a smarter alternative for code completion. Writing entire functions or programs is better suited to much larger (and slower) instruct/chat-tuned models. [0]: https://huggingface.co/replit/replit-code-v1-3b/discussions/24 https://huggingface.co/replit/replit-code-v1-3b/discussions/... [1]: https://refact.ai/blog/2023/introducing-refact-code-llm/ https://refact.ai/blog/2023/introducing-refact-code-llm/
- amelius 3y agoThe main feature I'm looking for is to train it on my own code also (in the UI it should have a switch).