6 ms·
Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonne
by banddk 2y ago
Hi HN! We worked at OpenAI and Anthropic and believe we can provide much higher quality code generation by fine-tuning an LLM on your codebase compared to Sonnet-3.5 or o1 but not fine-tuned. Let me know if you are interested and we can fine-tune for you for free to test.
- dr_kiszonka 2y agoI wish you posted more evaluation details on your page as text. What exactly was your accuracy vs. Sonnet? (Right now, we can only tell that Sonnet's was ≤ 1/4.3.) Why the Discourse repo? Providing more detailed information would help folks trust your claims more.
- samatdav 2y agoI agree, we need to post more data. Since we are very early (<1 month) we just shared the initial results. Discourse repo was just a good option since it is a big public repo that could benefit from fine-tuning. We plan to add more benchmarks to the website as we progress.
- WhatsName 2y agoTalk is cheap, benchmarks please. Also why did you decide for LLama? AFAIK deepseek always had a slight edge over llama when it comes to coding performance, or is this no longer the case?
- pdimitar 2y agoSome more details that programmers can inspect would be very useful.
- samatdav 2y agoI agree, we plan to publish more benchmarks and metrics. We also want to publicly host our fine-tuned model for one of the open-source repos so that people can try themselves agains SOTA models.
- siscia 2y agoI am getting quite deep into coding with AI and cost of tokens is a bit of an issue indeed. Trivial issue because it saves me A LOT of time, but it could be an issue for new people testing it. I would love to test this approach. Are you guys fine tuning for each codebase?
- manishsharan 2y ago>>cost of tokens is a bit of an issue indeed Their cost is $0.7 per 1M token. DeepSeek is $0.14 / 1M tokens ( cache miss)
- siscia 2y agoDeepSeek is an amazing product but has few issues: 1. Data is used for training 2. Context window is rather small and doesn't fit as well large codebase I keep saying this over and over in all the content I create, the valu of coding with AI will come from working on big, complex, legacy codebases. Not from flashy demo where you create a to-do app. For that you need solid models with big context and private inference.
- MacsHeadroom 2y agoDeepSeek is open source and has a context length of 128k tokens.
- siscia 2y agoCommercial service have a context of 64k tokens, which I find quite limiting. https://api-docs.deepseek.com/quick_start/pricing https://api-docs.deepseek.com/quick_start/pricing Running it locally is quite a bit beyond the scope of being productive while coding with AI. Beside that 128k is still significantly less than Claude
- elashri 2y agoShouldn't we be comparing with other open source model? In particular since this is about llama3.3 then they have the exact context limit which is 128k [1]. Also [1] https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct
- doubtfuluser 2y agoAny plans on distilling it down to an 8b model to enable it for pure local usage on most consumer hardware?
- samatdav 2y agoCould be done in the future. Our current focus is highest accuracy. But there are no limitations on the models - just would depend on user preference of size/performance tradeoff.
- jbellis 2y agoI'm interested. I submitted my email to your landing page form.
- samatdav 2y agoThank you! Will email you within a couple of days:)
- tucnak 2y agoI'm not saying you're an imposter... but you're making it really easy to assume that; it doesn't seem you have learnt much while you guys were there. Are you sure you weren't hired by mistake?
- ryao 2y agoI would be interested in a fine tune on OpenZFS: https://github.com/openzfs/zfs https://github.com/openzfs/zfs
- samatdav 2y agoThank you for the suggestion, we will take a look!
- futureshock 2y agoIt seems an interesting fine-tuning idea. Drawing from reasoning models, I wonder if it’s effective to 10x or 100x the fine-tune dataset by having a larger reasoning model create documentation and reasoning COTs about the code base’s current state and speculation about future state updates. Maybe have it output some verbose execution flow analysis.
- samatdav 2y agoThank you for the idea! We are also considering upsampling and distillation. But on high level, correctly setting up the data for simple fine-tuning can already produce great results.
- redman25 2y agoWhat is the metric for LLMs? Shouldn't more than just accuracy be measured? If something has high accuracy but low recall, won't it be overfit and fail to generalize? Your metrics would give you false confidence in how effective your model is. Just wondering because the announcement only seems to mention accuracy.
- samatdav 2y agoGood point, we should provide more detailed metrics. Since we are very early, we focus on the main metric in our view: higher accuracy of changes to be more practically usable. We will do more testing on overfitting and how the model performance on different types of tasks. On high level we believe in the idea of "a well fine-tuned model should be much better than a large general model". But we need more metrics, I agree.
- rtfeldman 2y agoInterested! Our large Rust code base at https://zed.dev https://zed.dev is open-source at https://github.com/zed-industries/zed https://github.com/zed-industries/zed and I'd be curious to try this out on it. My email is richard at our website's domain if you'd like to get in touch!
- samatdav 2y agoLooks like a great repo to try the fine-tuning! I will email you, thanks!