samatdav

4 ms·

1.
by samatdav 2y ago
cool! So the user does not need to use any additional tools?
2.
by samatdav 2y ago
Haha, yes it is a pattern. However, the claim here is that "our tiny model beats best model" is applicable for highly specific tasks.
3.
by samatdav 2y ago
Yes, you can download and host the fine-tuned open-source model like Llama. The fine-tuning is easy once you have the data, but gathering and cleaning data is challenging. There are also optimizations like upsampling and distillation that c
4.
by samatdav 2y ago
Thank you, we will!:) This was a quick landing page for us to start the conversation and gather feedback. We are trying to make sure we are not building something that nobody needs.
5.
by samatdav 2y ago
We used a single file for the context. It is a cherry-picked example, you are right. I wanted to demonstrate a simple visual change that our model did correctly unlike Sonnet-3.5. Since we are just getting started, we don't have many f
6.
by samatdav 2y ago
Good point, I agree, we haven't shared enough details. Since we are very early, we only got high level results and want to get feedback on what direction would be most applicable and useful. We plan to add more metrics and data to the
7.
by samatdav 2y ago
I agree. Our local early results were promising were a higher percentage of code change requests produced a functionally correct output. We will post more metrics and data in the future.
8.
by samatdav 2y ago
Not yet, but we plan to publicly host a fine-tuned model so anyone can try.
9.
by samatdav 2y ago
We run a set of change requests on the discourse repo. Good point, we plan to publish more detailed testing benchmarks and metrics on the website.
10.
by samatdav 2y ago
Good point, we plan to publish more benchmarks and also publicly host a model for anyone to try. We think Llama is a good option but as we progress we will test other open source models too like deepseek.
11.
by samatdav 2y ago
Thank you! Will email you within a couple of days:)
12.
by samatdav 2y ago
Yes, we fine-tune for each codebase. Now we are focusing on larger enterprise codebases that would: 1. benefit from the fine-tuning the most. 2. have the budget to pay us for the service. For smaller projects that are price-sensitive we are
13.
by samatdav 2y ago
Thank you for the idea! We are also considering upsampling and distillation. But on high level, correctly setting up the data for simple fine-tuning can already produce great results.
14.
by samatdav 2y ago
I agree, we plan to publish more benchmarks and metrics. We also want to publicly host our fine-tuned model for one of the open-source repos so that people can try themselves agains SOTA models.
15.
by samatdav 2y ago
Good point, we should provide more detailed metrics. Since we are very early, we focus on the main metric in our view: higher accuracy of changes to be more practically usable. We will do more testing on overfitting and how the model perfor
16.
by samatdav 2y ago
Thank you for the suggestion, we will take a look!
17.
by samatdav 2y ago
Looks like a great repo to try the fine-tuning! I will email you, thanks!
18.
by samatdav 2y ago
Could be done in the future. Our current focus is highest accuracy. But there are no limitations on the models - just would depend on user preference of size/performance tradeoff.
19.
by samatdav 2y ago
I agree, we need to post more data. Since we are very early (<1 month) we just shared the initial results. Discourse repo was just a good option since it is a big public repo that could benefit from fine-tuning. We plan to add more bench
20.
by samatdav 2y ago
I understand the concern but we don't need anyone's IP. Unfortunately, it is hard to provide fine-tuning solution without access to the codebase. We just think that using a large general-purpose model for a highly specific codebas
21.
by samatdav 2y ago
At Asana we did not do any fine-tuning because it was too complicated even for our AI org of 40 engineers. We believe we can do it by setting up and cleaning data correctly.
22.
by samatdav 2y ago
Good point! We are just very early and our experience is our main selling point. We plan to remove it.
23.
by samatdav 2y ago
Hi HN! I'm Samat, the co-founder from the video. Thank you for the critical feedback, great points. 0. Is this a scam? No. We're very early (started <1 month ago) so our landing page is to validate our concept, gather initial f
24.
by samatdav 2y ago
Hi! Currently we generate a whole diff (like cmd+shift+k in Cursor). But plan to add there rest soon!:)
25.
by samatdav 2y ago
Exciting - will give it a try!:)
26.
Automating Meeting Summaaries
1 points by samatdav 4y ago|1 comments