4 ms·
Ask HN: Why doesn't Copilot for Business use your organization's actual repos?
I was just looking at the page for Copilot for Business ( https://docs.github.com/en/enterprise-cloud@latest/copilot/overview-of-github-copilot/about-github-copilot-for-business ), and while it does offer some useful things for business users versus the regular Copilot product, it seems to me that they could make it much more powerful by fine-tuning a smaller model on all the repos in an organization (even private ones) that could generate additional recommendations that take into account all the existing code a company has. Obviously, users would want assurances that these personalized models wouldn't be shared with anyone outside of the organization, but Microsoft has enough credibility that I think many businesses would try it in hopes of enhanced productivity.
Is it because it would be cost prohibitive? Maybe it only makes sense to fine tune a model if there are at least N users in the organization. Anyway, curious if anyone here has insights into this. Also interested in whether there are other companies offering this kind of product.
- PaulHoule 4y agoThe usual story is that fine tuning is not very expensive compared to building the foundation model.
- eigenvalue 4y agoRight, that's why I'm really surprised they aren't doing something like that. Especially because they presumably have some amount of idle compute at Azure that could be used to work on this when it's not needed for something else.
- theGnuMe 4y agoIt might not be stable. Sometimes fine-tuning can throw things out.
- eigenvalue 4y agoI think there are some good strategies for avoiding this, like setting an extremely low learning rate, or not allowing gradient updates that would result in more than N weights changing by more than K%, things like that.
- PaulHoule 4y agoThe right approach might be something retrieval based where your own code gets embedded and then searched with some combination of dense similarity and keyword index.
- travisjungroth 4y agoVery limited opinion. Having looked into fine tuning Whisper and GPT, it looked to be a fiddly thing. The training isn’t as robust as the inference. Makes sense, it’s running at a different scale. You can manually check models before you release them. But that would put the cost at a different scale than $10/month. My takeaway was that it’s very easy to mess up a model by fine tuning. You can overfit or have rapid degradation. Again, this is just reading about other experiences on other software, so maybe that’s not the case here.
- SteveDR 4y agoMy org shot down an idea like this because they don’t want OpenAI to train on (or have access to) our codebase. I don’t blame them.
- literalAardvark 4y agoNeither do I, but it's a dinosaur with an umbrella attitude.
- sqs 4y agoThey're working on it: https://githubnext.com/projects/copilot-view/ https://githubnext.com/projects/copilot-view/. We (Sourcegraph) are also working on it, and also bringing other kinds of code intelligence to the LLM so we can answer more kinds of questions: https://twitter.com/sourcegraph/status/1623339664428892162 https://twitter.com/sourcegraph/status/1623339664428892162.
- eigenvalue 4y agoAwesome, thanks for your reply.
- gtaylor 4y agoIs this going to be available for self-hosted enterprise users?
- sqs 4y agoYes, we're getting this out to existing customers first (including self-hosted customers). Anyone can sign up on the waitlist in my post above, and we're automatically prioritizing existing customers.
- dpiers 4y agoI’m an auditor at a large organization that uses Sourcegraph, and I love it. On one project I had to trace a compliance flow across 14 services written in 4 languages, and Sourcegraph made it easy to navigate everything and document our findings with links to code at specific revisions. I use it to dig through our codebases on a daily basis.
- louiskw 4y agoA couple of projects you can look at that might be good substitutes for what you’re asking for - https://www.codecomplete.ai/ https://www.codecomplete.ai/ - current YC batch, looks very promising and sounds like this could be a big part of their USP https://www.tabnine.com/ https://www.tabnine.com/ - the OG code completion, mentions personalised models on their enterprise plan It’s also on the roadmap for us (AI code search not completions) https://bloop.ai/ https://bloop.ai/
- aunch 4y agoCodeium has a self hosted enterprise solution that gives the enterprise the ability to fine tune on their repos within that self hosted instance!