6 ms·
CodeCompose: A large-scale industrial deployment of AI-assisted code authoring
- m3kw9 3y agoWho else’s is tired of these trillion dollar companies talk and talk and have no products?
- neuronexmachina 3y agoIt's a paper about a (for now) internal product: "In this paper we present CodeCompose, an AI-assisted code authoring tool developed and deployed at Meta internally. CodeCompose is based on the InCoder LLM that merges generative capabilities with bi-directionality. We have scaled up CodeCompose to serve tens of thousands of developers at Meta, across 10+ programming languages and several coding surfaces." Looks like the InCoder model it's based on can be downloaded here: https://huggingface.co/facebook/incoder-6B https://huggingface.co/facebook/incoder-6B
- Veuxdo 3y agoDoes companies publishing papers about internal tools serve any purpose other than PR?
- Keyframe 3y agoGoogle ones seemed to have sparked few revolutions on their own.
- williamstein 3y agoThis paper says: “Customized for the organization: CodeCompose is fine-tuned on Meta’s internal code repository, and is thus able to handle Meta-specific languages such as Hack and Flow.” If you work at an org that might want to build their own LLM trained on their own internal code base, then the lessons of this paper would be of value to you.
- Veuxdo 3y agoMakes sense. For Meta, though, by publishing papers like this are they hoping for something else other than PR? My only other guess would be attracting talent.
- bee_rider 3y agoLots of companies have research departments and release papers; the people working in these departments have some academic roots at least. The incentives for releasing papers are: * To raise your profile and reputation generally * The specific publish or perish incentives in academia * Because you really think you’ve done something interesting and novel, and want to share it with the world Only the middle one is removed when going to industry.
- chromatin 3y agoThere are many scholars working in industry (from software engineering to biotech) who believe in the ideals of information sharing and publication of research.
- deleted 3y ago[deleted]
- lmeyerov 3y agoIt was a useful read for me, esp seeing numbers on fine-tuning & use. We are piloting a DB analyst tool where users can use natural language to do DB queries, generate AI analyses, make interactive GPU charts, etc, so many nearby questions we think about a lot. Previously as a PhD publishing on program synthesis, most of our writeups were much smaller scale wrt live user evals, so all combiner... super cool to see. For FB... It probably helps keep the team valued internally + helps with retention & recruiting. For PhD trained types, this kind of paper is almost table stakes. Less obvious... FB has been laying off teams like this despite productivity ROI intuitions, so if I was there, I'd be careful to quantify current + future ROI - I'm sure there are key #'s not being shared.
- IshKebab 3y agoOf course it does. They give lots of details about the tools and their use which is obviously helpful to anyone wanting to do something similar. A great example is https://www.uber.com/blog/research/keeping-master-green-at-scale/ https://www.uber.com/blog/research/keeping-master-green-at-s... I think maybe Gitlab Merge Trains predate it, but it was definitely influential.
- ChatGTP 3y agoIt’s a funny game because they all need their own clones of each model / product. Feels like tech is making billions but is a little lost ?
- regularfry 3y agoI would very much like a local code assist tool. Assuming integration with editors is my problem, what's best in class this week if a) I have a respectable GPU; b) I don't, and need CPU-only?
- Filligree 3y agoNothing even comes close to copilot. I realise you said “local”, but if you insist on that you’re going to be disappointed.
- MisterAV 3y agoOn Visual Studio there's an extension (by Microsoft) called IntelliCode which is a small AI assistant that runs locally on the CPU. It doesn't come close to these new large GPU models but it's quite handy. It looks into what you're typing on the current line and the previous activity along with the current project and tries to predict the full line or even the same change on multiple lines if that makes sense.
- azhenley 3y agoWe recently published a paper on IntelliCode and share some of the usage numbers. https://austinhenley.com/pubs/Vaithilingam2023ICSE_IntelliCode.pdf https://austinhenley.com/pubs/Vaithilingam2023ICSE_IntelliCo... Disclaimer: I'm one of the co-authors.
- wsxiaoys 3y agoCheck out https://github.com/TabbyML/tabby https://github.com/TabbyML/tabby, which is fully self-hostable and comes with niche features. On M1/M2, it offers a convenient single binary deployment, thanks to Rust. You can find the latest release at https://github.com/TabbyML/tabby/releases/tag/latest https://github.com/TabbyML/tabby/releases/tag/latest (Disclaimer: I am the author)
- mormegil 3y agoI have just installed Fauxpilot <https://github.com/fauxpilot/fauxpilot https://github.com/fauxpilot/fauxpilot> (nVidia GPU-only) and it works... OK. Still evaluating and I'm basically sceptic on the whole concept, but... let's see.
- alan-stark 3y agoThe abstract says ..we present metrics from our large-scale deployment of CodeCompose that shows its impact on Meta's internal code authoring experience over a 15-day time window, where 4.5 million suggestions were made by CodeCompose. Quantitative metrics reveal that (i) CodeCompose has an acceptance rate of 22% across several languages, and (ii) 8% of the code typed by users of CodeCompose is through accepting code suggestions from CodeCompose. Qualitative feedback indicates an overwhelming 91.5% positive reception for CodeCompose. In other terms, out of 4.5 million suggestions about 80% were off, yet there is 91% positive reception. That's 3.6 million rejected suggestions that potentially distracted programmers from doing their work. Yet users are happy. Is there a contradiction in these figures?
- pavlov 3y agoI think the 8% number better explains why users were so overwhelmingly happy. Assuming the suggestions in general are not distractingly wrong, then 8% of code automatically written is a decent amount of time saved researching solutions.
- layer8 3y agoBut only 22% are accepted for those 8%, which means that the 78% code suggestions that are not accepted correspond to an equivalent of over 28% of all code written. Not sure that having to spend the time evaluating an additional 28% of code in vain amounts to an overall win. Though I guess the success rates when using Stack Overflow aren’t too dissimilar.
- shostack 3y agoWhat it doesn't tell us though is how useful the rejected recommendations were. Meaning, how many rejected solutions were sufficient to give the engineer enough context to turn a 30m task into a 5m task because they generated a recommendation, got an idea, rejected it, and rewrote it more efficiently or more correctly? There's a lot of "devil in the details" likely buried in here.
- visarga 3y ago
- wpride 3y agoEveryone in this space seems to be building on the LSP and classic auto-complete in particular as their UI. But I've found this to be non ideal. - As mentioned in this paper I definitely do not want the AI suggestion crowding out a suggestion generated directly from the type bindings - I often do want the AI to write an entirely new block of boilerplate. To do this you have to write a comment string targeted at the AI, then delete this afterwards - Sometimes I'd just like the AI to explain to me what some code does without writing anything - This isn't something I always want on; I find myself turning the plugin on and off depending on the context Overall I think we need a novel UX to really unlock the AI's helpfulness
- anotherpaulg 3y agoI have been enjoying a chat based AI coding modality. I built some tooling that gets rid of the need to cut & paste code between the chat and your files. This makes chatting about code changes much more ergonomic. My tool also integrates directly with git, which provides a safety net. It’s easy to undo changes if the AI does something silly. Here are some chat transcripts that give a flavor of what it’s like to code with AI this way: https://aider.chat/examples/ https://aider.chat/examples/ My tool is open source, and currently only works if you have a gpt-4 api key.
- tyree731 3y agoVery cool. I’ve been using ChatGPT quite manually for similar effect here, though I’m often using it for fragments of code less than whole projects/files, given I’m often dealing with an existing codebase.
- anotherpaulg 3y agoA lot of my recent work has been focused on making this chat style coding work better with larger, pre-existing codebases. I wrote up some notes on one effective approach: https://github.com/paul-gauthier/aider/blob/main/docs/ctags.md https://github.com/paul-gauthier/aider/blob/main/docs/ctags....
- 3y ago
- zeedude 3y agoLimit training to stackoverflow input and wham! we have automated modern programming ;)
- synthiq 3y agoFor anyone interested in related research, I used https://mirrorthink.ai https://mirrorthink.ai to find some background on the state-of-the-art. (disclaimer: this is AI generated, but grounded on contents of papers, with real references, so I'd say it is still constructive) The state-of-the-art in code generation has seen significant advancements with the deployment of large language models (LLMs) in various code authoring tools. One such example is the study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT [1], which evaluates the code quality of these AI-assisted code generation tools. The study reveals that ChatGPT generates correct code 65.2% of the time, while GitHub Copilot and Amazon CodeWhisperer achieve 46.3% and 31.1% correctness, respectively. These results indicate that LLMs have made substantial progress in generating high-quality code, but there is still room for improvement. Other research in the field has explored various techniques to enhance code generation and assistance. For instance, RepoCoder [2] focuses on repository-level code completion by integrating code generation and retrieval models in an iterative paradigm. This approach considers the repository-level context, including customized information such as API definitions and identifier names, to improve code completion suggestions. Serenity [3] leverages library-based Python code analysis for code completion and automated machine learning. The authors explore the potential of data flow analysis produced by Serenity to improve code completion when combined with neural models. In addition to these advancements, the field has seen progress in incorporating contextual information into code completion models. The paper on enriching source code with contextual data [4] investigates the impact of incorporating contextual information on the performance of code completion models. The authors conduct an empirical study to analyze the effectiveness of this approach. These achievements, along with the advancements in LLMs, contribute to the ongoing progress in code generation and assistance. As the field continues to evolve, it is expected that AI-assisted tools will become increasingly sophisticated and effective in assisting developers with various aspects of the software development process. [1] Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT - 2023: https://arxiv.org/abs/2304.10778 https://arxiv.org/abs/2304.10778 [2] RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation - 2023: https://arxiv.org/abs/2303.12570 https://arxiv.org/abs/2303.12570 [3] Serenity: Library Based Python Code Analysis for Code Completion and Automated Machine Learning - 2023: https://arxiv.org/abs/2301.05108 https://arxiv.org/abs/2301.05108 [4] Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study - 2023: https://arxiv.org/abs/2304.12269 https://arxiv.org/abs/2304.12269
- bolinfest 3y agoIf you want to skip the paper and watch the video: https://youtu.be/ANDJ0TKjyWw https://youtu.be/ANDJ0TKjyWw Disclaimer: I am the person in the video.
- muglug 3y agoIt was a great video, and a great paper. As someone who writes quite a lot of Hack, I'm selfishly interested in whether you plan to open-source this work (not the weights, obviously, but everything else).
- shostack 3y agoAre users finding usefulness and time savings in the suggestions even if they ultimately reject them? I'm thinking of scenarios where they might get an idea and then reject them and rewrite it better.
- Animats 3y agoHm. It seems to be like automated Stack Overflow. Only 8% of the code comes from the AI system, but it's useful for getting examples of how to do something. Hallucination about API calls was reported as a problem. I've seen that one. There's an amusing, and seriously annoying, tendency for these systems to make up some plausible API call that does what you need, but doesn't exist. Maybe something should collect up such suggestions as proposals for new API calls.
- freeone3000 3y agoThe future of API design — “yes it would make SENSE if that existed, but it doesn’t” => now it does
- fabmilo 3y agoI would like to work in this code copilot space, I think will be one of the fastest applications of LLms in the near future. I have been working on a tool to autogenerate docstrings from a python method in google format
- nforgerit 3y agoMy 2 cents on AI-assisted code authoring: First time I got my hands on Github Copilot some time ago, I intentionally chose something "odd" (so no CRUD API in X, Kafka stuff, sorting algorithm) so I chose to re-create some Excel sheet alike UI in React & Canvas. First everything proposed was misleading but after the first couple of lines, it astonished me by really proposing things like nested loops for iterating over a grid (surely, I used an according function name). I came quite far (for a quick test) with marking single and multiple cells and being able to drag the marked area and double-click for text input. Then I realized it was about time to invent a proper data model and stopped it there. All in all, I'm using it for like 50% of daily coding, but it's frequently misleading in subtle tricky ways that I expect to be quite dangerous especially if using it for tests as well (need to double-check if it's not BSing me twice). Debugging code is already a big drag in our work and I'm somehow not happy with debugging AI-generated code.
- throwaway349893 3y agoThis work started as "Big Code" and the team responsible (Probability) was laid off: https://www.linkedin.com/posts/erikmeijer1_meta-layoffs-hit-an-entire-ml-research-team-activity-6996955327560650752-QFjV?trk=public_profile_share_view https://www.linkedin.com/posts/erikmeijer1_meta-layoffs-hit-...
- 6510 3y agoAs the tools improve I suspect a whole new breed of developers to spawn who cant really code but are great at rejecting code that isn't extremely easy to read. After that very little new code is written and all of the focus will be on the development of stronger less toxic glue. Then everything goes dark, a few companies will absorb and confuse the art or otherwise find ways to exclude everyone else from the process.
- nologic01 3y agoThen results are not really surprising. If you have access to a large body of code the mere ability to query it efficiently and re-use boilerplate or snippets of code performing some standard function and/or adding documentation is a productivity booster. Now we know its 8%. Small but measurable. Arguably if you optimise the system to do precisely that rather than pretend to hold civilized conversation you could get quite a bit more bang for you gazillion parameter buck.
- onesphere 3y agoAI-driven JIT https://thetaplane.com/ai/jit https://thetaplane.com/ai/jit