4 ms·
> The largest issue is that state-of-the-art models don’t understand the structure of code, such as non-local context. Depending on how local he's talking, thi
by lolinder 4y ago
> The largest issue is that state-of-the-art models don’t understand the structure of code, such as non-local context.
Depending on how local he's talking, this isn't really true of Copilot. In my experience it will use context all the way up to the top of the file, even in very long files. And at least the Rust version even seems to look at the imports—if you have a use declaration it will actually correctly build and use structs in other files regardless of whether you've yet used them in the current file.
- ShamelessC 4y agoI had a similar thought. I think in this instance, they mean something like a labeled/supervised training approach where the model is given not just the tokenized code, but also perhaps is grounded in the possible structure (indentation vs. semicolons, function-scope, etc.). My understanding is that copilot is largely a self-supervised approach. They feed a massive body of (somewhat noisy) code into the model. The model really does learn a lot of structure on its own and this is a testament to deep learning on noisy datasets. I'm guessing the "hooks" that they have already from IDE's, language-servers, etc. _are_ quite "structure-aware" - so they want the predicted structure as well as the code, so they can improve the typing experience beyond line-completion. I think the estimation of 100 million for such a task is maybe too high? I don't know - it feels like you could actually get quite close to such a system by simply using thousands of custom prompt engineering tricks that prepend structure examples to the prompt?
- adamsmith 4y agoYes, this was precisely what I was referring to. In small-enough programs (e.g. one file) Copilot has all the context. The other extreme would be something like the Chromium codebase. Because of this, Copilot looks better in quick demos than real-world use. (Though of course it is very impressive and this tech will get there, hopefully very soon!)
- lolinder 4y agoBut what I'm saying is that it does use imports, at least in Rust. I'm assuming that somehow behind the scenes they're concatenating the contents of the imports into the prompt. I can imagine this is easier in a language like Rust that has a really strict module system, and to be fair the project that I've been using it on is a side project that isn't over 10,000 lines of code yet. If I were up to 30 imports per file I can imagine concatenating would become much less effective.
- adamsmith 4y agoDoes it seem to only understand imports of public libraries? If so, it's likely that, rather than understanding the contents of those libraries, it's learning from others' use of those library APIs. If not, it is likely just understanding the words in the API at a shallow depth.
- lolinder 4y agoNo, it's imports from other files in my project. It's either using the import or the fact that I have another tab open. There are definitely times where it produces a close approximation that's obviously just statistical, but there are other times where there's no question that it picked up something from a different source file that couldn't have possibly been in its training set. I haven't yet decided if it's using imports or opened files in the editor, but it's definitely not just using the single file I have active.
- adamsmith 4y agoIt could be doing some "fine tuning" based on the repo. That would be cool! That said, what I meant when referring to 'understanding' the non-local nature of code was in a more principled way. For example, if an object defined in another file has a function called `rename` that takes zero arguments, when calling it from another file Copilot will likely suggest arguments if there are variables like `old` and `new` near the cursor, even though `rename` actually doesn't take any, just because functions called `rename` typically take arguments. This behavior is in contrast to a tool like an IDE that can trace through the way non-local code references work.