3 ms·
I've seen Copilot do the same thing - if you leave a comment like "// see this StackOverflow post", it'll often complete the comment with a link that looks a lo
by johnfn 4y ago
I've seen Copilot do the same thing - if you leave a comment like "// see this StackOverflow post", it'll often complete the comment with a link that looks a lot like a SO post but is actually nonsense.
My guess is that GPT-3 has seen a lot of SO links (or other links) and understands that sometimes engineers find it handy to have a long url-looking-thing with some numbers at the end, but it doesn't understand the fact that it links to something unique in the world outside of code, so it just tries to forge them as appropriate. Same thing with your URLs to bogus publications, I'd imagine.
- andrepd 4y agoOf course, it's a language model with 0 semantic knowledge about its output.
- phire 4y agoYeah, I once saw Copilot add a comment that talked about a workaround for bug in the win32 API it was calling, referencing a stackoverflow post. Obviously, the stackoverflow post didn't exit. I googled, and the bug it described didn't exist. Even widening the search, I couldn't find any documentation about any bug in that API. The emitted code after the comment was pretty good and didn't contain a workaround. It just hallucinated the comment about a workaround because it's seen other code do similar things and thought it would look nice here. I find Copilot to be a useful tool... But it is not a knowledge engine and should not be trusted. In my opinion, you only accept it's suggestions if you were about to type the same thing.