I think we're on the cusp -- which side of the cusp I'm not sure -- of the Uncanny Valley of language models. They're good enough that a surprising number of pe

I think we're on the cusp -- which side of the cusp I'm not sure -- of the Uncanny Valley of language models. They're good enough that a surprising number of people believe they're capable of human levels of cognition, but bad enough to prove that's not quite the case yet. Having played with GPT-3 (and a bit with Codex) for quite awhile now, and keeping up with the announcements of all the major breakthroughs, I can see both sides of it. They're certainly not at human level yet. I originally got access to GPT-3 with a proposal of using it to generate unit tests for JavaScript code, which would be a massive time saver for everyone on my dev team at work. Turns out, even Codex, which is trained specifically on code, can't produce a coherent unit test for even a function as simple as one averaging an array of numbers -- and half the time, the unit test it produces isn't even syntactically valid and would just fail to run if used. Codex also had trouble understanding references to external data: when asked to produce the HTML/JS code for a table containing links to specific data, it often generated nonsensical links (and, amusingly and also not at all surprisingly, links to Rickrolls). On the other hand, when I've given it a decent prose prompt, it's been able to write stories that could easily have been human-written 85% of the time. (The other 15% covers cases such as when it decides dogs can talk in an otherwise realistic story, or describes a character's outfit completely differently from one sentence to the next, or mixes up which names refer to which characters, etc.) It's certainly an extremely impressive AI, one that sounds closer to human than any other AI I've used (including GPT clones with smaller models, like GPT-NeoX or GPT-J). And of course, GPT-3 is "so last year"; PaLM, Flamingo, and other models are now outperforming it regularly, and doing so with increasingly multi-modal capabilities. Though those aren't public for me to play with :( So, yeah, I can see how people who don't know much about AI could think we've reached human levels, but I also see how we clearly have not. I obviously don't have access to LaMDA to put it through its paces as I have with GPT-3, and it's intriguing that this was an AI engineer who jumped the shark here, but I still doubt we're quite at human levels yet. Tantalizingly close, but not there. But I could be wrong. Without access to these models, who knows? That said, LaMDA is from last year, and is the same sort of transformer architecture as GPT-3, only trained exclusively on conversations between humans. It's got 137B parameters, which is slightly smaller than GPT-3. Together, this leads me to assume it's probably comparable in performance to GPT-3, albeit with more conversational results, since that's the data set it trained on. So, yeah. I see both sides, and I'm excited by the pace of progress, but we're not there yet, even if the Uncanny Conversation has fooled some people.