7 ms·
Well, I take the the opposite stand: GPT-3 = Give a 3rd grader wikipedia and some paper. While it's certainly fun how it's relatively coherent in writing text,
by fock 6y ago
Well, I take the the opposite stand: GPT-3 = Give a 3rd grader wikipedia and some paper. While it's certainly fun how it's relatively coherent in writing text, I couldn't see yet, how it links syntactically correct text to actual facts. Which in my opinion is the difference between 1e100 apes with typewriters and aforementioned 3rd grader.
- throwaway4666 6y agoI think gwern's point (whose opinion on deep learning I respect a lot more than his on genetics) is that GPT-3 is nearly not the end of the story. OpenAI released a GPT pretty much every year and each one is a spectacular improvement on the previous one with no sign of that trend ever stopping. If size is really what matters at all, there's no telling what GPT-4 or GPT-5 might be capable of, let alone a giant GPT run by state-sized actors.
- ipsum2 6y agoThe machine that trained GPT-3 is one of the largest GPU clusters in the world at the moment. (10k v100 GPUs hooked up with fast interconnect). For comparison, Nvidia's "Selene" supercomputer only has 2k a100 GPUs and is #7 fastest supercomputer in the world: https://www.hpcwire.com/2020/06/22/nvidia-nabs-7-spot-on-top500-with-selene-launches-a100-pcie-cards/ https://www.hpcwire.com/2020/06/22/nvidia-nabs-7-spot-on-top... a100 GPUs are around 3x faster than v100 GPUs. It would take some time for governments to catch up :)
- qeternity 6y ago1. This is a list of publicly known supercomputers (it’s likely that the intelligence community have larger secret assets). 2. Most of the largest super computers are owned by states.
- throwawaygh 6y agoLots of large companies don't include their clusters on those supercomputer lists. Including Google. IDK about DoD, but I wouldn't be surprised if they have top 10 clusters we don't know about. The list of largest supercomputers is definitely only a list of the largest disclosed supercomputers. That said, only so many actors could've secretly sourced 10K+ v100s with fast interconnect...
- pharke 6y agoI think the point Gwern makes is that if a government wanted to they could easily allocate enough resources to do this. More bluntly I think he's saying if the US decided tomorrow to begin a Manhattan Project for AGI there is a non-zero chance that they might succeed in 7 years.
- dwohnitmok 6y ago> GPT-3 = Give a 3rd grader wikipedia and some paper. I don't know if you're being hyperbolic here, but if not, I consider that a massive step forward for AI.
- bildung 6y agoI think fock's point is not about the 3rd grader's capability of reasoning, but only about their proficiency in writing syntactically correct sentences. GPT-3 does zero reasoning.
- AgentME 6y agoWhat are you calling reasoning? Is reasoning something entirely separate from choosing what word is expected next in a sentence based on the logic of the words? You can give GPT-3 things like "Alice was friends with Bob. Alice went to visit her friend _____" and "10 + 10 = 20. 20 + 20 = ___" and get right answers. You can tell it the definition of a made-up word and ask it to use the word in a sentence, and it can come up with a sentence that uses it in a relevant context. You can give it a short story and ask it questions about the story. All of these go far beyond basic syntactic concerns like putting subjects and verbs in the right spots.
- p1esk 6y agoI think he meant GPT-3 does zero human reasoning.
- empath75 6y agoDo we know that humans do reasoning? People talk much much faster than they could work out anything like what people consider to be logical reasoning. Remember that logic had to be invented by Aristotle. It was a mechanical system meant to approximate how humans make decisions naturally.
- Sharlin 6y agoWell, of course. It’s not human. Even a superintelligent AGI would likely do zero ”human reasoning” unless it wanted to do authentic emulation of a meatbrain for whatever reason. Like keeping uploaded humans running.
- ipsum2 6y agoIf the AI is at a 3rd grader level, that means it's already reached human intelligence! In my opinion, in some regards the AI performs well above a third grade level in some areas (being able to write a complex paragraph, terminal completion, writing code) and below in some other areas (not knowing basic multiplication, doesn't understand spatial context).
- dash2 6y agoA confused idea people have about AI: that the goal is to make a computer that thinks like a human. There's not much point in that: we have 7 billion humans, and making more is cheap and fun. At least for now, AI is interesting when it complements, not substitutes for, human intelligence. We want Google to be able to search a billion web pages. We don't care if it can't make a decent cup of tea.
- Erlich_Bachman 6y agoYou are missing a huge part of the picture. If we can make an AI with an intelligence of a human, we can train that to be very good at some problem (like a human expert in some area). And then we can clone those intelligences into thousands in order to progress in that area much quicker. A human-level intelligence is also a very important goal because it, even by definition, produces the singularity point. (AIs that can make better AIs, exponentially.) > We don't care if it can't make a decent cup of tea. We absolutely do, although it depends on the price at which they can do that. But once an AI can do that at all, it is only a matter of time before we can make them do it cheaply (we are very good at making things cheap once we know what things we need), at which point it becomes a driving force in our progress. Once there are self-driving cars, there will be a revolution in the labor force, for example, and driving is only a small portion of what a "human-level" intelligence is capable of.
- dash2 6y agoYes, I think that in the end, this would be true. AGI itself would be transformational - partly because it would be easily linked with computers' "superhuman" powers of memory, throughput etc. But, on the way to AGI - which seems to me a long way off, despite some of the interesting arguments made here - it's not really true that "being very humanlike" is a big commercial advantage. As long as we have (say) less than the intelligence of a 2 year old child, or a chimp, then we'd rather have computers be doing very un-human things.
- empath75 6y agoDoes a third grader not have intelligence?
- criddell 6y agoAnd in a few years it might be a forth grader. Then maybe a few years after that, a fifth grader. Slow progress is still progress.
- fock 6y agoand at some point the energy input is larger than producing the fifth grader and a generation of teachers. Which ... - seems inefficient, especially given that you can't teach this machine any new rules in a closed-model fashion; say, language finally adapts a nice grammatic form for non-binary-and-cis persons alike - how would you put it in this model?
- AndrewKemendo 6y ago>Give a 3rd grader wikipedia and some paper If that's your perspective, that would be considerably staggering progress. There is an implicit goalpost for AI that laypersons have that assumes if it can't kill us all then it's not "real AI."
- teruakohatu 6y agoBut if they kill us by accident its not AI eg. Tesla Autopilot.
- AndrewKemendo 6y agoRight...which is consistent cause the "AI" doesn't have volition.