6 ms·
Wow. This is amazing. Although I wish they documented how much compute and data was used to get these results. I absolutely believe we'll crack the fundamental
by nojvek 6y ago
Wow. This is amazing. Although I wish they documented how much compute and data was used to get these results.
I absolutely believe we'll crack the fundamental principles of intelligence in our lifetimes. We now have capability to process all public data available on internet (of wikipedia is a huge chunk). We have so many cameras and microphones (one in each pocket).
It's also scary to think if it goes wrong (the great filter for fermi paradox). However I'm optimistic.
The brain only uses 20 watts of power to do all its magic. The entire human body is built from 700MB of data in the DNA. The fundamental principles of intelligence is within reach if we look from that perspective.
Right now GPT3 and DALL-E seem to be using an insane amount of computation to achieve what they are doing. My prediction is that in 2050, we'll have pretty good intelligence in our phones that has deep understanding (language and visual) of the world around us.
- SoSoRoCoCo 6y ago> Although I wish they documented how much compute and data was used to get these results. I'm hearing astonishing numbers, in the tens of megawatts range for training these billion-parameter models. And I wish they showed us all the rejected images. If those images (like the snail harp) were the FIRST pass of the release candidate model.... wow... but how much curating did they do? EDIT: Units. Derp.
- nullc 6y ago> tens of mega-joule range do you mean tera-joules? A hundred megajoules is about three bucks at 10 cents per kwh. I routinely do giga-joule level computations using just a rack of computers in my garage, they're no big deal.
- ajnin 6y ago> in the tens of megawatts But for how long? 1 second, 1 hour, 1 month? The energy matters more than the power.
- thom 6y agoDo you think we’ll crack the principles, or do you think we’ll just have very powerful models without really knowing what makes them clever?
- nojvek 6y agoMy bet is understanding the fundamental principles. Like building an airbus plane or starship requires fundamental understanding of aerodynamic principles, chemistry, materials and physics. DNNs will definitely not get us there in their current form.
- WitCanStain 6y agoI am very curious to see if concepts from cognitive science and theoretical linguistics (like the Language of Thought paradigm as a framework of cognition or the Merge function as a fundamental cognitive operation) will be applied to machine learning. They seem to be some of the best candidates for the fundamental principles of cognition.
- adriang133 6y agoThey don't need to solve the problem of reasoning, they only need to simulate reasoning well enough. They are getting pretty good, people already have to "try" a little bit to find examples where GPT-3 or DALL-E are wrong. Give it a few more billion parameters and training data, and GPT-10 might still be as dumb as GPT-3 but it'll be impossible/irrelevant to prove.
- pontus 6y agoMaybe a nitpick, but there's a difference between energy consumption during training and inference. If you want to talk about the energy necessary to train a human brain, it involves years of that 20W power consumption. What is the power consumption for inference time for Dall-E?
- burrows 6y agoNot to mention the billions of years spent in the evolutionary pipeline.
- littlestymaar 6y agoWell, if you want to go that route, you'd need to count all the energy spent to build computers of all kind since the 50s, and also all the energy spent to sustain the lives of people working on AI. And, well, all the millions of years spent in the evolutionary pipeline before these people where born ;)
- littlestymaar 6y agoI did some kick napkin math for Lee Sedol vs AlphaGo a few weeks ago: https://news.ycombinator.com/item?id=25493358 https://news.ycombinator.com/item?id=25493358
- nojvek 6y agoWow. Good to know that entire lifetime energy consumption is 50 MWh. At $0.1/KWh, you’re looking at $5000 in equivalent electric energy consumption over entire lifetime of a human being. The brain uses 20W of power. For a life time of ~80 years, that is 14MWh of energy usage. Suppose we say the brain trains for the first 25 years then that is 4.38 MWh. Equivalent electric energy consumption is only at $438. So yeah, the brain is quite efficient both in hardware and software.
- pontus 6y agoThat's still really surprising I think. I would have imagined that it would have been off by many orders of magnitude. The fact that these models are within a factor of 3-10 of the human consumption is pretty impressive. That being said, these models are training only for very specific tasks whereas obviously the human brain is far more sophisticated in terms of its capabilities.
- martamorena9 6y ago> I absolutely believe we'll crack the fundamental principles of intelligence in our lifetimes. I tend to agree. However this looks a lot like the beginning of the end for the human race as well. Perhaps we are really just a statistical approximation device.
- nojvek 6y agoYeah why i mentioned the great filter of Fermi paradox. I also believe humans in our current species form won’t become a space bearing species. We’re pretty awful as space travelers. It is very likely that we’ll have robots with human like intelligence, sensor and motor capabilities sent as probes to other planets to explore and carry on the human story. But future is hard to predict. I do know that if the intelligence algorithm is only in the hands of Google and Facebook, we are doomed. This is a thing that ought to be open source and equally beneficial to everyone.
- jackric 6y ago> The entire human body is built from 700MB of data in the DNA. I think this notion is misleading. It doesn't relate to the ease of simulation it on our current computers. You'll need a quantum computer to emulate this ROM in anything like realtime. The DNA program was optimised to execute in an environment offering quantum tunnelling, multi component chemical reactions etc.
- nojvek 6y agoMy point wasn’t to emulate the DNA in a virtual machine. The point was that between humans and chimps (our closest DNA relatives) we’re 99% same to them. So high order intelligence that gives rise to written language and tool building is somewhere in that 700MB of DNA code. And 1% of that (just 7MB) is responsible for creating the smartest intelligence we know (humans). In that sense intelligence artchitecture isn’t very complicated. The uniformity of isocortex which we have the most relative to our brain size compared to any animal says we ought to replicate its behavior in a machine. The isocortex/neocortex is where the gold is. It’s very uniform when seen under microscopes. Brain cells from one region can be put in another region and they work just fine. All of ^ says intelligence is some recursive architecture of information processing. That’s why I’m optimistic we’ll crack it.
- ZeikJT 6y agoI think what the parent was saying is important though, that 700mb of "data" isn't complete. It's basically just really really good compression that requires the runtime of our universe to work properly. The way proteins form and interact, the way physics works, etc are all requirements for that DNA to be able to realize itself as complex thinking human beings.
- jackric 6y agoYes you rephrased that very well. If your plan to build intelligence is by copying how nature does it, well then you'll need to build a "nature" runtime that can emulate the universe. You can either do that slowly or inaccurately