9 ms·
Deep Neural Nets: 33 years ago and 33 years from now (2022)
- version_five 3y agoThis was really good. The only thing I didn't see explicitly discussed, although I guess it's obvious, is that what's different 33 years later is the inputs the models operate on. The '89 sota model used 16x16 greyscale images, today we have single digit megapixel color images, in 30 years, a desktop will be able to train Clip in 90 seconds, but what will the sota models be trained on?
- retrac 3y agoHuman behaviour in a way far more general than which token we might next type. To mimick humans as closely as might be possible with the basic deep learning method, train something that can predict human behaviour in general. Training would require billions to quadrillions of hours of video and audio and probably many other inputs, from many different people, engaged in the full variety of human activity.
- reducesuffering 3y agoWhy? An adult by 25 only has 146k hours of video experience “training,” most of it repeated, derivative, and unproductive. And their encoded genes can be observed in their genome, so don’t need to be retrained by millions of years of evolution.
- deleted 3y ago[deleted]
- alanbernstein 3y agoHow better to learn to do menial physical tasks like house cleaning, and produce picking?
- thaw13579 3y agoMuch of that time also includes physical interaction with the world, which makes it far more valuable because it can improve performance in a focused way.
- canjobear 3y agoNeural nets seem to learn much slower than humans. Even GPT-2 has seen orders of magnitude more tokens of language than a human experiences in a lifetime. At least as far as language is concerned, humans are able to extract a lot more information from their training data.
- mkaic 3y agoHumans are also extensively pretrained by billions of years of evolution, so by starting from scratch GPT is admittedly disadvantaged from the get-go.
- georgeg23 3y agoBut the human genome is only 3 gigabytes and the vast majority of that is unlikely to be encoding brain structure.
- retrac 3y ago"Only" 3 gigabytes. The lambda calculus (a system we know is capable of infinite self-complexity, learning, etc. with the right program) can be described in a few hundred bits. And a neural net can be described in the lambda calculus in perhaps a few thousand bits. Also, we have no idea how "compressed" the genome is.
- georgeg23 3y agoBasic structure can be encoded yes (and obviously is given brains have consistent structure), but the weights or parameters, presuming that brains learn via synaptic weights, obviously do not fit in the genome. Compression still must obey information theory.
- canjobear 3y agoTrue, but we don’t know how to recreate that kind of pretraining.
- thaw13579 3y ago
- edgyquant 3y agoHumans do not train on video, the idea of a video, or even a frame, is a high level abstraction within the human brain.
- quickthrower2 3y agoHumans have a brain that physically changes especially in childhood, so that is potentially a massive advantage.
- ramblerman 3y agoMillions of hours of data captured by headsets like the vision pro? Not sure all the things it captures, but a model could be trained on the combination of audio/video/spatial/iris/what have you...
- eigenvalue 3y agoWe might have megapixel images that we can easily get with phone cameras, but virtually all vision models in common use take 224x224 resolution images as input, or maybe 384x384. Anything higher resolution than that just gets resampled down. It seems that you are better off using your compute budget on a bigger “brain” than on better “eyes” for now.
- version_five 3y agoI don't think that's current. Certainly the object detection models work on bigger images, and the datasets they're pretrained on e.g. coco are not 224x224. I think standard models pretrained on imagenet, like the Resnets usually have everything resized to 224x224, and so they favor this kind of scaling.
- alexmuro 3y agoIt's crazy how little has changed and how much had changed. I remember what a revelation "the unreasonable effectiveness of RNNs" was when I was read it and it feels like we live in a different world.
- benreesman 3y agoI think we could collectively more constructive and sober conversation if we kept that 2015 bit of work as a sort of baseline. The new stuff is better, by a lot, and with implications more to come. But those of us paying attention then had a frame of reference where “so much better it’s crazy” still stops short of “it’s out of control”. It’s a lot better.
- dang 3y agoRelated: Deep Neural Nets: 33 years ago and 33 years from now - https://news.ycombinator.com/item?id=30673821 https://news.ycombinator.com/item?id=30673821 - March 2022 (5 comments)
- bilsbie 3y agoIt’s interesting in that time we almost completely lost interest in neural networks and then came back around to them.
- sroussey 3y agoI had to retake my AI class at university several times because I just didn’t agree on the “AI is symbolic search” aspect. Now though, I’m sure people are taking LLMs and putting them together to do forward and backward chaining.
- jacquesm 3y agoThank Hinton for that. It's a pity we don't have a Nobel for software. But a Turing award is pretty neat as well.
- Sharlin 3y agoIn this case there are good reasons for the resurgence, but that's really the case with pretty much anything software-related. Except the fashion cycles tend to be shorter with more mainstream technologies.
- Funtech007 3y ago[flagged]
- Nevermark 3y agoThe most fundamental change is the difference in what models are being trained on. Little images of characters is a trivia type problem, very different from training on the linguistic and visual communication of essentially the whole human race. Another 33 years of expanded computing resources won’t be training models to mimic the behavior and knowledge of humanity. That problem (us!) will have been reduced to a toy problem long before then.
- visarga 3y agoI think AI models will evolve by generating synthetic data, filtering and improving it, and then retraining. Possibly with external systems in the loop - code execution, search, human, simulation or robot. Quality won't degrade because there will be a lot of effort put into data filtering and diversity. We can always improve on a model by giving it more time. Model architecture doesn't matter compared to the dataset. Any model from a class can learn the same skills from the same data, but change the data and they all change their abilities - the intelligence is in the data. The future is data engineering, not model architecturing. Human culture, by analogy, evolves faster than human biology. The data is evolving faster than the model. And we are seeing a drastic reduction in novel architectures in AI, diverse datasets applied to the same transformer models in recent years. Even among the transformers, very few variants are largely used, thousands of them abandoned. I like to think of it as language evolution by memetics being the real engine behind intelligence. We and AI are riding the language exponential together.
- eru 3y ago> Model architecture doesn't matter compared to the dataset. Any model from a class can learn the same skills from the same data, but change the data and they all change their abilities - the intelligence is in the data. You might be right in the same sense that big-O notation is 'right'. Constant factor can matter; especially once you have to take energy use into account.
- ResearchCode 3y agoCome close to solving the toy problem of autonomous driving first, we're still waiting.
- fnord77 3y ago> Our datasets and models today [2055] look like a joke. Both are somewhere around 10,000,000X larger. will there really be 10 million times 400 million images floating around then?
- sroussey 3y agoGenerate as many as you need.
- sroussey 3y agoOh, also curious… today, how many individual image frames from video are there just from Tesla vehicles?
- alpaca128 3y agoTraining models from generated content degrades them over time.
- pyinstallwoes 3y agoYet science fiction pushes civilization towards novelty
- Philpax 3y agoThe generated results can come from other means - for example, pretraining on rendered CG imagery is quite popular in the computer vision world, especially for problems where acquiring ground truth data in the real world is quite difficult.
- ramblerman 3y agoI think you are limiting yourself by thinking of the dataset of the future as just being more and bigger images. Perhaps it will be trained on whole videos, or a combination of different inputs from agents that move about in the real world / or a video game.
- 3y ago
- mk_stjames 3y agoSomething else I find exciting, starting with one of the reflections- The original training took 3 days on a Sun 4/260 workstation; I can't find specifics but I believe that era of early SPARC workstations would likely pull about 200 watts in total (the CPU wasn't super high powered but the whole system, running with the disks and the monitor etc would pull about that). So 200 watts * 72 hours = 14400 watt-hours of energy. Karpathy trained the equivalent on a Macbook, not even fully utilized, in 90 seconds. Likely something around 20 watts * 0.025 hours = 0.5 watt-hours. An energy efficiency improvement of nearly 30000x.
- fouronnes3 3y agoThis is very interesting, because I've always thought that all NN performance should be measured in a unit with energy in the denominator.
- eru 3y agoIt totally depends on what you want to use a measure for. Just like neither height or volume alone will tell you what will fit in your car. By any measure that puts energy used by the brain in the denominator, humans are probably dumber than ants. But that doesn't mean those measures are always accurate. (For contemporary neural networks, you also have to distinguish training costs from inference costs.)
- nojvek 3y agoTo add more context, humans are 100W biological machines. Brain is ~20% of that power - 20W. The greatest form of general intelligence at 20W. A MacBook Air is ~30W. https://www.jackery.com/blogs/knowledge/how-many-watts-a-laptop-uses https://www.jackery.com/blogs/knowledge/how-many-watts-a-lap...
- renonce 3y agoIt is “the greatest” because we only appreciate intelligence that we ourselves understand. A 0.0001W calculator calculates arithmetic faster than any human brain.
- deleted 3y ago[deleted]
- dsign 3y agoIt's not clear that compute will scale as it did for the next 33 years. But it doesn't really need to. I read the article and I was thinking "my God, I remember I used MSE that weekend in my pet ML project and it really didn't work out that well; wrong loss function." Our current crop of LLMs, or the one next year, will be perfectly able to tell me how I can improve my code and graphs, which means that I can deploy some expert-level techniques that otherwise would be "locked" to me by 50000 hours of "mastery acquisition". A part of me is telling me that we humans are doomed, and that in 33 years we would have created a world in which we humans are irrelevant. But another part tells me that if we avoid that fate and all the other dooms, the future might just be quite bright.
- somewhereoutth 3y ago> or the one next year We have heard, and will continue to hear, this sort of thing rather a lot. The last 5 yards are the hardest, but without them the previous 5 miles are of limited utility.
- NumberWangMan 3y agoI think there’s going to be a point where we need to slow a AI way, way down in order to avoid bad outcomes. I’m with Zvi Mowshowitz here: we should encourage progress and risk taking in every area except those where there are extinction risks. Applying today’s LLMs to all sorts of problems won’t end us. But I think we may only be a few years away from AGI that is conscious and can plan, and we don’t know the upper limit of how smart we’ll be able to make them. And I think that we have a responsibility to any intelligent being we bring into the world. Some lament that there’s no test to become a parent-what about creating a million copies of a new virtual brain from scratch? And basically so they can be born into lifelong servitude.
- ilaksh 3y agoMaybe a better idea of the future is to look at what LeCun is working on now as a future program. He wants to change quite a lot in order to move towards more animal-like cognitive abilities. LeCun is not even really interested in supervised learning anymore, for example. https://youtu.be/vyqXLJsmsrk?si=8n0ylC6qdLX06CmY https://youtu.be/vyqXLJsmsrk?si=8n0ylC6qdLX06CmY Note that the talk is not really primarily about ChatGPT even though that's in the title. The new ideas are a little bit in. The beginning of the talk is just him explaining how unimpressed he is with LLMs. Which I think is a misjudgement but that doesn't mean his plan doesn't have merit.
- golol 3y agoI don't understand why Yann is so focused on his "Animals are smarter than AI" analogy. If compute wasn't so limited, couldn't we just train a transformer on video, audio and text data? I don't see why it would not learn the basic physical structure of our world just like a language transformer learns the grammatical and other structures of language. Then with this prettaines transformer you can build an agent and use some reinforcement learning too. I feel very confident that this would mirror the level of intelligence of non-human animals quite well.
- jerpint 3y agoThere have been many attempts to do multi modal pre training, the difficulty is finding the right combination of data for it to be “useful” and “scalable”. It’s not trivial to just train a transformer on video, text, audio, etc. mainly due to O(N^2) on the token counts, time components with video, etc
- golol 3y agoI don't really see it. O(N^2) for context window length is not an issue as you don't need particularly much longer context windows than for text. You don't have to run this at 30 FPS, 12 would already be enough to understand what's going on. The dimensionality is much higher of course, but how many dimensions does the latent space of TikTok videos really have? Train an autoencoder and take only 1000 dimensions for the frames. Of course it's not that simple but my mental models atm make me feel like this should work. What do I mean by work? A grainy low fps video+subtitles transformer that looks terrible but as a plus actually has some decent physical consistency. I guess that is Phenaki etc., but I'd hope for much better understanding if the "grammar" of the physical world. That should be rather orthogonal to FPS or resolution (or number of dimensions you take).
- RivieraKid 3y agoPossible future development I'm excited about is something like GPT but for real world interaction - i.e. robots that take input from sensors and are able to physically navigate and manipulate the world. Fine-tuning would then be used for specific environments (human hair, an apartment) and robots (robotic barber, cleaning robot).
- ultra_nick 3y agoGoogle put an LLM into a robot a few months ago. This is the second paper I've seen on it. https://arxiv.org/abs/2306.08647 https://arxiv.org/abs/2306.08647
- antirez 3y agoAlways refreshing reading Andrej Karpathy: the more he knows stuff, the more he explores the fundamentals of the science of ML in a direct and simple way. The field is full of papers that for a very hard to reproduce gain in some new convoluted architecture (in the hope to beat some state of art result) will happily fill 50 useless pages trying to make their works "serious".
- m-i-l 3y agoSo should we spend the next 33 years doing the same things, just with more data and more compute power? That would be the logical conclusion of the breathless "I can't believe it is finally happening in my lifetime" and "we just need bigger models and more data" enthusiasm for LLMs when they first appeared. But can we really simply brute force our way to AGI? Remember, 33 years ago "connectionist AI" wasn't the dominant AI paradigm, and "symbolic AI" wasn't the only other approach either - there were others, like "robotic functionalism" (the idea that you couldn't have true intelligence with interacting with the physical world). Maybe in 33 years some of these other approaches will have a resurgence, perhaps in combination with connectionist approaches. Or maybe they'll even be some entirely new approach.
- phkahler 3y ago>> The original network trained for 3 days on a SUN-4/260 workstation. This is exactly why I didn't start experimenting with this stuff back then. I read some articles and had the interest, but having no access to existing training data or "fast" computers was really a show stopper. This article really convinced me that the amazing results today are mostly due to hardware advances. I will add my own view that 1) hardware will not be advancing anywhere near so much in the future. And 2) training and inference have to be done together like real brains do. Then the AI will learn from experience while deployed and you can clone the best ones later.
- 0xDEF 3y ago>This article really convinced me that the amazing results today are mostly due to hardware advances. For LLMs that is true. But many other things like Whisper, Stable Diffusion etc. could in theory have been made a decade earlier.
- breckinloggins 3y agoI really enjoyed this article. My only critique is that the 2055 predictions are "meta-linear". In other words: the author avoids the (probable) mistake of taking our current tech and linearly regressing the numbers 33 years forward, but the predictions still suggest a kind of "worldline symmetry" with the present date at the origin. It's quite possible that none of these predictions will come true simply because the timeframe is large enough for many unanticipated breakthroughs and roadblocks. Maybe someone will figure out a much, much simpler foundational architecture than "perceptrons++", maybe we'll all be training clouds of 3D gaussians, maybe quantum computers will finally take off and we don't even have the nouns for the building blocks we'll use. On the negative side perhaps we hit a hard scaling limit (in hardware or training) that we didn't see coming. Or a civilizational setback. All that said, though, if I were a betting man I wouldn't exactly wager against the article's conclusions; they're probably the best we can extrapolate knowing only the past and present state of affairs.
- Nevermark 3y agoI think you are right, the next 33 years are likely to be very different. I would lean to them being even more dramatic, due to the opportunity to advance algorithms, not just resources. On the more obvious side, most libraries are not yet taking full advantage of many known gradient optimization techniques. It’s been so much easier to just add data & processing that there is an overhangs of tools to still apply. And large successful models are telling us important things. For instance, it is clear that language models are learning a kind of logic of language similar to how we process thoughts, allowing highly disparate types information to be woven together sensibly. At some point, identifying the nature of that processing could radically simplify language processing. That is just one opportunity for radical architecture and algorithm advances, and it would be revolutionary.
- mark_l_watson 3y agoGreat article. I lived through the early days of artificial neural networks. I was on a DARPA advisory panel for neural network tooling in the mid 1980s, wrote the first version of the SAIC ANSim commercial product, and created the simple back-prop model that was deployed in the bomb detector my company built under contract to the FAA. I also managed a ‘conventional’ deep learning team at Capital One 5-6 years ago. My world has been very exciting in the last 18 months. I spend as much time as I can exploring self hosted LLMs, APIs from Hugging Face, OpenAI, etc. My mind is blown even thinking about tech 33 years from now!
- vendiddy 3y agoAI layperson here. Is there something like an MRI scan for a neural network? Imagining I could take a foundation model, run it on my specialized task, and measure which regions of the neural network light up. Then I could carve out unused regions of the network to create a more lightweight model. Or is this a silly idea?