12 ms·
The deep learning boom caught almost everyone by surprise
- arcmechanica 2y agoIt was basically useful to average people and wasn't just some way to steal and resell data or dump ad after ad on us. A lot of dark patterns really ruin services.
- teknover 2y ago“Nvidia invented the GPU in 1999” wrong on many fronts. Arguably the November 1996 launch of 3dfx kickstarted GPU interest and OpenGL. After reading that, it’s hard to take author seriously on the rest of the claims.
- ahofmann 2y agoWow, that is harsh. The quoted claim is in the middle of a very long article. The background of the author seems to be more on the scientific side, than the technical side. So throw out everything, because the author got one (not very important) date wrong?
- RicoElectrico 2y agoRevisionist marketing should not be given a free pass.
- twelve40 2y agoyet it's almost the norm these days. Sick of hearing Steve Jobs invented smartphones when I personally was using a device with web and streaming music years before that.
- kragen 2y agoYou don't remember when Bill Gates and AOL invented the internet, Apple invented the GUI, and Tim Berners-Lee invented hypertext?
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- santoshalper 2y agoPossibly technically correct, but utterly irrelevant. The 3dfx chips accelerated parts of the 3d graphics pipeline and were not general-purpose programmable computers the way a modern GPU is (and thus would be useless for deep learning or any other kind of AI). If you are going to count 3dfx as a proper GPU and not just a geometry and lighting accelerator, then you might as well go back further and count things like the SGI Reality Engine. Either way, 3dfx wasn't really first to anything meaningful.
- FeepingCreature 2y agoBut the first NVidia GPUs didn't have general-purpose compute either. Google informs me that the first GPU with user-programmable shaders was the GeForce 3 in 2001.
- rramadass 2y agoAfter actually having read the article i can say that your comment is unnecessarily negative and clueless. The article is a very good historical one showing how 3 important things came together to make the current progress possible viz; 1) Geoffrey Hinton's back-propagation algorithm for deep neural networks 2) Nvidia's GPU hardware used via CUDA for AI/ML and 3) Fei-Fei Li's huge ImageNet database to train the algorithm on the hardware. This team actually used "Amazon Mechanical Turk"(AMT) to label the massive dataset of 14 million images. Excerpts; “Pre-ImageNet, people did not believe in data,” Li said in a September interview at the Computer History Museum. “Everyone was working on completely different paradigms in AI with a tiny bit of data.” “That moment was pretty symbolic to the world of AI because three fundamental elements of modern AI converged for the first time,” Li said in a September interview at the Computer History Museum. “The first element was neural networks. The second element was big data, using ImageNet. And the third element was GPU computing.”
- Someone 2y agoI wound not call it invent”, but it seems Nvidia defined the term GPU. See https://www.britannica.com/technology/graphics-processing-unit https://www.britannica.com/technology/graphics-processing-un... and https://en.wikipedia.org/wiki/GeForce_256#Architecture https://en.wikipedia.org/wiki/GeForce_256#Architecture: “GeForce 256 was marketed as "the world's first 'GPU', or Graphics Processing Unit", a term Nvidia defined at the time as "a single-chip processor with integrated transform, lighting, triangle setup/clipping, and rendering engines that is capable of processing a minimum of 10 million polygons per second"” They may have been the first with a product that fitted that definition to market.
- kragen 2y agoThat sounds like marketing wank, not a description of an invention. I don't think you can get a speedup by running neural networks on the GeForce 256, and the features listed there aren't really relevant (or arguably even present) in today's GPUs. As I recall, people were trying to figure out how to use GPUs to get faster processing in their Beowulfs in the late 90s and early 21st century, but it wasn't until about 02005 that anyone could actually get a speedup. The PlayStation 3's "Cell" was a little more flexible.
- KevinMS 2y agoCan confirm. I was playing Unreal on my dual Voodoo2 SLI rig back in 1998.
- kragen 2y agoArguably the November 01981 launch of Silicon Graphics kickstarted GPU interest and OpenGL. You can read Jim Clark's 01982 paper about the Geometry Engine in https://web.archive.org/web/20170513193926/http://excelsior.biosci.ohio-state.edu/~carlson/history/PDFs/geometry-engine.pdf https://web.archive.org/web/20170513193926/http://excelsior..... His first key point in the paper was that the chip had a "general instruction set", although what he meant by it was quite different from today's GPUs. IRIS GL started morphing into OpenGL in 01992, and certainly when I went to SIGGRAPH 93 it was full of hardware-accelerated 3-D drawn with OpenGL on Silicon Graphics Hardware. But graphics coprocessors date back to the 60s; Evans & Sutherland was founded in 01968. I mean, I certainly don't think NVIDIA invented the GPU—that's a clear error in an otherwise pretty decent article—but it was a pretty gradual process.
- binarybits 2y agoDefining who "really" invented something is often tricky. For example I mentioned in the article that there is some dispute about who discovered backpropagation. A According to Wikipedia, Nvidia released its first product, the RV1, in November 1995, the same month 3dfx released its first Voodoo Graphics 3D chip. Is there reason to think the 3dfx card was more of a "true" GPU than the RV1? If not, I'd say Nvidia has as good a claim to inventing the GPU as 3dfx does.
- in3d 2y agoNV1, not RV1. 3dfx Voodoo cards were initially more successful, but I don’t think anything not actually used for deep learning should count.
- vdvsvwvwvwvwv 2y agoLesson: ignore detractors. Especially if their argument is "dont be a tall poppy"
- jakeNaround 2y agoThe lesson is reality is not the dialectics and symbolic logic but all the stuff in it. Study story problems and you end up with string theory. Study data computed from endless world of stuff, find utility. What a shock building the bridge is more useful than a drawer full of bridge designs.
- aleph_minus_one 2y ago> What a shock building the bridge is more useful than a drawer full of bridge designs. Here, opinions will differ.
- psd1 2y agoAlso: look for fields that have stagnated, where progress is enabled by apparently-unrelated innovations elsewhere
- xanderlewis 2y agoUnfortunately, they’re usually right. We just don’t hear about all the time wasted.
- blitzar 2y agoOn several occasions I have heard "they said it couldn't be done" - only to discover that yes it is technically correct, however, "they" was on one random person who had no clue and anyone with any domain knowledge said it was reasonable.
- friendzis 2y agoUsually when I hear "they said it couldn't be done", it is used as triumphant downplay of legitimate critique. If you dig deeper that "couldn't be done" usually is in relation to some constraints or performance characteristics, which the "done" thing still does not meet, but the goalposts have already been moved.
- madaxe_again 2y agoI can’t be the only one who has watched this all unfold with a sense of inevitability, surely. When the first serious CUDA based ML demos started appearing a decade or so ago, it was, at least to me, pretty clear that this would lead to AGI in 10-15 years - and here we are. It was the same sort of feeling as when I first saw the WWW aged 11, and knew that this was going to eat the world - and here we are. The thing that flummoxes me is that now that we are so obviously on this self-reinforcing cycle, how many are still insistent that AI will amount to nothing. I am reminded of how the internet was just a fad - although this is going to have an even greater impact on how we live, and our economies.
- BriggyDwiggs42 2y agoDownvoters are responding to a perceived arrogance. What does agi mean to you?
- nineteen999 2y agoCould be arrogance, or could be the delusion.
- madaxe_again 2y agoWhy is it a delusion, in your opinion?
- andai 2y agoIt's a delusion on the part of the downvoters.
- BriggyDwiggs42 2y agoIndeed, it sure could be arrogance.
- xen0 2y agoWhat makes you think AGI is either here or imminent? For me the current systems still clearly fall short of that goal.
- 2sk21 2y agoI'm surprised that the article doesn't mention that one of the key factors that enabled deep learning was the use of RELU as the activation function in the early 2010s. RELU behaves a lot better than the logistic sigmoid that we used until then.
- sanxiyn 2y agoGeoffrey Hinton (now a Nobel Prize winner!) himself did a summary. I think it is the single best summary on this topic. Our labeled datasets were thousands of times too small. Our computers were millions of times too slow. We initialized the weights in a stupid way. We used the wrong type of non-linearity.
- imjonse 2y agoThat is a pithier formulation of the widely accepted summary of "more data + more compute + algo improvements"
- sanxiyn 2y agoNo, it isn't. It emphasizes importance of Glorot initialization and ReLU.
- helltone 2y agoI'm curious and it's not obvious to me: what changed in terms of weight initialisation?
- HarHarVeryFunny 2y agoAlso: nets too small (not enough layers) gradients not flowing (residual connections) layer outputs not normalized training algorithms and procedures not optimal (Adam, warm-up, etc)
- cma 2y agoAs compute has outpaced memory bandwidth most recent stuff has moved away from ReLU. I think Llama 3.x uses SwiGLU. Still probably closer to ReLU than logistic sigmoid, but it's back to being something more smooth than ReLU.
- DeathArrow 2y agoI think neural nets are just a subset of machine learning techniques. I wonder what would have happened if we poured the same amount of money, talent and hardware into SVMs, random forests, KNN, etc. I don't say that transformers, LLMs, deep learning and other great things that happened in the neural network space aren't very valuable, because they are. But I think in the future we should also study other options which might be better suited than neural networks for some classes of problems. Can a very large and expensive LLM do sentiment analysis or classification? Yes, it can. But so can simple SVMs and KNN and sometimes even better. I saw some YouTube coders doing calls to OpenAI's o1 model for some very simple classification tasks. That isn't the best tool for the job.
- ldjkfkdsjnv 2y agoThis is such a terrible opinion, im so tired of reading the LLM deniers
- Meloniko 2y agoAnd based on what though do you think that? I think neural networks are fundamental and we will focus/experiment a lot more with architecture, layers and other parts involved but emerging features arise through size
- mentalgear 2y agoKANs (Kolmogorov-Arnold Networks) are one example of a promising exploration pathway to real AGI, with the advantage of full explain-ability.
- astrange 2y ago"Explainable" is a strong word. As a simple example, if you ask a question and part of the answer is directly quoted from a book from memory, that text is not computed/reasoned by the AI and so doesn't have an "explanation". But I also suspect that any AGI would necessarily produce answers it can't explain. That's called intuition.
- 2y ago
- macrolime 2y agoI took some AI courses around the same time as the author, and I remember the professors were actually big proponents of neural nets, but they believed the missing piece was some new genius learning algorithm rather than just scaling up with more data.
- rramadass 2y ago> rather than just scaling up with more data. That was the key takeaway for me from this article. I didn't know of Fei-Fei Li's ImageNet contribution which actually gave all the other researchers the essential data to train with. Her intuition that more data would probably make the accuracy of existing algorithms better i think is very much under appreciated. Key excerpt; So when she got to Princeton, Li decided to go much bigger. She became obsessed with an estimate by vision scientist Irving Biederman that the average person recognizes roughly 30,000 different kinds of objects. Li started to wonder if it would be possible to build a truly comprehensive image dataset—one that included every kind of object people commonly encounter in the physical world.
- aithrowawaycomm 2y agoI think there is a slight disconnect here between making AI systems which are smart and AI systems which are useful. It’s a very old fallacy in AI: pretending tools which assist human intelligence by solving human problems must themselves be intelligent. The utility of big datasets was indeed surprising, but that skepticism came about from recognizing the scaling paradigm must be a dead end: vertebrates across the board require less data to learn new things, by several orders of magnitude. Methods to give ANNs “common sense” are essentially identical to the old LISP expert systems: hard-wiring the answers to specific common-sense questions in either code or training data, even though fish and lizards can rapidly make common-sense deductions about manmade objects they couldn’t have possibly seen in their evolutionary histories. Even spiders have generalization abilities seemingly absent in transformers: they spin webs inside human homes with unnatural geometry. Again it is surprising that the ImageNet stuff worked as well as it did. Deep learning is undoubtedly a useful way to build applications, just like Lisp was. But I think we are about as close to AGI as we were in the 80s, since we have made zero progress on common sense: in the 80s we knew Big Data can poorly emulate common sense, and that’s where we’re at today.
- j_bum 2y ago> vertebrates across the board require less data to learn new things, by several orders of magnitude. Sometimes I wonder if it’s fair to say this. Organisms have had billions of years of training. We might come online and succeed in our environments with very little data, but we can’t ignore the information that’s been trained into our DNA, so to speak. What’s billions of years of sensory information that drove behavior and selection, if not training data?
- aithrowawaycomm 2y agoMy primary concern is the generalization to manmade things that couldn’t possibly be in the evolutionary “training data.” As a thought experiment, it seems very plausible that you can train a transformer ANN on spiderwebs between trees, rocks, bushes, etc, and get “superspider” performance (say in a computer simulation). But I strongly doubt this will generalize to building webs between garages and pantries like actual spiders, no matter how many trees you throw at it, so such a system wouldn’t be ASI. This extends to all sorts of animal cognitive experiments: crows understand simple pulleys simply by inspecting them, but they couldn’t have evolved to use pulleys. Mice can quickly learn that hitting a button 5 times will give them a treat: does it make sense to say that they encountered a similar situation in their evolutionary past? It makes more sense to suppose that mice and crows have powerful abilities to reason causally about their actions. These abilities are more sophisticated than mere “Pavlovian” associative reasoning, which is about understanding stimuli. With AI we can emulate associative reasoning very well because we have a good mathematical framework for Pavlovian responses as a sort of learning of correlations. But causal reasoning is much more mysterious, and we are very far from figuring out a good mathematical formalism that a computer can make sense of. I also just detest the evolution = training data metaphor because it completely ignores architecture. Evolution is not just glomming on data, it’s trying different types of neurons, different connections between them, etc. All organisms alive today evolved with “billions of years of training,” but only architecture explains why we are so much smarter than chimps. In fact I think the “evolution” preys on our misconception that humans are “more evolved” than chimps, but our common ancestor was more primitive than a chimp.
- kleiba 2y ago> “Pre-ImageNet, people did not believe in data,” Li said in a September interview at the Computer History Museum. “Everyone was working on completely different paradigms in AI with a tiny bit of data.” That's baloney. The old ML adage "there's no data like more data" is as old as mankind itself.
- FrustratedMonky 2y agoNot really. This is referring back to the 80's. People weren't even doing 'ML'. And back then people were more focused on teasing out 'laws' in as few data points as possible. The focus was more on formulas and symbols, and finding relationships between individual data points. Not the broad patterns we take for granted today.
- criddell 2y agoI would say using backpropagation to train multi-layer neural networks would qualify as ML and we were definitely doing that in 80's.
- icf80 2y agologic is data and data is logic
- hollerith 2y agoThe deep learning boom caught deep-learning researchers by surprise because deep-learning researchers don't understand their craft well enough to predict essential properties of their creations. A model is grown, not crafted like a computer program, which makes it hard to predict. (More precisely, a big growth phase follows the crafting phase.)
- lynndotpy 2y agoI was a deep learning researcher. The problem is that accuracy (+ related metrics) were prioritized in research and funding. Factors like interpretability, extrapolation, efficiency, or consistency were not prioritized, but were clearly important before being implemented. Dall-E was the only big surprising consumer model-- 2022 saw a sudden huge leap from "txt2img is kind of funny" to "txt2img is actually interesting". I would have assumed such a thing could only come in 2030 or earlier. But deep learning is full of counterintuitive results (like the NFL theorem not mattering, or ReLU being better than sigmoid). But in hindsight, it was naive to think "this does not work yet" would get in the way of the products being sold and monetized.
- nxobject 2y agoI'm still very taken aback by how far we've been able to take prompting as somehow our universal language to communicate with AI of choice.
- TheRealPomax 2y agoIt wasn't "almost everyone", it was straight up everyone.
- vl 2y agoSo the AI boom of the last 12 years was made possible by three visionaries who pursued unorthodox ideas in the face of widespread criticism. I argue that Mikolov with word2vec was instrumental in current AI revolution. This demonstrated the easy of extracting meaning in mathematical way from text and directly lead to all advancements we have today with LLMs. And ironically, didn’t require GPU.
- MichaelZuo 2y agoHow much easier was it compared to the next best method at the time?
- barrenko 2y agoYup, and it all started with a Master's thesis.
- gregw2 2y agoThe article credits two academics (Hinton, Fei Fei Li) and a CEO (Jensen Huang). But really it was three academics. Jensen Huang, reasonably, was desperate for any market that could suck up more compute, which he could pivot to from GPUs for gaming when gaming saturated its ability to use compute. Screen resolutions and visible polygons and texture maps only demand so much compute; it's an S-curve like everything else. So from a marketing/market-development and capital investment perspective I do think he deserves credit. Certainly the Intel guys struggled to similarly recognize it (and to execute even on plain GPUs.) But... the technical/academic insight of the CUDA/GPU vision in my view came from Ian Buck's "Brook" PhD thesis at Stanford under Pat Hanrahan (Pixar+Tableau co-founder, Turing Award Winner) and Ian promptly took it to Nvidia where it was commercialized under Jensen. For a good telling of this under-told story, see one of Hanrahan's lectures at MIT: https://www.youtube.com/watch?v=Dk4fvqaOqv4 https://www.youtube.com/watch?v=Dk4fvqaOqv4 Corrections welcome.
- markhahn 2y agoJensen embraced AI as a way to recover TAM after ASICs took over crypto mining. You can see that between-period in NVidia revenue and profit graphs. By that time, GP-GPU had been around for a long, long time. CUDA still doesn't have much to do with AI - sure, it supports AI usage, even includes some AI-specific features (low-mixed precision blocked operations).
- cameldrv 2y agoJensen embraced AI way before that. CuDNN was released back in 2014. I remember being at ICLR in 2015, and there were three companies with booths: Google and Facebook who were recruiting, and NVIDIA was selling a 4 GPU desktop computer.
- dartos 2y agoWell as soon as matmul has a marketable use (ML predictive algorithms) nvidia was on top of it. I don’t think they were thinking of LLMs in 2014, tbf.
- AvAn12 2y agoThree legs to the stool - the NN algorithms, the data, and the labels. I think the first two are obvious but think about how much human time and care went into labeling millions of images...
- fragmede 2y agoAnd the compute power!
- hyperific 2y agoThe article mentions Support Vector Machines being the hot topic in 2008. Is anyone still using/researching these? I often wonder how many useful technologies could exist if trends went a different way. Where would we be if neural nets hadn't caught on and SVMs and expert systems had.
- spencerchubb 2y agoin insurance we use older statistical methods that are easily interpretable, because we are required to have rates approved by departments of insurance
- bob1029 2y agoI've been looking at SVMs for use with a binary classification experiment. Training and operating these models is quite cheap. The tooling is ubiquitous and will run on a toaster. A binary decision made well can be extremely powerful. Multiple binary decisions underly... gestures broadly. Obvious contextual/attention caveats aside, a few thousand binary classifiers operating bitwise over the training & inference sets would get you enough bandwidth for a half-ass language model. 2^N can be a very big place very quickly.
- Legend2440 2y agoExpert systems did catch on and do see widespread use - they're just not called AI anymore. It's 'business logic' or 'rules engine' now. The issue with SVMs is that they get intractably expensive for large datasets. The cost of training a neural network scales linearly with dataset size, while the kernel trick for SVMs scales with dataset size squared. You could never create an LLM-sized SVM.
- est 2y agoI think the gpt-3 vs BERT story should be more interesting. I was told that OpenAI practically tried to "label every human knowledge" prior to ChatGPT, sort of similar to what Li Feifei was doing. At that time, no one seems to understand how to exact the massive capabilities out of gpt-3 (remember that gpt3 troll bot on 4chan?), until some joking threads on Twitter proposed "Lets think step by step..." That seems to be the wakeup prompt to a new AI era.
- PeterStuer 2y agoYou have to understand how academia functions. It is not about what works, it is all about what is new or different in theory, regardless of what works in practice. So especially when an already explored (and thus claimed) theory relies on scale and emergence, it becomes near impossible for academia to reach, not because of the scale of investment needed you only have to look at physics to see funding is not a problem, but because the underlying theory is already done. NN theory is not very different from what we had in the 80s. The scale is many many orders of magitude beyond when I was collecting floppies from all the lab's Macs in the morning to get the results of my overnight run.
- datavirtue 2y agoIt was more like: "Oh crap, here it is." It's not like the community was surprised. Many of us were educated on the principles and understood the possibilities. I haven't seen anything this big developed so fast in my life. I have used more Copilot UIs than I can remember. Drastic sweeping changes to how it works and integrates. Just amazing. The pretrained LLM community is hot and a lot of people are tinkering again. It reminds me of the PC explosion of the 1980s-1990s.