13 ms·

> Convnets had been invented but hadn't yet been popularized. > Given the right code, the right recipe, you could have made it work on that 90s supercomputer a

> Convnets had been invented but hadn't yet been popularized. > Given the right code, the right recipe, you could have made it work on that 90s supercomputer and with the compute budget you had, but the right recipe just wasn't known at the time. We could expect the best of the best given access to these powerful computing machines, surely they would know their lenet (1989) http://karpathy.github.io/2022/03/14/lecun1989/ http://karpathy.github.io/2022/03/14/lecun1989/ and could have studied statistics of gradients and activations to implement something like ReLu - which is one line of code.It was a low-hanging fruit. > Someone needed to sit down and figure out that vanishing gradients were actually a problem we needed to think about. If Sepp Hochreiter could do it, surely someone from CalTech could do it as well, given the caliber of people who study there. Again, looks like a simple problem of misapplication of best and brightest (who, surely, know what they want to work on in life, but are still influenced by academic prestige and their advisors' advice which would repel them from connectionism then).

snek_case 4y ago
I took a machine learning class in ~2006 and granted the prof wasn't specializing in studying NNs, but he seemed to have little awareness of the impact of dataset size, convnets were never mentioned, the suggested approach was two-layer MLPs and sigmoid activation functions. The course was based on the book "Artificial Intelligence: A Modern Approach" 2nd edition, which barely just mentions neural networks in passing. I think there just weren't enough people looking at neural networks back then. At the time, they were still considered a niche machine learning technique, just one tool in the toolbox, and not necessarily the best one. Also, it's one thing to say that the best of the best had access to computation, but if your university has just one supercomputer, which is shared among everyone, and you have a limited number of hours during which you can use it, that's not increasing your chances at running very thorough experiments. If anything, it's not that we didn't have the compute to do deep learning in the 90s and early 2000s, it's that more access to compute makes a broader range of experimentation a lot more accessible. That makes it a lot easier to find the right training recipe more quickly. Back in 2009 the enthusiasm around neural nets was starting to grow, as there were some early successes with deep neural nets, but I remember some people were still trying to argue that support vector machines were just as effective for image classification. It seemed laughable to me at the time, but I think that everyone who wasn't a connectionist was starting to feel threatened and wanted to justify their chosen research specialty, which they would keep doing until they couldn't anymore.

snek_case 4y ago

I took a machine learning class in ~2006 and granted the prof wasn't specializing in studying NNs, but he seemed to have little awareness of the impact of dataset size, convnets were never mentioned, the suggested approach was two-layer MLPs and sigmoid activation functions. The course was based on the book "Artificial Intelligence: A Modern Approach" 2nd edition, which barely just mentions neural networks in passing. I think there just weren't enough people looking at neural networks back then. At the time, they were still considered a niche machine learning technique, just one tool in the toolbox, and not necessarily the best one. Also, it's one thing to say that the best of the best had access to computation, but if your university has just one supercomputer, which is shared among everyone, and you have a limited number of hours during which you can use it, that's not increasing your chances at running very thorough experiments. If anything, it's not that we didn't have the compute to do deep learning in the 90s and early 2000s, it's that more access to compute makes a broader range of experimentation a lot more accessible. That makes it a lot easier to find the right training recipe more quickly. Back in 2009 the enthusiasm around neural nets was starting to grow, as there were some early successes with deep neural nets, but I remember some people were still trying to argue that support vector machines were just as effective for image classification. It seemed laughable to me at the time, but I think that everyone who wasn't a connectionist was starting to feel threatened and wanted to justify their chosen research specialty, which they would keep doing until they couldn't anymore.