2 ms·
I learned about neural nets in roughly the late 80s and early 90s and thought they would transform science. I struggled to understand backprogation and gradien
by dekhn 6y ago
I learned about neural nets in roughly the late 80s and early 90s and thought they would transform science. I struggled to understand backprogation and gradient descent. My undergrad work was training the weights of a linear model using gradient descent to predict gene encoding regions. By the time I got to grad school in '95, a very small number of people were using MLPs to predict protein secondary structure and had reached about 75% accuracy and got stuck. I was told, at the time, there wasn't enough data or CPU cycles to train good networks, so why bother?
A few years later SpamAssassin came out and I tried to use it to train a spam classifier without any luck. And I tried my hand at a few protein structure SVM classifiers (failng miserably, I didn't really understand it was critical to have a balanced false and true training set).
A few years after that I landed at Google (around 2007) and very few people were doing machine learning, other than ads and search and the work that was being done was far from what I knew about (mainly supervised training with SGD on batched data). Eventually Google adopted the paradigm I enjoy (synchronized SGD using allreduce).
Nowadays, we have ample CPU and data to train amazing models (with the most interesting working being around large language models). It took a lot longer to get there than I expected.