7 ms·

Applications of Deep Neural Networks v2 [pdf]

157 points by tanelpoder 6y ago

jonbaer 6y ago
Same guy? https://www.youtube.com/user/HeatonResearch https://www.youtube.com/user/HeatonResearch
- tanelpoder 6y ago
  Yes, looks like the same person. The Code tab on arxiv page points to the same github account as on the youtube page.
melling 6y ago
What book are people recommending for Deep Neural Networks? I’m working through ISLR (starting ch8) so I’ll be done in a few weeks. This topic isn’t covered so another textbook, with exercises, would be ideal. Just noticed that this paper is a book. Maybe I have a winner?
- rsfern 6y ago
  This paper seems really interesting, and it’s geared towards applications I guess For more fundamental material I like the CS231n course notes [0] and Goodfellow, Bengio, and Courville [1] 0: https://cs231n.github.io/ https://cs231n.github.io/ 1: https://www.deeplearningbook.org/ https://www.deeplearningbook.org/
- manojlds 6y ago
  Have been hearing good things about this - http://d2l.ai/ http://d2l.ai/ Have to get to it soon.
- max_ 6y ago
  The Deep Learning book is great & I have a copy, but its not honestly something that I read cover to cover. To me, Its more of a reference book. But if you want a "Fyneman" type book that describes the underlying structure & workings in a non academic way, I would recommend; Michael Nielsen's Neural Networks & Deep Learning[0] Jeff Heaton's Introduction To The Math Of Neural Networks[1] [0]:http://neuralnetworksanddeeplearning.com/index.html http://neuralnetworksanddeeplearning.com/index.html [1]:https://www.amazon.com/Introduction-Math-Neural-Networks-Heaton-ebook/dp/B00845UQL6 https://www.amazon.com/Introduction-Math-Neural-Networks-Hea...
- visarga 6y ago
  The Hundred-Page Machine Learning Book by Andriy Burkov - is quite highly regarded and to the point. http://themlbook.com/ http://themlbook.com/
sillysaurusx 6y ago
After 1.5 years of self study in neural networks, my advice would be to internalize the fact that you can train a neural network to do anything that you can encode as a loss function. Networks try to minimize loss. If you want something to happen less frequently, add it to the loss. Literally addition. It was a mind-bending “there is no spoon” moment for me. Also, loss is one of the worst names imaginable. Kerfluffle would’ve been better, because at least it’s mostly meaningless. Whenever you see “loss”, substitute with “penalty” and things will become much clearer. Secondary advice: if you don’t have patience, you won’t get anywhere. In the same way that the stock market is a lever for transferring money from the impatient to the patient, neural networks are a lever to transfer advantage to the patient. By that I mean, I can’t count the number of times I almost wrote off some small tweak as “doesn’t work”, only to leave the network training for another week or so and discovering it worked fine. In fact, it was almost always equivalent, or had no advantage, I.e. a placebo. It’s not like code; you can do so much fucked-up shit to a neural network, and it will still work. It’s unlike anything you’re used to. Beyond that, just remember that this stuff is hard. Coding the network is easy. Getting it right is hard. And getting it perfect, well, took me a year. Google’s official biggan model at google/compare_gan never achieved the same FID as real biggan. Why? I immersed myself in this mystery, eventually reverse engineering the official tensorflow graph. I discovered their implementation was missing a crucial + 1, so their gamma was centered around zero instead of one. And in batchnorm, gamma is a multiplier — so the network was basically multiplied by zero and no one noticed for years. (Remember how I said you can do a lot to a network without causing problems? Sometimes the problems are so subtle they’ll drive you nuts. You know something is wrong, but you don’t know what or why, and it’s almost impossible to debug.)
- sjg007 6y ago
  I’d be interested in reading your formal write up. It’d be interesting to see the issue as you understand it.
  - sillysaurusx 6y ago
    Sure thing! https://github.com/google/compare_gan/issues/54 https://github.com/google/compare_gan/issues/54 It’s not much of a writeup. It’s basically saying, hey, this is zero when it should be one. The results were dramatic. It went from blobs to replicating the biggan paper almost perfectly. I think we’re at a FID of 11 or so on imagenet. Here's a screenshot I just pulled from our current run: https://i.imgur.com/k1RuWEG.png https://i.imgur.com/k1RuWEG.png Stole a year of my life to track it down. But it was a puzzle I couldn’t put down. It haunted my dreams. I was tossing and turning like, but why won’t it work... why won’t it work...
shimonabi 6y ago
I came across Jeff Heaton's books a long time ago, when I was looking for material on how to implement my own NN for my AI class. If I remember correctly, he published C# and Java implementations. I eventually settled for Python and implemented the NN with the help of the book Make Your Own Network by Tariq Rashid. Numpy is really magic.
NalNezumi 6y ago
I remember taking my first Deep Learning course in University 5 years ago; We had to implement the neural network, gradient computation, Batch Normalization, Drop out and all other details from scratch without external libraries in either Java/C++/Matlab. Apparently this was the old way DL used to be taught, and even in 2016 the professor insisted everyone had to do it this way. I didn't study ML in general so what I gained from the course was a deep understanding of the fundamentals & math behind it, but since I didn't get to familiarize myself with any of the existing libraries (Tensorflow/Keras back then) I had a hard time convincing anyone in industry of my skills in the field :/ Also: Why does the book only cover Deep Q-Network on Reinforcement Learning? Sure it is the most notable deep learning step in the field but, there are some relevant versions such as Actor-Critic & Maximum Entropy RL that can be very relevant too. If one includes YOLO, ResNet and newer architectures for Computer Vision application, I don't know why same things are not on RL.
gillesjacobs 6y ago
I don't like that OP is using Arxiv to upload his course material. Arxiv is supposed to be a pre-print scientific publication server. A place to post your nearly finished or ideally submitted journal or conference manuscript so you can reference it while it is being reviewed. The purpose of Arxiv is in itself a patch for a too common, too long in duration publication process. Now it is often the first place to publish ML research and an obligatory source of literature in ML. No peer-review or any quality assurance makes for dubious work appearing there that can waste a lot of research time. Hosting your elementary course there because that's where the researchers are, muddies the quality of work on Arxiv further.
- deleted 6y ago
  [deleted]
- deleted 6y ago
  [deleted]
- hoseja 6y ago
  Your criticism depends on peer-review actually being functional in the first place.
  - gillesjacobs 6y ago
    I fail to see how it does. The popularity of Arxiv is indeed a sign of a slow-functioning peer-review system, but it has its valid uses. My criticism pertains to OP using Arxiv as a PDF host for course material irrelevant to Arxiv's userbase, i.e. expert researchers. It is already hard enough to find the quality manuscripts in Arxiv. Hence why Karpathy made Arxiv Sanity Preserver [1]. I would rather not have to drudge through pages of tutorial pdf's when searching "neural graph methods for NLP" for instance. 1. https://www.arxiv-sanity.com https://www.arxiv-sanity.com
- cambalache 6y ago
  You are around 25 years too late. Since the very beginning Arxiv has been used to publish not only cutting-edge research but also divulgation pieces, reviews, commentaries,workshop's accompanying materials,lecture notes and books.
dragandj 6y ago
If I may drop in with a bit of shameless self-promotion. My "Deep Learning for Programmers: A Tutorial with CUDA, OpenCL, DNNL, Java, and Clojure" book explains and executes every single line of code interactively, from low level operations to high-level networks that do everything automatically. The code is built on the state of the art performance operations of oneDNN (Intel, CPU) and cuDNN (CUDA, GPU). Very concise readable and understandable by humans. https://aiprobook.com/deep-learning-for-programmers/ https://aiprobook.com/deep-learning-for-programmers/ Here's the open source library built throughout the book: https://github.com/uncomplicate/deep-diamond https://github.com/uncomplicate/deep-diamond Some chapters from the beginning of the book are available on my blog, as a tutorial series: https://dragan.rocks https://dragan.rocks
gnrlst 6y ago
Is there an equivalent book/pdf for PyTorch? In general, any recommended courses/books? I've already done a first pass of fastai's 2020 course which was very eye-opening and a good segue from the introductory courses on Kaggle.
aceon48 6y ago
I've been really enjoying learning Deep Learning. What are job titles where you would get to work with this? and how much knowledge is expected already?

157 points by tanelpoder 6y ago

jonbaer 6y ago

Same guy? https://www.youtube.com/user/HeatonResearch https://www.youtube.com/user/HeatonResearch

tanelpoder 6y ago
Yes, looks like the same person. The Code tab on arxiv page points to the same github account as on the youtube page.

melling 6y ago

What book are people recommending for Deep Neural Networks? I’m working through ISLR (starting ch8) so I’ll be done in a few weeks. This topic isn’t covered so another textbook, with exercises, would be ideal. Just noticed that this paper is a book. Maybe I have a winner?

rsfern 6y ago
This paper seems really interesting, and it’s geared towards applications I guess For more fundamental material I like the CS231n course notes [0] and Goodfellow, Bengio, and Courville [1] 0: https://cs231n.github.io/ https://cs231n.github.io/ 1: https://www.deeplearningbook.org/ https://www.deeplearningbook.org/
manojlds 6y ago
Have been hearing good things about this - http://d2l.ai/ http://d2l.ai/ Have to get to it soon.
max_ 6y ago
The Deep Learning book is great & I have a copy, but its not honestly something that I read cover to cover. To me, Its more of a reference book. But if you want a "Fyneman" type book that describes the underlying structure & workings in a non academic way, I would recommend; Michael Nielsen's Neural Networks & Deep Learning[0] Jeff Heaton's Introduction To The Math Of Neural Networks[1] [0]:http://neuralnetworksanddeeplearning.com/index.html http://neuralnetworksanddeeplearning.com/index.html [1]:https://www.amazon.com/Introduction-Math-Neural-Networks-Heaton-ebook/dp/B00845UQL6 https://www.amazon.com/Introduction-Math-Neural-Networks-Hea...
visarga 6y ago
The Hundred-Page Machine Learning Book by Andriy Burkov - is quite highly regarded and to the point. http://themlbook.com/ http://themlbook.com/

sillysaurusx 6y ago

After 1.5 years of self study in neural networks, my advice would be to internalize the fact that you can train a neural network to do anything that you can encode as a loss function. Networks try to minimize loss. If you want something to happen less frequently, add it to the loss. Literally addition. It was a mind-bending “there is no spoon” moment for me. Also, loss is one of the worst names imaginable. Kerfluffle would’ve been better, because at least it’s mostly meaningless. Whenever you see “loss”, substitute with “penalty” and things will become much clearer. Secondary advice: if you don’t have patience, you won’t get anywhere. In the same way that the stock market is a lever for transferring money from the impatient to the patient, neural networks are a lever to transfer advantage to the patient. By that I mean, I can’t count the number of times I almost wrote off some small tweak as “doesn’t work”, only to leave the network training for another week or so and discovering it worked fine. In fact, it was almost always equivalent, or had no advantage, I.e. a placebo. It’s not like code; you can do so much fucked-up shit to a neural network, and it will still work. It’s unlike anything you’re used to. Beyond that, just remember that this stuff is hard. Coding the network is easy. Getting it right is hard. And getting it perfect, well, took me a year. Google’s official biggan model at google/compare_gan never achieved the same FID as real biggan. Why? I immersed myself in this mystery, eventually reverse engineering the official tensorflow graph. I discovered their implementation was missing a crucial + 1, so their gamma was centered around zero instead of one. And in batchnorm, gamma is a multiplier — so the network was basically multiplied by zero and no one noticed for years. (Remember how I said you can do a lot to a network without causing problems? Sometimes the problems are so subtle they’ll drive you nuts. You know something is wrong, but you don’t know what or why, and it’s almost impossible to debug.)

sjg007 6y ago
I’d be interested in reading your formal write up. It’d be interesting to see the issue as you understand it.
- sillysaurusx 6y ago
  Sure thing! https://github.com/google/compare_gan/issues/54 https://github.com/google/compare_gan/issues/54 It’s not much of a writeup. It’s basically saying, hey, this is zero when it should be one. The results were dramatic. It went from blobs to replicating the biggan paper almost perfectly. I think we’re at a FID of 11 or so on imagenet. Here's a screenshot I just pulled from our current run: https://i.imgur.com/k1RuWEG.png https://i.imgur.com/k1RuWEG.png Stole a year of my life to track it down. But it was a puzzle I couldn’t put down. It haunted my dreams. I was tossing and turning like, but why won’t it work... why won’t it work...

shimonabi 6y ago

I came across Jeff Heaton's books a long time ago, when I was looking for material on how to implement my own NN for my AI class. If I remember correctly, he published C# and Java implementations. I eventually settled for Python and implemented the NN with the help of the book Make Your Own Network by Tariq Rashid. Numpy is really magic.

NalNezumi 6y ago

I remember taking my first Deep Learning course in University 5 years ago; We had to implement the neural network, gradient computation, Batch Normalization, Drop out and all other details from scratch without external libraries in either Java/C++/Matlab. Apparently this was the old way DL used to be taught, and even in 2016 the professor insisted everyone had to do it this way. I didn't study ML in general so what I gained from the course was a deep understanding of the fundamentals & math behind it, but since I didn't get to familiarize myself with any of the existing libraries (Tensorflow/Keras back then) I had a hard time convincing anyone in industry of my skills in the field :/ Also: Why does the book only cover Deep Q-Network on Reinforcement Learning? Sure it is the most notable deep learning step in the field but, there are some relevant versions such as Actor-Critic & Maximum Entropy RL that can be very relevant too. If one includes YOLO, ResNet and newer architectures for Computer Vision application, I don't know why same things are not on RL.

gillesjacobs 6y ago

I don't like that OP is using Arxiv to upload his course material. Arxiv is supposed to be a pre-print scientific publication server. A place to post your nearly finished or ideally submitted journal or conference manuscript so you can reference it while it is being reviewed. The purpose of Arxiv is in itself a patch for a too common, too long in duration publication process. Now it is often the first place to publish ML research and an obligatory source of literature in ML. No peer-review or any quality assurance makes for dubious work appearing there that can waste a lot of research time. Hosting your elementary course there because that's where the researchers are, muddies the quality of work on Arxiv further.

deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
hoseja 6y ago
Your criticism depends on peer-review actually being functional in the first place.
- gillesjacobs 6y ago
  I fail to see how it does. The popularity of Arxiv is indeed a sign of a slow-functioning peer-review system, but it has its valid uses. My criticism pertains to OP using Arxiv as a PDF host for course material irrelevant to Arxiv's userbase, i.e. expert researchers. It is already hard enough to find the quality manuscripts in Arxiv. Hence why Karpathy made Arxiv Sanity Preserver [1]. I would rather not have to drudge through pages of tutorial pdf's when searching "neural graph methods for NLP" for instance. 1. https://www.arxiv-sanity.com https://www.arxiv-sanity.com
cambalache 6y ago
You are around 25 years too late. Since the very beginning Arxiv has been used to publish not only cutting-edge research but also divulgation pieces, reviews, commentaries,workshop's accompanying materials,lecture notes and books.

dragandj 6y ago

If I may drop in with a bit of shameless self-promotion. My "Deep Learning for Programmers: A Tutorial with CUDA, OpenCL, DNNL, Java, and Clojure" book explains and executes every single line of code interactively, from low level operations to high-level networks that do everything automatically. The code is built on the state of the art performance operations of oneDNN (Intel, CPU) and cuDNN (CUDA, GPU). Very concise readable and understandable by humans. https://aiprobook.com/deep-learning-for-programmers/ https://aiprobook.com/deep-learning-for-programmers/ Here's the open source library built throughout the book: https://github.com/uncomplicate/deep-diamond https://github.com/uncomplicate/deep-diamond Some chapters from the beginning of the book are available on my blog, as a tutorial series: https://dragan.rocks https://dragan.rocks

gnrlst 6y ago

Is there an equivalent book/pdf for PyTorch? In general, any recommended courses/books? I've already done a first pass of fastai's 2020 course which was very eye-opening and a good segue from the introductory courses on Kaggle.

aceon48 6y ago

I've been really enjoying learning Deep Learning. What are job titles where you would get to work with this? and how much knowledge is expected already?