8 ms·

How Convolutional Neural Networks Work

403 points by eternalban 10y ago

jimkittridge 10y ago
Great explanation. Thanks for sharing.
oh_sigh 10y ago
Really great write up. Ive been trying to wrap my not too mathematically talented head around convolutional filters and this really helped in visualizing what is happening.
MrFeynmannsJoke 10y ago
So ist works just like i thought it would. Why are CNN so hyped? Wasnt all this already known decades ago? Or is it just because we can afford the computing power?
- bbctol 10y ago
  The underlying math was figured out a long time ago, but it's only been in recent years that we've had the computing power to test these out on lots of complicated, real-world classification problems, and had some incredible success.
- gcp 10y ago
  Computing power, but also some implementation tricks that turn out to make things a lot better. For example, activation via ReLU instead of sigmoid/tanh significantly improves the performance of deep neural networks. Then there's stuff like BatchNorm, Pooling, Dropout etc...
- nkozyra 10y ago
  1. Computing power 2. Data availability 3. Fast, large local storage
- zwieback 10y ago
  Decades ago I played around with neural nets but was frustrated because I either had to preprocess and normalize my inputs to the point where I didn't need a network anymore or I had to train a large network with so much data that it was not practical. Having a cookbook approach with a catchy name and orders of magnitude more processing power have revived neural nets and now they are finally doing something useful. Now everyone is jumping on the bandwagon so the field is progressing very quickly. Just because it's hyped doesn't mean it's not worth giving it a second look (although I'm still on the sidelines myself.)
- dougabug 10y ago
  The basic CNN structure was in place, but as the saying goes, "The Devil's in the details." Early CNN's were applied to problems such as handwritten character recognition with rows of small grayscale image cells as inputs, and were much shallower, smaller models. Today's CNN's operate on full resolution, multi-channel images and video, and can be orders of magnitudes deeper and larger. For instance, ResNets have been proven to demonstrate monotonic performance improvements out to 1200 layers on benchmark datasets. This would have been unthinkable even a couple years ago. By way of comparison, even the state of the art VGG network architecture of a couple years ago originally had to be trained in stages to reach 16 and 19 layers for submission to ILSVRC 2014 (Xavier / MSRA initialization makes this unnecessary now). At the time, VGG and GoogleNet (22 layers) were considered to be extraordinarily deep CNN's.
- zackmorris 10y ago
  I argued back in 2000 (a year after I got my computer engineering degree) that AI wouldn't take off until computing moved from single threaded/single core to multithreaded/ multicore processing. The fact that we are only hearing about this stuff 15 years later makes me feel that that assertion was largely right. The biggest problem I see in AI is that the algorithms are generally fairly straightforward, but people haven't had the computing power to explore the problem space. We are seeing drastic improvement in things like video cards (routinely 1000+ cores) and data processing locality (map reduce). But processors have stagnated. If we really want AI in any reasonable timescale, we need large arrays of general-purpose cores with a sane communication protocol that doesn't fixate on things like caching, we need a hybrid between Go and Erlang to do concurrent functional programming in a readable way with automagic scaling over a network, and we need all this yesterday. The fancy schmancy AI algorithms will become apparent when processing power is no longer the primary limitation, and at that point we can optimize them.
- bilbobeer 10y ago
  In 1965 Cooley-Tukey published their famous FFT paper, funny thing is that at Standard Oil they were using FFT in the 1950's, because it could be used for analyzing 'Russian' atomic bomb seismic signatures in real-time it took some 10+ years before it became public ( openly published ), but it was routine in BIG-OIL in 1950's. Now jump to CNN, we were doing CNN on Cray's in the 1970's on Seismic 3d data ( acoustic sound waves pressure/sheer ) in order to find oil deposits, its the same stuff I see in CNN algo's of today, and its the exact same stuff we were doing in the 1970's in visualizing seismic data (P-S wave ratios were the color that told you what kind of rock/substance you propagated ). We used to call this 'around the wheel', I suspect that its good to make the kids think this stuff is all something new, most of the foundation of computational ML was designed in 1950's, hell Von Neuman wrote the book on Cybernetic's, automating the human mind with computers, again 1950's. Another perspective is given the quality ( poor ) that's being leaked, and given corporate history, I suspect that BIG-OIL, and BIG-NSA of today have stuff that is super good and advanced and most of what they leak to GIT-HUB is just garbage. In summary keep it simply, learn how it really works, and then dial in your own algorithm, and you too will discover the holy-grail, but remember that edoocation is always 15-20 years lagging industry. It's always been this way, and always will. NSA/CIA has always mandated a competitive edge over the foreign gov's, never assume that anything you get for free, or academia is cutting-edge. ( Lastly George Green developed the Math here, back in 1816-ish, thus the Convolutional in CNN is 200 years old to be exact. )
elcct 10y ago
This is brilliant, so far one of the easiest to understand explanations.
spiderfarmer 10y ago
Question: I have a database with 1.000.000 vehicle pictures, organized by make and model. What would be the easiest way to play with this data, so that I can train it to predict the make / model? I don't want to reinvent the wheel now so much tutorials are written and software is being released. What would be the easiest way to start?
- stared 10y ago
  With that number you can train a model from scratch. Alternatively, you can retrain the last layers (it requires much less data). The easiest way is to use some well-known architecture (e.g. VGG16) and go. See: https://github.com/leriomaggio/deep-learning-keras-euroscipy2016 https://github.com/leriomaggio/deep-learning-keras-euroscipy...
  - misiti3780 10y ago
    do you have any examples of scripts/examples where only specific layers of a network are being trained?
- polite_cancer 10y ago
  Oh, if you don't want to get into the nitty gritty of it, you should use Digits (https://github.com/NVIDIA/DIGITS https://github.com/NVIDIA/DIGITS). This is the easiest way to setup a CNN and train it with your sample images (at least compared to Caffe, Tensorflow, and Theano). I say that because it's all GUI based! Real convenient.
  - mrfusion 10y ago
    Wow that is cool! What if I wanted to make a model to detect cats? Can I just load a bunch of pictures from the web? Do I need negative examples?
    - polite_cancer 10y ago
      Good question. Detecting just one class (in this case, cats) will require negative examples. Finding good negatives is somewhat of challenging task because they should be pretty comprehensive, but if you create an account on image-net (http://image-net.org/ http://image-net.org/), then you can download thousands of images. Any questions, feel free to pm me.
nstj 10y ago
Did someone say burritos?
drewm1980 10y ago
I am supportive of clear explanations of some of the building blocks, but I worry repeatedly describing things as being grade school math level gives the wrong impression about the actual learning curve for getting up to speed on working with CNN's. Yes, the building blocks are easy to understand, but actually understanding why a given network structure, or optimization technique isn't working, is a black art. And if you don't have a workstation with a $2k gpu or two, you're probably not going to have a good time.
- lettergram 10y ago
  I setup an automated script to setup a AWS g2 instance, train my neural net using tensorflow, copy my model to my personal computer, and spin down. It costs like $5-$10 to train/test most neural network models. My most expensive model cost like $100 and required a ton of time and resources. It took like 4 days or something. You really dont need $2k workstation.. Of course, for personal use I do have a gtx 1080 because I like to game and play tensorflow/caffe
  - danieltillett 10y ago
    Would you mind sharing this script? I am thinking of investing in the 2K workstation, but this sounds like a much more efficient way to go.
    - lettergram 10y ago
      I tried to get it released before, but was shut down by the "open source office". So I can't give you the exact script. However, h2o has a script that launches a cluster, that's very similar: https://github.com/ledell/h2oEnsemble-benchmarks/blob/master/ec2/h2o-cluster-launch-instances.py https://github.com/ledell/h2oEnsemble-benchmarks/blob/master... You can also do stuff like following in python to execute some code: os.system("scp script_demo.py " + sys.argv[1] + ":/home/ubuntu/") os.system("ssh " + sys.argv[1] + " sudo chmod 777 /home/ubuntu/script_demo.py") os.system("ssh " + sys.argv[1] + " './script_demo.py'") os.system("scp " + sys.argv[1] + ":/home/ubuntu/output/* ./output/") Then it will execute, after it executes - shut it down.
partycoder 10y ago
I think the most intuitive example of a neural network in action is this: http://swaption.net http://swaption.net This is not convolutional though.
nilved 10y ago
Good post, but the author needs to read this article. I interpret the tone in some places to be condescending. https://css-tricks.com/words-avoid-educational-writing/ https://css-tricks.com/words-avoid-educational-writing/
dicroce 10y ago
This video is a great introduction to convolutions and pooling. The other best resource, IMHO, is http://karpathy.github.io/neuralnets/ http://karpathy.github.io/neuralnets/.
armandtamzarian 10y ago
Great post. I like how he didn't go into too much detail on the math of backprop etc. I find the conceptual understanding of ML is more interesting as a lay person.
- banned4life 10y ago
  If you google "Hinton machine learning" on youtube, you will find hinton's lecture's they are non-mathematical, he is a psychologist/math guy, and he is the inventor of almost all this stuff, backprop, drop-out, You will find his lectures to be very entertaining and easy to understand, being a psychologist whose desire is to make a computer operate like a human brain, he's more interested in how the brain actually works, than hacking ML code. Hinton describes backprop, why he invented it, and exactly how it emulates the way the human brain works. Hinton now works at microsoft, he is considered the modern day 'godfather' of DeepLearning/ML
  - eli_gottlieb 10y ago
    >You will find his lectures to be very entertaining and easy to understand, being a psychologist whose desire is to make a computer operate like a human brain, he's more interested in how the brain actually works, than hacking ML code. > >Hinton describes backprop, why he invented it, and exactly how it emulates the way the human brain works. Of course, basically no actual neuroscientists or cognitive scientists think the brain actually works via supervised backpropagation. So he actually has a bit of a holy war going on with the people who properly work on human learning rather than machine learning.
bilbobeer 10y ago
Try them all! And then tell us what you think. Caffe CNTK Deeplearning4j TensorFlow Theano Torch Many others About 5+ years ago I was using ML for music transcription problem's, I largely completed my project. I was satisfied I could take any foreign track and generate a visual music sheet so I could play any song I wish. ( I'm rather tone death and not good enough to play by ear, but I'm very good at playing from sheet music. Five years ago I used MATLAB all can be gotten from torrent, all of matlab has people who provide all the library's for CNN,RNN, and all the Algo's you could want. Almost all the algo's in MATLAB can be read in one page, which makes it easy to understand the algo, and makes it easy to dial-in the fine tuning required for ML. In the past month I ventured back into ML, as I have a desire to apply ML to code generation and dynamic analysis. I have spent over a month on all of the 'free stuff' available... Try them all! And then tell us what you think. Caffe CNTK Deeplearning4j TensorFlow Theano Torch Many others I have found that for almost all of them you have to spend hours on 'Stack' just to get the installation problems dialed-in, I'm using UBUNTU 16.04, so please ... I have found that NONE of the stuff works out of the box, even the KAPARTHY stuff fails without hours of fine-tuning the library's. When using MATLAB 5+ years ago I found it to be highly productive, in a few days I was able to choose my ALGO because I could take the dozen most popular models apply them to my data, choose which generated the best results, then I dialed in my +98% and I was good to go. Now after spending a MONTH on all these Lib's and packages, I still don't even have a single model that can easily migrate to my data. All is about say tensorflow, but let's be honest its a black box, Theano is great, but its clear they dropped the ball years ago, CAFFE is a pain. The best stuff seems to revolve around PYTHON, which is appropriate for fast modelling, as PYTHON is excellent with abstraction just at the right level for math people. Most of the stuff coming down from Github appears to be from 2015, the only stuff that appears new is TensorFlow, which of course is GOOGLE which means sure its fun to play and see what their doing, at the end of the day you don't want to be in bed with Google (CIA/NSA) The rule of thumb is to KEEP-IT-SIMPLE. Keep it Fast, Keep it clear. Most of the models are all based on the MNIST data, which is ok for a very limited problem set. IMHO the stuff that's being DUMPED today on the market is a GREAT step backword in productivity. There is NOTHING new that wasn't available in 2012. All that we have now is a dozen different packages that are all hyped by BIG players. Another item don't even bother to get anything working on windows, as nobody is supporting anything for windows. CUDA has always worked great with MATLAB even 5+ years ago, all you had to do is "USE CUDA", and your python like MATLAB took care of all the business and off-loaded you work to you GPU, which made the computation 10X. Now the NVIDIA lib's are great, but you have to explicitly use CUDA array's in your code. Even KARPATHY models require filters to go back & forth from GPU/CPU data, as they compress the Pickels/CV differently which is a pain. I would prefer to work in GPU 100%. Things have NOT gotten more simple. Things have gotten burdensome. Then there are all these languages LUA, JIT, Go, everybody has to start at zero, PYTHON is now 25 years old and the support for MATH is deep, and PYTHON is almost "APL" like in its power over abstract types. There are people abstracting the GPU but its tough to get the stuff working. The model seems to be to PUSH people off their private hardware and onto a GOOGLE/AMAZON server for $1/hr to do the training, but anybody that actually has done this stuff knows that it can take weeks of 16+ hour days to find the right training ALGO, then another week to dial it in ( find the right loss-function, and Kernel ), and set the hidden-size,drop-out, and learning-rate, thus in actuality you could eat up 100's of dollars all being FED to google/Amazon. This is just 'rent' folks instead of OWNING your own computer, you would just become a renter, and of course if your poor then you can't play. I don't like this model. Even the new laptops have MX980 Nvid Gpu's which are plenty enough. Tensorflow and its brother's are pretty much designed to be services, which means computation in the cloud. The real developer thing for the little guy is to get you training model done, and then generate a matrix, and then put it into C, and down load to firmware and now you have an expert system on a $1 CPU, then you can do as you wish, the Amazon model is your trained matrix is held in the CLOUD an everybody uses services from say iPhone/android, ok again its the old nickel paradigm of Oracle, where they tax all transaction by service. Memory is cheap, simplicity is best.
j1vms 10y ago
> CNNs can be used to categorize other types of data too. The trick is, whatever data type you start with, to transform it to make it look like an image. This is an interesting point, and I assume that 'make it look look like an image' means the same thing as 'think of it as an image'. Can others here who works with CNNs regularly or professionally, comment on whether the author's intuition is essentially correct (give or take some details of course)?
- sprobertson 10y ago
  One thing that might work well with a CNN is geospatial data (easy to consider as an image).
- arketyp 10y ago
  It comes down to the characterizing architecture of convolutional nets, that is weight sharing, and the assumption on data this makes. If by image one means something where you can expect any pattern (at some level in the hierarchy) being equally likely to occur anywhere across an input dimension, then yes this is true. Personally I would say that this is too narrow of a definition of an image (too great of an assumption), and, interestingly enough, perhaps too broad too. I am not a pro. [Edit] Too broad in the sense that, intuitively, there is perhaps an implied assumption of continuity of the input function defining the image. Note that such assumptions can be made explicit with various so-called statistical priors incorporated in the network.
banned4life 10y ago
In 1965 Cooley-Tukey published their famous FFT paper, funny thing is that at Standard Oil they were using FFT in the 1950's, because it could be used for analyzing 'Russian' atomic bomb seismic signatures in real-time it took some 10+ years before it became public ( openly published ), but it was routine in BIG-OIL in 1950's. Now jump to CNN, we were doing CNN on Cray's in the 1970's on Seismic 3d data ( acoustic sound waves pressure/sheer ) in order to find oil deposits, its the same stuff I see in CNN algo's of today, and its the exact same stuff we were doing in the 1970's in visualizing seismic data (P-S wave ratios were the color that told you what kind of rock/substance you propagated ). We used to call this 'around the wheel', I suspect that its good to make the kids think this stuff is all something new, most of the foundation of computational ML was designed in 1950's, hell Von Neuman wrote the book on Cybernetic's, automating the human mind with computers, again 1950's. Another perspective is given the quality ( poor ) that's being leaked, and given corporate history, I suspect that BIG-OIL, and BIG-NSA of today have stuff that is super good and advanced and most of what they leak to GIT-HUB is just garbage. In summary keep it simply, learn how it really works, and then dial in your own algorithm, and you too will discover the holy-grail, but remember that edoocation is always 15-20 years lagging industry. It's always been this way, and always will. NSA/CIA has always mandated a competitive edge over the foreign gov's, never assume that anything you get for free, or academia is cutting-edge. ( Lastly George Green developed the Math here, back in 1816-ish, thus the Convolutional in CNN is 200 years old to be exact. ) ... Try them all! And then tell us what you think. Caffe CNTK Deeplearning4j TensorFlow Theano Torch Many others About 5+ years ago I was using ML for music transcription problem's, I largely completed my project. I was satisfied I could take any foreign track and generate a visual music sheet so I could play any song I wish. ( I'm rather tone death and not good enough to play by ear, but I'm very good at playing from sheet music. Five years ago I used MATLAB all can be gotten from torrent, all of matlab has people who provide all the library's for CNN,RNN, and all the Algo's you could want. Almost all the algo's in MATLAB can be read in one page, which makes it easy to understand the algo, and makes it easy to dial-in the fine tuning required for ML. In the past month I ventured back into ML, as I have a desire to apply ML to code generation and dynamic analysis. I have spent over a month on all of the 'free stuff' available... Try them all! And then tell us what you think. Caffe CNTK Deeplearning4j TensorFlow Theano Torch Many others I have found that faor almost all of them you have to spend hours on 'Stack' just to get the installation problems dialed-in, I'm using UBUNTU 16.04, so please ... I have found that NONE of the stuff works out of the box, even the KAPARTHY stuff fails without hours of fine-tuning the library's. When using MATLAB 5+ years ago I found it to be highly productive, in a few days I was able to choose my ALGO because I could take the dozen most popular models apply them to my data, choose which generated the best results, then I dialed in my +98% and I was good to go. Now after spending a MONTH on all these Lib's and packages, I still don't even have a single model that can easily migrate to my data. All is about say tensorflow, but let's be honest its a black box, Theano is great, but its clear they dropped the ball years ago, CAFFE is a pain. The best stuff seems to revolve around PYTHON, which is appropriate for fast modelling, as PYTHON is excellent with abstraction just at the right level for math people. Most of the stuff coming down from Github appears to be from 2015, the only stuff that appears new is TensorFlow, which of course is GOOGLE which means sure its fun to play and see what their doing, at the end of the day you don't want to be in bed with Google (CIA/NSA) The rule of thumb is to KEEP-IT-SIMPLE. Keep it Fast, Keep it clear. Most of the models are all based on the MNIST data, which is ok for a very limited problem set. IMHO the stuff that's being DUMPED today on the market is a GREAT step backword in productivity. There is NOTHING new that wasn't available in 2012. All that we have now is a dozen different packages that are all hyped by BIG players. Another item don't even bother to get anything working on windows, as nobody is supporting anything for windows. CUDA has always worked great with MATLAB even 5+ years ago, all you had to do is "USE CUDA", and your python like MATLAB took care of all the business and off-loaded you work to you GPU, which made the computation 10X. Now the NVIDIA lib's are great, but you have to explicitly use CUDA array's in your code. Even KARPATHY models require filters to go back & forth from GPU/CPU data, as they compress the Pickels/CV differently which is a pain. I would prefer to work in GPU 100%. Things have NOT gotten more simple. Things have gotten burdensome. Then there are all these languages LUA, JIT, Go, everybody has to start at zero, PYTHON is now 25 years old and the support for MATH is deep, and PYTHON is almost "APL" like in its power over abstract types. There are people abstracting the GPU but its tough to get the stuff working. The model seems to be to PUSH people off their private hardware and onto a GOOGLE/AMAZON server for $1/hr to do the training, but anybody that actually has done this stuff knows that it can take weeks of 16+ hour days to find the right training ALGO, then another week to dial it in ( find the right loss-function, and Kernel ), and set the hidden-size,drop-out, and learning-rate, thus in actuality you could eat up 100's of dollars all being FED to google/Amazon. This is just 'rent' folks instead of OWNING your own computer, you would just become a renter, and of course if your poor then you can't play. I don't like this model. Even the new laptops have MX980 Nvid Gpu's which are plenty enough. Tensorflow and its brother's are pretty much designed to be services, which means computation in the cloud. The real developer thing for the little guy is to get you training model done, and then generate a matrix, and then put it into C, and down load to firmware and now you have an expert system on a $1 CPU, then you can do as you wish, the Amazon model is your trained matrix is held in the CLOUD an everybody uses services from say iPhone/android, ok again its the old nickel paradigm of Oracle, where they tax all transaction by service. Memory is cheap, simplicity is best.

403 points by eternalban 10y ago

Great explanation. Thanks for sharing.

Really great write up. Ive been trying to wrap my not too mathematically talented head around convolutional filters and this really helped in visualizing what is happening.

MrFeynmannsJoke 10y ago

So ist works just like i thought it would. Why are CNN so hyped? Wasnt all this already known decades ago? Or is it just because we can afford the computing power?

bbctol 10y ago
The underlying math was figured out a long time ago, but it's only been in recent years that we've had the computing power to test these out on lots of complicated, real-world classification problems, and had some incredible success.
gcp 10y ago
Computing power, but also some implementation tricks that turn out to make things a lot better. For example, activation via ReLU instead of sigmoid/tanh significantly improves the performance of deep neural networks. Then there's stuff like BatchNorm, Pooling, Dropout etc...
nkozyra 10y ago
1. Computing power 2. Data availability 3. Fast, large local storage
zwieback 10y ago
Decades ago I played around with neural nets but was frustrated because I either had to preprocess and normalize my inputs to the point where I didn't need a network anymore or I had to train a large network with so much data that it was not practical. Having a cookbook approach with a catchy name and orders of magnitude more processing power have revived neural nets and now they are finally doing something useful. Now everyone is jumping on the bandwagon so the field is progressing very quickly. Just because it's hyped doesn't mean it's not worth giving it a second look (although I'm still on the sidelines myself.)
dougabug 10y ago
The basic CNN structure was in place, but as the saying goes, "The Devil's in the details." Early CNN's were applied to problems such as handwritten character recognition with rows of small grayscale image cells as inputs, and were much shallower, smaller models. Today's CNN's operate on full resolution, multi-channel images and video, and can be orders of magnitudes deeper and larger. For instance, ResNets have been proven to demonstrate monotonic performance improvements out to 1200 layers on benchmark datasets. This would have been unthinkable even a couple years ago. By way of comparison, even the state of the art VGG network architecture of a couple years ago originally had to be trained in stages to reach 16 and 19 layers for submission to ILSVRC 2014 (Xavier / MSRA initialization makes this unnecessary now). At the time, VGG and GoogleNet (22 layers) were considered to be extraordinarily deep CNN's.
zackmorris 10y ago
I argued back in 2000 (a year after I got my computer engineering degree) that AI wouldn't take off until computing moved from single threaded/single core to multithreaded/ multicore processing. The fact that we are only hearing about this stuff 15 years later makes me feel that that assertion was largely right. The biggest problem I see in AI is that the algorithms are generally fairly straightforward, but people haven't had the computing power to explore the problem space. We are seeing drastic improvement in things like video cards (routinely 1000+ cores) and data processing locality (map reduce). But processors have stagnated. If we really want AI in any reasonable timescale, we need large arrays of general-purpose cores with a sane communication protocol that doesn't fixate on things like caching, we need a hybrid between Go and Erlang to do concurrent functional programming in a readable way with automagic scaling over a network, and we need all this yesterday. The fancy schmancy AI algorithms will become apparent when processing power is no longer the primary limitation, and at that point we can optimize them.
bilbobeer 10y ago
In 1965 Cooley-Tukey published their famous FFT paper, funny thing is that at Standard Oil they were using FFT in the 1950's, because it could be used for analyzing 'Russian' atomic bomb seismic signatures in real-time it took some 10+ years before it became public ( openly published ), but it was routine in BIG-OIL in 1950's. Now jump to CNN, we were doing CNN on Cray's in the 1970's on Seismic 3d data ( acoustic sound waves pressure/sheer ) in order to find oil deposits, its the same stuff I see in CNN algo's of today, and its the exact same stuff we were doing in the 1970's in visualizing seismic data (P-S wave ratios were the color that told you what kind of rock/substance you propagated ). We used to call this 'around the wheel', I suspect that its good to make the kids think this stuff is all something new, most of the foundation of computational ML was designed in 1950's, hell Von Neuman wrote the book on Cybernetic's, automating the human mind with computers, again 1950's. Another perspective is given the quality ( poor ) that's being leaked, and given corporate history, I suspect that BIG-OIL, and BIG-NSA of today have stuff that is super good and advanced and most of what they leak to GIT-HUB is just garbage. In summary keep it simply, learn how it really works, and then dial in your own algorithm, and you too will discover the holy-grail, but remember that edoocation is always 15-20 years lagging industry. It's always been this way, and always will. NSA/CIA has always mandated a competitive edge over the foreign gov's, never assume that anything you get for free, or academia is cutting-edge. ( Lastly George Green developed the Math here, back in 1816-ish, thus the Convolutional in CNN is 200 years old to be exact. )

elcct 10y ago

This is brilliant, so far one of the easiest to understand explanations.

spiderfarmer 10y ago

Question: I have a database with 1.000.000 vehicle pictures, organized by make and model. What would be the easiest way to play with this data, so that I can train it to predict the make / model? I don't want to reinvent the wheel now so much tutorials are written and software is being released. What would be the easiest way to start?

stared 10y ago
With that number you can train a model from scratch. Alternatively, you can retrain the last layers (it requires much less data). The easiest way is to use some well-known architecture (e.g. VGG16) and go. See: https://github.com/leriomaggio/deep-learning-keras-euroscipy2016 https://github.com/leriomaggio/deep-learning-keras-euroscipy...
- misiti3780 10y ago
  do you have any examples of scripts/examples where only specific layers of a network are being trained?
polite_cancer 10y ago
Oh, if you don't want to get into the nitty gritty of it, you should use Digits (https://github.com/NVIDIA/DIGITS https://github.com/NVIDIA/DIGITS). This is the easiest way to setup a CNN and train it with your sample images (at least compared to Caffe, Tensorflow, and Theano). I say that because it's all GUI based! Real convenient.
- mrfusion 10y ago
  Wow that is cool! What if I wanted to make a model to detect cats? Can I just load a bunch of pictures from the web? Do I need negative examples?
  - polite_cancer 10y ago
    Good question. Detecting just one class (in this case, cats) will require negative examples. Finding good negatives is somewhat of challenging task because they should be pretty comprehensive, but if you create an account on image-net (http://image-net.org/ http://image-net.org/), then you can download thousands of images. Any questions, feel free to pm me.

nstj 10y ago

Did someone say burritos?

drewm1980 10y ago

I am supportive of clear explanations of some of the building blocks, but I worry repeatedly describing things as being grade school math level gives the wrong impression about the actual learning curve for getting up to speed on working with CNN's. Yes, the building blocks are easy to understand, but actually understanding why a given network structure, or optimization technique isn't working, is a black art. And if you don't have a workstation with a $2k gpu or two, you're probably not going to have a good time.

lettergram 10y ago
I setup an automated script to setup a AWS g2 instance, train my neural net using tensorflow, copy my model to my personal computer, and spin down. It costs like $5-$10 to train/test most neural network models. My most expensive model cost like $100 and required a ton of time and resources. It took like 4 days or something. You really dont need $2k workstation.. Of course, for personal use I do have a gtx 1080 because I like to game and play tensorflow/caffe
- danieltillett 10y ago
  Would you mind sharing this script? I am thinking of investing in the 2K workstation, but this sounds like a much more efficient way to go.
  - lettergram 10y ago
    I tried to get it released before, but was shut down by the "open source office". So I can't give you the exact script. However, h2o has a script that launches a cluster, that's very similar: https://github.com/ledell/h2oEnsemble-benchmarks/blob/master/ec2/h2o-cluster-launch-instances.py https://github.com/ledell/h2oEnsemble-benchmarks/blob/master... You can also do stuff like following in python to execute some code: os.system("scp script_demo.py " + sys.argv[1] + ":/home/ubuntu/") os.system("ssh " + sys.argv[1] + " sudo chmod 777 /home/ubuntu/script_demo.py") os.system("ssh " + sys.argv[1] + " './script_demo.py'") os.system("scp " + sys.argv[1] + ":/home/ubuntu/output/* ./output/") Then it will execute, after it executes - shut it down.

partycoder 10y ago

I think the most intuitive example of a neural network in action is this: http://swaption.net http://swaption.net This is not convolutional though.

nilved 10y ago

Good post, but the author needs to read this article. I interpret the tone in some places to be condescending. https://css-tricks.com/words-avoid-educational-writing/ https://css-tricks.com/words-avoid-educational-writing/

dicroce 10y ago

This video is a great introduction to convolutions and pooling. The other best resource, IMHO, is http://karpathy.github.io/neuralnets/ http://karpathy.github.io/neuralnets/.

armandtamzarian 10y ago

Great post. I like how he didn't go into too much detail on the math of backprop etc. I find the conceptual understanding of ML is more interesting as a lay person.

banned4life 10y ago
If you google "Hinton machine learning" on youtube, you will find hinton's lecture's they are non-mathematical, he is a psychologist/math guy, and he is the inventor of almost all this stuff, backprop, drop-out, You will find his lectures to be very entertaining and easy to understand, being a psychologist whose desire is to make a computer operate like a human brain, he's more interested in how the brain actually works, than hacking ML code. Hinton describes backprop, why he invented it, and exactly how it emulates the way the human brain works. Hinton now works at microsoft, he is considered the modern day 'godfather' of DeepLearning/ML
- eli_gottlieb 10y ago
  >You will find his lectures to be very entertaining and easy to understand, being a psychologist whose desire is to make a computer operate like a human brain, he's more interested in how the brain actually works, than hacking ML code. > >Hinton describes backprop, why he invented it, and exactly how it emulates the way the human brain works. Of course, basically no actual neuroscientists or cognitive scientists think the brain actually works via supervised backpropagation. So he actually has a bit of a holy war going on with the people who properly work on human learning rather than machine learning.

bilbobeer 10y ago

Try them all! And then tell us what you think. Caffe CNTK Deeplearning4j TensorFlow Theano Torch Many others About 5+ years ago I was using ML for music transcription problem's, I largely completed my project. I was satisfied I could take any foreign track and generate a visual music sheet so I could play any song I wish. ( I'm rather tone death and not good enough to play by ear, but I'm very good at playing from sheet music. Five years ago I used MATLAB all can be gotten from torrent, all of matlab has people who provide all the library's for CNN,RNN, and all the Algo's you could want. Almost all the algo's in MATLAB can be read in one page, which makes it easy to understand the algo, and makes it easy to dial-in the fine tuning required for ML. In the past month I ventured back into ML, as I have a desire to apply ML to code generation and dynamic analysis. I have spent over a month on all of the 'free stuff' available... Try them all! And then tell us what you think. Caffe CNTK Deeplearning4j TensorFlow Theano Torch Many others I have found that for almost all of them you have to spend hours on 'Stack' just to get the installation problems dialed-in, I'm using UBUNTU 16.04, so please ... I have found that NONE of the stuff works out of the box, even the KAPARTHY stuff fails without hours of fine-tuning the library's. When using MATLAB 5+ years ago I found it to be highly productive, in a few days I was able to choose my ALGO because I could take the dozen most popular models apply them to my data, choose which generated the best results, then I dialed in my +98% and I was good to go. Now after spending a MONTH on all these Lib's and packages, I still don't even have a single model that can easily migrate to my data. All is about say tensorflow, but let's be honest its a black box, Theano is great, but its clear they dropped the ball years ago, CAFFE is a pain. The best stuff seems to revolve around PYTHON, which is appropriate for fast modelling, as PYTHON is excellent with abstraction just at the right level for math people. Most of the stuff coming down from Github appears to be from 2015, the only stuff that appears new is TensorFlow, which of course is GOOGLE which means sure its fun to play and see what their doing, at the end of the day you don't want to be in bed with Google (CIA/NSA) The rule of thumb is to KEEP-IT-SIMPLE. Keep it Fast, Keep it clear. Most of the models are all based on the MNIST data, which is ok for a very limited problem set. IMHO the stuff that's being DUMPED today on the market is a GREAT step backword in productivity. There is NOTHING new that wasn't available in 2012. All that we have now is a dozen different packages that are all hyped by BIG players. Another item don't even bother to get anything working on windows, as nobody is supporting anything for windows. CUDA has always worked great with MATLAB even 5+ years ago, all you had to do is "USE CUDA", and your python like MATLAB took care of all the business and off-loaded you work to you GPU, which made the computation 10X. Now the NVIDIA lib's are great, but you have to explicitly use CUDA array's in your code. Even KARPATHY models require filters to go back & forth from GPU/CPU data, as they compress the Pickels/CV differently which is a pain. I would prefer to work in GPU 100%. Things have NOT gotten more simple. Things have gotten burdensome. Then there are all these languages LUA, JIT, Go, everybody has to start at zero, PYTHON is now 25 years old and the support for MATH is deep, and PYTHON is almost "APL" like in its power over abstract types. There are people abstracting the GPU but its tough to get the stuff working. The model seems to be to PUSH people off their private hardware and onto a GOOGLE/AMAZON server for $1/hr to do the training, but anybody that actually has done this stuff knows that it can take weeks of 16+ hour days to find the right training ALGO, then another week to dial it in ( find the right loss-function, and Kernel ), and set the hidden-size,drop-out, and learning-rate, thus in actuality you could eat up 100's of dollars all being FED to google/Amazon. This is just 'rent' folks instead of OWNING your own computer, you would just become a renter, and of course if your poor then you can't play. I don't like this model. Even the new laptops have MX980 Nvid Gpu's which are plenty enough. Tensorflow and its brother's are pretty much designed to be services, which means computation in the cloud. The real developer thing for the little guy is to get you training model done, and then generate a matrix, and then put it into C, and down load to firmware and now you have an expert system on a $1 CPU, then you can do as you wish, the Amazon model is your trained matrix is held in the CLOUD an everybody uses services from say iPhone/android, ok again its the old nickel paradigm of Oracle, where they tax all transaction by service. Memory is cheap, simplicity is best.

j1vms 10y ago

> CNNs can be used to categorize other types of data too. The trick is, whatever data type you start with, to transform it to make it look like an image. This is an interesting point, and I assume that 'make it look look like an image' means the same thing as 'think of it as an image'. Can others here who works with CNNs regularly or professionally, comment on whether the author's intuition is essentially correct (give or take some details of course)?

sprobertson 10y ago
One thing that might work well with a CNN is geospatial data (easy to consider as an image).
arketyp 10y ago
It comes down to the characterizing architecture of convolutional nets, that is weight sharing, and the assumption on data this makes. If by image one means something where you can expect any pattern (at some level in the hierarchy) being equally likely to occur anywhere across an input dimension, then yes this is true. Personally I would say that this is too narrow of a definition of an image (too great of an assumption), and, interestingly enough, perhaps too broad too. I am not a pro. [Edit] Too broad in the sense that, intuitively, there is perhaps an implied assumption of continuity of the input function defining the image. Note that such assumptions can be made explicit with various so-called statistical priors incorporated in the network.

banned4life 10y ago

In 1965 Cooley-Tukey published their famous FFT paper, funny thing is that at Standard Oil they were using FFT in the 1950's, because it could be used for analyzing 'Russian' atomic bomb seismic signatures in real-time it took some 10+ years before it became public ( openly published ), but it was routine in BIG-OIL in 1950's. Now jump to CNN, we were doing CNN on Cray's in the 1970's on Seismic 3d data ( acoustic sound waves pressure/sheer ) in order to find oil deposits, its the same stuff I see in CNN algo's of today, and its the exact same stuff we were doing in the 1970's in visualizing seismic data (P-S wave ratios were the color that told you what kind of rock/substance you propagated ). We used to call this 'around the wheel', I suspect that its good to make the kids think this stuff is all something new, most of the foundation of computational ML was designed in 1950's, hell Von Neuman wrote the book on Cybernetic's, automating the human mind with computers, again 1950's. Another perspective is given the quality ( poor ) that's being leaked, and given corporate history, I suspect that BIG-OIL, and BIG-NSA of today have stuff that is super good and advanced and most of what they leak to GIT-HUB is just garbage. In summary keep it simply, learn how it really works, and then dial in your own algorithm, and you too will discover the holy-grail, but remember that edoocation is always 15-20 years lagging industry. It's always been this way, and always will. NSA/CIA has always mandated a competitive edge over the foreign gov's, never assume that anything you get for free, or academia is cutting-edge. ( Lastly George Green developed the Math here, back in 1816-ish, thus the Convolutional in CNN is 200 years old to be exact. ) ... Try them all! And then tell us what you think. Caffe CNTK Deeplearning4j TensorFlow Theano Torch Many others About 5+ years ago I was using ML for music transcription problem's, I largely completed my project. I was satisfied I could take any foreign track and generate a visual music sheet so I could play any song I wish. ( I'm rather tone death and not good enough to play by ear, but I'm very good at playing from sheet music. Five years ago I used MATLAB all can be gotten from torrent, all of matlab has people who provide all the library's for CNN,RNN, and all the Algo's you could want. Almost all the algo's in MATLAB can be read in one page, which makes it easy to understand the algo, and makes it easy to dial-in the fine tuning required for ML. In the past month I ventured back into ML, as I have a desire to apply ML to code generation and dynamic analysis. I have spent over a month on all of the 'free stuff' available... Try them all! And then tell us what you think. Caffe CNTK Deeplearning4j TensorFlow Theano Torch Many others I have found that faor almost all of them you have to spend hours on 'Stack' just to get the installation problems dialed-in, I'm using UBUNTU 16.04, so please ... I have found that NONE of the stuff works out of the box, even the KAPARTHY stuff fails without hours of fine-tuning the library's. When using MATLAB 5+ years ago I found it to be highly productive, in a few days I was able to choose my ALGO because I could take the dozen most popular models apply them to my data, choose which generated the best results, then I dialed in my +98% and I was good to go. Now after spending a MONTH on all these Lib's and packages, I still don't even have a single model that can easily migrate to my data. All is about say tensorflow, but let's be honest its a black box, Theano is great, but its clear they dropped the ball years ago, CAFFE is a pain. The best stuff seems to revolve around PYTHON, which is appropriate for fast modelling, as PYTHON is excellent with abstraction just at the right level for math people. Most of the stuff coming down from Github appears to be from 2015, the only stuff that appears new is TensorFlow, which of course is GOOGLE which means sure its fun to play and see what their doing, at the end of the day you don't want to be in bed with Google (CIA/NSA) The rule of thumb is to KEEP-IT-SIMPLE. Keep it Fast, Keep it clear. Most of the models are all based on the MNIST data, which is ok for a very limited problem set. IMHO the stuff that's being DUMPED today on the market is a GREAT step backword in productivity. There is NOTHING new that wasn't available in 2012. All that we have now is a dozen different packages that are all hyped by BIG players. Another item don't even bother to get anything working on windows, as nobody is supporting anything for windows. CUDA has always worked great with MATLAB even 5+ years ago, all you had to do is "USE CUDA", and your python like MATLAB took care of all the business and off-loaded you work to you GPU, which made the computation 10X. Now the NVIDIA lib's are great, but you have to explicitly use CUDA array's in your code. Even KARPATHY models require filters to go back & forth from GPU/CPU data, as they compress the Pickels/CV differently which is a pain. I would prefer to work in GPU 100%. Things have NOT gotten more simple. Things have gotten burdensome. Then there are all these languages LUA, JIT, Go, everybody has to start at zero, PYTHON is now 25 years old and the support for MATH is deep, and PYTHON is almost "APL" like in its power over abstract types. There are people abstracting the GPU but its tough to get the stuff working. The model seems to be to PUSH people off their private hardware and onto a GOOGLE/AMAZON server for $1/hr to do the training, but anybody that actually has done this stuff knows that it can take weeks of 16+ hour days to find the right training ALGO, then another week to dial it in ( find the right loss-function, and Kernel ), and set the hidden-size,drop-out, and learning-rate, thus in actuality you could eat up 100's of dollars all being FED to google/Amazon. This is just 'rent' folks instead of OWNING your own computer, you would just become a renter, and of course if your poor then you can't play. I don't like this model. Even the new laptops have MX980 Nvid Gpu's which are plenty enough. Tensorflow and its brother's are pretty much designed to be services, which means computation in the cloud. The real developer thing for the little guy is to get you training model done, and then generate a matrix, and then put it into C, and down load to firmware and now you have an expert system on a $1 CPU, then you can do as you wish, the Amazon model is your trained matrix is held in the CLOUD an everybody uses services from say iPhone/android, ok again its the old nickel paradigm of Oracle, where they tax all transaction by service. Memory is cheap, simplicity is best.