4 ms·

C++ Neural Network in a Weekend (2020)

140 points by mbustamanter 4y ago

deleted 4y ago
[deleted]
rramadass 4y ago
For anybody looking to understand/implement Neural Networks in C++, the books by Timothy Masters are indispensable - http://www.timothymasters.info/ http://www.timothymasters.info/
- MuffinFlavored 4y ago
  What's something useful to do with neural networks?
  - dplavery92 4y ago
    NNs are potentially very powerful arbitrary function approximators, but you have very limited control (or, arguably, insight) into the precise nature of the solutions their optimization arrives at. Because of that, they've been especially well suited to problems in vision and NLP where we have basic intuition about the phenomenology but can't practically manage a formal description of that intuition (and enumerating that description is probably not of great intellectual interest): what, in pixel space, makes a cat a cat or a dog a dog? What, in patterns of natural words, indicates sarcasm or positive/negative sentiment? They also get tons of use in results-oriented modeling of lots of other statistics questions in structured data (home prices, resource allocation, voter turnouts, etc.) but in this luddite's opinion, these sorts of applications tend to be pretty fraught if they short-change the convenience of the model training paradigm for a deeper understanding of the data phenomenology.
    - MuffinFlavored 4y ago
      I've never understood how to add "layers" when training a neural network. For example, I could train a model on historical home prices by area + date/time, but I'd also like to add for example, the unemployment rate, the trailing rate of inflation past 12 months, the current 30 year mortgage rate for the time, etc.
      - nanidin 4y ago
        The things you mention after “For [the(sic)] example” are called features and they are considered as dimensions of the data. Layers are not strictly related to dimensions.
        MuffinFlavored 4y ago
        Is there a typical limit to number of features (that have almost nothing to do with each other/very little correlation) you can add to a basic "hack it together real quick from Google/copy paste/tutorials" Python neural network before the results are pretty bleh?
- hrgiger 4y ago
  There are plenty of resources on search engine results , what did you find in particular those books stays top of the others?
hrgiger 4y ago
You might wanna validate your back propogation *or the final cost against a hand written very small network to see what went wrong
readthenotes1 4y ago
After I took a C++ class, I built a small neural net in 2 weeks (no C beforehand, nets only from school a few years ago). A few months later, I replicated the neural net in Smalltalk in 2 half days. Fortunately, our project went with Smalltalk...
- taneq 4y ago
  Did you have any prior Smalltalk experience? Or if not, any programming experience at all? Sounds like it might have been “I learned to program in C++ in two weeks, then ported my program to Smalltalk in 2 days”.
- nine_k 4y ago
  How fast the implementations are, in comparison?
ilaksh 4y ago
Here is my simplistic and dumb NN in C++ https://github.com/runvnc/mlp/blob/master/neuralnetwork.cpp https://github.com/runvnc/mlp/blob/master/neuralnetwork.cpp based on the Hinton paper https://github.com/runvnc/nnpapers/blob/master/hinton86.pdf https://github.com/runvnc/nnpapers/blob/master/hinton86.pdf
jheriko 4y ago
"shit you did in your bedroom as a kid before the internet, in a few hours, done in a weekend with enormous resources!" you people suck.
nnq 4y ago
this is cool, but try C (modern, C99) instead: all I see in this C++ version is obsfucation and confusingly named things... (sure, for an "industrial strength" version C++'s features are worth the cost, but for learning/education I never saw the appeal of classes and objects and smart pointers and stuff)
freecodyx 4y ago
Forget about the c++ code, and appreciate the beautifully put explanation of the concept. Then you can implement yours even in php if you want
d_tr 4y ago
I did my CNN implementation in C++ a couple of years back using this: http://cs231n.stanford.edu http://cs231n.stanford.edu Around 600 lines of free functions in a header file for the core functionality, including convolution, dropout, batch normalization, etc. Maybe half of it is actual code. Around 2000 lines for an executable training program including a couple of array template classes, data loaders, and convenience functions. Slow as shit but it works :P An enjoyable exercise.

140 points by mbustamanter 4y ago

deleted 4y ago

[deleted]

For anybody looking to understand/implement Neural Networks in C++, the books by Timothy Masters are indispensable - http://www.timothymasters.info/ http://www.timothymasters.info/

MuffinFlavored 4y ago
What's something useful to do with neural networks?
- dplavery92 4y ago
  NNs are potentially very powerful arbitrary function approximators, but you have very limited control (or, arguably, insight) into the precise nature of the solutions their optimization arrives at. Because of that, they've been especially well suited to problems in vision and NLP where we have basic intuition about the phenomenology but can't practically manage a formal description of that intuition (and enumerating that description is probably not of great intellectual interest): what, in pixel space, makes a cat a cat or a dog a dog? What, in patterns of natural words, indicates sarcasm or positive/negative sentiment? They also get tons of use in results-oriented modeling of lots of other statistics questions in structured data (home prices, resource allocation, voter turnouts, etc.) but in this luddite's opinion, these sorts of applications tend to be pretty fraught if they short-change the convenience of the model training paradigm for a deeper understanding of the data phenomenology.
  - MuffinFlavored 4y ago
    I've never understood how to add "layers" when training a neural network. For example, I could train a model on historical home prices by area + date/time, but I'd also like to add for example, the unemployment rate, the trailing rate of inflation past 12 months, the current 30 year mortgage rate for the time, etc.
    - nanidin 4y ago
      The things you mention after “For [the(sic)] example” are called features and they are considered as dimensions of the data. Layers are not strictly related to dimensions.
      - MuffinFlavored 4y ago
        Is there a typical limit to number of features (that have almost nothing to do with each other/very little correlation) you can add to a basic "hack it together real quick from Google/copy paste/tutorials" Python neural network before the results are pretty bleh?
hrgiger 4y ago
There are plenty of resources on search engine results , what did you find in particular those books stays top of the others?

hrgiger 4y ago

You might wanna validate your back propogation *or the final cost against a hand written very small network to see what went wrong

readthenotes1 4y ago

After I took a C++ class, I built a small neural net in 2 weeks (no C beforehand, nets only from school a few years ago). A few months later, I replicated the neural net in Smalltalk in 2 half days. Fortunately, our project went with Smalltalk...

taneq 4y ago
Did you have any prior Smalltalk experience? Or if not, any programming experience at all? Sounds like it might have been “I learned to program in C++ in two weeks, then ported my program to Smalltalk in 2 days”.
nine_k 4y ago
How fast the implementations are, in comparison?

ilaksh 4y ago

Here is my simplistic and dumb NN in C++ https://github.com/runvnc/mlp/blob/master/neuralnetwork.cpp https://github.com/runvnc/mlp/blob/master/neuralnetwork.cpp based on the Hinton paper https://github.com/runvnc/nnpapers/blob/master/hinton86.pdf https://github.com/runvnc/nnpapers/blob/master/hinton86.pdf

jheriko 4y ago

"shit you did in your bedroom as a kid before the internet, in a few hours, done in a weekend with enormous resources!" you people suck.

nnq 4y ago

this is cool, but try C (modern, C99) instead: all I see in this C++ version is obsfucation and confusingly named things... (sure, for an "industrial strength" version C++'s features are worth the cost, but for learning/education I never saw the appeal of classes and objects and smart pointers and stuff)

freecodyx 4y ago

Forget about the c++ code, and appreciate the beautifully put explanation of the concept. Then you can implement yours even in php if you want

d_tr 4y ago

I did my CNN implementation in C++ a couple of years back using this: http://cs231n.stanford.edu http://cs231n.stanford.edu Around 600 lines of free functions in a header file for the core functionality, including convolution, dropout, batch normalization, etc. Maybe half of it is actual code. Around 2000 lines for an executable training program including a couple of array template classes, data loaders, and convenience functions. Slow as shit but it works :P An enjoyable exercise.