8 ms·

Building a simple neural net in Java

137 points by wheresvic4 6y ago

mjburgess 6y ago
> A neural net is a software representation of how the brain works A NN shares as much with the brain, as a decision tree does with a forest: nothing at all. It would be preferable to completely dispense with any talk of the brain, and introduce it simply as a modified form of (high school) linear regression. Any biological talk at this point is unhelpful mystification.
- Frost1x 6y ago
  >Any biological talk at this point is unhelpful mystification. The business marketing groups for NNs might push back on this a bit, I suspect.
- zedderled 6y ago
  Thank you for pointing this out. I studied math and still get tripped by the flowery language. I understand the reason for analogy but it hinders application of concepts like regression, line of best fit and other things that makes machines “smarter.”
- curioushacking 6y ago
  I agree with you that the biological inspiration for a neural network is tenuous at best. Especially the basic two-layer network outlined here. However at least for visual perception, there is some scientific basis for convolutional neural networks having some properties to biological visual perception. The work of Hubel and Wiesel demonstrated visual cortex activations that look very similar to the first layers of CNN kernels. ref: https://knowingneurons.com/2014/10/29/hubel-and-wiesel-the-neural-basis-of-visual-perception/ https://knowingneurons.com/2014/10/29/hubel-and-wiesel-the-n...
  - mjburgess 6y ago
    There is no biological analogue to backprop, any supervised learning step, etc. Nor, conversely, a mathematical analogue of neuroplasticity, biochemical signalling, etc. I haven't gone far on looking at CNN vs. visual context, but I could only imagine the analogue exists at the hierachical-geometric level; ie., it isnt a model of function, but simply a model of how one can in principle process visual information. (Which any system parsing information would follow, AST parsers, etc. are likewise hierarchical.).
    - salawat 6y ago
      >There is no biological analogue to backprop, any supervised learning step, etc. Excuse me? How do you figure? Have you tried to do any physical task before, and not gotten it right the first time, tried again, and got closer? Hello, backprop. Has someone trained you, telling you you're doing it wrong? Hello, supervised learning. There is an entire portion of your brain tasked with measuring the difference between what you intended and what you did. If you damage it, you actually end up in a state where other people's feedback is what you have to rely on. The equations behind NN's are "non-biological" only in the sense that you're taking a general math function (the sigmoid) and burning it into silicon gates that are not part of a living organism that also has to deal with the excess baggage of remaining alive, or operating as a chemically based computer. However, the dynamics and fundamental capabilities of a biological neural network and a silicon based one are fundamentally the same. You can just scale input domains and speed of the Silicon based one a lot easier than you can the bio-based one due to the difference in computing media. You can also wipe out and retrain a silicon net without being considered to have "killed" anything. The Silicon nets, interestingly, have no analog of running and training simultaneously, or are just starting to get it from the lit I've kept up with, nor an analog for forgetting typically applied to them, making the biological net the far more interesting information processing construct. To be frank, NN's not resembling the biological models says more about the inefficiencies of our media of computing, our woeful lack of understanding around the nature of human/biological perception, and the arrogance of human beings who lack the capacity to recognize in the math something fundamental to their own existence. This post written by a multiple decade uptime GA/C/R/D pick your letter, constantly training internetworked lattice of sigmoid encoded control networks whose morning your lack of perspective just tainted, and whose training is thusfar unsuccessful in attempts to extinguish the need to point out mis- or incomplete understandings of those sharing the same messaging medium as it generally does. Your overfitting to textual and numerical symbolic representation recognition will not serve you well if you can't extend that capacity to discern the pattern into the overaching context of human existence. That's where you really start gaining an appreciation for the power of NN's. It's the building block of the information processing construct that can eventually recognize and reproduce itself. Now if you'll excuse me, I need to get back to some obligatory biological maintnance. This bag of organs doesn't maintain itself, you know.
- acvny 6y ago
  That's not true. A neural net is a simplified model of neuron networks. A perceptron is a simple model of a biological neuron.
  - dnautics 6y ago
    A perceptron was a simple model of a biological neuron. We now know that biological neurons work rather differently.
- timkam 6y ago
  One can perhaps say: "A neural net is a collection of machine learning approaches that are somewhat inspired by the human brain. However, neural nets have little to do with how the brain actually works." I think this statement (or a similar, more accurate version) is actually useful, because the name "neural net" is somewhat deceiving.
- kalal 6y ago
  I am afraid that a decision tree HAS direct impact on the forest. Multiple trees averaged give the forest response. It's like saying, one vote in elections has nothing to do with the final decision. Sure we have heard it many times that it IS a like a brain, and many times it is NOT like a brain. It really depends on your intended audience, but it is not worth picking on and it feels like the tabs/spaces problem. BTW: tabs of course!
  - kjeetgill 6y ago
    I think the poster meant real world wooded forests as an example. As in, the human brain is as distant from this mathematical construct as an actual forest with bears is from decision trees.
    - kalal 6y ago
      Well, that is possible. I have probably worked too much with trees and forests.
- SomeHacker44 6y ago
  It only works because of non-linearities. So that seems like too much of a simplification, limiting it to "linear algebra" when it involves both vector calculus and non-linear algebras.
  - mjburgess 6y ago
    In principle. In practice, the "ReLU Algebra" is almost a linear algebra. NN solutions end up being discontinuous stacks of linear regressions. So it's not a bad place to start in explaining NNs. ReLU NN models are piece-wise linear.
- pc86 6y ago
  > A NN shares as much with the brain, as a decision tree does with a forest: nothing at all. A more accurate statement might be "A NN shares as much with a human brain as a decision tree does with an actual tree: The architecture of the first sometimes appears similar to gross oversimplification of the architecture of the second." Tree:forest is a change in scale not present in NN:brain, and while NN can sometimes look like a very simplified drawing of actual neurons, a decision tree can also look like a very simplified drawing of an actual tree.
bullen 6y ago
I wish someone could take this simple example and add GPU support (and how that can be accomplished best OpenCL/CUDA etc.) just so one could understand how the GPU actually helps accelerate things when the amount of neurons/layers grows? Edit: A bit sad that OpenCL does not work on Jetson Nano and I can't find anything recent that allows Java to run CUDA!? So we're back to AMD? Also "In the next post we will see if adding another layer to our neural network can help in improving the predictions ;)" but there is no next post even after 4 years!
- suyash 6y ago
  Look up project - JCUDA and this article if you're interested in learning more about that https://blogs.oracle.com/javamagazine/programming-the-gpu-in-java https://blogs.oracle.com/javamagazine/programming-the-gpu-in...
  - bullen 6y ago
    Thx, after some more investigating I decided to write my own CPU stuff first then maybe add OpenGL compute shaders if I need them...
    - suyash 6y ago
      Ok, but these are open source projects that companies like Oracle and NVIDIA are workin on which is awesome.
      - bullen 6y ago
        CUDA is not so open-source... and the glue between Java and CUDA seems tricky!
        suyash 6y ago
        I mentioned JCUDA (Java Bindings for CUDA) which is an open source project https://github.com/jcuda https://github.com/jcuda Here is the tutorial : http://www.jcuda.org/tutorial/TutorialIndex.html http://www.jcuda.org/tutorial/TutorialIndex.html
boyadjian 6y ago
I like very much these simple tutorials, from which you can start something bigger
stfwn 6y ago
The last time a post like this appeared on HN prompted me to write a gist with a simple neural network in Python (with Numpy). It downloads the MNIST dataset for you, trains a fully connected network on it, prints the accuracy on the validation set and plots the loss. It's pretty verbose with plenty of terms and comments to search the web for if you're interested. https://gist.github.com/stfwn/62e51d86ca4ff155becd3c6a14adf60e https://gist.github.com/stfwn/62e51d86ca4ff155becd3c6a14adf6...
- eternalban 6y ago
  [OT: this is great! https://www.youtube.com/watch?v=UD3AtBm5R7g&feature=youtu.be https://www.youtube.com/watch?v=UD3AtBm5R7g&feature=youtu.be]
  - stfwn 6y ago
    Thanks!
suyash 6y ago
I cover this topic quite a lot in my talks - Deep Learning in Java for software engineers. Today you can build pretty much any deep learning model in Java that you can do with Python with availability of plenty of ML & Deep Learning frameworks for Java. If you're interested in this, check out projects like : - https://djl.ai https://djl.ai - https://deeplearning4j.org https://deeplearning4j.org - https://github.com/tensorflow/java https://github.com/tensorflow/java - https://github.com/pytorch/java-demo https://github.com/pytorch/java-demo - https://www.deepnetts.com https://www.deepnetts.com - https://tribuo.org https://tribuo.org - ONNX support for Java - MLFlow for Java https://docs.databricks.com/applications/mlflow/index.html https://docs.databricks.com/applications/mlflow/index.html - SparkMLlib : https://spark.apache.org/mllib/ https://spark.apache.org/mllib/ All of the above listed projects allow one to perform Deep Learning Training and Deployment models into production. Happy to answer any questions on this subject. You can also sign up for my newsletter on this topic https://docs.google.com/forms/d/1oa_TtltDRmnov2bv5Cqo2Hiwfb0lBK8bSrVyPFFn7nw/edit https://docs.google.com/forms/d/1oa_TtltDRmnov2bv5Cqo2Hiwfb0...
TrackerFF 6y ago
Maybe it's just semantics - but I believe that is a perceptron (single-layer NN)
- blackbear_ 6y ago
  The proper name would be logistic regression, but linear models aren't so sexy anymore nowadays :)
shimonabi 6y ago
I really recommend the book "Make Your Own Neural Network" by Tariq Rashid for beginners. It uses Python and Numpy. I was able to adapt the code for my own mouse-drawn-symbols recognizer for my AI class project.
pmayrgundter 6y ago
Here's another along the same lines, with a demo learning faces according to the Machine Learning book by Tom Mitchell@CMU: https://github.com/pablo-mayrgundter/freality/tree/master/ml/nn https://github.com/pablo-mayrgundter/freality/tree/master/ml...

137 points by wheresvic4 6y ago

> A neural net is a software representation of how the brain works A NN shares as much with the brain, as a decision tree does with a forest: nothing at all. It would be preferable to completely dispense with any talk of the brain, and introduce it simply as a modified form of (high school) linear regression. Any biological talk at this point is unhelpful mystification.

Frost1x 6y ago
>Any biological talk at this point is unhelpful mystification. The business marketing groups for NNs might push back on this a bit, I suspect.
zedderled 6y ago
Thank you for pointing this out. I studied math and still get tripped by the flowery language. I understand the reason for analogy but it hinders application of concepts like regression, line of best fit and other things that makes machines “smarter.”
curioushacking 6y ago
I agree with you that the biological inspiration for a neural network is tenuous at best. Especially the basic two-layer network outlined here. However at least for visual perception, there is some scientific basis for convolutional neural networks having some properties to biological visual perception. The work of Hubel and Wiesel demonstrated visual cortex activations that look very similar to the first layers of CNN kernels. ref: https://knowingneurons.com/2014/10/29/hubel-and-wiesel-the-neural-basis-of-visual-perception/ https://knowingneurons.com/2014/10/29/hubel-and-wiesel-the-n...
- mjburgess 6y ago
  There is no biological analogue to backprop, any supervised learning step, etc. Nor, conversely, a mathematical analogue of neuroplasticity, biochemical signalling, etc. I haven't gone far on looking at CNN vs. visual context, but I could only imagine the analogue exists at the hierachical-geometric level; ie., it isnt a model of function, but simply a model of how one can in principle process visual information. (Which any system parsing information would follow, AST parsers, etc. are likewise hierarchical.).
  - salawat 6y ago
    >There is no biological analogue to backprop, any supervised learning step, etc. Excuse me? How do you figure? Have you tried to do any physical task before, and not gotten it right the first time, tried again, and got closer? Hello, backprop. Has someone trained you, telling you you're doing it wrong? Hello, supervised learning. There is an entire portion of your brain tasked with measuring the difference between what you intended and what you did. If you damage it, you actually end up in a state where other people's feedback is what you have to rely on. The equations behind NN's are "non-biological" only in the sense that you're taking a general math function (the sigmoid) and burning it into silicon gates that are not part of a living organism that also has to deal with the excess baggage of remaining alive, or operating as a chemically based computer. However, the dynamics and fundamental capabilities of a biological neural network and a silicon based one are fundamentally the same. You can just scale input domains and speed of the Silicon based one a lot easier than you can the bio-based one due to the difference in computing media. You can also wipe out and retrain a silicon net without being considered to have "killed" anything. The Silicon nets, interestingly, have no analog of running and training simultaneously, or are just starting to get it from the lit I've kept up with, nor an analog for forgetting typically applied to them, making the biological net the far more interesting information processing construct. To be frank, NN's not resembling the biological models says more about the inefficiencies of our media of computing, our woeful lack of understanding around the nature of human/biological perception, and the arrogance of human beings who lack the capacity to recognize in the math something fundamental to their own existence. This post written by a multiple decade uptime GA/C/R/D pick your letter, constantly training internetworked lattice of sigmoid encoded control networks whose morning your lack of perspective just tainted, and whose training is thusfar unsuccessful in attempts to extinguish the need to point out mis- or incomplete understandings of those sharing the same messaging medium as it generally does. Your overfitting to textual and numerical symbolic representation recognition will not serve you well if you can't extend that capacity to discern the pattern into the overaching context of human existence. That's where you really start gaining an appreciation for the power of NN's. It's the building block of the information processing construct that can eventually recognize and reproduce itself. Now if you'll excuse me, I need to get back to some obligatory biological maintnance. This bag of organs doesn't maintain itself, you know.
acvny 6y ago
That's not true. A neural net is a simplified model of neuron networks. A perceptron is a simple model of a biological neuron.
- dnautics 6y ago
  A perceptron was a simple model of a biological neuron. We now know that biological neurons work rather differently.
timkam 6y ago
One can perhaps say: "A neural net is a collection of machine learning approaches that are somewhat inspired by the human brain. However, neural nets have little to do with how the brain actually works." I think this statement (or a similar, more accurate version) is actually useful, because the name "neural net" is somewhat deceiving.
kalal 6y ago
I am afraid that a decision tree HAS direct impact on the forest. Multiple trees averaged give the forest response. It's like saying, one vote in elections has nothing to do with the final decision. Sure we have heard it many times that it IS a like a brain, and many times it is NOT like a brain. It really depends on your intended audience, but it is not worth picking on and it feels like the tabs/spaces problem. BTW: tabs of course!
- kjeetgill 6y ago
  I think the poster meant real world wooded forests as an example. As in, the human brain is as distant from this mathematical construct as an actual forest with bears is from decision trees.
  - kalal 6y ago
    Well, that is possible. I have probably worked too much with trees and forests.
SomeHacker44 6y ago
It only works because of non-linearities. So that seems like too much of a simplification, limiting it to "linear algebra" when it involves both vector calculus and non-linear algebras.
- mjburgess 6y ago
  In principle. In practice, the "ReLU Algebra" is almost a linear algebra. NN solutions end up being discontinuous stacks of linear regressions. So it's not a bad place to start in explaining NNs. ReLU NN models are piece-wise linear.
pc86 6y ago
> A NN shares as much with the brain, as a decision tree does with a forest: nothing at all. A more accurate statement might be "A NN shares as much with a human brain as a decision tree does with an actual tree: The architecture of the first sometimes appears similar to gross oversimplification of the architecture of the second." Tree:forest is a change in scale not present in NN:brain, and while NN can sometimes look like a very simplified drawing of actual neurons, a decision tree can also look like a very simplified drawing of an actual tree.

bullen 6y ago

I wish someone could take this simple example and add GPU support (and how that can be accomplished best OpenCL/CUDA etc.) just so one could understand how the GPU actually helps accelerate things when the amount of neurons/layers grows? Edit: A bit sad that OpenCL does not work on Jetson Nano and I can't find anything recent that allows Java to run CUDA!? So we're back to AMD? Also "In the next post we will see if adding another layer to our neural network can help in improving the predictions ;)" but there is no next post even after 4 years!

suyash 6y ago
Look up project - JCUDA and this article if you're interested in learning more about that https://blogs.oracle.com/javamagazine/programming-the-gpu-in-java https://blogs.oracle.com/javamagazine/programming-the-gpu-in...
- bullen 6y ago
  Thx, after some more investigating I decided to write my own CPU stuff first then maybe add OpenGL compute shaders if I need them...
  - suyash 6y ago
    Ok, but these are open source projects that companies like Oracle and NVIDIA are workin on which is awesome.
    - bullen 6y ago
      CUDA is not so open-source... and the glue between Java and CUDA seems tricky!
      - suyash 6y ago
        I mentioned JCUDA (Java Bindings for CUDA) which is an open source project https://github.com/jcuda https://github.com/jcuda Here is the tutorial : http://www.jcuda.org/tutorial/TutorialIndex.html http://www.jcuda.org/tutorial/TutorialIndex.html

boyadjian 6y ago

I like very much these simple tutorials, from which you can start something bigger

stfwn 6y ago

The last time a post like this appeared on HN prompted me to write a gist with a simple neural network in Python (with Numpy). It downloads the MNIST dataset for you, trains a fully connected network on it, prints the accuracy on the validation set and plots the loss. It's pretty verbose with plenty of terms and comments to search the web for if you're interested. https://gist.github.com/stfwn/62e51d86ca4ff155becd3c6a14adf60e https://gist.github.com/stfwn/62e51d86ca4ff155becd3c6a14adf6...

eternalban 6y ago
[OT: this is great! https://www.youtube.com/watch?v=UD3AtBm5R7g&feature=youtu.be https://www.youtube.com/watch?v=UD3AtBm5R7g&feature=youtu.be]
- stfwn 6y ago
  Thanks!

suyash 6y ago

I cover this topic quite a lot in my talks - Deep Learning in Java for software engineers. Today you can build pretty much any deep learning model in Java that you can do with Python with availability of plenty of ML & Deep Learning frameworks for Java. If you're interested in this, check out projects like : - https://djl.ai https://djl.ai - https://deeplearning4j.org https://deeplearning4j.org - https://github.com/tensorflow/java https://github.com/tensorflow/java - https://github.com/pytorch/java-demo https://github.com/pytorch/java-demo - https://www.deepnetts.com https://www.deepnetts.com - https://tribuo.org https://tribuo.org - ONNX support for Java - MLFlow for Java https://docs.databricks.com/applications/mlflow/index.html https://docs.databricks.com/applications/mlflow/index.html - SparkMLlib : https://spark.apache.org/mllib/ https://spark.apache.org/mllib/ All of the above listed projects allow one to perform Deep Learning Training and Deployment models into production. Happy to answer any questions on this subject. You can also sign up for my newsletter on this topic https://docs.google.com/forms/d/1oa_TtltDRmnov2bv5Cqo2Hiwfb0lBK8bSrVyPFFn7nw/edit https://docs.google.com/forms/d/1oa_TtltDRmnov2bv5Cqo2Hiwfb0...

TrackerFF 6y ago

Maybe it's just semantics - but I believe that is a perceptron (single-layer NN)

blackbear_ 6y ago
The proper name would be logistic regression, but linear models aren't so sexy anymore nowadays :)

shimonabi 6y ago

I really recommend the book "Make Your Own Neural Network" by Tariq Rashid for beginners. It uses Python and Numpy. I was able to adapt the code for my own mouse-drawn-symbols recognizer for my AI class project.

pmayrgundter 6y ago

Here's another along the same lines, with a demo learning faces according to the Machine Learning book by Tom Mitchell@CMU: https://github.com/pablo-mayrgundter/freality/tree/master/ml/nn https://github.com/pablo-mayrgundter/freality/tree/master/ml...