8 ms·
Neural Network Architectures
- partycoder 10y agoNot directly related, but never a time waste: http://www.scholarpedia.org/article/Encyclopedia:Computational_neuroscience http://www.scholarpedia.org/article/Encyclopedia:Computation...
- rustyfe 10y agoCould anyone recommend a starting point on Neural Networks for the uninitiated? The parts of this I understood were fascinating, but I quickly realized I was looking up every third word, and not really absorbing much. If I could only read one thing to gain the technical grounding for this history, what should it be?
- deleted 10y ago[deleted]
- llSourcell 10y agohttps://www.youtube.com/watch?v=h3l4qz76JhQ https://www.youtube.com/watch?v=h3l4qz76JhQ
- billconan 10y agothis should get you started: http://neuralnetworksanddeeplearning.com/ http://neuralnetworksanddeeplearning.com/
- trophygeek 10y agoThis was the one that flipped the lightbulb for me. `Hacker's guide to Neural Networks` http://karpathy.github.io/neuralnets/ http://karpathy.github.io/neuralnets/
- trophygeek 10y agoActually, read that one 2nd. Start here: https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/ https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convn...
- rustyfe 10y agoHey, the premise was I could only read one thing! I kid, much obliged.
- trophygeek 10y agolol. point Read the 2nd one then.
- WhoBeI 10y agoThat one was very good and links to a nice demo http://scs.ryerson.ca/~aharley/vis/conv/flat.html http://scs.ryerson.ca/~aharley/vis/conv/flat.html
- eddiepierce 10y agoKarpathy's course notes for Stanford's Convolutional Neural Networks for Visual Recognition is a great intro [1]. It starts with with linear classification, then moves to neural nets, and then explains convolutional neural nets. [1]: http://cs231n.github.io http://cs231n.github.io
- ramblenode 10y agoThe other suggestions in this thread are quite good. I'll add "Machine Learning" by Murphy. It's not strictly about neural networks but it's an ML classic and a rigorous introduction to the subject that will give you a principled understanding of the statistical fundamentals. For actual NN implementation the Karpathy and Nielson sources are excellent.
- p1esk 10y agoDefinitely not the starting point for someone who has no clue. Murphy is probably the very last thing one would read before becoming an expert, and start publishing ML papers.
- imh 10y agoThis course is a great first step in learning about neural nets: https://www.cs.ox.ac.uk/people/nando.defreitas/machinelearning/ https://www.cs.ox.ac.uk/people/nando.defreitas/machinelearni...
- j_juggernaut 10y agoHow about the textbook http://www.deeplearningbook.org/ http://www.deeplearningbook.org/. It introduces you to some of the underlying principles which haven't changed much over time. I highly recommend it if you want to get deeper intuitions on the principles of CNN, LSTM/RNN, Restricted Boltzmann Machines etc. Also, Hinton's Coursera lectures, though not sure if you can access it anymore.
- Joof 10y agoIt was on academic torrents I believe.
- vonnik 10y ago[Disclosure: I wrote this.] http://deeplearning4j.org/neuralnet-overview.html http://deeplearning4j.org/neuralnet-overview.html Also, this book is coming: https://www.amazon.com/Deep-Learning-Practitioners-Adam-Gibson/dp/1491914254/ https://www.amazon.com/Deep-Learning-Practitioners-Adam-Gibs...
- taliesinb 10y ago> The NiN architecture used spatial MLP layers after each convolution, in order to better combine features before another layer. Again one can think the 1x1 convolutions are against the original principles of LeNet Why would it be against the original principles of LeNet?
- troyastorino 10y agoAs far as I understood from the description, in LeNet the convolutional layer lets you avoid training parameters that will effectively be doing the same thing as a convolution. Adjacent pixels are highly correlated, so convolutions can capture most of the information in groups of adjacent pixels without having to train a fully-connected layer of neurons. Effectively, you're kinda downsampling the image without losing information. So, if you're using 1x1 convolutions, I think you're basically having a neuron per pixel, so you're forcing your fully-connected layers to learn the spacial correlations of pixels, instead of capturing that information in a convolutional layer. In other words, you're wasting training on capturing spacial correlations of adjacent pixels instead of other correlations.
- taliesinb 10y ago> So, if you're using 1x1 convolutions, I think you're basically having a neuron per pixel, so you're forcing your fully-connected layers to learn the spacial correlations of pixels, instead of capturing that information in a convolutional layer. Saying "a neuron per pixel" doesn't mean anything, really, that way of thinking isn't helpful unless you're looking at small multi-layer perceptrons. The right way to think about things is that you have tensors and layers that compute new tensors from old tensors. A 1x1 convolution only 'sees' the feature channels of a pixel, and does the same thing to each pixel. So a 1x1 convolution on a grayscale input (e.g. a 1x28x28 tensor in the case of MNIST) does nothing, basically, other than scale and bias every pixel by the same linear function. It doesn't "force the network to learn" anything, it's just totally pointless. One of the uses of 1x1 convolutions is to collapse the feature dimension when you're deeper in the network (e.g. 100 channels to 10 channels) to reduce number of parameters subsequent layers need operate on. It's a "channelwise fully connected layer". I think you're thinking of (and perhaps what the author was thinking of) is the practice prior to convnets of collapsing the image into a vector and then doing a fully connected layer on it. That indeed doesn't exploit translation invariance of natural images, requires the net to learn the same features in every required spatial position at great expense, and so on. But that has nothing to do with 1x1 convolutions.
- phodo 10y agoCheck out: http://lazyprogrammer.me http://lazyprogrammer.me [his courses and books are accessible, good, and relatively cheap]
- phodo 10y agoOops that was meant as a reply to "rustyfe" on a starting point resource! Can i edit the response?
- gok 10y agoA more accurate title might be "Convolutional Neural Network Architectures" or "Neural Network Architectures for Computer Vision" but still a nice overview!
- AgentME 10y agoOne thing I'm confused about is that everyone seems to treat "Convolutional Neural Networks" as synonymous with or as being the thing that enabled "Deep Learning", but convolutional neural networks are only for image processing, right? Are many layer ("deep") networks useless outside of image processing? Were there other break-through techniques besides convolutional nets that are necessary for deep networks to work well?
- gok 10y agoConvolution neural networks and many-layered networks are useful for things outside of image processing. CNNs are used for acoustic modeling in speech recognition, and character-convolutional layers are used in language modeling. And pretty much all neural networks in use today anywhere are many layered. As mentioned in the article, using convolutional layers in ANNs was an idea from the 1980s, but networks that could be trained on the hardware available at the time were never all that competitive until recently. Once we figured out how to train big/deep networks (use GPUs, have lots of data, maybe use pre-training), CNNs started to perform really well. This did make a positive feedback loop: as CNNs started to work better, deeper networks in general started to get more attention, which got more people into CNNs, etc.
- Question1101 10y agoIsn't data and processing power the most important thing with neural networks? Even if I knew how they worked I would have no idea what to do with them that hasn't been done already as a hobbyst without access to huge amounts of data like companies do.
- eegilbert 10y agoYes. This is often, perhaps willfully given the incentives, overlooked by computer scientists (and people trained in that discipline). For a complete, cogent argument see: http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf http://static.googleusercontent.com/media/research.google.co...
- naveen99 10y agoData is cheap and all around us. Processing power yes, but you can get started with a gaming pc.
- p1esk 10y agoNo. As a researcher, you can make it your goal to find/invent a smallest possible architecture for a given task (in terms of number parameters, or number of operations). Alternatively, you can try to invent an architecture to learn faster from data (or require less data to achieve state of the art results).
- tree_of_item 10y ago> As a researcher But the post you replied to specifically said "as a hobbyist", so it doesn't really sound like there's much hope.
- Joof 10y agoHobbyist researcher? Seems like a more accessible plan anyway; fuck bothering with huge datasets and long training times and just focus on optimizing small architecture.
- 10y ago