3 ms·
A few comments on some of these projects: Keras is pretty much the best way to do almost anything these days. If you are starting out learning, use ConvNet JS,
by nl 10y ago
A few comments on some of these projects:
Keras is pretty much the best way to do almost anything these days. If you are starting out learning, use ConvNet JS, but after that switch to Keras.
TFLearn is really nice if you are already using Scikit.
There's lot of frameworks on there: TensorFlow, Caffe, CNTK (that's a lot of stars for something no one outside MS uses!) Theano, Torch etc. But I think the sleeper there is MXNet. I haven't used it, but I hear good things about it, and DLMC have good track record in producing some pretty nice software (XGBoost!).
Also, DeepDetect! I keep trying to find that and never can remember the name.
- pilooch 10y ago> Also, DeepDetect! I keep trying to find that and never can remember the name. Ah, thanks for the reference, DD author here, the funny fact about the name is that translated to my mother tongue (French), it sounds like porn ;)
- denzil_correa 10y ago> Keras is pretty much the best way to do almost anything these days What makes Keras take the advantage?
- stephanheijl 10y agoVery extensible API, accessible and widely used programming language (Python), the ability to use both Theano and Tensorflow as a backend and easy to implement non-linear neural networks (where data is split and merged at will) all contribute to this. Using Keras means you will almost never need to implement some custom layer or function, whilst sacrificing very little performance-wise.
- nl 10y agoIt's a well designed API for using deep neural networks rather than an API for doing optimized mathematical operations. Compare how you build some vaguely comparable models in Keras[1] and raw TensorFlow[2]. Keras uses TensorFlow (or Theano) underneath, so there is no performance penalty. It's like in Python machine learning, most people use Scikit instead of implementing things in numpy. [1] https://github.com/fchollet/keras/blob/master/examples/mnist_mlp.py#L39 https://github.com/fchollet/keras/blob/master/examples/mnist... [2] https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/mnist/mnist_softmax.py#L39 https://github.com/tensorflow/tensorflow/blob/master/tensorf...
- denzil_correa 10y agoThanks! Your example to explain the differences makes me more leaning towards Keras. It feels like Scikit for Deep Learning.
- nl 10y agoYes, that's a decent analogy. If you are already familiar with Scikit then TFLearn is worth looking at too.
- chestervonwinch 10y agoTheano itself is more like a language, not a deep learning framework. There is no NeuralNetworkClassifier class, for example. Although, you could write a neural network library / framework using Theano, and it would have all the benefits of Theano (code compiled for the GPU, various common neural net ops available, etc.), which is what it looks like the Keras folks have done. I took a stab at this a while ago (1), but I didn't keep up on it. I haven't used Keras much, but it looks like it fills a much needed gap, which I'm glad for. (1): https://github.com/notmatthancock/neural_network https://github.com/notmatthancock/neural_network
- cedricblack 10y agoI ported ConvNet JS to C# in order to really understand what's going on: https://github.com/cbovar/ConvNetSharp https://github.com/cbovar/ConvNetSharp
- redcalx 10y agoBrilliant, I've been looking for projects like this. I'm currently working through a couple of RBM C# projects but will add this to my list of reference code. Top tip: If you use the matrix and vector classes in math.net then you can optionally configure it to use optimised version of e.g. matrix multiplication, that map through to one of the providers, such as Intel Math Kernel Lib, OpenBLAS, and I think there's a CUDA provider too. I tried the Intel MKL one and the dense matrix multiplication was about 60x faster than a plain C# version. https://github.com/mathnet/mathnet-numerics https://github.com/mathnet/mathnet-numerics https://www.nuget.org/packages/MathNet.Numerics/ https://www.nuget.org/packages/MathNet.Numerics/
- cedricblack 10y agoThanks for the tip. I'll see where I can apply it. Most of the time is usually spent in the convolution layers. Convolution is not a matrix multiplication in the current implementation. I guess it would be a matrix multiplication in frequency domain or by using a Toeplitz matrix. I've implemented a CPU Parallel version and gave a try at GPU implementation. But I'm not satisfied at all by the GPU version :) https://github.com/cbovar/ConvNetSharp/blob/master/src/ConvNetSharp/ConvLayer.cs#L54 https://github.com/cbovar/ConvNetSharp/blob/master/src/ConvN... https://github.com/cbovar/ConvNetSharp/blob/Gpu/src/ConvNetSharp/ConvLayer.cs#L88 https://github.com/cbovar/ConvNetSharp/blob/Gpu/src/ConvNetS... Pull requests more than welcome!
- redcalx 10y ago> Convolution is not a matrix multiplication in the current implementation I figure there's a code re-organisation task since propagating node activations through a layer of weights is essentially a matrix multiplication (fully connected => fully dense matrix). The optimised routines make use of vectorised CPU instructions and the FMA instruction (fused multiply and add), all of which are perfect fits for [dense] matrix multiplcation. Not so great for sparse matrices, but they help, usually unless it's very sparse it's faster to use a dense matrix format with zeros for the missing weights. > Pull requests more than welcome! Duly noted :)
- IshKebab 10y agoCNTK is actually pretty damn good. It just lacks a good scripting interface. They're adding one though. The network description language (now 'BrainScript') is a far nicer way to specify networks than the approaches used by any other network. Especially for recurrent networks. In CNTK you can just say `X = FutureValue(Y)` or `X = PastValue(Y)`. It's so convoluted in TensorFlow I actually never worked out how to do it. It also has their fancy 1-bit SGD stuff, but I doubt many people use that, and it has a more restrictive license anyway.
- Eridrus 10y agoKeras support for recurrent models leaves a bit to be desired at this point, so it's great if it has what you want, but otherwise you have to start peeking under the hood, which may be harder than just learning the underlying framework.
- shostack 10y agoWhen you say starting out, do you mean a true beginner to machine learning, or someone new to these frameworks?