4 ms·
Interesting, Having done a small amount of machine learning, I can see how the advice here is "true". And by "true", I mean appropriate for the way that machin
by joe_the_user 13y ago
Interesting,
Having done a small amount of machine learning, I can see how the advice here is "true". And by "true", I mean appropriate for the way that machine learning exists and operates in present-day space. Algorithms are difficult, temperamental and requires expert "tuning".
The sequence seems to be:
- First you learn the formal theory, the math and statistics.
- Then you learn the "squinting", the ad-hoc rules for how to apply which algorithm.
- Then implement the thing
This works better than just starting your editor and piecing code. However, I would claim that this doesn't actually work well in the sense that this is kind of where AI/ML have bogged down. I mean, there are only 5 main approaches, 20 main algorithms and whatever subsidiaries and random stuff. They don't work great and the only progress is incremental (though there is progress and throwing more computer power around at the same time enhances - while masking the low amount of conceptual progress).
What's lacking is any modularity in combining algorithms. The power of ordinary programming is, essentially, using function calls to put together what you want. ML doesn't do that and for all the magic, that makes it weak and fragile - when one magic algorithm doesn't work well, rather than improving it, it really is better, at present, in the interest of getting stuff done, to start with a different magic algorithm. This is true, I'm a realistic in the sense of accepting the present but an idealist in the sense of saying "that kind of sucks, we should be able to fix problems, not surrender and regroup".
Yes, I'm happy to denigrate the good and proper in my question for the best. But I'm an idealist, I suppose it's a matter of taste.
- agibsonccc 13y agoI would seriously look in to deep learning. http://deeplearning.net/ http://deeplearning.net/ I am doing everything with it now. This includes principal component analysis/compression, face detection,hand writing recognition, named entity recognition, clustering, topic modeling, semantic role labeling, among other things. There is a very common structure to this. Despite neural nets having their own baggage, they're worth understanding. The structure you're wanting is definitely in there. Edit: Yes a bit of self promotion here. Just making a point with patterns I've found as I've built this out. See: https://github.com/agibsonccc/java-deeplearning/blob/master/deeplearning4j-parent/deeplearning4j-core/src/main/java/com/ccc/deeplearning/nn/BaseNeuralNetwork.java https://github.com/agibsonccc/java-deeplearning/blob/master/... https://github.com/agibsonccc/java-deeplearning/blob/master/deeplearning4j-parent/deeplearning4j-core/src/main/java/com/ccc/deeplearning/nn/BaseNeuralNetwork.java https://github.com/agibsonccc/java-deeplearning/blob/master/... Half the battle is understanding the linear algebra going on here. Beyond that you can pretty much do everything with one set of algorithms and terminology. For those who go WTF java are you insane? The core idea I'm linking to here is the fact that deep nets are composed of singular neural networks with slight variations having a very common structure for both the singular layer as well as the deep nets themselves.
- grovulent 13y agoAdvice question - how far into the linear algebra should I go? I'm currently working my way through a book - and it's not too bad, but I really don't want to invest more than I really need. Any suggestions?
- agibsonccc 13y agoThe main thing to understand is the basics. For example element wise multiplication vs full blown matrix operations. Think something like this: http://en.wikipedia.org/wiki/Matrix_multiplication http://en.wikipedia.org/wiki/Matrix_multiplication Matlab/Octave is a great way to practice this due to the native data types. If python is your thing numpy's arrays are also pretty easy to digest. Subtle little tricks like this: https://www.youtube.com/watch?v=evF-3ykjRU0 https://www.youtube.com/watch?v=evF-3ykjRU0 And understanding the dynamics of scalar operations vs matrix - vector operations. The machine learning class has some good fundamentals if you need a refresher on how something works. There will be more complex things like some optimization algorithms have different uses for eigen values: http://see.stanford.edu/materials/lsocoee364b/11-conj_grad_slides.pdf http://see.stanford.edu/materials/lsocoee364b/11-conj_grad_s... See: http://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors http://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors One other thing might be understanding different ways you can manipulate data. In this case, numerical representation here is an example per row when I toss in one matrix for training. This is applicable to many machine learning problems.
- grovulent 13y agoVery much appreciated. That's really helpful and feels very doable. :)
- fretless 13y agoYou could look at the appendixes in texts by Bishop, MacKay, Barber and Rasmussen/Williams to see what they expect (and they expect a pretty thorough understanding. The last 3 are freely available content http://www.inference.phy.cam.ac.uk/itila/book.html http://www.inference.phy.cam.ac.uk/itila/book.html http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf http://www.gaussianprocess.org/gpml/chapters/RW.pdf http://www.gaussianprocess.org/gpml/chapters/RW.pdf Also, an excellent list of ML resources: http://www.reddit.com/r/MachineLearning/comments/1jeawf/machine_learning_books/ http://www.reddit.com/r/MachineLearning/comments/1jeawf/mach...
- allenlavoie 13y agoThis is essentially the goal of probabilistic programming[1]: let the programmer specify and tweak the model, and have the analog of a compiler handle inference. Finding a good model is then analogous to debugging. You are mostly stuck with Bayesian models, since that's what we have general-purpose inference algorithms for. And in practice an understanding of Bayesian statistics is a prerequisite for writing useful probabilistic programs. [1] e.g. http://probabilistic-programming.org/ http://probabilistic-programming.org/
- jlees 13y agoWell, you can kind of use decision forests, and similar approaches, to combine algorithms. The lack of human-readability makes it pretty tough unless you're in a domain where you can effectively apply weighted heuristics (which I found to work pretty well in NLP, albeit with a lot of experimental tweaking to arrive at the right model for my domain).