8 ms·
Machine learning systems are stuck in a rut
- deleted 7y ago[deleted]
- Arbalest 7y agoIs the fix here really just, lets build more infrastructure? If so, who is going to fund it? AI/ML is already mostly a marketing term at this stage, it's all just people who want results now now now. Maybe this is the reason that AI winters have occured: Companies expect all this stuff to be plug and play, but it really isn't. You've got to put in the hard yards, and that means the dollars.
- deleted 7y ago[deleted]
- AstralStorm 7y agoWere it just dollars. Actual breakthroughs are not exactly predictable.
- Arbalest 7y agoFair enough, I was going to say time as well, but money is time as well, just means you have to multiply it a bit more.
- jsty 7y ago> If so, who is going to fund it? Considering the difficulty outlined in the article is in optimising the more 'cutting-edge' approaches (i.e. those without manually optimised kernels), there are numerous AI research centres (in industry and academia) that would be quite keen to reduce the compute cost of exploring new approaches / architectures. The vast majority of companies applying ML are unlikely to step outside of the "mainstream" in terms of model architecture (at least currently), and thus the conclusions aren't particularly relevant to them.
- DEADBEEFC0FFEE 7y agoNot rut, local minima, surely.
- beobab 7y agoWhat's the difference?
- stevesimmons 7y agoit's a ML joke...
- quickthrower2 7y agoOne makes a more clickable article title.
- gwern 7y agoBut surely a rut is a saddlepoint in 3 dimensions?
- angel_j 7y ago4. Spinning in place...
- eitland 7y agoIt's worse [0]. I have some fantastic recommendations from Google Now and recently from Amazon as well :-/ Here is one brilliant example of something that must be a machine learning thing that has gotten a little too much freedom: http://erik.itland.no/more-fun-with-google-mixing-images-from-different-sources http://erik.itland.no/more-fun-with-google-mixing-images-fro... If you are lucky you can still see them for yourself by searching Google for mat-table. (It is a component of Angular Material, which the AI/ML thing kind of realizes but not without fuzzing it to include an actual table with a mat on as well :-) Edit: meanwhile, for the first time I can remember for years Google isn't pushing some dumb dating site front and center but rather text ads for some things that could be relevant for a software engineer with a lovely wife and small kids: mule integration, robotics stuff and holiday suggestions. Congrats to whoever managed to convince their boss (or the AI) to try some other options. I’d also be delighted if next year I’d get a well placed ad for a couple of the conferences I missed out on this year. Edit 2: Finally got around to posting a couple of screens showing off Amazons understanding of Law books, Engineering books and Humor and Entertainment: https://erik.itland.no/fun-with-amazons-ai-machine-learning https://erik.itland.no/fun-with-amazons-ai-machine-learning [0]: since AI is now making things actively worse.
- YeGoblynQueenne 7y agoThat's it? We need better frameworks? I thought this was going to be about this "stuck in a rut": GH: One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas. Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad. What we should be going for, particularly in the basic science conferences, is radically new ideas. Because we know a radically new idea in the long run is going to be much more influential than a tiny improvement. That's I think the main downside of the fact that we've got this inversion now, where you've got a few senior guys and a gazillion young guys. https://www.wired.com/story/googles-ai-guru-computers-think-more-like-brains/ https://www.wired.com/story/googles-ai-guru-computers-think-... Which is the granddaddy of the other one. You don't really need better frameworks that make it easier to explore innovative ideas if you can't ever hope to publish those innovative ideas, even if you manage to make them work. NB: "GH" is Geoff Hinton.
- mantap 7y agoDon't people just read the arXiv? I think Hinton is being a bit old fashioned here. A paper doesn't need to be published to a journal at all to be influential.
- YorkshireSeason 7y agoHow do you filter arXiv? There's way too many papers on arXiv, it's impossible to keep track of everything, especially since the quality on arXiv is ... heterogenous.
- 7y ago
- tim_sw 7y agoProbably bec of tensorflow, it’s optimized for deep learning and encourages certain kinds of models
- IshKebab 7y agoI definitely agree that if you stray outside the "traditional" CNN, LSTM, etc. it is very hard to implement custom layers in most frameworks. Especially if they are recurrent. I tried to implement a custom RNN in Tensorflow and gave up. All of the documentation is "just call tf.Lstm()" or whatever. The exception is CNTK. That makes it really really easy to describe any network including recurrent ones. Sadly it doesn't seem to have caught on at all.
- acollins1331 7y agoTensorflow has pretty poor documentation. Looks like you've been reading the higher level stuff. It's pretty easy to make custom layers and such with tf if you read some of the source. Though, honestly, numpy might be the simplest method to create a custom layers inside an architecture. If you use TF 2.0 this is simple because it doesn't even have to be part of the graph, per se.
- sdenton4 7y agoYou should be able to just inherit from the rnn layer interface, and then use it directly. https://www.tensorflow.org/api_docs/python/tf/keras/layers/RNN https://www.tensorflow.org/api_docs/python/tf/keras/layers/R...
- orbifold 7y agoI agree that it is super painful and confusing to implement RNNs in TensorFlow and had a similar experience starting out. You should take a look at https://www.tensorflow.org/api_docs/python/tf/scan https://www.tensorflow.org/api_docs/python/tf/scan, that is one of the easier ways of implementing one I think. Other than that Pytorch is also great for experimentation and even has a nice example https://pytorch.org/tutorials/advanced/cpp_extension.html https://pytorch.org/tutorials/advanced/cpp_extension.html of implementing a variant of an LSTM. Contributing to this problem is that there is a ton of low quality blog posts on all of these topics.
- cs702 7y agoDirect link to the Google Brain paper, which is well worth a read: https://dl.acm.org/citation.cfm?id=3321441 https://dl.acm.org/citation.cfm?id=3321441 (click on "PDF" link to read) Abstract: "In this paper we argue that systems for numerical computing are stuck in a local basin of performance and programmability. Systems researchers are doing an excellent job improving the performance of 5-year-old benchmarks, but gradually making it harder to explore innovative machine learning research ideas. We explain how the evolution of hardware accelerators favors compiler back ends that hyper-optimize large monolithic kernels, show how this reliance on high-performance but inflexible kernels reinforces the dominant style of programming model, and argue these programming abstractions lack expressiveness, maintainability, and modularity; all of which hinders research progress. We conclude by noting promising directions in the field, and advocate steps to advance progress towards high-performance general purpose numerical computing systems on modern accelerators." The main example the authors use to illustrate these issues is capsule networks, first proposed two years ago.[a] To date, no one has been able to develop a high-performance implementation of capsule networks. At present, the best-performing implementations in Tensorflow and PyTorch must copy, rearrange, and materialize to memory two orders of magnitude more data than necessary, due to the issues raised by the authors. See sections 1 and 2 of the paper for the gory details. Two orders of magnitude. That is pathetic. [a] https://arxiv.org/abs/1710.09829 https://arxiv.org/abs/1710.09829
- cr0sh 7y agoNote: I am not an expert on ML, ANNs, etc - most of what I have done has been mainly "hobby level" and MOOC learning. From what I understand, though, what GH is trying to accomplish with capsule networks (which I have tried to understand, and have yet to succeed) is optimizing (or possible removal of) backpropagation. He has noted in the past how - so far (from what I know) - there isn't a biological equivalent to backpropagation for learning. Backprop is a completely artificial mathematical construct that doesn't happen in natural systems. It is also extremely energy intensive - at least in the manner that it is currently done. So the question is - that he's hoping to answer I think - what is a proper working framework for implementing an artificial neural network that can learn, without using backpropagation (or using it differently)? I think whoever can solve this will fundamentally remake the field of ML/AI - much in the same way that backprop (and later "deep learning") did.
- Farradfahren 7y agoCan a NN be crafted that behaves like a learning lifefform? 1) Generate Noise with a actor and a sensor. 2) Classify Noise reaction of the system into reproduceable and chaotic responses. 3) Cull all chaotic responses, but re-evaluate ocassionally against a simplified model for hidden reproduceablity. 4) Repeat algorithm for a reproduceable system with a diffrent (more fine-tuned) actor and (same/diffrent) sensor until nothing new is learned. If nothing new is learned, re-evaluate chaotic responding systems.
- PaulHoule 7y agoIt is not just academia that is stuck with systems that are fast as some things and slow at other things. Branching, pointer indirection, and variable length data structures all kill performance. That is where the serialization tax comes from. Arrays of numbers can be loaded into RAM, even memory mapped, and be used right away. Commercial products are very much limited by what we know how to make fast.
- platz 7y agoI think languages like http://unisonweb.org/posts/ http://unisonweb.org/posts/ are a bit early, but in a couple decades there could be a strong institutional push to languages with features like it.
- angel_j 7y agoI have it all figured out, not gonna lie.
- Tarq0n 7y agoIsn't this the kind of application where Julia would shine? If you're trying to develop something new or cutting edge it makes sense that frameworks don't necessarily suit you. Frameworks try to make common actions easy (provide a 'happy path'), very much the opposite of developing something new. Something like Julia which offers useful primitives, but can also let you compile general purpose code for CUDA seems like a better fit, rather than expecting frameworks to cater to you.
- cs702 7y agoThe authors of the paper discuss how Julia fares with capsule networks in section 4.1:[a] > There are frameworks, such as Julia, which nominally use the same language to represent both the graph of operators and their implementations, but back-end designs can diminish the effectiveness of such a front end. In Julia, while 2D convolution is provided as a native Julia library, there is an overloaded conv2d function for GPU inputs which calls NVidia’s cuDNN kernel. Bypassing this custom implementation in favor of the generic code essentially hits a “not implemented” case and falls back to a path that is many orders of magnitude slower. [a] The paper uses capsule networks as its main example. See my earlier comment for context and a link to the paper: https://news.ycombinator.com/item?id=20304576 https://news.ycombinator.com/item?id=20304576
- ChrisRackauckas 7y agoJulia makes this substantially easier, but yes in many cases you might need to come up with a cache-optimized tensor operation kernel. The Julia Lab is building some tooling for the automatic construction of this stuff which is compatible with the AD frameworks, kind of like Halide, but in the Julia language and compatible with all of its generics. The real key though is that packages play nicely in the Julia sphere, so if someone writes a good Julia code for the operation and puts a package up, then you can take that package and utilize it in Flux even if the person did not write for ML purposes. There's a lot of quantum particle folks writing esoteric tensor operations in Julia that I personally have been pulling stuff from.
- 7y ago
- gambler 7y agoYes. If you have an idea about machine learning that doesn't fit the mold, you're pretty much forced to write your own mini-framework before trying it. And I'm not even talking about CUDA compilation. Mainstream languages don't have syntax for trivial math like graphs and matrices. Even 2d array support is horrible everywhere. Try to write the following algorithm: 1. Given an image in greyscale, chose two n*n regions at random. 2. Compute the median difference between pixels in those regions. 3. Add both regions to a graph (unless they are already there) and set the edge value to that difference. Trivial to visualize in your head, a nightmare to implement in most languages. And what if you want to parallelize this to work on large amounts of data? I have hopes that AMD's monster processors will make experimentation a little bit more practical, since you wouldn't be in such a desperate need to push everything onto GPU.
- ilaksh 7y agoThis does seem to be a big problem. More than just frameworks not being amenable to alternative approaches, people just don't know than non-mainstream approaches exist. But if they find out about something different and try to use it, good luck. It's not impossible though. For example, I have been thinking about learning OpenCL. And there are other GPGPU approaches like CUDA and a lot of less popular stuff.