16 ms·
The Unreasonable Effectiveness of Recurrent Neural Networks
- jgmmo 11y agoVery neat, and funny article. I love the PG generator.
- tshadwell 11y agoI'm not sure how "unreasonable" the effectiveness of RNNs are if the corpus output at 2000 iterations isn't significantly better than a simple prefix based markov chain implementation [1] (and for the regular languages, with some extra bracket-checking), but I found the evolution visualizations really interesting. [1] http://thinkzone.wlonk.com/Gibber/GibGen.htm http://thinkzone.wlonk.com/Gibber/GibGen.htm
- darkmighty 11y agoIt's quite unreasonable. He could have optimized it more for fooling humans in Gibberish generation, but that would not show the general effectiveness of the approach. The power shows (quantifiably) in compression: 1.57 bits per character wikipedia is quite hard to beat. Of course, Markov Chains are essentially universal models, so the training algorithm is the crucial distinction. I believe Markov Chains as a model quickly become inefficient (specially memory-wise) as you increase the complexity (long range correlations) of your prediction. It's an unnecessarily restrictive model for high complexity behavior that state of the art RNNs skip entirely.
- new299 11y agoThe state of the art in compressing wikipedia is 1.278bits (on a certain subset) [1]. So that does seem pretty good. [1] http://prize.hutter1.net/ http://prize.hutter1.net/
- jsprogrammer 11y agoExcept this NN isn't really a compression of Wikipedia since it can only generate Wikipedia-like nonsense.
- gwern 11y agoIt's a compression of Wikipedia in the sense that the NN generates probability estimates of the next character given the previous; the gibberish is simply greedily asking the NN repeatedly what the most-likely next character is. However, plug it into an arithmetic coder and start feeding in an actual Wikipedia corpus, and hey presto! a pretty high performance Wikipedia compressor, which works well on Wikipedia text but not so well on other texts (like this one, with its lack of brackets).
- teraflop 11y agoThere's very little difference between a contextual predictive model like this and the guts of a compressor. If your prediction is good enough that you can always come up with two possible predictions for each character, each of which has a 50% chance of being correct, then obviously you can compress your input down to one bit per character by storing just enough information to tell you which choice to pick. More generally, you can use arithmetic coding to do the same thing with an arbitrary set of letter probabilities, which is exactly what you get as the output of a neural network. When the blog post says the model achieved a performance of "1.57 bits per character", that's just another way of saying "if we used the neural network as a compressor, this is how well it would perform."
- jsprogrammer 11y agoI'd be interested in seeing this NN perform a lossless compression of Wikipedia at 1.57 bits per character.
- murbard2 11y agoIt balances parentheses and keeps track of other long range dependencies, something markov chain implementations cannot do.
- deleted 11y ago[deleted]
- pohl 11y agoWelcome to the unbearable forced-ness of titles. Everyone's making a nod to Milan Kundera these days.
- jsprogrammer 11y agohttp://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences http://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_...
- coldtea 11y agoFirst, it's not a nod to Kundera, but to a classic math related work that predates Kundera's book. Second, even if it was, really? As if we see plays on Kundera titles regularly on the web?
- tim333 11y agoI doubt it's a reference to Kundera. I was thinking that both Eugene Wigner's 1960 article 'The Unreasonable Effectiveness of Mathematics in the Natural Sciences'[0] and Karpathy's 'The Unreasonable Effectiveness of Recurrent Neural Networks' probably touch deep aspects of the nature of existence. The first on why the universe exists and is mathematical - because at the fundamental level it is mathematical[1], and in Karpathy's case the RNNs are probably effective because they are close to the mechanisms of human consciousness. [0] Wigner's article: http://www.dartmouth.edu/~matc/MathDrama/reading/Wigner.html http://www.dartmouth.edu/~matc/MathDrama/reading/Wigner.html [1] 'physical world is completely mathematical' theory: http://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences#Max_Tegmark http://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_...
- samizdatum 11y agoThat Markov Chain model operates on 4-grams by default. The RNN featured in the article generates output character-by-character, which is significantly more impressive. Here's a sample from the Markov Chain model operating on 4-grams: Ther deat is more; for in thers that undiscorns the unwortune, the pangs against a life, the law's we know no trave, the hear, thers thus pause. The only reason why it seems like the model can occasionally spell, and create anglo-sounding neologisms, is because it operates on 4-grams. Here's some character-by-character output from the same Markov Chain model. T,omotsuo ait pw,, l f,s teo efoat t hoy tha fm nwo bs rs a h enwcbr lwntikh wqmaohaaer ah es aer mkazeoltl.etnhhifcmfeifnmeeoddssmusoat irca do'ltyuntos sih i etsoatbrbdl
- 0xdeadbeefbabe 11y agoI'm getting the funny impression that what distinguishes an algorithm from an AI algorithm isn't about the algorithm, but how people treat the algorithm. It's an AI algorithm if they describe it behaving intelligently i.e. painting numbers on a house, learning english first, being born, being tricked into painting a fence, etc. Otherwise its just an algorithm.
- deleted 11y ago[deleted]
- Jtsummers 11y agoThis is an old problem in AI. Chess was an AI problem, until a computer beat a grandmaster. Vision was an AI problem, now we have OpenCV. Many AI problems get shifted out of "AI" once they're solved.
- dumitrue 11y agoI don't think OpenCV really solved computer vision to be fair. There's definitely no model out there that can do image-based question & answering as well as a human can, or interpret the contents of an image (parse it, if you will) in an accurate way, with the exception of very few special cases.
- im3w1l 11y agoOne explanation for this could be that we think that some problem is so hard that any solution to it is necessarily so complicated that it could be adapted to solve pretty much anything. When we realize that that isn't the case, we stop calling it AI.
- TheLoneWolfling 11y agoIt stems from our definition of an AI. An AI is a computer doing those things a computer cannot do. As such, anything that a computer cannot do isn't AI, and anything a computer can do isn't AI either.
- 11y ago
- pcmonk 11y agoThe code generator is awesome. There's hardly a syntax error. The file headers are the best. Nitpick: although tty == tty is, as you say, vacuously true in this case, that's just because tty is a pointer. If tty were a float, this wouldn't be the case, since it could be NaN. I wouldn't be surprised if it learned to test a variable for equality against itself from some floating point code.
- joelthelion 11y agoCould be interesting to plug this kind of generator into American Fuzzy Lop.
- zxyzzxxx 11y agoThe code is nonsense. Their method is good for fuzzy logic like recognition, but this approach with code will never work for anything other than an art project.
- teraflop 11y agoCurrently it doesn't work, but saying it'll never work is pretty strong. This kind of demo shows that deep neural networks can capture the structure of language, if not the semantics, in a very general way. And we have separate evidence that they can (in principle) capture semantic meaning and algorithmic reasoning as well, for example: http://arxiv.org/pdf/1410.5401v2.pdf http://arxiv.org/pdf/1410.5401v2.pdf (the "neural Turing machines" paper from DeepMind)
- kpil 11y agoThis is better, but you get pretty far with just markov chains with probabilities for letters actually.
- mikkom 11y agoShow me markov chain implementation that can write code letter by letter and I'll give you a car. (And I mean plain markov chain, not something with additional logic that understands code structure) comment by samizdatum shows pretty well how well markov chains work without some tweaking.
- fpgaminer 11y agoI'm in the middle of reading this article (very much appreciate Karpathy's writings), but I also wanted to brain dump some of my musings on modern machine learning; RNNs in particular. Sorry if this is redundant to anything the article talks about. Deep learning has made great strides in recent years, but I don't think architectures which aren't recurrent will ever give rise to mammalian "thought". In my opinion, thought is equivalent to state, and feed forward networks do not have immediate state. Not in any relevant sense. So therefore they can never have thought. RNNs, on the other hand, do have state, and therefore are a real step towards building machines that posses the capacity to think. That said, modern deep learning architectures based around feed forward networks are still very important. They aren't thinking machines, but they are helping us to build all those important pre-processing filters mammalian brains have (e.g. the visual cortex). This means we won't have to copy the mammalian versions, which would be rather tedious. We can just "learn" a V1, V2, etc from scratch. Wonderful. And they'll be helpful for building machine with senses different than biology has yet evolved. But, again, these feed forward networks won't lead to thought. My second musing is where I think the next leap in machine learning will occur. To-date efforts have been focused on how to build algorithms that optimize the NN architecture (i.e. optimize weights, biases, etc). But mammalian brains seem to posses the ability to problem solve on the fly, far faster than I imagine tweaks to architecture could account for. We solve problem in-thought, rather than in-architecture; we think through a problem. Machine Learning doesn't posses this ability. It can only learn by torturing its architecture. So, I believe there is this distinction to the learning that mammalian brains are able to do on the fly, using just their thoughts, and the learning they do long term by adjusting synaptic connections/response. It seems as if they solve a problem in the short term, and then store the way they solved it in the underlying architecture over the long term. Tweaking the architecture then makes solving similar problems in the future easier. The synaptic weights lead to what we call intuition, understanding, and wisdom. They make it so we don't have to think about a class of problems; we just know the solutions without thought. (Note how I say class of problems; this isn't just long term memory). Along those lines, I come to my final musing. That mammalian brains are motivated by optimization of energy expenditure. Like anything in biologically evolved systems, energy efficiency is key, since food is often scarce. So why wouldn't brains also be motivated to be energy efficient? To that end, I believe tweaking synaptic weights, that kind of learning that machine learning does so well, is a result of the brain trying to reduce energy expenditure. Thoughts are expensive. Any time you have a thought running through your brain, it has some associated neuronal activity associated with it. That activity costs energy. So minimizing the amount we have to think on a day-to-day basis is important. And that, again, is where architecture changes come in. They are not the basis for learning; they are the basis for making future problem solving more efficient. Like I said, once a class of problems has been carved into your synaptic weights, you no longer have to think about that class of problems. The solutions come immediately. You don't think about walking; you just do it. But when you were a baby, I'll bet the bank that your young mind thought about walking a lot. Eventually all the mechanics of it were carved into your brain's architecture and now it requires many orders of magnitude less energy expenditure by your brain to walk. So, the obvious question is ... how do mammalian brains problem solve using just thoughts. The answer to that, as I mentioned, is likely to lead to the next leap in machine learning. And it will, more likely than not, come from research on RNNs. What we need to do is find a way to train RNNs that are able to adapt to new problems immediately without tweaking their weights (which should be a slower, longer term process). P.S. Yes, I know this was probably a bit off-topic and quite a bit wandering. I've had these musing percolating for awhile and don't really have an outlet for them at the moment. I hope it's on topic enough, and at least stimulates some interesting discussion. Machine learning is fascinating.
- waterlesscloud 11y agoSide note: The title is in reference to this famous paper from 1960- http://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences http://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_... The form of the title has become a common trope.
- cs702 11y agoNice. Andrej Karpathy deserves some kind of award for demystifying deep learning and making the subject so accessible to a wider audience. If you're a developer who knows little about the subject and want to learn more, a great starting point is the home page for his ConvNetJS project.[1] -- [1] http://cs.stanford.edu/people/karpathy/convnetjs/ http://cs.stanford.edu/people/karpathy/convnetjs/
- choppaface 11y agoAnd if you're more comfortable with Python, I strongly recommend the CS231n assignments / labs: http://cs231n.github.io/ http://cs231n.github.io/ Assignments 1 and 2 alone give a solid intro to implementing these algorithms, and the lab-oriented iPython-based format gives you a very high probability of writing a correct implementation even if you're clueless at the start.
- myth_buster 11y agoThis is quite incredible. The stylistic similarities of generated Shakespearean saga, Linux code etc was quite startling. Perhaps we can train a Haiku/Fortune cookie generator which could occasionally be quite profound.
- speechduh 11y agoThat particular stuff is actually pretty typical. I have a textbook that shows similar results on Shakespeare using N-grams from years ago.
- gipp 11y agoCapturing writing style with ngram-based input and individual-character input are very, very different tasks. That's several ballparks higher in difficulty. With ngrams, Markov models are perfectly sufficient. With individual characters, complex concepts need to be remembered across many, many characters of input.
- seiji 11y ago> Linux code etc People are always worried about "computers taking factory jobs" resulting in mass unemployment, but the truth is, a rudimentary AI with acceptance tests on output will obsolete every programmer alive. Hell, half the programming people do these days is just gluing APIs together then seeing if it actually works. It doesn't take 16 years of rich inner human life experience to accomplish that, just exhaustive combinational parameter searching on the subset of API interactions you're interested in evaluating.
- myth_buster 11y agoDouglas Crockford touches on this aspect in this entertaining and insightful talk [0]. I'm guilty of what you state and I think a large part of "programming" is rudimentary boiler plate coding/configuration and staring into the Abyss. I think our role will be to design algorithms and come up with creative solutions/hacks (which would be difficult for a program) and designing a workflow/flow chart and feeding it into a program which spits out binaries and flag for edge cases. A whole swat industries and economies (read outsourcing) will become redundant and only outsourcing done would be to the generator. [0]: https://www.youtube.com/watch?v=taaEzHI9xyY https://www.youtube.com/watch?v=taaEzHI9xyY
- deleted 11y ago[deleted]
- snikeris 11y agoIn the spirit of: https://www.dartmouth.edu/~matc/MathDrama/reading/Wigner.html https://www.dartmouth.edu/~matc/MathDrama/reading/Wigner.htm... http://www.researchgate.net/profile/Derek_Abbott/publication/256838918_The_Reasonable_Ineffectiveness_of_Mathematics/links/00b7d523d5bd289428000000.pdf http://www.researchgate.net/profile/Derek_Abbott/publication...
- iyn 11y agoVery short video about the topic: https://www.youtube.com/watch?v=ZBkzqLJPkmM https://www.youtube.com/watch?v=ZBkzqLJPkmM
- TheLoneWolfling 11y agoMy question, and something this doesn't get into, is this: how do you train a RNN?
- deepnet 11y agoYou need an error signal - a target value is compared with the networks prediction. That error is carefully assigned proportionally to the network weights that contributed to it and the weights adjusted a small amount in that direction. This is repeated many times. Backpropagation suffers from vanishing gradients on very deep neural nets. Recurrent Neural Nets can be very deep in time. Or the weights could be evolved using Genetic Programming.
- raverbashing 11y ago> Backpropagation suffers from vanishing gradients on very deep neural nets. Especially when using saturating functions (tanh/sigmoid) > Or the weights could be evolved using Genetic Programming GA, not GP http://en.wikipedia.org/wiki/Genetic_algorithm http://en.wikipedia.org/wiki/Genetic_algorithm
- skorgu 11y agoIt would be interesting to occasionally train the generated C against a compiler.
- Houshalter 11y agoIt would also be ideal to use a higher level interpreted language, and have it try to generate one page scripts rather than giant mega projects like linux.
- deepnet 11y agoIn "Learning to Execute" by Zaremba & Sutskever http://arxiv.org/abs/1410.4615 http://arxiv.org/abs/1410.4615 An RNN learns snippets of python Their next paper is "Reinforcement Learning Neural Turing Machines" http://arxiv.org/abs/1505.00521 http://arxiv.org/abs/1505.00521 based on Graves "Neural Turing Machines" http://arxiv.org/abs/1410.5401 http://arxiv.org/abs/1410.5401, which attempts to infer algorithms from the result. In a lost BBC interview from 1951 Turing reputedly spoke of evolving cpu bitmasks for computation.
- clickok 11y agoI love stuff like this, and I think "unreasonable" is almost an understatement. It's "unreasonable" mainly because it occasionally captures subtle aspects of the data source for "free". If you've worked with procedurally generated content, Markov chains, and so on, you probably have had to perform a few tweaks in order to get plausible results[1]. From the article, an excerpt of the output from an RNN trained on Shakespeare: Second Lord: They would be ruled after this chamber, and my fair nues begun out of the fact, to be conveyed, Whose noble souls I'll have the heart of the wars. Clown: Come, sir, I will make did behold your worship. VIOLA: I'll drink it. Sure, the individual blocks are similar to what you'd get from a Markov text generator-- but it gets that after a full stop, there comes a newline, a new character name, and a new text block. To my eyes, this is a qualitative leap in performance. It suggests that the model has figured out some things about the data stream that you'd normally have to add in by hand[2]. It's also unreasonable that the same framework works well for so many different data sources. My experience with other generative methods has been that they were fragile and prone to pathological behaviour, and that getting them to work required for a specific use case required a bunch of unprincipled hacks[3]. It used to be that when a talk started to veer towards generative models, I'd start looking around the room, wondering whether I could survive the drop from any outside-facing windows. But with RNNs using LSTM (or neural Turing machines!) you can consider incorporating a generative model in the solution you're putting together without having to spend a huge chunk of time massaging it into usefulness and purchasing time on a supercomputer[4] 1. I once wrote quick a Reddit bot with the aim of learning to repost frequent highly upvoted comments and trained it using a simple k-Markov model... it was not good at first, and in order to get it to work I had to do a lot of non-fun stuff like sanitizing input, adding heuristics for when/where to post, and at the end it was mediocre. 2. Alex Graves (from DeepMind) has a demo about using RNNs to "hallucinate" the evolution of Atari games, using the pixels from the screen as inputs. It's interesting because it shows that same sort of tendency to capture the subtle stuff: https://youtu.be/-yX1SYeDHbg?t=2968 https://youtu.be/-yX1SYeDHbg?t=2968 3. As in occult knowledge and rules-of-thumb, but you might also read this as a double entendre about myself and my colleagues. 4. Well, you still might need an AWS GPU instance if you don't have a fancy graphics card.
- jameshart 11y ago
- swalsh 11y agoAs a father, the output feels really familiar. It's like a child learning to talk. At first, though the words they say are actual words (and mean something to you), they themselves have no idea what the meaning is. Eventually though they start understanding the meaning, which combined with the syntax creates a person who can communicate. I wonder if all that's missing is just a few more layers, and another source of input. Maybe a list of requirements/output/input matched with the code so it understands why what was written was written. I wonder what would happen if you ran the program, took the output, and fed it back in as input. Really cool stuff here.
- deleted 11y ago[deleted]
- cristianpascu 11y agoA machine will never get the meaning of a word, unlike a very small child. I am simply amazed by the fact that a child can learn a language, catch what a question is, offer an answer, say no (and how they like to say no), and all. As much as I wish it was possible, that much I believe it's not. The best we can do is put our knowledge of our ability to infer meaning of words into machine code.
- hanspeter 11y agoWhat makes the human brain not a machine?
- coldtea 11y agoPhysical properties? What if those kind of properties of physical materials are needed in cognition? The problem is simulations of the brain are not "machines", they are algorithms, e.g. they assume everything is happening at the information processing level. To use your own example, we can design an algorithm to simulate making coffee. But the algorithm can never make coffee -- unless it's fitted and connected to a coffee making apparatus. Or take something being "wet" for example. We can emulate the motions and powers in play in liquids, but not "wetness" in the sense of the physical property (moisture etc). If something depends on it, e.g. the emulation actually watering some actual flowers, then it will fail. An emulation can only water emulated flowers.
- narrator 11y agoThe thing about neural nets is that they are pretty opaque from an analyst point of view. It's hard to figure out why they do what they do, except that they have been trained to optimize a particular cost function. I think Strong AI will never happen because the people in charge will not give control over to a system that makes important decisions without explaining why. They will certainly not give control over the cost function to a strong AI because control of determination of the cost function is the axis upon which all power will rest.
- relate 11y agoThis is a common criticism. However, almost all ML methods have some built in heuristic choices, that are the result of finding something that both works and is mathematically nice. Each of these choices restricts us to some family of functions where it's hard to justify why it's really relevant to the problem at hand, e.g. convex loss functions (l1, l2, ..), convex regularizers (l1,l2,..), gaussian priors, linear classifiers, some mathematically nice kernel functions, e.t.c. In the end, people usually statistically estimate the performance of the methods and use what works.
- __Joker 11y agoI kind of drifted into the camp of transhumanism as future where human is enhanced by all the smart sub AI problem solver but generally the humans take the decision at the end of the day. Also I think other problem is for strong AI to exist we are not sure what the "objective function" for the AI to work for.
- pjc50 11y agothe people in charge will not give control over to a system that makes important decisions without explaining why They will if it gives the answers they want to hear. History is full of critical decisions based on ridiculous pretexts or unclear processes.
- deleted 11y ago[deleted]
- dimatura 11y agoOur life is dominated by systems we don't understand. I have some understanding of how my cell phone works at the software level, but when it comes to details at the hardware level I just trust the electrical engineers knew what they're doing. I have virtually no understanding of how the engine in the bus operates beyond what I learned in thermodynamics 101. Sure, you might say - someone understands these things. But for some systems, it's hard to pinpoint these people. And for some other complex systems, like the stock market, nobody really understands them or (completely) controls them. But we still use them every day. I think once AI becomes useful enough, people will gladly hand control over.
- stcredzero 11y agoSo, if Neural Networks can be thought of as just an optimized way of implementing unreasonably large dictionaries, Recurrent Neural Networks could be thought of as an optimized way of implementing unreasonably large Markov chains.
- maaaats 11y agoIsn't the author's definition of RNNs wrong? I thought the difference is that a RNN allows connection back to previous layers, compared to a feed-forward net. Not this talk about "fixed sizes" and "accepting vectors". Or am I wrong?
- fpgaminer 11y agoKarpathy usually talks about machine learning topics from multiple viewpoints, and usually (in my experience with his writings) prefers more loose, non-traditional interpretations (that ultimately lead to better understanding of the underlying mechanics of the approach). In this case, his point was that one way RNNs differ from FFNNs is their ability to accept arbitrarily sized inputs and generated arbitrarily sized outputs. That's pretty important, which is likely why he emphasizes it. But the rest of the article shows the salient point; RNNs are NNs that hold a state vector. Saying that RNNs are NNs that allow connections back to previous layers is true, but that's only one way of looking at it. Holding state is another, since it implies backwards connections. Feedback is another term. And because they have backwards connections, state, feedback, etc, they also posses the capacity to handle non-fixed sized inputs and outputs. In summary; it's different viewpoints of the same mathematical object. Karpathy focuses on the ability of RNNs to handle arbitrarily long inputs and outputs, because that's something FFNNs cannot do.
- j2kun 11y agoWhat's unreasonable about neural networks (in general, not just recurrent ones) is that we don't really have any theoretical understanding of why they work. In fact, we don't even really understand what sorts of functions neural networks compute.
- tormeh 11y agoI've thought a bit about RNNs, and I can see an obvious problem: Fixed amount of memory. Is there any chance someone's come up with an RNN that has dynamic amounts of memory?
- varelse 11y agoThere's a huge degree of data re-use in the weights. This should be exploited. Second, one could envision paging the hidden units back to system memory on a coprocessor-based implementation (GPUs/FPGAs/not Xeon Phi, gag me). 256 GB servers are effectively peanuts these days relative to developer salaries and university grants (datapoint: my grad school work system was ~$100K in 1990 dollars) so unless you're trying to create the first strong AI, I don't think this is a serious constraint. Good luck with that no matter what Stephen Hawking, Elon Musk, and Nick Bostrom harp on about: we have no idea what the error function for strong AI ought to be and even if we did, it's over a MW using current technology to achieve the estimated FLOPS of a human cerebrum.
- tormeh 11y agoI meant that the state vector has constant size and just setting it at the maximum available might give you problems with training.
- varelse 11y agoNothing you can't work around if you're willing to roll your own code. That said, I agree 100% if you're dependent on someone else's framework...
- tormeh 11y agoI meant that the state vector has constant size and just setting it at the maximum available might give you problems with training.
- exgrv 11y agoThere is this paper by Joulin & Mikolov: Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (http://arxiv.org/abs/1503.01007 http://arxiv.org/abs/1503.01007). In this case, the memory of the RNN is an ensemble of differentiable stacks.
- Smerity 11y agoKarpathy is one of my favourite authors - not only is he deeply involved in technical work (audit the CS231n course for more[1]!), he spends much of his time demystifying the field itself, which is a brilliant way to encourage others to explore it :) If you enjoyed his blog posts, I highly recommend watching his talk on "Automated Image Captioning with ConvNets and Recurrent Nets"[2]. In it he raises many interesting points that he hasn't had a chance to get around to fully in his articles. He humbly says that his captioning work is just stacking image recognition (CNN) on to sentence generation (RNN), with the gradients effectively influencing the two to work together. Given that we've powerful enough machines now, I think we'll be seeing a lot of stacking of previously separate models, either to improve performance or to perform multi-task learning[3]. A very simple concept but one that can still be applied to many other fields of interest. [1]: http://cs231n.stanford.edu/ http://cs231n.stanford.edu/ [2]: https://www.youtube.com/watch?v=xKt21ucdBY0 https://www.youtube.com/watch?v=xKt21ucdBY0 [3]: One of the earliest - "Parsing Natural Scenes and Natural Language with Recursive Neural Networks" http://nlp.stanford.edu/pubs/SocherLinNgManning_ICML2011.pdf http://nlp.stanford.edu/pubs/SocherLinNgManning_ICML2011.pdf
- wonderingwhere 11y ago> he spends much of his time demystifying the field itself, which is a brilliant way to encourage others to explore it :) yup. this is the first time I understood someone from this field. Honestly, this dude just broken down the wall. What's more important, passion flows through his writing. And it can be felt. I got so excited while reading it.
- pigscantfly 11y agoAndrej is also a great lecturer; his CS231n class in the winter was both the most enjoyable and educational I've taken all year. All of the materials are available at cs231n.stanford.edu, although I can't seem to find the lecture videos online. It may not have been recorded. As a bonus, there's an ongoing class on deep learning architectures for NLP which covers Recurrent (and Recursive) Neural nets in depth (as well as LSTM's and GRU's). Check out cs224d.stanford.edu for lecture notes and materials. The lectures are definitely being recorded, but I don't think they're publicly available yet.
- divs1210 11y agoThis felt like watching Ex Machina. Thanks a lot, this was extremely informative and super fun.
- higherpurpose 11y agoIf neural networks are the way to build strong AI and neural nets are all about optimization, wouldn't a quantum computer be ideal to power an AI? (assuming we can get one to work)
- Houshalter 11y agoI don't think so. NNs have millions of parameters, and making a quantum computer that large, and with that many complex interactions, would be very difficult. Optimization of NNs isn't really that bad. Stochastic gradient descent is extremely powerful and roughly linear with the number of parameters, possibly better.
- hgibbs 11y agoI did have a bit of a chuckle when they got to Algebraic Geometry. That's incredible.
- rsp1984 11y agoI wonder what would happen if you train an RNN like described with, say, the scores of all of Mozart's Chamber Music and then let it generate new music from the learned pieces. How would it sound? Would it figure out beat? Chords? Harmonies? May it even sound a bit like Mozart?
- Nadya 11y agoThere's a few such projects in existence. Perhaps not RNN-Mozart inspired, but I'm sure that exists too. Emily Howell https://www.youtube.com/watch?v=QEjdiE0AoCU https://www.youtube.com/watch?v=QEjdiE0AoCU Here's a Bach-inspired computer-generated song: https://www.youtube.com/watch?v=PczDLl92vlc https://www.youtube.com/watch?v=PczDLl92vlc
- kastnerkyle 11y agoThe work of Nicolas Boulanger-Lewandowski was extensively focused on this topic, see his work [1]. He wrote a Theano deep learning tutorial on this topic [2], and several people (Kratarth Goel) [3][4] have advanced the work to use LSTM and deep belief networks. For a brief while RNN-NADE made an appearance as well, though I do not know of an open source implementation There are also a few of us who are working on more advanced versions of this model for speech synthesis, versus operating on the MIDI sequence. Stay tuned in the near future! I can say from experience that some of the samples from the LSTM-DBN are shockingly cool, and drove me to spend about a week using K-means coded speech. It made robo-voices at least but our research moved past that pretty fast. [1] http://www-etud.iro.umontreal.ca/~boulanni/ http://www-etud.iro.umontreal.ca/~boulanni/ [2] http://deeplearning.net/tutorial/rnnrbm.html http://deeplearning.net/tutorial/rnnrbm.html [3] http://arxiv.org/pdf/1412.6093.pdf http://arxiv.org/pdf/1412.6093.pdf [4] https://github.com/kratarth1203/NeuralNet/blob/master/rnndbn.py https://github.com/kratarth1203/NeuralNet/blob/master/rnndbn...
- JonnieCache 11y agoIs the robot-voice code published anywhere? You can make money out of that kind of thing btw! https://soniccharge.com/bitspeek https://soniccharge.com/bitspeek (Obviously not the same thing but the point is that silly robo-voice code is marketable :)
- wonderingwhere 11y agothis is quite possibly the most interesting item I've read on HN
- lqdc13 11y agoDoes anyone know if these are/can be good for named entity recognition? I am stuck implementing second order CRFs right now for the lack of a good implementation, and this seems a lot easier.
- syllogism 11y agoI'm not aware of any strong RNN results for NER, no. You'd probably find the paper here: http://aclweb.org/anthology/ http://aclweb.org/anthology/ (everything in CL is open access). You want the proceedings of CL, TACL, ACL, EMNLP, EACL, and NAACL. Don't bother with the workshops.
- syllogism 11y agoI'm not aware of any strong RNN results for NER, no. You'd probably find the paper here: http://aclweb.org/anthology/ http://aclweb.org/anthology/ (everything in CL is open access). You want the proceedings of CL, TACL, ACL, EMNLP, EACL, and NAACL. Don't bother with the workshops.
- noahmbarr 11y agoWould the returned samples from PG/Shakespeare/Wikipedia examples be of higher quality if you used a word-level language model instead of character model with similar parameters? I was curious if the overhead of learning how to spell words (vs a pure task of sentence construction with word objects) out weigh the reduction in sample set size? (Awesome article for a RNN newbie)
- fpgaminer 11y agoKarpathy states in the blog post that word-level models currently tend to beat character models, across the broad field of NLP related RNNs. But he argues that character models will eventually overtake (much in the same way that ConvNets have "replaced" manual feature extraction). That said, I think the RNNs here are limited by the corpus. They need to be exposed to more writing. Even if all you want is a Shakespeare generator, you still need to expose it to other literature. That will give it greater context, and more freedom of expression and, dare I say, creativity. I mean, imagine if all you were exposed to your whole life was Shakespeare. Nothing else (no other senses). Even with your superior mind, I doubt you'd generate anything better than what this RNN spits out. So yeah, it needs a large corpus to build a broader model. Then we need a way to instruct the broadly trained RNN to generate only Shakespeare-like text. Perhaps by adding an "author" or "style" input.
- kylebgorman 11y agoI fail to see how word-based models are character-based models with manual feature extraction. Word boundaries are read directly from deterministically tokenized inputs. And, as I mentioned upthread, it has been known for about ten years, long before the current neural net revival, that high-order character-based models are competitive with word-based models (at least in terms of perplexity).
- kylebgorman 11y ago"Old-school" Markovian language models (the vast majority of what's being used in production today) are mostly word-based but for text applications with tons of data, high-order character models are competitive with word-based models. (http://www.aclweb.org/anthology/W05-1107 http://www.aclweb.org/anthology/W05-1107)
- ux-app 11y agoI'm an absolute layman with regard to AI, so I'd be keen to hear some explanations with regard to the possibility of creating strong AI in silicon. Might there be properties of our biological brain that silicon can't capture? Is this related to the concept of computability? I'm not suggesting that there is a spiritual or metaphysical component to thinking. I'm not, I'm a materialist through and through. I just wonder if maybe there is some component of non-deterministic behavior occurring inside a brain that our current silicon-based computing does not capture. Another way to ask this is will we need to incorporate some form of wetware to achieve strong AI?
- aamar 11y agoThese are not fully settled questions, though the answer is probably no. Most researchers believe that brains are Turing machine equivalent, therefore can be simulated by any other equivalents. Even Gödel believed this, though he believed the mind had more capabilities than the brain.[1] As a materialist, you would share the commonly-accepted view and reject his latter claim. There is a small minority of philosophers and physicists who believe that there are meaningful quantum reactions happening in the brain, distinguishing them from classical computers.[2] Some recent computer simulations have shown this to be plausible, but the general impression is that it seems unlikely, and we don't have specific evidence of effects of this sort. Quantum effects of certain sorts are computationally infeasible to perform with classical computers. And it's theoretically plausible that such effects can not be conducted at scale with in-development quantum computer technology, and is only practical with organic chemistry, but again, this is quite a minority view. It's also possible that classical brain features, such as its massive concurrence or various clever algorithms, prove difficult to replicate or simulate. If these are easy problems to solve, then strong AI may arrive in decades; if very difficult, centuries. In the latter case, it seems plausible that incorporating wetware would be a useful shortcut. But there's good reason to believe that the practical disadvantages of wetware (e.g. keeping it alive, coordinating with its slow "clock speed") overwhelm the computational conveniences. -- [1] http://www.hss.cmu.edu/philosophy/sieg/onmindTuringsMachines.pdf http://www.hss.cmu.edu/philosophy/sieg/onmindTuringsMachines... [2] http://en.wikipedia.org/wiki/Quantum_mind http://en.wikipedia.org/wiki/Quantum_mind
- viraptor 11y agoI found the learning progress great. I was thinking some time ago how to generate english-sounding words which don't exist. Well, here they are: (from iteration 700) Aftair, unsuch, hearly, arwage, misfort, overelical, ... (although I admit, some of them may be just old words I haven't heard of before)
- deleted 11y ago[deleted]
- efnx 11y agoI've only read the first section but it seems RNNs are very close in concept to Mealy machines. http://hackage.haskell.org/package/machines-0.4.1/docs/Data-Machine-Mealy.html http://hackage.haskell.org/package/machines-0.4.1/docs/Data-... > They accept an input vector x and give you an output vector y. However, crucially this output vector's contents are influenced not only by the input you just fed in, but also on the entire history of inputs you've fed in in the past.
- teraflop 11y agoIf it helps, you can think of a RNN as being analogous to a finite state machine. But instead of a single discrete state, it's a continuous, high-dimensional vector. That has the extremely important effect that the output is a continuous function of the input, which is necessary for training using gradient descent.
- mangeletti 11y agoImagine a conversion-optimizing genetic algorithm for spam (web and/or email) generation, using a tool like this (e.g., when users perform the intended actions, DNA is passed on to the next iteration). That would be one positive feedback loop to rule them all.
- fdej 11y ago"This sample from a relatively decent model illustrates a few common mistakes. For example, the model opens a \begin{proof} environment but then ends it with a \end{lemma} ... By the time the model is done with the proof it has forgotten whether it was doing a proof or a lemma. Similarly, it opens an \begin{enumerate} but then forgets to close it." Ah, so strong AI is finally here. A computer program that makes just the same mistakes as humans when writing in TeX.
- Patryk 11y agoThis same thing (i.e., using recurrent neural networks to predict characters (and even words)) was done by Elman in 1990 in a paper called "Finding Structure in Time"[1]. In that paper, Elman goes several steps further and carries out some analysis to show what kind of information the recurrent neural network maintains about the on-going inputs. It's an excellent read for anyone interested in learning about recurrent neural networks. [1] http://crl.ucsd.edu/~elman/Papers/fsit.pdf http://crl.ucsd.edu/~elman/Papers/fsit.pdf
- sushirain 11y agoIt's amazing how much was already known decades ago. Elman and others did much more, and hopefully, now the field will take the next step (which was long delayed), with the help of today's computer power.
- dools 11y agoWeb spam 2.0: 1) Take the entire works of several popular content creators in a given field, complete with links out to articles etc. 2) Concatenate them into a single file 3) Train this thing to generate new articles 4) Create a map of popular articles that other people have written, to articles you have written on similar topics 5) Replace the originals with your articles 6) Publish millions of articles that can't be detected as spam automatically by Google It's like bot wars: Spammers can train their robots to try and defeat Google's robots.
- stefs 11y agowell, i don't see how they - the spammers - would fake google's valuation system of valuing incoming links from valuable sources. it's not like many valuable sites outside this relatively insular system would link to those generated nonsense pages. that'd practically create an insular babblenet that could be relatively easily identified. i mean, it's not like that's exactly what's happening right now.
- dools 11y agoOkay so in the system I'm hypothesising, I pick a topic -- say content marketing. I go to Neil Patel's and KissMetrics blog and get all their articles on content marketing, and train this thingy with them. I then buy, say, 1,000 domains. Doesn't matter what they are -- Or I buy 100 domains and setup 300 tumblr blogs, and 300 blogger blogs and 300 wordpress.com blogs. Now I drip feed content to each of those blogs, but instead of linking to the articles on content marketing that kissmetrics and neil patel originally reference, I link to articles I have created instead. How can Google tell the difference between a tonne of nobody bloggers link to Neil Patel's articles, and my bots linking to my articles? The fact is that if you blog on niche topics, with good article titles reflecting low competition long tail keywords, you'll get some traffic from Google pretty easily -- how can Google possible tell that links are coming from shitty bot generated pages versus from a tonne of obscure bloggers with virtually no audiences (of which there are thousands)? The way they can tell the difference is Panda (or Penguin? I think it's Panda ... ) so as long as your pet robot can learn from Neil Patel and Kissmetrics well enough to produce content that cannot be penalised by Panda, and so long as you don't do it stupidly by like, having the same anchor text for all the articles and doing 1,000 articles overnight and actually phase it in so that it looks as though you're getting some reasonable organic spread, you'll be able to game Google's rankings pretty reliably for your real articles that you're trying to promote, and get higher volumes of traffic to those articles than you would be able to by just focusing on niche, long tail articles (for example because you'd be able to get on page #1 or in the top 5 for much higher volume keywords). You would then get shares etc. for your actual content -- just because those "spam farms" don't have social shares or backlinks from PR6 blogs doesn't mean Google completely disregards them, just means that you need a lot more of them to make the same impact as lots of shares/backlinks from PR6 blogs. This strategy is old, and was killed by Panda, but if you could beat Panda using a RNN then this would work again.
- thewarrior 11y agoI have a dumb question. How is a recurrent neural network different from a Markov Model ?
- mikecmpbll 11y agoThis is my deep learning enlightenment moment. 22/05/15
- phyalow 11y agome to, mesmerised.
- oggy 11y agoIn all the examples on the page, the RNN is first trained and then used to generate the text. Is there a way to use RNNs for something interactive? For instance, can one train an RNN to mimic Paul Graham in a discussion, and not only in writing an essay?
- evc123 11y agoSomeone should train an RNN on neural network source code to see if it's possible to get neural networks to generate neural networks.
- danans 11y agoI'm curious to know if, since these networks can learn syntax, whether they can also be re-purposed as syntax checkers, not just syntax generators. That is, can the syntactical knowledge learned by these models be run in a static classification mode on some input text to recognize the anomalies within and suggest fixes.