7 ms·
Andrew Ng and the Quest for the New AI
- kinofcain 13y agoThere is a very quick reference to the person who inspired him, Jeff Hawkins, whose book is worth a read: http://www.amazon.com/On-Intelligence-Jeff-Hawkins/dp/0805078533/ http://www.amazon.com/On-Intelligence-Jeff-Hawkins/dp/080507... Edit: update link
- netrus 13y agoLink does not load CSS for me, but this does: http://www.amazon.com/Intelligence-Jeff-Hawkins/dp/B000GQLCVE http://www.amazon.com/Intelligence-Jeff-Hawkins/dp/B000GQLCV...
- kinofcain 13y agoThanks, had stripped the tracking codes off the mobile link, but it doesn't load right on desktop.
- blhack 13y agoThat book will change the way that you look at yourself.
- lomendil 13y agoI just saw Jeff Hawkins give a talk and it was quite interesting. I was a bit worried, however, that he is basing his theory of intelligence on the human neocortex, while claiming to go after general principles. This is guaranteed not to be terribly general, considering the many bits of matter on this planet that exhibit intelligence without a neocortex. By many, I mean ones that hugely outnumber humans. So very interesting stuff, but not the answer that I think he wants it to be.
- Lambdanaut 13y agoIn "On Intelligence" he postulates that he's not necessarily going for a "human-like" intelligence or even a "life-like" one. Basically he just wants something that's very good at recognizing patterns over time, which I can imagine the neocortex would be great at. Though, he also references the thalamus and hippocampus in the books a lot, as very important parts of the brain to his framework. [http://en.wikipedia.org/wiki/Memory-prediction_framework#Neurophysiological_implementation http://en.wikipedia.org/wiki/Memory-prediction_framework#Neu...]
- thomaspaine 13y agoGrok (formerly Numenta) has a slightly more technical white paper that goes into more detail on the actual algorithms from On Intelligence: https://www.groksolutions.com/htm-overview/education/HTM_CorticalLearningAlgorithms.pdf https://www.groksolutions.com/htm-overview/education/HTM_Cor... There are some subtle differences between HTMs and straight up deep learning, mainly the requirement for HTM data to be temporal and spatial. I know Andrew used to sit on an advisory committee at Numenta, I don't know if he still does.
- jhartmann 13y agoI've said this before, but deep learning is terribly powerful precisely because you don't have to spend lots of time doing feature engineering. Multi layer networks that are trained in semi-supervised, unsupervised and supervised fashions all can now produce networks that meet or beat the state of the art hand created models for speech, handwriting recognition, ocr, and object recognition. We are only just beginning to see what is possible with these sorts of techniques. I predict within a few years time we will see a huge renaissance in AI research and Neural Network research specifically as these techniques are applied more broadly in industry. My startup is building some cool stuff around this technology, and I know there are hundreds like me out there. This is going to be a fun ride.
- wookietrader 13y agoThis has been said about neural nets two times already. Sadly, they did never deliver. There are still applications where e.g. random forests beat the crap out of all kinds of deep learning algorithms in (a) training time (b) predictive quality (c) prediction time. We should stop hyping this. I am a researcher working in deep learning myself, but the current deep learning hype is actually what makes me worry that I will have trouble getting a job because industry will be disappointed a third time.
- jhartmann 13y agoWell I think a big part of this is that right now we have finally gotten to where the algorithms + the required computing power are starting to become more widely available. Cheap graphics cards or things like Intel's Phi + techniques like drop out to prevent overfitting are really enabling much more sophisticated things to be done in a reasonable wall time. Granted multilayer neural networks aren't a free lunch that will solve everything, but there are large classes of problems that are falling to these techniques all the time. We are also finding that Neural Networks scale very well to very large architectures, better than some of the other techniques. I understand we should be careful not to overhype since we have seen previous excitement cause a mass exodus from this research before. I however think people like Hinton were always right, and this was awesome stuff. We just couldn't really take advantage of it because we could never train it for long enough and we hadn't learned how to do things efficiently yet.
- hvs 13y agoOT: His online Machine Learning class last year was great. He is the best professor I've ever had, and explains things so clearly that you understand them the first time. You are lucky if you ever get to work or study under him.
- deleted 13y ago[deleted]
- rzendacott 13y agoThat's awesome to hear! I'm taking the Coursera class now, and it's been great so far. It just started a couple weeks ago, so it definitely isn't too late to join! https://www.coursera.org/course/ml https://www.coursera.org/course/ml
- psbp 13y agoI wish Coursera followed the Udacity model. I always find out about these classes after they're already weeks in progress or over.
- berberous 13y agoCheck out class-central.com for a list of all current and upcoming classes. Coursera also lets you star a specific class and get notified if they are repeated in a new cycle.
- nkraft 13y agoUnless you're attached to getting a certificate of completion, you can pretty much follow the Udacity model. As long as the course hasn't finished, sign up and get around to the videos and assignments when you get to them. There isn't the same discussion forum interchange, and your homework isn't graded, but they don't drop the class from your list even if you do nothing during the run.
- packetslave 13y agoNote that you'll want to be careful to cache the materials offline if you do this, especially if you plan on "catching up" after the formal end date for the class. Some of the courses (notably the Princeton algorithms ones) disable access to the materials once the official course ends.
- aespinoza 13y agoI thought the man behind the Google Brain was Ray Kurzweil (http://www.wired.com/business/2013/04/kurzweil-google-ai/ http://www.wired.com/business/2013/04/kurzweil-google-ai/).
- drcross 13y agoKurzweil is a hack. (Source: I've an Msc in Robotics and read all his books)
- konstruktor 13y agoPlease think about what a source is. This isn't reddit.
- snikolic 13y agoSeconded. I'm curious to hear your reasoning, and your actual arguments will sway me much more than your credentials anyway.
- Cushman 13y agoBullshit. There is nothing wrong with using personal experience for reference so long as it is properly cited. It beats "This guy is wrong (source: random blog post that I didn't really read)", anyway. Edit: To be clear, I don't love the Reddit snowclones, but there's nothing wrong with the sentiment behind "I'm a scholar in this field, and I think this guy is a hack."
- konstruktor 13y agoSo misapplying the term source and saying that one has a degree gives one enough credibility as a scholar to turn a dismissive and colloquial statement into a worthwhile contribution to a discussion about AI?
- krichman 13y agoIn this case the term "source" is obviously a rhetoric device, and we can avoid a lot of pedantry by noting that the poster was totally transparent about his reasoning thus achieving the ends we want via means that aren't awful. PS having a degree by definition implies some credibility as a scholar.
- gavanwoolery 13y agoHmm...I don't mean to be a skeptic, but I do not see any new theories here. Neural networking has been around for a long time, as have an abundance of theories and implementations around it...some people have gone so far as to build an actual brain replica (a digital version of the bio/analog thing). Neural networking is extremely powerful, but to be of any use, you need a lot of computing power. As it turns out, our brains are really good at massively parallel tasks like image and motion processing; these things can be done explicitly on a computer with some ease, but having a computer learn on its own from scratch how to do them is not easy.
- raverbashing 13y agoExactly Neural networks of course "mimic" the way the brain works, but it stops there. Not to mention several modern AI techniques have nothing to do with mimicking biology (like SVMs) This looks a reporter pushing his own agenda to make for a colorful story.
- xtacy 13y agoI think the requirement on enormous computing power is exactly the point. Since not many have access to such computing power, Andrew Ng and folks at Google (incl. Jeff Dean!) built a large scale neural network system called DistBelief (nice name!). This system allows programmers to think of really large scale neural networks without worrying about how to scale them, handling fault tolerance, etc. You can think of it as the MapReduce for neural networks. They demonstrated how a large scale neural network can do interesting stuff on its own (e.g. recognising cats and human faces from unlabeled youtube videos).
- jbelanich 13y agoYou're correct in that neural networks as a model have been around for a long time. However, those networks were restricted to be shallow because backpropogation didn't work well on networks with many hidden layers. Only recently have researchers developed learning procedures that can learn these deep architectures efficiently, using some clever unsupervised learning techniques. And surprisingly, they are finding that these deep networks perform remarkably well, beating the state of the art in a number of benchmarks. You are also right that you do need a lot of processing power to get neural networks to work well. But that is changing rapidly. Hinton's convolutional neural network has the state of the art in the ImageNet benchmark, yet was trained using significantly less power than google brain. Regardless, you don't need google scale computation to get deep networks to work well. The point of google brain is to see how far one could push neural networks.
- EthanHeilman 13y agoWe should stop trying to claim every new method is "like the brain". We don't have any clear understanding of how the brain works. One can be inspired by a particular and likely wrong cognitive theory, but one can not say one is building "machines that can process data in much the same way the brain does" truthfully without a deeper, and currently unavailable, understanding of the functioning of the human brain.
- Jach 13y agoI feel the same way, that researchers should stop trying to mimic the brain, but not because we don't understand the brain. While I think there are still several decades before we'll be able to have mind uploads, I also think a lot of people underestimate the quality of modern brain science. In any case, I have the same reason as Dijkstra for why I think mimicking the brain isn't that great an idea. In http://www.cs.utexas.edu/~EWD/transcriptions/EWD10xx/EWD1036.html http://www.cs.utexas.edu/~EWD/transcriptions/EWD10xx/EWD1036... (really a great read to branch all sorts of thoughts off of) Dijkstra said, "The effort of using machines to mimic the human mind has always struck me as rather silly: I'd rather use them to mimic something better." It's probably a harder problem, creating smarter-than-human intelligence on a machine, but research isn't as constrained by laws and ethics (they don't have to bemoan not being able to experiment with living human brains). I wish more people were active in the area.
- visarga 13y agoThey are trying to find THE ONE ALGORITHM that solves all A.I. problems. The brain has an implementation of it, but it is in wetware, hard to extract. Deep learning makes some pretty good approximations of the visual areas, though.
- EthanHeilman 13y agoYou assume that human cognition has an algorithmic component. Maybe you are right, but we still have a pretty shaky understanding of how Neurons work, let alone how the brain works on a large scale. Who knows, lets investigate by trying possibilities but lets understand that we are still in a position of ignorance. We have some probabilistic models of that successfully predict various future states of the brain from past states or stimuli. This is not the same as understanding it or even approximating it.
- yankoff 13y agoGreat and inspiring professor. Taking his ML course on coursera and trying to follow his talks.
- wookietrader 13y agoIf an article says that Andrew Ng is "the man at the center of deep learning" it's just not right. Geoffrey Hinton's and Yoshua Bengio's impact were at least as high as his, if not much higher.
- wookietrader 13y agoIf an article says that Andrew Ng is "the man at the center of deep learning" it's just not right. Geoffrey Hinton's and Yoshua Bengio's impact were at least as high as his, if not far bigger.
- why-el 13y agoSo I am a little confused. Where are we on the learning part of AI? As I understand it, the current consensus is to throw as much data as you can at your model (millions of cat pictures in this article's example) to make it pick up patterns and yet still claim that we are closing in on how the brain works? As far as I can tell no human brain would need that many pictures to see a pattern. In fact, and this is probably more apparent in language, we humans tend to work with degenerate data and still end up with perfect models.
- DennisP 13y agoBear in mind that the neural network is starting at the same point as a newborn baby.
- piglop 13y agoYou may not have seen millions of cats, but how many images have your brain processed since you can see? A five years old brain has been trained on billions of images (along with many other simultaneous inputs).
- cscurmudgeon 13y agoAh. The good old AI cycle. Scientist: X can help us get full AI! You: Why? Scientist: Because of reason R. You: But, reason R is a non sequitur... More seriously, reasons similar to that for deep learning have been repeated multiple times in AI with failure (e.g. Thinking Machines). I would suggest that these folks remain calm and build something on the scale of IBM's Watson using just deep learning..
- niklaslogren 13y agoVery interesting article, it makes me hopeful. This might be slightly off-topic, but I'll try it here anyway: can anyone recommend any books/other learning resources for someone who wants to grasp neural networks? I'm a CS student who finds the idea behind them really exciting, but I'm not sure where to get started.
- pilooch 13y agoIt seems the relationship between the brain and deep learning has evolved in such a way that the later can help with insights into how the former works. In this regard, I thought I would mention the extraordinary simple and elegant talk by G. Hinton last summer: http://www.youtube.com/watch?v=DleXA5ADG78 http://www.youtube.com/watch?v=DleXA5ADG78 It starts from a simple and clever improvement to an existing deep learning method and ends up with beautiful (and simple!) insights on why neurons are using simple spikes to communicate.