4 ms·

A Beginner’s Guide to Recurrent Networks and LSTMs

211 points by vonnik 11y ago

hailpixel 11y ago
If this feels like jumping in the deep end, I highly recommend Stanford's online course on this subject: http://coursera.org/learn/machine-learning/ http://coursera.org/learn/machine-learning/
- vonnik 11y ago
  I agree. Andrew Ng's course is a great place to start with ML. We've got a list of great online ML/DL courses here: http://deeplearning4j.org/deeplearningpapers.html http://deeplearning4j.org/deeplearningpapers.html
yohann305 11y ago
Machine Learning is in my bucket list of things to focus on at some point in my life, just like many of you. I know it's a bit off topic, does anyone of you know any practical resource that teaches ML while creating some kind of robot with AI? Thanks in advance.
- vonnik 11y ago
  Sebastian Thrun teaches a course called AI for Robotics on Udacity: https://www.udacity.com/course/artificial-intelligence-for-robotics--cs373 https://www.udacity.com/course/artificial-intelligence-for-r...
  - tgflynn 11y ago
    I took that class several years ago and it was definitely not about machine learning, mostly mapping/localization and route planning. It was a good class but the claim: "Learn how to program all the major systems of a robotic car" is vastly overblown. That would only be true if things like traffic sign recognition and pedestrian detection are not considered major systems of a robotic car.
- argonaut 11y ago
  To address what you and tgflynn are saying. Machine learning != robotics. Robotics is mostly probabilistic inference / search, control theory, etc. The Sebastian Thrun course actually looks great - for robotics. Even computer vision is mostly not related to machine learning. And when machine learning is used, it has usually been simpler, faster "dumber" algorithms. Deep learning has only recently made advances in computer vision and robotics - it's still definitely an open area of research. Even in production it's only going to be a component of a larger system. I would bet the vast majority of the work in Google's self-driving car is related to probabilistic inference / mapping / search / feature-engineering, and mostly does not involve machine learning.
  - tgflynn 11y ago
    The future of both computer vision and robotics is in machine learning, that is where the state of the art will advance. No autonomous vehicle that isn't integrally dependent on machine learning algorithms will ever hit the market, unless the entire road system is redesigned and human drivers are outlawed.
    - argonaut 11y ago
      Maybe. Maybe not. Impossible to tell. People were saying this about SVMs and kernels 10 years ago (which, by the way, is why it's important to have a broad knowledge of probability and machine learning).
spdustin 11y ago
I've been meaning to set aside more time to better understand ML as it applies to NLP. I've wanted for a while to understand how to, say, perform unsupervised training on my iMessage, Facebook and Slack history (which spans many years now) to tag messages, threads, and people with topics, sentiment, frequency, etc. I want to correlate that data with other time-series metrics that I either already have it can acquire easily enough, such as weather, medications taken, apps I have open on my desktop (which could signal context), to see if there are any interesting correlations. I have always thought that depression, anxiety, and AD/HD have had their own impacts on my life that are impossible to quantify without this sort of thing exercise, which could help me understand myself better, and help create a tool chain that would let others get the same kind of insight into their own data. I had always planned to make an open source kit of scripts or maybe Python notebooks out of the endeavor. Things that I get stuck on, which then takes the wind out of my sails: tokenization (specifically handling the Unicode Emoji entities and ascribing meaning to them - Do I use them as tags/signals, or replace with synonyms?), lemmatizing (do I spend time going down the rabbit hole of simplifying all lines of chat to their most basic words?), grouping likes of dialog (if my reply was within ten minutes, consider it part of a "conversation" object), and how best to time stamp things (everything is individually stamped, but for some correlations, the time of day is the important bucket - for others, it may be the calendar day/season/busy work day). It's such a huge domain, I keep spinning my wheels trying to feel like I'm going down a path that will lead to some form of success, no matter how small. Another topic I've tried getting into is using ML to process my Hearthstone logs (live, not historical) to try my own approach on an unreleased project I recently read about that sought to predict opponent's cards. My thought was to create a series of dicts from popular "net decking" sites and compute cosine similarity between the cards an opponent has already played - the other project used game histories to predict the "next card", and I was seeking to predict which archetype my opponent is likely playing, since my own domain knowledge would figure out their likely "signature moves" once I had that clue. I'd maybe expand on it to predict how likely it is the opponent can go lethal on their next turn, given the cards they likely have and the ones on the board. With that topic, I've been trying to figure out state machines and various data structures in Python. Computing similarity I've figured out, with Counters seems to work, but the mechanics of doing so against potentially hundreds of "net decks" is challenging. Is a comprehension the way to go? A matrix function? I have no idea. I guess I'm making this big dump of my own thoughts to see if anyone has any pointers, guidance, example projects, or general knowledge to share or direct me to that could help me figure this stuff out, because I'm excited to learn and I learn best when I have a pet project to which I can apply my newfound knowledge. Anyone in the same boat? Or have anything to suggest?

211 points by vonnik 11y ago

If this feels like jumping in the deep end, I highly recommend Stanford's online course on this subject: http://coursera.org/learn/machine-learning/ http://coursera.org/learn/machine-learning/

vonnik 11y ago
I agree. Andrew Ng's course is a great place to start with ML. We've got a list of great online ML/DL courses here: http://deeplearning4j.org/deeplearningpapers.html http://deeplearning4j.org/deeplearningpapers.html

yohann305 11y ago

Machine Learning is in my bucket list of things to focus on at some point in my life, just like many of you. I know it's a bit off topic, does anyone of you know any practical resource that teaches ML while creating some kind of robot with AI? Thanks in advance.

vonnik 11y ago
Sebastian Thrun teaches a course called AI for Robotics on Udacity: https://www.udacity.com/course/artificial-intelligence-for-robotics--cs373 https://www.udacity.com/course/artificial-intelligence-for-r...
- tgflynn 11y ago
  I took that class several years ago and it was definitely not about machine learning, mostly mapping/localization and route planning. It was a good class but the claim: "Learn how to program all the major systems of a robotic car" is vastly overblown. That would only be true if things like traffic sign recognition and pedestrian detection are not considered major systems of a robotic car.
argonaut 11y ago
To address what you and tgflynn are saying. Machine learning != robotics. Robotics is mostly probabilistic inference / search, control theory, etc. The Sebastian Thrun course actually looks great - for robotics. Even computer vision is mostly not related to machine learning. And when machine learning is used, it has usually been simpler, faster "dumber" algorithms. Deep learning has only recently made advances in computer vision and robotics - it's still definitely an open area of research. Even in production it's only going to be a component of a larger system. I would bet the vast majority of the work in Google's self-driving car is related to probabilistic inference / mapping / search / feature-engineering, and mostly does not involve machine learning.
- tgflynn 11y ago
  The future of both computer vision and robotics is in machine learning, that is where the state of the art will advance. No autonomous vehicle that isn't integrally dependent on machine learning algorithms will ever hit the market, unless the entire road system is redesigned and human drivers are outlawed.
  - argonaut 11y ago
    Maybe. Maybe not. Impossible to tell. People were saying this about SVMs and kernels 10 years ago (which, by the way, is why it's important to have a broad knowledge of probability and machine learning).

spdustin 11y ago

I've been meaning to set aside more time to better understand ML as it applies to NLP. I've wanted for a while to understand how to, say, perform unsupervised training on my iMessage, Facebook and Slack history (which spans many years now) to tag messages, threads, and people with topics, sentiment, frequency, etc. I want to correlate that data with other time-series metrics that I either already have it can acquire easily enough, such as weather, medications taken, apps I have open on my desktop (which could signal context), to see if there are any interesting correlations. I have always thought that depression, anxiety, and AD/HD have had their own impacts on my life that are impossible to quantify without this sort of thing exercise, which could help me understand myself better, and help create a tool chain that would let others get the same kind of insight into their own data. I had always planned to make an open source kit of scripts or maybe Python notebooks out of the endeavor. Things that I get stuck on, which then takes the wind out of my sails: tokenization (specifically handling the Unicode Emoji entities and ascribing meaning to them - Do I use them as tags/signals, or replace with synonyms?), lemmatizing (do I spend time going down the rabbit hole of simplifying all lines of chat to their most basic words?), grouping likes of dialog (if my reply was within ten minutes, consider it part of a "conversation" object), and how best to time stamp things (everything is individually stamped, but for some correlations, the time of day is the important bucket - for others, it may be the calendar day/season/busy work day). It's such a huge domain, I keep spinning my wheels trying to feel like I'm going down a path that will lead to some form of success, no matter how small. Another topic I've tried getting into is using ML to process my Hearthstone logs (live, not historical) to try my own approach on an unreleased project I recently read about that sought to predict opponent's cards. My thought was to create a series of dicts from popular "net decking" sites and compute cosine similarity between the cards an opponent has already played - the other project used game histories to predict the "next card", and I was seeking to predict which archetype my opponent is likely playing, since my own domain knowledge would figure out their likely "signature moves" once I had that clue. I'd maybe expand on it to predict how likely it is the opponent can go lethal on their next turn, given the cards they likely have and the ones on the board. With that topic, I've been trying to figure out state machines and various data structures in Python. Computing similarity I've figured out, with Counters seems to work, but the mechanics of doing so against potentially hundreds of "net decks" is challenging. Is a comprehension the way to go? A matrix function? I have no idea. I guess I'm making this big dump of my own thoughts to see if anyone has any pointers, guidance, example projects, or general knowledge to share or direct me to that could help me figure this stuff out, because I'm excited to learn and I learn best when I have a pet project to which I can apply my newfound knowledge. Anyone in the same boat? Or have anything to suggest?