5 ms·

The Promise of Hierarchical Reinforcement Learning

84 points by ghosthamlet 8y ago

guineamax2000 8y ago
I think I read about three articles like this per day. Still haven't seen anyone actually using machine learning in any practical application yet, anywhere. Lots of neat toy demos, no real work seems to be getting done.
- jononor 8y ago
  All the useful machine learning is under the hood of a product. Google Search and Google Translate alone are used by hundreds of millions daily. Probably millions use voice control of Siri/Alexa/etc weekly. Amazon, Facebook, Twitter all use machine learning for recommending content. Billions of people use those monthly. It is not just practical, it is a huge industry.
  - guineamax2000 8y ago
    Converging everything towards the median (google results) is not machine learning, and is making search results and the internet far worse. Speech recognition on my phone hasn't improved much since Dragon in the 90s. I'd argue the tech today is worse because it never works without a network connection, so it basically never works unless you're in a big city. Google translate is a running joke. I don't think that I'm allowed to comment further on Google's use of machine learning without violating NDAs, but manually fiddling with weights until you get the magic number you want is not machine learning. Again, not really seeing any benefit. Lots of hype though.
    - whymauri 8y ago
      You're just pulling a No True Scotsman. I'm surprised someone replied to you in good faith.
      - scottlocklin 8y ago
        He's very much not engaging in any rhetorical fallacy: it's an uncommonly clear observation of the actual reality of the current year in machine learning. Which probably means he labors in the field. The main businesses which seems to truly depend on ML (other than maybe FICO) is that of tech journalist and PR dweeb.
        jononor 8y ago
        Sure Facebook, Google and Amazon (for example) would probably operate fine without machine learning, ie their businesses don't "truly depend on it". But that is far from that there has not been "any practical application yet".
        wenc 8y ago
        Are we talking about ML or DL in particular? Also, does ML have to be sophisticated in order for it to "count"? Just off the top of my head, 3D/HD mapping companies like HERE rely on DL-based image recognition to recognize objects at reasonable levels of accuracy -- totally infeasible to do manually at scale. Also, one of the triggers of the renaissance of NNs is that it was shown to outperform traditional computer vision techniques around the early 2010s. USPS does handwriting recognition every single day -- maybe not with DL -- but definitely with some ML algorithm (I was at a talk given by one of the originators of said algorithm). ML is more than just DL. If take the definition of ML encompass to statistical learning -- which it traditionally does -- production ML deployments is extremely pervasive, from industries as varied as finance to chemical manufacturing. The exact FICO algorithm is proprietary, but algorithms of its ilk are pervasive. As you know, at least two other companies (Transunion and Experian) also have their own algorithms. And credit scoring algorithms are among the least sophisticated ML deployments. I'm not able to talk about the ML models I work on, but they're in production and the business relies on them. I guess I'm not seeing the finer points of the argument -- as it stands without further qualification or refinement, it does not seem to be a valid conclusion.
        scottlocklin 8y ago
        I haven't spoken to the Here guys lately, but while they did hire a bunch of DL weenies, it looked like they mostly put them to work doing more useful things. I'll say it again: no company depends on machine learning, other than, maybe Fair Isaac. Many use such things. They don't depend on ML. IMO the trend is in the other direction; many companies will begin to realize what they've spent on models isn't worth the returns.
    - openasocket 8y ago
      I mean, isn't pretty much all image recognition going to involve some form of machine learning at some level? I'm not even sure how you'd be able to do that without some sort of ML system. And that's used plenty in production.
      - scottlocklin 8y ago
        Lots of stuff which can be done here; cigarette companies used to recognize tax stamps using optics hardware.
    - obastani 8y ago
      Considering that Dragon is based on machine learning (hidden Markov models, or HMMs, to be exact) [1], I'm not sure what point you're trying to make. In any case, I used Dragon in the late 90's, and it was terrible, whereas Google's voice recognition on Pixel devices works great for me. One benefit of using deep learning is that they are much less brittle, e.g., working much better for people with accents, in noisy environments, etc. [1] https://en.wikipedia.org/wiki/Dragon_NaturallySpeaking https://en.wikipedia.org/wiki/Dragon_NaturallySpeaking
  - Will_Parker 8y ago
    The snake oil part of it is also a huge industry. A simple statistical algorithm or set of heuristics can be sold for much more if you can pass it off as sophisticated ML.
    - mlthoughts2018 8y ago
      This is not generally true. When you overtly brand products with machine learning or AI, there’s often a customer perception that it’s overly complicated or something they could never understand, which often drives them away. For example, you don’t see too many of the most successful or long-term quant finance products touting machine learning or AI. They do use these techniques (sometimes directly for trading, sometimes only for auxiliary problems), but don’t make a big deal out of it. I worked previously in both adtech & quant finance, and have seen products where ML is the absolute core of the whole thing, yet it wasn’t marketed that way (even recently). I’ve also seen products fail to gain market traction due to overly aggressive ML/AI branding that customers (mostly other enterprise business marketing teams) found alienating and overly complex.
      - loganfrederick 8y ago
        Marketing to actual customers versus marketing to potential investors and acquirers are two different things, which I think explains the difference between the parent and grandparent comments.
    - cheeko1234 8y ago
      Exactly. Case in point: IBM Watson https://www.computerworld.com/article/3321138/did-ibm-put-too-much-stock-in-watson-health-too-soon.html https://www.computerworld.com/article/3321138/did-ibm-put-to... IBM is going around selling this to corporations and agencies as a solution to pretty much everything that involves data analysis.
- b_tterc_p 8y ago
  Why do you keep reading the articles?
  - willvarfar 8y ago
    Not the poster you are replying to, but I enjoy reading everything about ML that turns up on HN even though I know deep down that its not very applicable to my day job. I work on big databases with gazillions of rows that need to be classified etc and hand-made rules-based approaches win every time. And I still read this stuff on HN.
    - deleted 8y ago
      [deleted]
- Will_Parker 8y ago
  Maybe chess isn't practical, but I'm getting a lot of entertainment in following the progress of Leela Chess Zero. https://groups.google.com/forum/#!forum/lczero https://groups.google.com/forum/#!forum/lczero It's right at the critical point where it is essentially equal with the best tree search engine (Stockfish) on high end hardware, and new test results are posted daily in the discord server. Many games are commented on Youtube by Kingscrusher and others. (And yes I know reality isn't a game with a small finite move set, simple deterministic rules, and perfect information.)
- obastani 8y ago
  Machine learning has been used for more than two decades in ATMs to perform optical character recognitions for automatic processing of checks [1]. I believe it is used by the post office as well. It is used by PayPal and other financial services companies for the purposes of fraud detection [2]. It is widely used to filter spam emails as well [3]. It is also widely used in recommender systems, starting with Netflix movie recommendations [4]. While currently more academic, it also has tremendous promise in healthcare diagnostic tasks [5,6]. It has also been used in economics applications such as measuring the global impact of fishing [7] and in poverty prediction in third world countries [8]. [1] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.26.7360&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.26.... [2] https://www.infoworld.com/article/2907877/how-paypal-reduces-fraud-with-machine-learning.html https://www.infoworld.com/article/2907877/how-paypal-reduces... [3] https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering [4] https://en.wikipedia.org/wiki/Netflix_Prize https://en.wikipedia.org/wiki/Netflix_Prize [5] https://jamanetwork.com/journals/jama/fullarticle/2588763 https://jamanetwork.com/journals/jama/fullarticle/2588763 [6] https://www.nature.com/articles/nature21056 https://www.nature.com/articles/nature21056 [7] http://science.sciencemag.org/content/359/6378/904 http://science.sciencemag.org/content/359/6378/904 [8] http://science.sciencemag.org/content/353/6301/790 http://science.sciencemag.org/content/353/6301/790
- plaguuuuuu 8y ago
  I mean, I can go to my google photos and search for "cat" and it will show me pics I've taken of my cat using image classifiers. Most of the algo traders on the stock market are running ML now, and that's the quintessential results oriented environment Internet ads are auctioned off in realtime to ad buyers who use ML to value impressions based on user profile data, demographics etc. Anything involving prediction or classification is likely to already have ML solutions in development or production.
canjobear 8y ago
What I find more promising is methods where hierarchical perception and control emerge naturally from information bottlenecks. Eg https://www.frontiersin.org/articles/10.3389/frobt.2015.00027/full https://www.frontiersin.org/articles/10.3389/frobt.2015.0002...

84 points by ghosthamlet 8y ago

guineamax2000 8y ago

I think I read about three articles like this per day. Still haven't seen anyone actually using machine learning in any practical application yet, anywhere. Lots of neat toy demos, no real work seems to be getting done.

jononor 8y ago
All the useful machine learning is under the hood of a product. Google Search and Google Translate alone are used by hundreds of millions daily. Probably millions use voice control of Siri/Alexa/etc weekly. Amazon, Facebook, Twitter all use machine learning for recommending content. Billions of people use those monthly. It is not just practical, it is a huge industry.
- guineamax2000 8y ago
  Converging everything towards the median (google results) is not machine learning, and is making search results and the internet far worse. Speech recognition on my phone hasn't improved much since Dragon in the 90s. I'd argue the tech today is worse because it never works without a network connection, so it basically never works unless you're in a big city. Google translate is a running joke. I don't think that I'm allowed to comment further on Google's use of machine learning without violating NDAs, but manually fiddling with weights until you get the magic number you want is not machine learning. Again, not really seeing any benefit. Lots of hype though.
  - whymauri 8y ago
    You're just pulling a No True Scotsman. I'm surprised someone replied to you in good faith.
    - scottlocklin 8y ago
      He's very much not engaging in any rhetorical fallacy: it's an uncommonly clear observation of the actual reality of the current year in machine learning. Which probably means he labors in the field. The main businesses which seems to truly depend on ML (other than maybe FICO) is that of tech journalist and PR dweeb.
      - jononor 8y ago
        Sure Facebook, Google and Amazon (for example) would probably operate fine without machine learning, ie their businesses don't "truly depend on it". But that is far from that there has not been "any practical application yet".
      - wenc 8y ago
        Are we talking about ML or DL in particular? Also, does ML have to be sophisticated in order for it to "count"? Just off the top of my head, 3D/HD mapping companies like HERE rely on DL-based image recognition to recognize objects at reasonable levels of accuracy -- totally infeasible to do manually at scale. Also, one of the triggers of the renaissance of NNs is that it was shown to outperform traditional computer vision techniques around the early 2010s. USPS does handwriting recognition every single day -- maybe not with DL -- but definitely with some ML algorithm (I was at a talk given by one of the originators of said algorithm). ML is more than just DL. If take the definition of ML encompass to statistical learning -- which it traditionally does -- production ML deployments is extremely pervasive, from industries as varied as finance to chemical manufacturing. The exact FICO algorithm is proprietary, but algorithms of its ilk are pervasive. As you know, at least two other companies (Transunion and Experian) also have their own algorithms. And credit scoring algorithms are among the least sophisticated ML deployments. I'm not able to talk about the ML models I work on, but they're in production and the business relies on them. I guess I'm not seeing the finer points of the argument -- as it stands without further qualification or refinement, it does not seem to be a valid conclusion.
        scottlocklin 8y ago
        I haven't spoken to the Here guys lately, but while they did hire a bunch of DL weenies, it looked like they mostly put them to work doing more useful things. I'll say it again: no company depends on machine learning, other than, maybe Fair Isaac. Many use such things. They don't depend on ML. IMO the trend is in the other direction; many companies will begin to realize what they've spent on models isn't worth the returns.
  - openasocket 8y ago
    I mean, isn't pretty much all image recognition going to involve some form of machine learning at some level? I'm not even sure how you'd be able to do that without some sort of ML system. And that's used plenty in production.
    - scottlocklin 8y ago
      Lots of stuff which can be done here; cigarette companies used to recognize tax stamps using optics hardware.
  - obastani 8y ago
    Considering that Dragon is based on machine learning (hidden Markov models, or HMMs, to be exact) [1], I'm not sure what point you're trying to make. In any case, I used Dragon in the late 90's, and it was terrible, whereas Google's voice recognition on Pixel devices works great for me. One benefit of using deep learning is that they are much less brittle, e.g., working much better for people with accents, in noisy environments, etc. [1] https://en.wikipedia.org/wiki/Dragon_NaturallySpeaking https://en.wikipedia.org/wiki/Dragon_NaturallySpeaking
- Will_Parker 8y ago
  The snake oil part of it is also a huge industry. A simple statistical algorithm or set of heuristics can be sold for much more if you can pass it off as sophisticated ML.
  - mlthoughts2018 8y ago
    This is not generally true. When you overtly brand products with machine learning or AI, there’s often a customer perception that it’s overly complicated or something they could never understand, which often drives them away. For example, you don’t see too many of the most successful or long-term quant finance products touting machine learning or AI. They do use these techniques (sometimes directly for trading, sometimes only for auxiliary problems), but don’t make a big deal out of it. I worked previously in both adtech & quant finance, and have seen products where ML is the absolute core of the whole thing, yet it wasn’t marketed that way (even recently). I’ve also seen products fail to gain market traction due to overly aggressive ML/AI branding that customers (mostly other enterprise business marketing teams) found alienating and overly complex.
    - loganfrederick 8y ago
      Marketing to actual customers versus marketing to potential investors and acquirers are two different things, which I think explains the difference between the parent and grandparent comments.
  - cheeko1234 8y ago
    Exactly. Case in point: IBM Watson https://www.computerworld.com/article/3321138/did-ibm-put-too-much-stock-in-watson-health-too-soon.html https://www.computerworld.com/article/3321138/did-ibm-put-to... IBM is going around selling this to corporations and agencies as a solution to pretty much everything that involves data analysis.
b_tterc_p 8y ago
Why do you keep reading the articles?
- willvarfar 8y ago
  Not the poster you are replying to, but I enjoy reading everything about ML that turns up on HN even though I know deep down that its not very applicable to my day job. I work on big databases with gazillions of rows that need to be classified etc and hand-made rules-based approaches win every time. And I still read this stuff on HN.
  - deleted 8y ago
    [deleted]
Will_Parker 8y ago
Maybe chess isn't practical, but I'm getting a lot of entertainment in following the progress of Leela Chess Zero. https://groups.google.com/forum/#!forum/lczero https://groups.google.com/forum/#!forum/lczero It's right at the critical point where it is essentially equal with the best tree search engine (Stockfish) on high end hardware, and new test results are posted daily in the discord server. Many games are commented on Youtube by Kingscrusher and others. (And yes I know reality isn't a game with a small finite move set, simple deterministic rules, and perfect information.)
obastani 8y ago
Machine learning has been used for more than two decades in ATMs to perform optical character recognitions for automatic processing of checks [1]. I believe it is used by the post office as well. It is used by PayPal and other financial services companies for the purposes of fraud detection [2]. It is widely used to filter spam emails as well [3]. It is also widely used in recommender systems, starting with Netflix movie recommendations [4]. While currently more academic, it also has tremendous promise in healthcare diagnostic tasks [5,6]. It has also been used in economics applications such as measuring the global impact of fishing [7] and in poverty prediction in third world countries [8]. [1] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.26.7360&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.26.... [2] https://www.infoworld.com/article/2907877/how-paypal-reduces-fraud-with-machine-learning.html https://www.infoworld.com/article/2907877/how-paypal-reduces... [3] https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering [4] https://en.wikipedia.org/wiki/Netflix_Prize https://en.wikipedia.org/wiki/Netflix_Prize [5] https://jamanetwork.com/journals/jama/fullarticle/2588763 https://jamanetwork.com/journals/jama/fullarticle/2588763 [6] https://www.nature.com/articles/nature21056 https://www.nature.com/articles/nature21056 [7] http://science.sciencemag.org/content/359/6378/904 http://science.sciencemag.org/content/359/6378/904 [8] http://science.sciencemag.org/content/353/6301/790 http://science.sciencemag.org/content/353/6301/790
plaguuuuuu 8y ago
I mean, I can go to my google photos and search for "cat" and it will show me pics I've taken of my cat using image classifiers. Most of the algo traders on the stock market are running ML now, and that's the quintessential results oriented environment Internet ads are auctioned off in realtime to ad buyers who use ML to value impressions based on user profile data, demographics etc. Anything involving prediction or classification is likely to already have ML solutions in development or production.

canjobear 8y ago

What I find more promising is methods where hierarchical perception and control emerge naturally from information bottlenecks. Eg https://www.frontiersin.org/articles/10.3389/frobt.2015.00027/full https://www.frontiersin.org/articles/10.3389/frobt.2015.0002...