3 ms·

That’s zero reason to reach for a bazooka like reinforcement learning though. You could use simple Thompson Sampling or many other multi-armed bandit methods.

by mlthoughts2018 6y ago

That’s zero reason to reach for a bazooka like reinforcement learning though. You could use simple Thompson Sampling or many other multi-armed bandit methods. Balancing learning vs serving the optimal result is fine - many companies have approached ranking and recommendation that way for decades - but reinforcement learning would still not be justified unless you present some extremely compelling evidence.

mlthoughts2018 6y ago
Also to be clear - I think it’s great to hear this perspective and putting together the interview is well done. I’m not trying to criticize you nor the merit of talking about this topic - it is well worth it. I was just asking, since you develop these kinds of interviews, what do you think would be a good way to get the other side of the story and talk to big tech practitioners who do not agree with the leap to reinforcement learning?
agbell 6y ago
Isn't multi-armed bandit a simple reinforcement learning algo? It is used in reagent's introductory tutorial: https://reagent.ai/rasp_tutorial.html https://reagent.ai/rasp_tutorial.html Replying here to sibling: Thank you for the feedback. I think it is fair to say this interview does not explore in depth the issues that these techniques can cause and it certainly only presents one side. I think the recommendation to get more than one perspective is a good one. Let me know if there is anyone specific you recommend I talk to.
- mlthoughts2018 6y ago
  That’s a lot of semantic hairsplitting. They are both “reinforcement learning” in the same way a Honda Civic and an aircraft carrier are both “vehicles.”
- srean 6y ago
  > Isn't multi-armed bandit a simple reinforcement learning algo It is indeed. Its a restricted form of it. In RL one can allow a state change after an action which in turn can make the same 'arms'|'actions' behave differently because their behavior is tied to the state. The state one lands in can also exercise control over which state you end up next. Its for this reason some extra book keeping is necessary for full-fledged RL. But you are absolutely right that bandits are considered a simplified version of RL. By controlling the size of the state space one can control how bandit like the solution will behave. There is also something called a contextual bandit that sits between pure bandits and RL. CBs do not have state change, but they do have access to a side information that can affect the 'arms'. In RL one needs to think not only about the reward but also about the possibility of ending up in a 'dead-end| hard-to-recover-from' state because the immediate reward was high. CBs do not have such 'traps' but have more modeling power because the reward of an arm can depend on this side-information, usually called context. The heat that you are getting from some comments is unwarranted. EDIT: Holy mother of monkey milk you have a ton of super interesting interviews ! Glad I ran into your content. Better late than never.
  - agbell 6y ago
    Bandits being reinforcement makes sense! Thank you, if you are looking for a recommendation "Software That Doesn't Suck" is a personal fav: https://corecursive.com/software-that-doesnt-suck-with-jim-blandy/ https://corecursive.com/software-that-doesnt-suck-with-jim-b...
    - srean 6y ago
      You had me at Brian Kernighan's interview. I don't think I have met a more modest man. Once upon a time I had my open cube just behind his open cube. I had no idea who he was and his modesty certainly did not make it any easier. Once he had got locked out of the floor and I had to let him in. Its only after that I came to notice his name tag
  - PartiallyTyped 6y ago
    I'd say that in CBs the action does not affect the distribution of future states.
  - mlthoughts2018 6y ago
    This comment appears to take an unwarranted parochial approach to how to taxonomically call something “reinforcement learning.” This is just not relevant to the content of the interview and the discussion of the large-scale recommendation problem as RL.

mlthoughts2018 6y ago

Also to be clear - I think it’s great to hear this perspective and putting together the interview is well done. I’m not trying to criticize you nor the merit of talking about this topic - it is well worth it. I was just asking, since you develop these kinds of interviews, what do you think would be a good way to get the other side of the story and talk to big tech practitioners who do not agree with the leap to reinforcement learning?

agbell 6y ago

Isn't multi-armed bandit a simple reinforcement learning algo? It is used in reagent's introductory tutorial: https://reagent.ai/rasp_tutorial.html https://reagent.ai/rasp_tutorial.html Replying here to sibling: Thank you for the feedback. I think it is fair to say this interview does not explore in depth the issues that these techniques can cause and it certainly only presents one side. I think the recommendation to get more than one perspective is a good one. Let me know if there is anyone specific you recommend I talk to.

mlthoughts2018 6y ago
That’s a lot of semantic hairsplitting. They are both “reinforcement learning” in the same way a Honda Civic and an aircraft carrier are both “vehicles.”
srean 6y ago
> Isn't multi-armed bandit a simple reinforcement learning algo It is indeed. Its a restricted form of it. In RL one can allow a state change after an action which in turn can make the same 'arms'|'actions' behave differently because their behavior is tied to the state. The state one lands in can also exercise control over which state you end up next. Its for this reason some extra book keeping is necessary for full-fledged RL. But you are absolutely right that bandits are considered a simplified version of RL. By controlling the size of the state space one can control how bandit like the solution will behave. There is also something called a contextual bandit that sits between pure bandits and RL. CBs do not have state change, but they do have access to a side information that can affect the 'arms'. In RL one needs to think not only about the reward but also about the possibility of ending up in a 'dead-end| hard-to-recover-from' state because the immediate reward was high. CBs do not have such 'traps' but have more modeling power because the reward of an arm can depend on this side-information, usually called context. The heat that you are getting from some comments is unwarranted. EDIT: Holy mother of monkey milk you have a ton of super interesting interviews ! Glad I ran into your content. Better late than never.
- agbell 6y ago
  Bandits being reinforcement makes sense! Thank you, if you are looking for a recommendation "Software That Doesn't Suck" is a personal fav: https://corecursive.com/software-that-doesnt-suck-with-jim-blandy/ https://corecursive.com/software-that-doesnt-suck-with-jim-b...
  - srean 6y ago
    You had me at Brian Kernighan's interview. I don't think I have met a more modest man. Once upon a time I had my open cube just behind his open cube. I had no idea who he was and his modesty certainly did not make it any easier. Once he had got locked out of the floor and I had to let him in. Its only after that I came to notice his name tag
- PartiallyTyped 6y ago
  I'd say that in CBs the action does not affect the distribution of future states.
- mlthoughts2018 6y ago
  This comment appears to take an unwarranted parochial approach to how to taxonomically call something “reinforcement learning.” This is just not relevant to the content of the interview and the discussion of the large-scale recommendation problem as RL.