3 ms·

Would it be feasible for me to run their learning avatar at home? giving it tasks to learn that i assign it? Or do you really need a server farm for this?

Maybestring 9y ago
If you allow untrusted users to define tasks for shared agents, then you must assume you will have malicous or incompetent users defining 'poison' tasks. These would be tasks that bias agents toward taking unproductive or harmful actions. Afaik, this problem hasn't been solved.
- radarsat1 9y ago
  While you make a fair point, who mentioned anything about untrusted users? You seem to be off on a tangent..
  - Maybestring 9y ago
    The question is about running a node in a distributed learning algorithm. The algorithm depends on trusted nodes to function.
- lawlessone 9y ago
  "untrusted users" I mean using this at home on my own machine, not uploading it..
  - Maybestring 9y ago
    I don't understand the question then. You want to run an actor disconnected from the learner, but you want it to learn? Edit: Or... You mean running the whole distributed learning algorithm on one machine. I don't see why not, but I doubt it would outperform something intended to run on a single machine.
nazgul17 9y ago
I don't think it will work well unless you have several actors. IIRC, one of the main factors giving A3C an edge compared to previous solutions, is that multiple learners from different episodes contribute experience at the same time. This avoids weights being updated too much in one direction. This is similar to what using batches does in supervised learning.
gwern 9y ago
The author addresses this on Reddit: https://www.reddit.com/r/MachineLearning/comments/7vkvg5/r_impala_scalable_distributed_deeprl_with/dtu6sod/ https://www.reddit.com/r/MachineLearning/comments/7vkvg5/r_i... DRL is currently CPU-heavy and GPU-lightweight, since you spend most of your time exploring the environment and only a little bit of work updating the neural network on the GPU, so if you have a fairly typical home computer like 6-cores + a Nvidia 1080 GPU, your GPU will go under-utilized while it waits on the 6 actors. So it'll still work, but obviously your wallclock is going to be a lot longer than the numbers in OP and it'll be somewhat worse than you think because your system is unbalanced and has reduced throughput.
OkGoDoIt 9y ago
Alternatively, could this be some sort of Folding@Home model where a variable number of users can train actors for whatever learner they partner with?

Maybestring 9y ago

If you allow untrusted users to define tasks for shared agents, then you must assume you will have malicous or incompetent users defining 'poison' tasks. These would be tasks that bias agents toward taking unproductive or harmful actions. Afaik, this problem hasn't been solved.

radarsat1 9y ago
While you make a fair point, who mentioned anything about untrusted users? You seem to be off on a tangent..
- Maybestring 9y ago
  The question is about running a node in a distributed learning algorithm. The algorithm depends on trusted nodes to function.
lawlessone 9y ago
"untrusted users" I mean using this at home on my own machine, not uploading it..
- Maybestring 9y ago
  I don't understand the question then. You want to run an actor disconnected from the learner, but you want it to learn? Edit: Or... You mean running the whole distributed learning algorithm on one machine. I don't see why not, but I doubt it would outperform something intended to run on a single machine.

nazgul17 9y ago

I don't think it will work well unless you have several actors. IIRC, one of the main factors giving A3C an edge compared to previous solutions, is that multiple learners from different episodes contribute experience at the same time. This avoids weights being updated too much in one direction. This is similar to what using batches does in supervised learning.

gwern 9y ago

The author addresses this on Reddit: https://www.reddit.com/r/MachineLearning/comments/7vkvg5/r_impala_scalable_distributed_deeprl_with/dtu6sod/ https://www.reddit.com/r/MachineLearning/comments/7vkvg5/r_i... DRL is currently CPU-heavy and GPU-lightweight, since you spend most of your time exploring the environment and only a little bit of work updating the neural network on the GPU, so if you have a fairly typical home computer like 6-cores + a Nvidia 1080 GPU, your GPU will go under-utilized while it waits on the 6 actors. So it'll still work, but obviously your wallclock is going to be a lot longer than the numbers in OP and it'll be somewhat worse than you think because your system is unbalanced and has reduced throughput.

OkGoDoIt 9y ago

Alternatively, could this be some sort of Folding@Home model where a variable number of users can train actors for whatever learner they partner with?