5 ms·

Learning@home hivemind – train large neural networks across the internet

148 points by sebg 6y ago

smabie 6y ago
One of my friends has a startup where individuals can sell computer time to the highest bidder. I've told him that I didn't think it was a good idea, but this library could change that. I wonder what the performance overhead is like. He is focused on the gaming space, but with this, the data science space might make more sense.
- Nerada 6y ago
  Chessbase actually does something similar, where you can rent other user's computers to run analysis on positions[1]. The users offering up their machines set the price though, as opposed to an auction. [1] https://en.chessbase.com/post/tutorial-how-does-the-engine-cloud-work https://en.chessbase.com/post/tutorial-how-does-the-engine-c...
Nerada 6y ago
I have no idea how ANNs work, but those GPT-3 numbers make it look like the barrier to better AI is an expense issue (compute/financial), whereas I always just assumed we lacked understanding or some better algorithm.
- TaylorAlexander 6y ago
  “The scaling hypothesis” is the name given to the idea that the existing algorithms might be all we need if we just throw more compute at it. Certainly GPT-3 is a very interesting data point here. However we definitely also need better algorithms. It’s a mix of scaling a new algorithms that will get us to AGI.
  - Reelin 6y ago
    There's actually a paper that attempts to quantify the scaling characteristics of NLP models from early this year. (https://arxiv.org/abs/2001.08361 https://arxiv.org/abs/2001.08361) Even if scaling alone ends up solving everything (I doubt it), I'd still feel that very significant improvements ought to be possible from an algorithmic perspective. (I realize that's largely baseless, but for some reason I just can't escape the feeling that current algorithms leave a huge amount of potential on the table.)
  - ricklamers 6y ago
    I think that 'just' scaling todays algorithms is quite a naive approach as it would imply the need of huge amounts of training samples for simple tasks (simple to humans). Given humans tend to need an order of magnitude less samples before being able to generalize I think we need more than just scaled up versions of todays NNs, SVMs, Trees, what-have-you.
    - Reelin 6y ago
      I agree that there are clearly algorithmic improvements remaining to be made. However, a counterpoint to your specific example would be the lottery ticket hypothesis and related weight agnostic neural networks. https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635 https://ai.facebook.com/blog/understanding-the-generalization-of-lottery-tickets-in-neural-networks/ https://ai.facebook.com/blog/understanding-the-generalizatio... https://ai.googleblog.com/2019/08/exploring-weight-agnostic-neural.html https://ai.googleblog.com/2019/08/exploring-weight-agnostic-...
      - ricklamers 6y ago
        The way I interpret the lottery ticket hypothesis is that you don't actually need the full sized networks (with their structure and parameters) in order to perform well at some tasks (when comparing performance against larger networks). I think everyone agrees that most neural networks are highly overparameterized as successful distillation efforts have shown. However, this doesn't directly make my point about sample efficiency of today's algorithms compared to humans less valid. Although what I'll give you is that with smaller networks the required sample size is expected to shrink (due to curse of dimensionality). Although the expressiveness is clearly harmed by the reduced parameter count/altered network structure which possibly reduces the ability of the network to perform well for certain tasks. I think it's important to clearly make a distinction between the required amount of computation and the number of data samples that are necessary when talking about scaling up existing methods. Compute is "cheap", while data isn't. As a side note, I think the usefulness of the lottery ticket hypothesis is mostly about the ability of random initialization to already give a hint about the quality of the 'prior' that is encoded by the network structure. Useful for less computationally intense architecture search as also suggested by the papers and a paper by Andrew Ng on this topic.
  - aiNohY6g 6y ago
    $dF.
awalton 6y ago
Lots of talk about fault tolerance, not a lot of talk about trusting peers and preventing them from introducing bad data into your presumably precious model... So if you're forced to trust all of the peers, how is this better than a cloud? Who out there is training models for purely benevolent reasons (i.e. non-profit seeking) and can trust random nodes? If not for purely benevolent reasons, who out there is going to donate CPU time to training your model, essentially writing you a blank check?
- FeepingCreature 6y ago
  Small to moderate groups of friends that all know each other and are excited about deep learning? edit: Ah, just saw the "what it isn't for" section - apparently not.
- deleted 6y ago
  [deleted]
- hnjst 6y ago
  Shameless plug... http://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent http://papers.nips.cc/paper/6617-machine-learning-with-adver...
  - awalton 6y ago
    That's a damned timely shameless plug :). I'll add it to the reading list.
    - Iv 6y ago
      Thanks :-)
- miedpo 6y ago
  Generally, from what I've read on SETI @ Home, the way these work is they run the same calculations on multiple computers. It's still possible to fool the system but it gets increasingly harder the less portion of computers on the network you own (assuming everybody else has an honest computer)
CleanItUpJanny 6y ago
If Facebook had a community-sourced network for training combat drones, you guys would trip over each other to volunteer your computing resources
- drusepth 6y ago
  I probably would, yeah.
zabhi 6y ago
Why would an average user want to participate in such a network? The only reasons I can think of are tied to benevolence and altruism of the participants. We saw this being successful in the SETI project. I doubt paying participants would ever be profitable enough for either the operator (why not rent a few machines over cloud) or the participants (training would need power and cpu). How about tying the training and consumption of the model together. An internet scale tool with a focused goal, like Alexa/Mycroft for speech and intention recognition, that trains a distributed model while pushing improvements back might be more successful in getting adoption.
- 0-_-0 6y ago
  Lots of people have machines sitting idle that they could rent for neural network training as long as the money they receive is more than the cost of electricity. Cloud computing is significantly more expensive than just the cost of electricity.
- Grimm1 6y ago
  I mean https://minecraftathome.com/minecrafthome/ https://minecraftathome.com/minecrafthome/ is a thing that people do so I kind of expect people will do this as well. And tons of people participate on Folding@home which also uses GPU these days.
- driscoll42 6y ago
  This project will probably be more successful if it can tie into BOINC, which grew out of SETI@Home, and has several hundred thousand computers working on various projects. It'll be a lot easier to get people already donating time/resources to donate to this than starting from the ground up. But you are right about the why? I admit I personally do Folding@Home most of the time, because EVGA gives you up to $10 to spend at EVGA a month by participating, or Boardgamegeek gives badges and some currency to to spent on their site. Which in EVGAs case, has made it so I basically only buy EVGA products now as I have a bunch of EVGA bucks to spend there.
botwriter 6y ago
Wonder how long it takes someone to incorporate this into a botnet.
sdenton4 6y ago
Curious how this deals work moving training data around... If you're dataset is a few gbs moving it around is a good bit of overhead, and a decent chunk of local disk space for the host system. Probably not bad of there's a consistent task, but seems like a big problem if the tasks change often.
- justheuristic 6y ago
  /* hypothesizing */ If you're using it for NLP, your dataset (token ids) typically weighs much less than intermediate tensors. So, i see two scenarios here: (1) distribute data chunks as you train using more conventional bittorrent systems (e.g. https://academictorrents.com https://academictorrents.com but internal) (2) since you most likely use raw unlabeled data (e.g. just text), peers can crawl it straight from the web
  - sdenton4 6y ago
    Yeah, it's probably less of a concern for text tasks, where the data per example is relatively light (though there is a whole internet worth of text data...) I mostly work with audio, where individual examples are ~2MB, so the dataset sizes get very heavy quickly.

148 points by sebg 6y ago

One of my friends has a startup where individuals can sell computer time to the highest bidder. I've told him that I didn't think it was a good idea, but this library could change that. I wonder what the performance overhead is like. He is focused on the gaming space, but with this, the data science space might make more sense.

Nerada 6y ago
Chessbase actually does something similar, where you can rent other user's computers to run analysis on positions[1]. The users offering up their machines set the price though, as opposed to an auction. [1] https://en.chessbase.com/post/tutorial-how-does-the-engine-cloud-work https://en.chessbase.com/post/tutorial-how-does-the-engine-c...

Nerada 6y ago

I have no idea how ANNs work, but those GPT-3 numbers make it look like the barrier to better AI is an expense issue (compute/financial), whereas I always just assumed we lacked understanding or some better algorithm.

TaylorAlexander 6y ago
“The scaling hypothesis” is the name given to the idea that the existing algorithms might be all we need if we just throw more compute at it. Certainly GPT-3 is a very interesting data point here. However we definitely also need better algorithms. It’s a mix of scaling a new algorithms that will get us to AGI.
- Reelin 6y ago
  There's actually a paper that attempts to quantify the scaling characteristics of NLP models from early this year. (https://arxiv.org/abs/2001.08361 https://arxiv.org/abs/2001.08361) Even if scaling alone ends up solving everything (I doubt it), I'd still feel that very significant improvements ought to be possible from an algorithmic perspective. (I realize that's largely baseless, but for some reason I just can't escape the feeling that current algorithms leave a huge amount of potential on the table.)
- ricklamers 6y ago
  I think that 'just' scaling todays algorithms is quite a naive approach as it would imply the need of huge amounts of training samples for simple tasks (simple to humans). Given humans tend to need an order of magnitude less samples before being able to generalize I think we need more than just scaled up versions of todays NNs, SVMs, Trees, what-have-you.
  - Reelin 6y ago
    I agree that there are clearly algorithmic improvements remaining to be made. However, a counterpoint to your specific example would be the lottery ticket hypothesis and related weight agnostic neural networks. https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635 https://ai.facebook.com/blog/understanding-the-generalization-of-lottery-tickets-in-neural-networks/ https://ai.facebook.com/blog/understanding-the-generalizatio... https://ai.googleblog.com/2019/08/exploring-weight-agnostic-neural.html https://ai.googleblog.com/2019/08/exploring-weight-agnostic-...
    - ricklamers 6y ago
      The way I interpret the lottery ticket hypothesis is that you don't actually need the full sized networks (with their structure and parameters) in order to perform well at some tasks (when comparing performance against larger networks). I think everyone agrees that most neural networks are highly overparameterized as successful distillation efforts have shown. However, this doesn't directly make my point about sample efficiency of today's algorithms compared to humans less valid. Although what I'll give you is that with smaller networks the required sample size is expected to shrink (due to curse of dimensionality). Although the expressiveness is clearly harmed by the reduced parameter count/altered network structure which possibly reduces the ability of the network to perform well for certain tasks. I think it's important to clearly make a distinction between the required amount of computation and the number of data samples that are necessary when talking about scaling up existing methods. Compute is "cheap", while data isn't. As a side note, I think the usefulness of the lottery ticket hypothesis is mostly about the ability of random initialization to already give a hint about the quality of the 'prior' that is encoded by the network structure. Useful for less computationally intense architecture search as also suggested by the papers and a paper by Andrew Ng on this topic.
- aiNohY6g 6y ago
  $dF.

awalton 6y ago

Lots of talk about fault tolerance, not a lot of talk about trusting peers and preventing them from introducing bad data into your presumably precious model... So if you're forced to trust all of the peers, how is this better than a cloud? Who out there is training models for purely benevolent reasons (i.e. non-profit seeking) and can trust random nodes? If not for purely benevolent reasons, who out there is going to donate CPU time to training your model, essentially writing you a blank check?

FeepingCreature 6y ago
Small to moderate groups of friends that all know each other and are excited about deep learning? edit: Ah, just saw the "what it isn't for" section - apparently not.
deleted 6y ago
[deleted]
hnjst 6y ago
Shameless plug... http://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent http://papers.nips.cc/paper/6617-machine-learning-with-adver...
- awalton 6y ago
  That's a damned timely shameless plug :). I'll add it to the reading list.
  - Iv 6y ago
    Thanks :-)
miedpo 6y ago
Generally, from what I've read on SETI @ Home, the way these work is they run the same calculations on multiple computers. It's still possible to fool the system but it gets increasingly harder the less portion of computers on the network you own (assuming everybody else has an honest computer)

CleanItUpJanny 6y ago

If Facebook had a community-sourced network for training combat drones, you guys would trip over each other to volunteer your computing resources

drusepth 6y ago
I probably would, yeah.

zabhi 6y ago

Why would an average user want to participate in such a network? The only reasons I can think of are tied to benevolence and altruism of the participants. We saw this being successful in the SETI project. I doubt paying participants would ever be profitable enough for either the operator (why not rent a few machines over cloud) or the participants (training would need power and cpu). How about tying the training and consumption of the model together. An internet scale tool with a focused goal, like Alexa/Mycroft for speech and intention recognition, that trains a distributed model while pushing improvements back might be more successful in getting adoption.

0-_-0 6y ago
Lots of people have machines sitting idle that they could rent for neural network training as long as the money they receive is more than the cost of electricity. Cloud computing is significantly more expensive than just the cost of electricity.
Grimm1 6y ago
I mean https://minecraftathome.com/minecrafthome/ https://minecraftathome.com/minecrafthome/ is a thing that people do so I kind of expect people will do this as well. And tons of people participate on Folding@home which also uses GPU these days.
driscoll42 6y ago
This project will probably be more successful if it can tie into BOINC, which grew out of SETI@Home, and has several hundred thousand computers working on various projects. It'll be a lot easier to get people already donating time/resources to donate to this than starting from the ground up. But you are right about the why? I admit I personally do Folding@Home most of the time, because EVGA gives you up to $10 to spend at EVGA a month by participating, or Boardgamegeek gives badges and some currency to to spent on their site. Which in EVGAs case, has made it so I basically only buy EVGA products now as I have a bunch of EVGA bucks to spend there.

botwriter 6y ago

Wonder how long it takes someone to incorporate this into a botnet.

sdenton4 6y ago

Curious how this deals work moving training data around... If you're dataset is a few gbs moving it around is a good bit of overhead, and a decent chunk of local disk space for the host system. Probably not bad of there's a consistent task, but seems like a big problem if the tasks change often.

justheuristic 6y ago
/* hypothesizing */ If you're using it for NLP, your dataset (token ids) typically weighs much less than intermediate tensors. So, i see two scenarios here: (1) distribute data chunks as you train using more conventional bittorrent systems (e.g. https://academictorrents.com https://academictorrents.com but internal) (2) since you most likely use raw unlabeled data (e.g. just text), peers can crawl it straight from the web
- sdenton4 6y ago
  Yeah, it's probably less of a concern for text tasks, where the data per example is relatively light (though there is a whole internet worth of text data...) I mostly work with audio, where individual examples are ~2MB, so the dataset sizes get very heavy quickly.