4 ms·

Surprised no one has commented this but the latency requires the model to be trained in tiny fragments on each device which is currently a field of research tha

Surprised no one has commented this but the latency requires the model to be trained in tiny fragments on each device which is currently a field of research that is being explored. As it stands now basically all of a model needs to be loaded into memory. There’s a whole field here and people exploring this problem, colloquially solving this would enable Federated Learning and whoever figures this out will far eclipse OpenAI (if it’s ever solved).

k2xl 4y ago
Where can I read more about this field of research?
- latchkey 4y ago
  https://arxiv.org/ https://arxiv.org/ has a ton of papers on it.
  - elicksaur 4y ago
    Do you have some good search terms to get started down the rabbit hole?
    - franl 4y ago
      I too would like to go down this rabbit hole. I am going to poke around using the terms “distributed learning” and “federated learning” (They’re different areas, but somewhat related as far as I understand).
    - franl 4y ago
      This one is a few years old, but seems interesting: https://arxiv.org/abs/1802.05799v2 https://arxiv.org/abs/1802.05799v2
    - vagabund 4y ago
      Probably the biggest recent result: https://arxiv.org/abs/2209.04836 https://arxiv.org/abs/2209.04836 (author thread: https://twitter.com/SamuelAinsworth/status/1569719494645526529 https://twitter.com/SamuelAinsworth/status/15697194946455265...) See also: https://github.com/learning-at-home/hivemind https://github.com/learning-at-home/hivemind and more to OP's incentive structure: https://docs.bittensor.com/ https://docs.bittensor.com/ Latter two intend to beat latency with Mixture-of-Expert models (MoEs). If the results of the former hold, it shows that with a simple algorithmic transformation you can merge two independently trained models in weight-space and have performance functionally equivalent to a model trained monolithically.
- eternalban 4y ago
  A broad covering term is MLOps (like DevOps). https://en.wikipedia.org/wiki/MLOps https://en.wikipedia.org/wiki/MLOps Armed with that term, we get (haven’t read): Machine Learning Operations (MLOps): Overview, Definition, and Architecture https://arxiv.org/abs/2205.02302 https://arxiv.org/abs/2205.02302 [+ps] Better resource: https://ml-ops.org/ https://ml-ops.org/ Based on my (limited) exposure to date, there is tremendous opportunity for software engineers and architects to make impact in ML systems. There is a pronounced lack of seasoned engineering talent (outside of big players like DataBricks, et al) and this knowledge gap sits behind an experience curve that mere IQ can’t jump over. Our experience as software architects and engineers is very valuable. Know this and recognize the value you will bring to the table.
- joaogui1 4y ago
  This initiative is getting some pretty good practical results https://arxiv.org/abs/2209.01188 https://arxiv.org/abs/2209.01188, taking a look at their citations should lead you to other examples in the field
Tostino 4y ago
Well, that explains the lack of a BOINC client for ML tasks at the moment.
cocochanel 4y ago
isn't this already implemented in iPhones for their autocomplete suggestions?
deleted 4y ago
[deleted]
Aromasin 4y ago
To piggy back this comment, Federated Learning actually has a lot of other uses beyond just crowdsourcing. It has benefits to data privacy also. The medical industry has struggled sharing their data with other institutions without compromising their information. They're using it to decentralize the model making process. Intel is doing some work with Penn on the subject now, if people want to read further: https://www.intel.com/content/www/us/en/newsroom/news/intel-penn-medicine-federated-learning-brain-tumor-detection.html#gs.mfttj3 https://www.intel.com/content/www/us/en/newsroom/news/intel-...
- alexmolas 4y ago
  AFAIK Federated Learning is not a magic solution. This post explains how you can exploit FL if there's some untrusted server (http://www.cleverhans.io/2022/04/17/fl-privacy.html http://www.cleverhans.io/2022/04/17/fl-privacy.html)
sfmike 4y ago
bitorrent for GPUs?

k2xl 4y ago

Where can I read more about this field of research?

latchkey 4y ago
https://arxiv.org/ https://arxiv.org/ has a ton of papers on it.
- elicksaur 4y ago
  Do you have some good search terms to get started down the rabbit hole?
  - franl 4y ago
    I too would like to go down this rabbit hole. I am going to poke around using the terms “distributed learning” and “federated learning” (They’re different areas, but somewhat related as far as I understand).
  - franl 4y ago
    This one is a few years old, but seems interesting: https://arxiv.org/abs/1802.05799v2 https://arxiv.org/abs/1802.05799v2
  - vagabund 4y ago
    Probably the biggest recent result: https://arxiv.org/abs/2209.04836 https://arxiv.org/abs/2209.04836 (author thread: https://twitter.com/SamuelAinsworth/status/1569719494645526529 https://twitter.com/SamuelAinsworth/status/15697194946455265...) See also: https://github.com/learning-at-home/hivemind https://github.com/learning-at-home/hivemind and more to OP's incentive structure: https://docs.bittensor.com/ https://docs.bittensor.com/ Latter two intend to beat latency with Mixture-of-Expert models (MoEs). If the results of the former hold, it shows that with a simple algorithmic transformation you can merge two independently trained models in weight-space and have performance functionally equivalent to a model trained monolithically.
eternalban 4y ago
A broad covering term is MLOps (like DevOps). https://en.wikipedia.org/wiki/MLOps https://en.wikipedia.org/wiki/MLOps Armed with that term, we get (haven’t read): Machine Learning Operations (MLOps): Overview, Definition, and Architecture https://arxiv.org/abs/2205.02302 https://arxiv.org/abs/2205.02302 [+ps] Better resource: https://ml-ops.org/ https://ml-ops.org/ Based on my (limited) exposure to date, there is tremendous opportunity for software engineers and architects to make impact in ML systems. There is a pronounced lack of seasoned engineering talent (outside of big players like DataBricks, et al) and this knowledge gap sits behind an experience curve that mere IQ can’t jump over. Our experience as software architects and engineers is very valuable. Know this and recognize the value you will bring to the table.
joaogui1 4y ago
This initiative is getting some pretty good practical results https://arxiv.org/abs/2209.01188 https://arxiv.org/abs/2209.01188, taking a look at their citations should lead you to other examples in the field

Tostino 4y ago

Well, that explains the lack of a BOINC client for ML tasks at the moment.

cocochanel 4y ago

isn't this already implemented in iPhones for their autocomplete suggestions?

deleted 4y ago

[deleted]

Aromasin 4y ago

To piggy back this comment, Federated Learning actually has a lot of other uses beyond just crowdsourcing. It has benefits to data privacy also. The medical industry has struggled sharing their data with other institutions without compromising their information. They're using it to decentralize the model making process. Intel is doing some work with Penn on the subject now, if people want to read further: https://www.intel.com/content/www/us/en/newsroom/news/intel-penn-medicine-federated-learning-brain-tumor-detection.html#gs.mfttj3 https://www.intel.com/content/www/us/en/newsroom/news/intel-...

alexmolas 4y ago
AFAIK Federated Learning is not a magic solution. This post explains how you can exploit FL if there's some untrusted server (http://www.cleverhans.io/2022/04/17/fl-privacy.html http://www.cleverhans.io/2022/04/17/fl-privacy.html)

sfmike 4y ago

bitorrent for GPUs?