5 ms·

This implies more centralization, as those with cheap access to vast compute gain a bigger relative edge.

sgillen 8y ago
Yes, unfortunately both data and compute will probably become more and more centralized. At least the algorithmic components have a chance to becone available to everyone.
- samstave 8y ago
  Here is an off the cuff thought: What if there was (or maybe there already is?) a system which is distributed such as was SETI back in the day and its a massively distributed general AI that can be used - and people on a mass scale allow for slices of their compute to be part of the system?
  - tlrobinson 8y ago
    So Skynet, but it lets you rent parts of itself? I'm sure someone is writing an ICO whitepaper for this right now, if they haven't already...
    - nostrademons 8y ago
      EOS: https://eos.io/ https://eos.io/ Raised $2.7B in its ICO, currently trading at a market cap of $10B. FileCoin: https://filecoin.io/ https://filecoin.io/ Raised $257M in its ICO. Tezos: https://tezos.com/ https://tezos.com/ Raised $232M in its ICO. Those are the 3 largest ICOs of all time, so yes, there is definitely a market for renting part of Skynet. The actual technology may or may not be vaporware or a scam. IMHO the way you build a decentralized P2P system is to give a single really smart programmer enough to live on for a couple years and see what he comes up with, not throw a billion dollars at a Cayman Islands corporation that may or may not use it for anything productive. Sorta like what Ethereum did.
      - tlrobinson 8y ago
        Are any of those platforms suitable for running deep learning algorithms? I think Golem is closer https://golem.network/ https://golem.network/ And some others: https://www.investinblockchain.com/distributed-computing-blockchain-projects/ https://www.investinblockchain.com/distributed-computing-blo... But I'm skeptical of distributed computing blockchains. I think a) it's unlikely a distributed compute network can compete with highly optimized datacenters running TPUs or whatever, b) people are unlikely to trust distributed compute networks with their proprietary data (maybe acceptable for CGI rendering and some other specific use-cases)
  - PeterisP 8y ago
    The current machine learning paradigms are hard to distribute efficiently - they can be parallelized but require significant ongoing communication between nodes, you can't really split a problem into separate subtasks and merge them only in the end, you need to transfer all the calculated values after, say, each iteration. I'd guess that having ten extra machines in the same rack would be more valuable than a thousand remote machines with limited network bandwidth.
  - TeMPOraL 8y ago
    > its a massively distributed general AI You've just described a centralized system. Centralization can happen at different layers - not all technical. The ultimate centralization is ownership, as defined legally.
  - hodgesrm 8y ago
    One problem with the SETI model is that a lot of the increase in available compute capacity is due to specialized GPU or TPU processors, which aren't widely available outside of purpose-built data centers. Trying to offload ML workloads to general purpose CPUs would likely be quite wasteful in terms of power consumption unless you can somehow get access to graphics processors.
vokep 8y ago
This is why I am working on raspberry pi based neural net things We have learned a lot using big computing which can still inform better efficiency of AI on smaller computing units. Raspi is pretty good because it is quite limiting, but also quite capable.
- tlrobinson 8y ago
  Is it really worth training on a Raspberry Pi? It seems vastly underpowered compared to even modest desktop hardware.
  - mc808 8y ago
    It's worth having a self-contained option available. For something like a drone, you don't need all that much computation for on-the-fly PID tuning in response to changing weather or different piloting styles, etc.
    - chillydawg 8y ago
      You've hooked a NN up to a PID? How's that going? It's hard enough tuning those things by hand using squishy human brain networks.
      - mc808 8y ago
        I haven't done it myself yet, it's on the to-do list. There is a lot of academic material on PID autotuning, not always with neural networks but that seems the most straightforward way. A Raspberry Pi is probably overkill for the job, actually.
    - tlrobinson 8y ago
      Inference on embedded hardware makes sense, but training no so much.

Yes, unfortunately both data and compute will probably become more and more centralized. At least the algorithmic components have a chance to becone available to everyone.

samstave 8y ago
Here is an off the cuff thought: What if there was (or maybe there already is?) a system which is distributed such as was SETI back in the day and its a massively distributed general AI that can be used - and people on a mass scale allow for slices of their compute to be part of the system?
- tlrobinson 8y ago
  So Skynet, but it lets you rent parts of itself? I'm sure someone is writing an ICO whitepaper for this right now, if they haven't already...
  - nostrademons 8y ago
    EOS: https://eos.io/ https://eos.io/ Raised $2.7B in its ICO, currently trading at a market cap of $10B. FileCoin: https://filecoin.io/ https://filecoin.io/ Raised $257M in its ICO. Tezos: https://tezos.com/ https://tezos.com/ Raised $232M in its ICO. Those are the 3 largest ICOs of all time, so yes, there is definitely a market for renting part of Skynet. The actual technology may or may not be vaporware or a scam. IMHO the way you build a decentralized P2P system is to give a single really smart programmer enough to live on for a couple years and see what he comes up with, not throw a billion dollars at a Cayman Islands corporation that may or may not use it for anything productive. Sorta like what Ethereum did.
    - tlrobinson 8y ago
      Are any of those platforms suitable for running deep learning algorithms? I think Golem is closer https://golem.network/ https://golem.network/ And some others: https://www.investinblockchain.com/distributed-computing-blockchain-projects/ https://www.investinblockchain.com/distributed-computing-blo... But I'm skeptical of distributed computing blockchains. I think a) it's unlikely a distributed compute network can compete with highly optimized datacenters running TPUs or whatever, b) people are unlikely to trust distributed compute networks with their proprietary data (maybe acceptable for CGI rendering and some other specific use-cases)
- PeterisP 8y ago
  The current machine learning paradigms are hard to distribute efficiently - they can be parallelized but require significant ongoing communication between nodes, you can't really split a problem into separate subtasks and merge them only in the end, you need to transfer all the calculated values after, say, each iteration. I'd guess that having ten extra machines in the same rack would be more valuable than a thousand remote machines with limited network bandwidth.
- TeMPOraL 8y ago
  > its a massively distributed general AI You've just described a centralized system. Centralization can happen at different layers - not all technical. The ultimate centralization is ownership, as defined legally.
- hodgesrm 8y ago
  One problem with the SETI model is that a lot of the increase in available compute capacity is due to specialized GPU or TPU processors, which aren't widely available outside of purpose-built data centers. Trying to offload ML workloads to general purpose CPUs would likely be quite wasteful in terms of power consumption unless you can somehow get access to graphics processors.

vokep 8y ago

This is why I am working on raspberry pi based neural net things We have learned a lot using big computing which can still inform better efficiency of AI on smaller computing units. Raspi is pretty good because it is quite limiting, but also quite capable.

tlrobinson 8y ago
Is it really worth training on a Raspberry Pi? It seems vastly underpowered compared to even modest desktop hardware.
- mc808 8y ago
  It's worth having a self-contained option available. For something like a drone, you don't need all that much computation for on-the-fly PID tuning in response to changing weather or different piloting styles, etc.
  - chillydawg 8y ago
    You've hooked a NN up to a PID? How's that going? It's hard enough tuning those things by hand using squishy human brain networks.
    - mc808 8y ago
      I haven't done it myself yet, it's on the to-do list. There is a lot of academic material on PID autotuning, not always with neural networks but that seems the most straightforward way. A Raspberry Pi is probably overkill for the job, actually.
  - tlrobinson 8y ago
    Inference on embedded hardware makes sense, but training no so much.