4 ms·

As I understand the whole point is to create decentralized marketplace without wasting energy on useless tasks (aside for proving energy use)

by gglon 7y ago

heyitsguay 7y ago
Wouldn't it waste even less energy to not use a PoW scheme at all though? Just set up a webpage that connects people with tasks to people with hardware. On another note, digging in closer, the proposed training scheme (Section 4.3) sounds like it would be incredibly inefficient for any large neural models - you're computing your own updates via gradient descent, but also getting compressed updates from peers on the network, and then broadcasting your own updates out. For reference, at work I might be training a model with, say, 30M parameters, and I get about 4.5 training iterations per second on a GTX 1080. What would that drop to with peer broadcasting added in? They mention only sending weight updates that under/over-flow a certain value, is that enough to avoid, e.g., a 5x drop in training speed? Given that training time is already a big bottleneck for ML work, this seems critical. Section 7.2 seems to address this, referencing Supplementary Materials (Appendix F). The numbers there don't look great - with the optimizations they mention, for GPU timing in Figure F.3 it looks like local training ops account for about 21ms, and the protocol overhead is something like 100ms. I also don't see an apples-to-apples comparison of "here's how long this example takes to train locally vs. with this protocol", which isn't a good sign.
- gglon 7y ago
  Webpage is also centralized. The whole point in creating PoW is to create a task that would be more expensive on the network than on the market. ie some of it would be "wasteful" - used only to secure decentralized network (see my other post)
  - heyitsguay 7y ago
    Right, I understand that. I'm saying from the point of view of the task submitters, the PoW approach is unnecessary and adds time and cost overhead to what is ostensibly the main goal of their participation: getting an ML model trained.
    - gglon 7y ago
      Yes, then the overhead(+ incentive) should be subsidized by Peers - normal users that just use the network eg. for transactions. As a tx fee or inflation.

As I understand the whole point is to create decentralized marketplace without wasting energy on useless tasks (aside for proving energy use)

by gglon 7y ago

As I understand the whole point is to create decentralized marketplace without wasting energy on useless tasks (aside for proving energy use)

heyitsguay 7y ago

Wouldn't it waste even less energy to not use a PoW scheme at all though? Just set up a webpage that connects people with tasks to people with hardware. On another note, digging in closer, the proposed training scheme (Section 4.3) sounds like it would be incredibly inefficient for any large neural models - you're computing your own updates via gradient descent, but also getting compressed updates from peers on the network, and then broadcasting your own updates out. For reference, at work I might be training a model with, say, 30M parameters, and I get about 4.5 training iterations per second on a GTX 1080. What would that drop to with peer broadcasting added in? They mention only sending weight updates that under/over-flow a certain value, is that enough to avoid, e.g., a 5x drop in training speed? Given that training time is already a big bottleneck for ML work, this seems critical. Section 7.2 seems to address this, referencing Supplementary Materials (Appendix F). The numbers there don't look great - with the optimizations they mention, for GPU timing in Figure F.3 it looks like local training ops account for about 21ms, and the protocol overhead is something like 100ms. I also don't see an apples-to-apples comparison of "here's how long this example takes to train locally vs. with this protocol", which isn't a good sign.

gglon 7y ago
Webpage is also centralized. The whole point in creating PoW is to create a task that would be more expensive on the network than on the market. ie some of it would be "wasteful" - used only to secure decentralized network (see my other post)
- heyitsguay 7y ago
  Right, I understand that. I'm saying from the point of view of the task submitters, the PoW approach is unnecessary and adds time and cost overhead to what is ostensibly the main goal of their participation: getting an ML model trained.
  - gglon 7y ago
    Yes, then the overhead(+ incentive) should be subsidized by Peers - normal users that just use the network eg. for transactions. As a tx fee or inflation.