4 ms·

SETI made sense because there is a lot of data where you download chunk and do expensive computation and return thin result. Model training is unlike that. It'

SETI made sense because there is a lot of data where you download chunk and do expensive computation and return thin result. Model training is unlike that. It's large state that is constantly updated and updates require full, up to date state. This means you cannot distribute it efficiently over slow network with many smaller workers. That's why NVIDIA is providing scalable clusters with specialized connectivity so they have ultra low latency and massive throughput. Even in those setups it takes ie. a month to train base model. Converted to distributed setup this same task would take billions of years - ie. it's not feasible. There aren't any known ways of contributing computation without access to the full state. This would require completely different architecture, not only "different than transformers" but "different than gradient descent", which would be basically creating new branch in machine learning and starting from zero. Safe bet is on "ain't going to happen" - better to focus on current state of art and keep advancing it until it builds itself and anything else we can dream of to reach this "mission fucking accomplished".

sigmoid10 3y ago
That's wrong. What you described is data parallelism and it would indeed be very tricky to e.g. sync gradients across machines. But this is not the only method of training neural nets (transformers or any other kind) in parallel. If we'd like to train, say, a human brain complexity level model with 10^15 parameters, we'd need a model parallelism approach anyways. It introduces a bit of complexity since you need to make sure that each distributed part of the model can run individually with roughly the same amount of compute, but you no longer need to worry about syncing anything (or have the entire state of anything on one machine). The real questions is if you can find enough people to run this who will never be able to run it themselves in the end, because inference alone will still require a supercluster. If you have access to that, you might as well train something on it today.
- mirekrusin 3y ago
  Lack of data parallelism is implied by computation that is performed. You gradient descend on your state. Each step needs to work on up to date state otherwise you're computing gradient descend from state that doesn't exist anymore and your computed gradient descent delta is nonsensical if applied to the most recent state (it was calculated on old one, direction that your computation calculated is now wrong). You also can't calculate it without having access to the whole state. You have to do full forward and backward pass and mutate weights. There aren't any ways of slicing and distributing that make sense in terms of efficiency. The reason is that too much data at too high frequency needs to be mutated and then made readable. That's also the reason why nvidia is focusing so much on hyper efficient interconnects - because that's the bottleneck. Computation itself is way ahead of in/out data transfer. Data transfer is the main problem and going in the direction of architecture that dramatically reduces it by several orders of magnitude is just not the way to go. If somebody solves this problem it'll mean they solved much more interesting problem – because it'll mean you can locally uptrain model and inject this knowledge into bigger one arbitrarily.
  - sigmoid10 3y ago
    Your gradient descent is an operation on a directed acyclic graph. The graph itself is stateless. You can do parts of the graph without needing to have access to the entire graph, particularly for transformers. In fact this is already done today for training and inference of large models. The transfer bottleneck is for currently used model sizes and architectures. There's nothing to stop you from building a model so complex that compute itself becomes the bottleneck rather than data transfer. Except its ultimate usability of course, as I already mentioned.
    - mirekrusin 3y ago
      Your DAG is big. It's stateless for single pass. Next one doesn't operate on it anymore, it operates on new, updated one from previous step. It has fully connected sub DAGs. There is nothing stopping you from distributing assembly/machine code for CPU instructions, yet nobody does it because it doesn't make sense from performance perspective. Or amazon driving truck from one depo to other to unload one package at a time to "distribute" unloading because "distributing = faster".
      - sigmoid10 3y ago
        Remember the famouse Tannenbaum quote: "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." It's only a question of scale and not of any principles.
  - ajb 3y ago
    Openai had a more distributable algorithm: https://openai.com/research/evolution-strategies https://openai.com/research/evolution-strategies But given that they don't seem to have worked on it since, I guess it wasn't too successful. But maybe there is a way
    - mirekrusin 3y ago
      Yes, if there was something interesting there you'd think since 2017 something would happen. Reinforcement Learning (that is compared with) is not particularly famous for its performance (it is it's biggest issue and reason for not being used that much). Also transformers don't use it at all.
mejutoco 3y ago
That is the challenging part indeed. But if we think of mixture of experts models outperforming "monolithic" models, why not? Maybe instead of 8 you can do 1000 and that is easy to paralellize. It sounds worth exploring to me.
- l33tman 3y ago
  I think the MoE models are trained together just like any other network though, including the dispatcher layer that has to learn which "expert" route each token to. Perhaps you could do some kind of technically worse model architecture that is trained separately and then a more complex dispatcher that then learns to utilize the individually trained experts as best as it can?
- nabakin 3y ago
  I don't think MoE allows for that either. You'd have to come up with a whole new architecture that allows parts to be trained independently and still somehow be merged together in the end.
  - nsagent 3y ago
    This paper addresses that issue and allows fully independent training of the experts: https://arxiv.org/abs/2303.14177 https://arxiv.org/abs/2303.14177
    - tkgally 3y ago
      This one, from a couple of days ago, might address that issue as well: https://arxiv.org/abs/2403.13187 https://arxiv.org/abs/2403.13187
      - mirekrusin 3y ago
        This approach works on merging finetunes. Base model was still trained in usual, non distributed way (by far the most cost). Fine tunes were also trained in usual, non distributed way. Proposed approach tries out several combinations to pick one that seems to perform better (where combination means ie. adhoc per layer operation). Merging is not distributed as well. There is not much distribution happening overall beyond the fact that fine tunes were trained independently. Taking weight averages, weighted weight averages, trimming low diffs, doing arithmetic (subtracting base model from fine tune) etc. are all ad hoc trials throwing something on the wall and seeing what sticks the most. None of those work well. For distributed training to work we'd have to have better algebra around this multidimentional/multilayer/multiconnectivity state. We don't have it and it has many problems, ie. evaluation is way too expensive. But solving "no need to rerun through whole training/benchmark corpus to see if my tiny change is better or not" problem will mean we solved problem of extracting essence of intelligence. If we do that, then hyper-efficient data centers will still keep beating out any distributed approach and it's all largely irrelevant because that's pure AGI already.
nsagent 3y ago
You're right that parameter updates typically require full state, but during my PhD I've explored some possibilities to address this limitation (unfortunately, my ideas didn't pan out in the time I had). That said, there is research that has explored this topic and made some progress, such as this paper: https://arxiv.org/abs/2303.14177 https://arxiv.org/abs/2303.14177
- mirekrusin 3y ago
  Unfortunately it's hardly progress. MoE expert models are still large, have to be trained in usual, linear way, this approach requires training set classification upfront, each expert model is completely independent, each has to relearn concepts, your overall model is as good as dedicated expert, scale is in low numbers ie. 8, not thousands (otherwise you'd have to run inference on beefed up cluster only, experts still have to be loaded when used) etc.

sigmoid10 3y ago

That's wrong. What you described is data parallelism and it would indeed be very tricky to e.g. sync gradients across machines. But this is not the only method of training neural nets (transformers or any other kind) in parallel. If we'd like to train, say, a human brain complexity level model with 10^15 parameters, we'd need a model parallelism approach anyways. It introduces a bit of complexity since you need to make sure that each distributed part of the model can run individually with roughly the same amount of compute, but you no longer need to worry about syncing anything (or have the entire state of anything on one machine). The real questions is if you can find enough people to run this who will never be able to run it themselves in the end, because inference alone will still require a supercluster. If you have access to that, you might as well train something on it today.

mirekrusin 3y ago
Lack of data parallelism is implied by computation that is performed. You gradient descend on your state. Each step needs to work on up to date state otherwise you're computing gradient descend from state that doesn't exist anymore and your computed gradient descent delta is nonsensical if applied to the most recent state (it was calculated on old one, direction that your computation calculated is now wrong). You also can't calculate it without having access to the whole state. You have to do full forward and backward pass and mutate weights. There aren't any ways of slicing and distributing that make sense in terms of efficiency. The reason is that too much data at too high frequency needs to be mutated and then made readable. That's also the reason why nvidia is focusing so much on hyper efficient interconnects - because that's the bottleneck. Computation itself is way ahead of in/out data transfer. Data transfer is the main problem and going in the direction of architecture that dramatically reduces it by several orders of magnitude is just not the way to go. If somebody solves this problem it'll mean they solved much more interesting problem – because it'll mean you can locally uptrain model and inject this knowledge into bigger one arbitrarily.
- sigmoid10 3y ago
  Your gradient descent is an operation on a directed acyclic graph. The graph itself is stateless. You can do parts of the graph without needing to have access to the entire graph, particularly for transformers. In fact this is already done today for training and inference of large models. The transfer bottleneck is for currently used model sizes and architectures. There's nothing to stop you from building a model so complex that compute itself becomes the bottleneck rather than data transfer. Except its ultimate usability of course, as I already mentioned.
  - mirekrusin 3y ago
    Your DAG is big. It's stateless for single pass. Next one doesn't operate on it anymore, it operates on new, updated one from previous step. It has fully connected sub DAGs. There is nothing stopping you from distributing assembly/machine code for CPU instructions, yet nobody does it because it doesn't make sense from performance perspective. Or amazon driving truck from one depo to other to unload one package at a time to "distribute" unloading because "distributing = faster".
    - sigmoid10 3y ago
      Remember the famouse Tannenbaum quote: "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." It's only a question of scale and not of any principles.
- ajb 3y ago
  Openai had a more distributable algorithm: https://openai.com/research/evolution-strategies https://openai.com/research/evolution-strategies But given that they don't seem to have worked on it since, I guess it wasn't too successful. But maybe there is a way
  - mirekrusin 3y ago
    Yes, if there was something interesting there you'd think since 2017 something would happen. Reinforcement Learning (that is compared with) is not particularly famous for its performance (it is it's biggest issue and reason for not being used that much). Also transformers don't use it at all.

mejutoco 3y ago

That is the challenging part indeed. But if we think of mixture of experts models outperforming "monolithic" models, why not? Maybe instead of 8 you can do 1000 and that is easy to paralellize. It sounds worth exploring to me.

l33tman 3y ago
I think the MoE models are trained together just like any other network though, including the dispatcher layer that has to learn which "expert" route each token to. Perhaps you could do some kind of technically worse model architecture that is trained separately and then a more complex dispatcher that then learns to utilize the individually trained experts as best as it can?
nabakin 3y ago
I don't think MoE allows for that either. You'd have to come up with a whole new architecture that allows parts to be trained independently and still somehow be merged together in the end.
- nsagent 3y ago
  This paper addresses that issue and allows fully independent training of the experts: https://arxiv.org/abs/2303.14177 https://arxiv.org/abs/2303.14177
  - tkgally 3y ago
    This one, from a couple of days ago, might address that issue as well: https://arxiv.org/abs/2403.13187 https://arxiv.org/abs/2403.13187
    - mirekrusin 3y ago
      This approach works on merging finetunes. Base model was still trained in usual, non distributed way (by far the most cost). Fine tunes were also trained in usual, non distributed way. Proposed approach tries out several combinations to pick one that seems to perform better (where combination means ie. adhoc per layer operation). Merging is not distributed as well. There is not much distribution happening overall beyond the fact that fine tunes were trained independently. Taking weight averages, weighted weight averages, trimming low diffs, doing arithmetic (subtracting base model from fine tune) etc. are all ad hoc trials throwing something on the wall and seeing what sticks the most. None of those work well. For distributed training to work we'd have to have better algebra around this multidimentional/multilayer/multiconnectivity state. We don't have it and it has many problems, ie. evaluation is way too expensive. But solving "no need to rerun through whole training/benchmark corpus to see if my tiny change is better or not" problem will mean we solved problem of extracting essence of intelligence. If we do that, then hyper-efficient data centers will still keep beating out any distributed approach and it's all largely irrelevant because that's pure AGI already.

nsagent 3y ago

You're right that parameter updates typically require full state, but during my PhD I've explored some possibilities to address this limitation (unfortunately, my ideas didn't pan out in the time I had). That said, there is research that has explored this topic and made some progress, such as this paper: https://arxiv.org/abs/2303.14177 https://arxiv.org/abs/2303.14177

mirekrusin 3y ago
Unfortunately it's hardly progress. MoE expert models are still large, have to be trained in usual, linear way, this approach requires training set classification upfront, each expert model is completely independent, each has to relearn concepts, your overall model is as good as dedicated expert, scale is in low numbers ie. 8, not thousands (otherwise you'd have to run inference on beefed up cluster only, experts still have to be loaded when used) etc.