3 ms·

Currently we do not explore sparsity. The next iteration of models will be much more compact by focusing on reducing effective tensor size.

bee_rider 3y ago
It seems like a ton of engineering effort has been put into these neural network frameworks. How didn’t they explore sparsity yet? With numerical linear algebra that’s, like, the most obvious thing to do (which is to say, you probably know beforehand if your problem can be mapped to sparse matrices). (Edit: just to be clear here, I’m not saying I expect the whole field is full of dummies who missed something obvious or something like that, I don’t know much at all about machine learning so I’m sure I’m missing something).
- taneq 3y ago
  From what I’ve read recently, most sparse methods just haven’t given that much improvement yet, and we’re only recently pushing up against the limits of the “just buy more RAM” approach. It sounds like there is a lot of work happening on sparse networks now, so it’ll be interesting to see how this changes in the near future.
- contravariant 3y ago
  It's not at all strange to get something to work before you start optimizing. I mean if you can only run small models then how would you even know what you're losing by optimizing for space? Heck you wouldn't even know how the model behaves, so you won't know where to start shaving away. I'm not saying it's impossible but if resources allow it makes a lot of sense to start with the biggest model you can still train. Especially since for whatever reason things seem to get a lot easier if you simply throw more computing power at it (kind of like how no matter how advanced your caching algorithm it's not going to be more than 2 times faster than the simplest LRU algorithm with double the amount of cache).
- qorrect 3y ago
  Sounds a bit like premature optimization ( to have done it _by_ now ) , I bet it's in the works now though.
- airstrike 3y ago
  "Do things that don't scale" – PG
- micro_cam 3y ago
  Gpus were built for dense math and they ran with it. To the point current best architectures are in part just the ones that run best using the subset of linear algebra gpus are really good at. There has been a lot of work on sparsity and discovering sparse subnetworks in trained dense networks. And intel even proposed some alternative cpu friendly architectures and torch/tf and gpus are starting to do okay with sparse matrixes so thing are changing.
kir-gadjello 3y ago
It is quite likely GPT-4 uses one or even two sparsity approaches on top of each other (namely, coarse grained switch transformer-like and fine grained intra-tensor block sparsity), if you look at the openly available contributors' research CVs. Google, in collaboration with OpenAI, has published an impressive tour de force where they have throughly developed and validated at scale a sparse transformer architecture, applied to general language modeling task: https://arxiv.org/abs/2111.12763 https://arxiv.org/abs/2111.12763 This happened in November of 2021, and there is a public implementation of this architecture on the Google's public github. Impressively, due to some reasons, other up-and-coming players are still not releasing models trained with this approach, even though it promises multiplicative payoff in inference economy. One boring explanation is conservatism for NN training at scale, where training runs cost O(yearly salary). Let's hope the open source side of things catches up.
- necroforest 3y ago
  not in collaboration with openai, one of the authors joined openai before the paper was written and arxiv'd
snovv_crash 3y ago
At least from my experience with sparse matrix libraries like Eigen, you need to get the sparsity down to about 5% before switching from a dense to a sparse algorithm gets you execution time benefits. Of course from a memory bandwidth and model size perspective maybe there are benefits long before that.

bee_rider 3y ago

It seems like a ton of engineering effort has been put into these neural network frameworks. How didn’t they explore sparsity yet? With numerical linear algebra that’s, like, the most obvious thing to do (which is to say, you probably know beforehand if your problem can be mapped to sparse matrices). (Edit: just to be clear here, I’m not saying I expect the whole field is full of dummies who missed something obvious or something like that, I don’t know much at all about machine learning so I’m sure I’m missing something).

taneq 3y ago
From what I’ve read recently, most sparse methods just haven’t given that much improvement yet, and we’re only recently pushing up against the limits of the “just buy more RAM” approach. It sounds like there is a lot of work happening on sparse networks now, so it’ll be interesting to see how this changes in the near future.
contravariant 3y ago
It's not at all strange to get something to work before you start optimizing. I mean if you can only run small models then how would you even know what you're losing by optimizing for space? Heck you wouldn't even know how the model behaves, so you won't know where to start shaving away. I'm not saying it's impossible but if resources allow it makes a lot of sense to start with the biggest model you can still train. Especially since for whatever reason things seem to get a lot easier if you simply throw more computing power at it (kind of like how no matter how advanced your caching algorithm it's not going to be more than 2 times faster than the simplest LRU algorithm with double the amount of cache).
qorrect 3y ago
Sounds a bit like premature optimization ( to have done it _by_ now ) , I bet it's in the works now though.
airstrike 3y ago
"Do things that don't scale" – PG
micro_cam 3y ago
Gpus were built for dense math and they ran with it. To the point current best architectures are in part just the ones that run best using the subset of linear algebra gpus are really good at. There has been a lot of work on sparsity and discovering sparse subnetworks in trained dense networks. And intel even proposed some alternative cpu friendly architectures and torch/tf and gpus are starting to do okay with sparse matrixes so thing are changing.

kir-gadjello 3y ago

It is quite likely GPT-4 uses one or even two sparsity approaches on top of each other (namely, coarse grained switch transformer-like and fine grained intra-tensor block sparsity), if you look at the openly available contributors' research CVs. Google, in collaboration with OpenAI, has published an impressive tour de force where they have throughly developed and validated at scale a sparse transformer architecture, applied to general language modeling task: https://arxiv.org/abs/2111.12763 https://arxiv.org/abs/2111.12763 This happened in November of 2021, and there is a public implementation of this architecture on the Google's public github. Impressively, due to some reasons, other up-and-coming players are still not releasing models trained with this approach, even though it promises multiplicative payoff in inference economy. One boring explanation is conservatism for NN training at scale, where training runs cost O(yearly salary). Let's hope the open source side of things catches up.

necroforest 3y ago
not in collaboration with openai, one of the authors joined openai before the paper was written and arxiv'd

snovv_crash 3y ago

At least from my experience with sparse matrix libraries like Eigen, you need to get the sparsity down to about 5% before switching from a dense to a sparse algorithm gets you execution time benefits. Of course from a memory bandwidth and model size perspective maybe there are benefits long before that.