5 ms·

Oh well, this is the death of democratic AI and an end of independent researchers :-( There goes any hope of a single Titan RTX producing meaningful commercial

by bitL 8y ago

Oh well, this is the death of democratic AI and an end of independent researchers :-( There goes any hope of a single Titan RTX producing meaningful commercial models.

ummonk 8y ago
Is there some scaling limit that prevents people from doing the same in several hours with a couple of GPUs?
- MasterScrat 8y ago
  If each experiment takes you literally 400 times longer than for a Google researcher, your chances of figuring out anything new drops dramatically. I was at an ML conference last year and I asked the panel: if I want to go forward in ML, should I rather do a PHD or work in the industry? a professor (!) answered that I should work in the industry as most research groups don't have the funding to be competitive.
  - p1esk 8y ago
    Most people who work in well funded ML labs in the industry have phds.
    - ousta 8y ago
      for now. wait 5 years.
      - p1esk 8y ago
        In 5 years there will be a lot more people with ML phds looking for work :)
p1esk 8y ago
If independent researchers don’t like that they should come up with a way to train resnet in 1 minute on a single GPU. Unless you think the ultimate best method to train neural nets has been discovered? Somehow 30 years ago researchers managed to invent cnns and rnns using hardware million times slower than what you can buy today for a few thousand bucks.
jdietrich 8y ago
GPUs and TPUs are readily available in the cloud. Big institutions have always had massive mainframes or compute clusters, but cloud services have democratised access to those resources. Of course a single Titan RTX can't match the performance of a huge compute cluster, but I can rent a 512 TPU pod with 11.5petaFLOPS of compute and 4TB of HBM for $6.40 per minute. Members of the TensorFlow Research Cloud programme can access TPU pods free of charge.
FakeComments 8y ago
Can independent researchers not afford $6? That’s what 512 GPUs for 1.5 minutes costs on EC2, using spot P2.16xl with 16GPUs each. The price would be $12 if you had to use on-demand instances. It does start to add up — if you wanted to do 2 runs per hour, 8 hours a day, 250 days per year (ie, a “full work year”), it would cost $48k. (And you’d have done 4000 such experiments.) All of this is to say, I think you might be exaggerating this as the “death of independent researchers”. A single card can still do meaningful inference on a model trained this way, and can do useful prototyping before you deploy your fanout for a few minutes and $20.
- nightski 8y ago
  You probably won't get a 56 Gbps network on AWS as used in the article. So it's probably going to take quite a bit longer than 2 minutes. This also ignores all data transfer & storage costs.
  - FakeComments 8y ago
    It’s 25Gbps between instances, and NVLINK between GPUs on the same instance. It’s really a pretty comparable setup. There’s no network cost within a region, and you’re probably not downloading terabytes of data out of AWS — so you’re talking about a few dollars a month, as opposed to tens a day in GPU cost. (You’d also have 23TB of RAM in your cluster, if that’s enough for you to use as storage.)
    - jph00 8y ago
      Also AWS has p3dn.24xlarge with 100Gbps :)
caenorst 8y ago
Sorry but even tho the big companies produce a lot of interesting research, I challenge you to not find any interesting model trained on a single GPU from recent publications (the majority coming from academia). Actually it's very rare to find a paper where largely distributed training is necessary (i.e: the training would fail or would be unreasonably too long). Yes having more money help you to scale your experiments, it's nothing new and it's not something specific to AI.
- bitL 8y ago
  A trivial example is BERT_large; won't fit into 12/16GB and takes ~year to train from the scratch on a single 15TFlops machine. It's now a base model for transfer learning for NLP.
  - caenorst 8y ago
    I'm not saying that there's no very big model, just saying that it's a minority of publications, for any trivia example of big model I can show you 10x trivia examples of relevant non-big models. Also you are talking about a model which is specifically designed for TPU (the dimensionality of the networks is especially fine-tuned). And even tho, BERT_large still fit in the memory of a single GPU (for very small batch), there is an implementation on Pytorch. I don't understand, are people complaining that Deep Learning is actually (reasonably) scaling ? Isn't it a good news ?
    - bitL 8y ago
      You need to study state-of-art a bit more. BERT_large can't reproduce results its authors achieved with TPUs on a Titan V/Tesla P100 as for getting there you need to use substantially larger batch sizes that won't fit into 12/16GB. If you get a V100/Titan RTX, it would fit, but you'd wait ~1 year for a single training session (40 epochs) to finish. MS already published another model based on BERT that is even better. It's unlikely memory x #GPUs would go down in foreseeable future; it's more like that everybody will start as large models as their infrastructure allows if they find something that improves target metrics.
- 8y ago

by bitL 8y ago

Oh well, this is the death of democratic AI and an end of independent researchers :-( There goes any hope of a single Titan RTX producing meaningful commercial models.

ummonk 8y ago

Is there some scaling limit that prevents people from doing the same in several hours with a couple of GPUs?

MasterScrat 8y ago
If each experiment takes you literally 400 times longer than for a Google researcher, your chances of figuring out anything new drops dramatically. I was at an ML conference last year and I asked the panel: if I want to go forward in ML, should I rather do a PHD or work in the industry? a professor (!) answered that I should work in the industry as most research groups don't have the funding to be competitive.
- p1esk 8y ago
  Most people who work in well funded ML labs in the industry have phds.
  - ousta 8y ago
    for now. wait 5 years.
    - p1esk 8y ago
      In 5 years there will be a lot more people with ML phds looking for work :)

p1esk 8y ago

If independent researchers don’t like that they should come up with a way to train resnet in 1 minute on a single GPU. Unless you think the ultimate best method to train neural nets has been discovered? Somehow 30 years ago researchers managed to invent cnns and rnns using hardware million times slower than what you can buy today for a few thousand bucks.

jdietrich 8y ago

GPUs and TPUs are readily available in the cloud. Big institutions have always had massive mainframes or compute clusters, but cloud services have democratised access to those resources. Of course a single Titan RTX can't match the performance of a huge compute cluster, but I can rent a 512 TPU pod with 11.5petaFLOPS of compute and 4TB of HBM for $6.40 per minute. Members of the TensorFlow Research Cloud programme can access TPU pods free of charge.

FakeComments 8y ago

Can independent researchers not afford $6? That’s what 512 GPUs for 1.5 minutes costs on EC2, using spot P2.16xl with 16GPUs each. The price would be $12 if you had to use on-demand instances. It does start to add up — if you wanted to do 2 runs per hour, 8 hours a day, 250 days per year (ie, a “full work year”), it would cost $48k. (And you’d have done 4000 such experiments.) All of this is to say, I think you might be exaggerating this as the “death of independent researchers”. A single card can still do meaningful inference on a model trained this way, and can do useful prototyping before you deploy your fanout for a few minutes and $20.

nightski 8y ago
You probably won't get a 56 Gbps network on AWS as used in the article. So it's probably going to take quite a bit longer than 2 minutes. This also ignores all data transfer & storage costs.
- FakeComments 8y ago
  It’s 25Gbps between instances, and NVLINK between GPUs on the same instance. It’s really a pretty comparable setup. There’s no network cost within a region, and you’re probably not downloading terabytes of data out of AWS — so you’re talking about a few dollars a month, as opposed to tens a day in GPU cost. (You’d also have 23TB of RAM in your cluster, if that’s enough for you to use as storage.)
  - jph00 8y ago
    Also AWS has p3dn.24xlarge with 100Gbps :)

caenorst 8y ago

Sorry but even tho the big companies produce a lot of interesting research, I challenge you to not find any interesting model trained on a single GPU from recent publications (the majority coming from academia). Actually it's very rare to find a paper where largely distributed training is necessary (i.e: the training would fail or would be unreasonably too long). Yes having more money help you to scale your experiments, it's nothing new and it's not something specific to AI.

bitL 8y ago
A trivial example is BERT_large; won't fit into 12/16GB and takes ~year to train from the scratch on a single 15TFlops machine. It's now a base model for transfer learning for NLP.
- caenorst 8y ago
  I'm not saying that there's no very big model, just saying that it's a minority of publications, for any trivia example of big model I can show you 10x trivia examples of relevant non-big models. Also you are talking about a model which is specifically designed for TPU (the dimensionality of the networks is especially fine-tuned). And even tho, BERT_large still fit in the memory of a single GPU (for very small batch), there is an implementation on Pytorch. I don't understand, are people complaining that Deep Learning is actually (reasonably) scaling ? Isn't it a good news ?
  - bitL 8y ago
    You need to study state-of-art a bit more. BERT_large can't reproduce results its authors achieved with TPUs on a Titan V/Tesla P100 as for getting there you need to use substantially larger batch sizes that won't fit into 12/16GB. If you get a V100/Titan RTX, it would fit, but you'd wait ~1 year for a single training session (40 epochs) to finish. MS already published another model based on BERT that is even better. It's unlikely memory x #GPUs would go down in foreseeable future; it's more like that everybody will start as large models as their infrastructure allows if they find something that improves target metrics.
8y ago