4 ms·

Also, this didn't come from OpenAI or DeepMind, or even industry. What are those guys even doing? :)

mmaunder 3y ago
Many labs doing foundational work like this and making progress don’t have the anything near the budget or compute to implement at scale. In other words they don’t have a Sam and his backers or a Zuck and his budget.
Micoloth 3y ago
They sure as hell have no incentives to make Neural Network faster and more accessible, for starters.. (Considering they right now make more money and have more control, the less accessible and the more computation-hungry AI models are) To be fair, this approach (claims to) only speed up inference, not training, so all the GPUs are needed anyway.
- WithinReason 3y ago
  They certainly have an incentive to keep these kinds of improvements in-house and not publish them, since they are commercial entities and this represents a competitive advantage.
  - lawlessone 3y ago
    I think Nvidia might have an incentive for this not to exist. edit: but you are right for the AI companies not open sourcing their models it's an advantage to have it when others don't
    - WithinReason 3y ago
      I'm actually not sure about Nvidia, due to https://en.wikipedia.org/wiki/Jevons_paradox https://en.wikipedia.org/wiki/Jevons_paradox
      - ForkMeOnTinder 3y ago
        But if things get too efficient for individual users, you won't need an Nvidia GPU anymore. People will use cheaper hardware instead. I'm looking forward to running good models at decent speed on a low-end CPU or whatever crappy GPU is in my phone.
    - jacobsimon 3y ago
      I had the same thought this morning and was debating selling my nvda stock when I saw this - feels like they are well-positioned right now, as with crypto a few years ago, but if there were an efficiency breakthrough that allowed commodity CPUs to do the inference instead, this advantage could vanish quickly.
    - godelski 3y ago
      Nvidia can't make GPUs fast enough. I doubt 10xing training and/or inference efficiency would result in a decrease in demand. I would be surprised if it didn't instead increase demand. Mind you, Nvidia is pushing hard on TensorRT which optimizes models at inference time and results in major increases in throughput (not 10x though lol).
      - rictic 3y ago
        Yeah, Jevons Paradox suggests that 10xing efficiency of training and inference would increase demand for GPUs.
- godelski 3y ago
  I wouldn't be so quick to conspiracy. I'm the author of a work and a famous blog post that trains a particular common architecture much faster (don't want to dox myself too much) and with far fewer parameters, but it has been rejected several times and is now arxiv only. Our most common complaint was "who would use this? Why not just take a large model and tune it?" That question alone held us back a year (had over a hundred citations by then and remains my most cited work) until it switched to "use more datasets" and "not novel" (by that time true, others had built off of us, cited us, and published in top venues). I don't think this was some conspiracy by big labs to push back against us (we're nobodies) but rather that people get caught up in hype and reviewers are lazy and incentivized to reject. You're trained to be critical of works and especially consider that post hoc most solutions appear far simpler than they actually are. But context matters because if you don't approach every paper with nuance it's easy to say "oh, it's just x." But if those ideas were so simple and obvious they would also be prolific. I see a lot of small labs suffer the same fate simply due to lack of compute. If you don't make your new technique work on many datasets it becomes the easiest thing to reject a paper by. ACs aren't checking that reviews are reasonable. I've even argued with fellow reviewers about papers in workshops -- papers I would have accepted in the main conference -- that are brushed off and the reviewers admit in their reviews that they do not work on these topics. I don't understand what's going on but at times it feels like a collective madness. A 10 page paper with 4 very different datasets that solves a problem, is clearly written, has no major flaws, and is useful to the community should not need defending when submitted to a workshop just because reviewers aren't qualified to review the work (this paper got in btw). We are moving into a "pay to play" ecosystem and that will only create bad science due to group think. (another aspect of "pay to play" is in the tuning. Spending $1M to tune your model to be the best doesn't mean it is better than a model that could not afford the search. Often more than half of resources are spent on tuning now)
  - wruza 3y ago
    Is there a place where you guys discuss... things? I'm layman interested in this topic akin to pop-physics/maths, but have no chance to just read papers and "get it". On the other hand, immediately available resources focus more on how-to part of it rather than on what's up overall. Also, do you have something like 3b1b/pbs/nph for it? Content that you can watch and say "well, yep, good job".
airgapstopgap 3y ago
…ETH Zurich is an illustrious research university that often cooperates with Deepmind and other hyped groups, they're right there at the frontier too, and have been for a very long time. They don't have massive training runs on their own but pound for pound I'd say they have better papers.
- godelski 3y ago
  ETH Zurich is one of the top labs in the world. Disney Research also works with them a lot. Another "sleeper" is University of Amsterdam that has rockstars like Max Welling and his students Kingma, Salimans,van den Berg, and Hoogeboom. It's easy to get hyped up on the big tech labs because they have the most compute, but the best papers come from smaller labs and unfortunately more lately face larger challenges in getting published. It's the smaller works that create the foundations that end up in these giant models. ML is in a really weird space right now.
  - brandall10 3y ago
    It's great to the school of Einstein/Von Neumann is doing highly innovative work in this space.
pr337h4m 3y ago
This from DeepMind: DiLoCo: Distributed Low-Communication Training of Language Models - https://arxiv.org/pdf/2311.08105.pdf https://arxiv.org/pdf/2311.08105.pdf From the first author on Twitter: "It could quite a big deal for people who don't have access to a colocated cluster of GPUs: e.g. with DiLoCo you could train your model, with data-parallelism, across all GPU providers, looking in real-time for the cheapest price, even if pre-emptable, even across continents" https://twitter.com/Ar_Douillard/status/1724839420820361687 https://twitter.com/Ar_Douillard/status/1724839420820361687
quickthrower2 3y ago
It is not surprising. The assumption is that they have the best people. That you can objectively search 8 billion people for the best people globally is folly of course. There are geniuses without US citizenship / visas / green cards. And so outside brains are going to figure this out. Mix in GDP of $rest_of_world has much more resources than any company, and the luck-driven nature of making AI discoveries, and I reckon most progress will be outside of OpenAI etc. Driven by a problem the big guys don't need to solve: how do I avoiding buying a $5k graphics card.

Many labs doing foundational work like this and making progress don’t have the anything near the budget or compute to implement at scale. In other words they don’t have a Sam and his backers or a Zuck and his budget.

Micoloth 3y ago

They sure as hell have no incentives to make Neural Network faster and more accessible, for starters.. (Considering they right now make more money and have more control, the less accessible and the more computation-hungry AI models are) To be fair, this approach (claims to) only speed up inference, not training, so all the GPUs are needed anyway.

WithinReason 3y ago
They certainly have an incentive to keep these kinds of improvements in-house and not publish them, since they are commercial entities and this represents a competitive advantage.
- lawlessone 3y ago
  I think Nvidia might have an incentive for this not to exist. edit: but you are right for the AI companies not open sourcing their models it's an advantage to have it when others don't
  - WithinReason 3y ago
    I'm actually not sure about Nvidia, due to https://en.wikipedia.org/wiki/Jevons_paradox https://en.wikipedia.org/wiki/Jevons_paradox
    - ForkMeOnTinder 3y ago
      But if things get too efficient for individual users, you won't need an Nvidia GPU anymore. People will use cheaper hardware instead. I'm looking forward to running good models at decent speed on a low-end CPU or whatever crappy GPU is in my phone.
  - jacobsimon 3y ago
    I had the same thought this morning and was debating selling my nvda stock when I saw this - feels like they are well-positioned right now, as with crypto a few years ago, but if there were an efficiency breakthrough that allowed commodity CPUs to do the inference instead, this advantage could vanish quickly.
  - godelski 3y ago
    Nvidia can't make GPUs fast enough. I doubt 10xing training and/or inference efficiency would result in a decrease in demand. I would be surprised if it didn't instead increase demand. Mind you, Nvidia is pushing hard on TensorRT which optimizes models at inference time and results in major increases in throughput (not 10x though lol).
    - rictic 3y ago
      Yeah, Jevons Paradox suggests that 10xing efficiency of training and inference would increase demand for GPUs.
godelski 3y ago
I wouldn't be so quick to conspiracy. I'm the author of a work and a famous blog post that trains a particular common architecture much faster (don't want to dox myself too much) and with far fewer parameters, but it has been rejected several times and is now arxiv only. Our most common complaint was "who would use this? Why not just take a large model and tune it?" That question alone held us back a year (had over a hundred citations by then and remains my most cited work) until it switched to "use more datasets" and "not novel" (by that time true, others had built off of us, cited us, and published in top venues). I don't think this was some conspiracy by big labs to push back against us (we're nobodies) but rather that people get caught up in hype and reviewers are lazy and incentivized to reject. You're trained to be critical of works and especially consider that post hoc most solutions appear far simpler than they actually are. But context matters because if you don't approach every paper with nuance it's easy to say "oh, it's just x." But if those ideas were so simple and obvious they would also be prolific. I see a lot of small labs suffer the same fate simply due to lack of compute. If you don't make your new technique work on many datasets it becomes the easiest thing to reject a paper by. ACs aren't checking that reviews are reasonable. I've even argued with fellow reviewers about papers in workshops -- papers I would have accepted in the main conference -- that are brushed off and the reviewers admit in their reviews that they do not work on these topics. I don't understand what's going on but at times it feels like a collective madness. A 10 page paper with 4 very different datasets that solves a problem, is clearly written, has no major flaws, and is useful to the community should not need defending when submitted to a workshop just because reviewers aren't qualified to review the work (this paper got in btw). We are moving into a "pay to play" ecosystem and that will only create bad science due to group think. (another aspect of "pay to play" is in the tuning. Spending $1M to tune your model to be the best doesn't mean it is better than a model that could not afford the search. Often more than half of resources are spent on tuning now)
- wruza 3y ago
  Is there a place where you guys discuss... things? I'm layman interested in this topic akin to pop-physics/maths, but have no chance to just read papers and "get it". On the other hand, immediately available resources focus more on how-to part of it rather than on what's up overall. Also, do you have something like 3b1b/pbs/nph for it? Content that you can watch and say "well, yep, good job".

airgapstopgap 3y ago

…ETH Zurich is an illustrious research university that often cooperates with Deepmind and other hyped groups, they're right there at the frontier too, and have been for a very long time. They don't have massive training runs on their own but pound for pound I'd say they have better papers.

godelski 3y ago
ETH Zurich is one of the top labs in the world. Disney Research also works with them a lot. Another "sleeper" is University of Amsterdam that has rockstars like Max Welling and his students Kingma, Salimans,van den Berg, and Hoogeboom. It's easy to get hyped up on the big tech labs because they have the most compute, but the best papers come from smaller labs and unfortunately more lately face larger challenges in getting published. It's the smaller works that create the foundations that end up in these giant models. ML is in a really weird space right now.
- brandall10 3y ago
  It's great to the school of Einstein/Von Neumann is doing highly innovative work in this space.

pr337h4m 3y ago

This from DeepMind: DiLoCo: Distributed Low-Communication Training of Language Models - https://arxiv.org/pdf/2311.08105.pdf https://arxiv.org/pdf/2311.08105.pdf From the first author on Twitter: "It could quite a big deal for people who don't have access to a colocated cluster of GPUs: e.g. with DiLoCo you could train your model, with data-parallelism, across all GPU providers, looking in real-time for the cheapest price, even if pre-emptable, even across continents" https://twitter.com/Ar_Douillard/status/1724839420820361687 https://twitter.com/Ar_Douillard/status/1724839420820361687

quickthrower2 3y ago

It is not surprising. The assumption is that they have the best people. That you can objectively search 8 billion people for the best people globally is folly of course. There are geniuses without US citizenship / visas / green cards. And so outside brains are going to figure this out. Mix in GDP of $rest_of_world has much more resources than any company, and the luck-driven nature of making AI discoveries, and I reckon most progress will be outside of OpenAI etc. Driven by a problem the big guys don't need to solve: how do I avoiding buying a $5k graphics card.