4 ms·
> I guess ML workloads are huge and here to stay. As a researcher, I think we'll go into a "winter" soon, but nowhere near the previous one. The hype will die
by godelski 4y ago
> I guess ML workloads are huge and here to stay.
As a researcher, I think we'll go into a "winter" soon, but nowhere near the previous one. The hype will die down but ML has shown to be extremely useful. Anything with graphic design, gaming, video, imaging, streaming, etc uses ML these days. It's been used in science for awhile (under statistical learning, e.g. regression) and we can't do without it. Though a lot of these things aren't following the main hype machine. But if you're in the weeds you'll see a lot of these works.
But I do think the hype will die. People even now are starting to notice that a lot of the diffusion work looks amazing at first glance but has major errors. There's also the issue of alignment. These are going to likely take a lot of mathematical experience to push forward[0] and will benefit less from a pure numbers game.
Of course, the other part (in Nvidia's favor) is that a lot of scientific work is GPU based now. Not just ML. Parallelism is king. You can do FEM, particle simulations, etc with GPUs now because the clock speed got high enough. So you can do "heavy" calculations with extreme parallelism. Previously you could only do this with lighter loads. (there's also major algorithmic improvements, so I don't want to undermine the fantastic work of those researchers).
[0] I want to admit my bias here. I'm a "math person" (not a mathematician, they will school me) and there's not a lot of ML researchers going down this path. To give an example, there are plenty of diffusion researchers I've talked to that don't know how to calculate the likelihood of their samples while they use an ELBO cost function. But this comes into the alignment issue because we need to talk about the smoothness of latent manifolds, interpolation, inversion (not just image inversion, but image/text inversion), and a lot more. I just don't think we can continue on this path of bigger networks, bigger data, search hyper-parameters, get benchmark result. (I fundamentally believe this is anti-scientific fwiw)
- 6gvONxR4sf7o 4y ago> Of course, the other part (in Nvidia's favor) is that a lot of scientific work is GPU based now. Not just ML. It’s not just the hardware like you’re talking about. The tooling has grown up insanely well too. You used to have to write CUDA or similar if you wanted a simulation on a GPU. Now you can write it in a high level language in a way that looks pretty close to the math.
- godelski 4y agoVery true! I should have appended this under the algorithmic improvements. This is also a reason ML has exploded. Tensorflow and Pytorch enabled easy GPU usage and so we can spend a lot less time programming and debugging. (even writing CUDA subroutines is easier!) I mean pytorch is basically numpy but with GPU access so it is fantastic for any optimization, even non ml. I haven't played around with JAX but I hear it is better for more statistical stuff because of this.