5 ms·

Diffusion Beats Autoregressive in Data-Constrained Settings

72 points by djoldman 1y ago

smokel 1y ago
I fail to understand why we would lack data. Sure, there is limited (historical) text, but if we just open up all available video, and send out interactive robots into the world, we'll drown in data. Then there is simulated data, and tons of sensors that can capture vast amounts of even more data. Edit: from the source [1], this quote pretty much sums it all up: "Our 2022 paper predicted that high-quality text data would be fully used by 2024, whereas our new results indicate that might not happen until 2028." [1] https://epoch.ai/blog/will-we-run-out-of-data-limits-of-llm-scaling-based-on-human-generated-data https://epoch.ai/blog/will-we-run-out-of-data-limits-of-llm-...
- Legend2440 1y ago
  >send out interactive robots into the world Easier said than done. Robotics tends to be even more data-constrained than NLP. The real world only runs at 1x speed, and if your robot breaks something it costs real money. Simulators are simplistic compared to reality and take a lot of manual effort to build. You will always need to make efficient use of the data you have.
- imtringued 1y ago
  Robotics data isn't labeled and if you build a robot, there ain't anyone who has collected data for your particular robot. There is also the problem that on-device learning is not yet practical.
deleted 1y ago
[deleted]
robots0only 1y ago
This paper was just too overhyped by the authors. Also, the initial evals were very limited and very strange. This blog post does a much better job at a similar observation -- goes into details and does proper evaluation (also better attribution): https://jinjieni.notion.site/Diffusion-Language-Models-are-Super-Data-Learners-239d8f03a866800ab196e49928c019ac https://jinjieni.notion.site/Diffusion-Language-Models-are-S...
thesz 1y ago
> This paper addresses the challenge by asking: how can we trade off more compute for less data? Autoregressive models are not matched by compute and this is the major drawback. There is evidence that training RNN models that compute several steps with same input and coefficients (but different state) lead to better performance. It was shown in a followup to [1] that performed ablation study. [1] https://arxiv.org/abs/1611.06188 https://arxiv.org/abs/1611.06188 They fixed number of time steps instead of varying it, and got better results. Unfortunately, I forgot the title of that ablation paper.
- kevinwang 1y ago
  Not sure if you meant this because it doesn't cite the paper you mention, but it's a similar work: "An Investigation of Model-Free Planning", Guez et Al. (Deepmind) 2019 https://arxiv.org/abs/1901.03559 https://arxiv.org/abs/1901.03559
  - astrange 1y ago
    Speaking of not citing, that one could go a bit further back. https://cdn.aaai.org/AAAI/1987/AAAI87-048.pdf https://cdn.aaai.org/AAAI/1987/AAAI87-048.pdf
- imtringued 1y ago
  It has already been proven that deep equilibrium models with a single layer are equivalent to models with a finite number of layers and the converse as well. That you can get the performance of a DEQ using a finite number of layers. The fixed point nature of DEQs means that they inherently have a concept of self assessment how close they are to the solution. If they are at the solution, they will simply stop changing it. If not, they will keep performing calculations.
blurbleblurble 1y ago
I have a feeling this technique might make waves: https://openreview.net/forum?id=c05qIG1Z2B#discussion https://openreview.net/forum?id=c05qIG1Z2B#discussion
- tripplyons 1y ago
  There are definitely parallels between diffusion and reasoning models, mostly being able to spend longer to get a better solution by using a more precise ODE solver for diffusion or using more tokens for reasoning. However, due to how diffusion models are trained, they never see their own predictions as input, so they cannot learn to store information across steps. This is the complete opposite for reasoning models.
  - yorwba 1y ago
    You can train a diffusion model using its own predictions as input, no problem at all.
    - tripplyons 1y ago
      At that point it is not following a diffusion training objective. I am aware of papers that do this, but I have not seen one that shows it as a better pretraining objective than something like v-prediction or flow matching.
      - mxwsn 1y ago
        Why is not the diffusion training objective? The technique is known as self-conditioning right? Is it an issue with conditional Tweedie's?
  - blurbleblurble 1y ago
    I'm probably not understanding your point but did you look at the paper? This explicitly does diffusion in an autoencoded latent space of the autoregressive prediction itself. The starting point is that prediction, but depending on how much noise is used, the diffusion model itself directly contributes to the prediction process to some degree or another. It should be trivial to make an encoder that has some memory of at least part of the prompt (say the tailing part) and do a diffusion step there too.

72 points by djoldman 1y ago

I fail to understand why we would lack data. Sure, there is limited (historical) text, but if we just open up all available video, and send out interactive robots into the world, we'll drown in data. Then there is simulated data, and tons of sensors that can capture vast amounts of even more data. Edit: from the source [1], this quote pretty much sums it all up: "Our 2022 paper predicted that high-quality text data would be fully used by 2024, whereas our new results indicate that might not happen until 2028." [1] https://epoch.ai/blog/will-we-run-out-of-data-limits-of-llm-scaling-based-on-human-generated-data https://epoch.ai/blog/will-we-run-out-of-data-limits-of-llm-...

Legend2440 1y ago
>send out interactive robots into the world Easier said than done. Robotics tends to be even more data-constrained than NLP. The real world only runs at 1x speed, and if your robot breaks something it costs real money. Simulators are simplistic compared to reality and take a lot of manual effort to build. You will always need to make efficient use of the data you have.
imtringued 1y ago
Robotics data isn't labeled and if you build a robot, there ain't anyone who has collected data for your particular robot. There is also the problem that on-device learning is not yet practical.

deleted 1y ago

[deleted]

robots0only 1y ago

This paper was just too overhyped by the authors. Also, the initial evals were very limited and very strange. This blog post does a much better job at a similar observation -- goes into details and does proper evaluation (also better attribution): https://jinjieni.notion.site/Diffusion-Language-Models-are-Super-Data-Learners-239d8f03a866800ab196e49928c019ac https://jinjieni.notion.site/Diffusion-Language-Models-are-S...

thesz 1y ago

> This paper addresses the challenge by asking: how can we trade off more compute for less data? Autoregressive models are not matched by compute and this is the major drawback. There is evidence that training RNN models that compute several steps with same input and coefficients (but different state) lead to better performance. It was shown in a followup to [1] that performed ablation study. [1] https://arxiv.org/abs/1611.06188 https://arxiv.org/abs/1611.06188 They fixed number of time steps instead of varying it, and got better results. Unfortunately, I forgot the title of that ablation paper.

kevinwang 1y ago
Not sure if you meant this because it doesn't cite the paper you mention, but it's a similar work: "An Investigation of Model-Free Planning", Guez et Al. (Deepmind) 2019 https://arxiv.org/abs/1901.03559 https://arxiv.org/abs/1901.03559
- astrange 1y ago
  Speaking of not citing, that one could go a bit further back. https://cdn.aaai.org/AAAI/1987/AAAI87-048.pdf https://cdn.aaai.org/AAAI/1987/AAAI87-048.pdf
imtringued 1y ago
It has already been proven that deep equilibrium models with a single layer are equivalent to models with a finite number of layers and the converse as well. That you can get the performance of a DEQ using a finite number of layers. The fixed point nature of DEQs means that they inherently have a concept of self assessment how close they are to the solution. If they are at the solution, they will simply stop changing it. If not, they will keep performing calculations.

blurbleblurble 1y ago

I have a feeling this technique might make waves: https://openreview.net/forum?id=c05qIG1Z2B#discussion https://openreview.net/forum?id=c05qIG1Z2B#discussion

tripplyons 1y ago
There are definitely parallels between diffusion and reasoning models, mostly being able to spend longer to get a better solution by using a more precise ODE solver for diffusion or using more tokens for reasoning. However, due to how diffusion models are trained, they never see their own predictions as input, so they cannot learn to store information across steps. This is the complete opposite for reasoning models.
- yorwba 1y ago
  You can train a diffusion model using its own predictions as input, no problem at all.
  - tripplyons 1y ago
    At that point it is not following a diffusion training objective. I am aware of papers that do this, but I have not seen one that shows it as a better pretraining objective than something like v-prediction or flow matching.
    - mxwsn 1y ago
      Why is not the diffusion training objective? The technique is known as self-conditioning right? Is it an issue with conditional Tweedie's?
- blurbleblurble 1y ago
  I'm probably not understanding your point but did you look at the paper? This explicitly does diffusion in an autoencoded latent space of the autoregressive prediction itself. The starting point is that prediction, but depending on how much noise is used, the diffusion model itself directly contributes to the prediction process to some degree or another. It should be trivial to make an encoder that has some memory of at least part of the prompt (say the tailing part) and do a diffusion step there too.