4 ms·

> The main idea is to train the model to generate sequences in a random order, which allow conditional density estimation, infilling and generating sequences by

> The main idea is to train the model to generate sequences in a random order, which allow conditional density estimation, infilling and generating sequences by burst using a novel rejection sampling method. > In exploring that idea, we aslo compared to a discrete diffusion baseline, which also allows to generate sequences in burst. We were surprised to see that diffusion models were able to solve path-finding task and we made a short Twitter thread The said thread: https://nitter.poast.org/ArnaudPannatier/status/1762864347397628396 https://nitter.poast.org/ArnaudPannatier/status/176286434739... And a showcase here: https://www.idiap.ch/~apannatier/sigma-gpt/ https://www.idiap.ch/~apannatier/sigma-gpt/ (excerpt taken from here: https://www.idiap.ch/~apannatier/ https://www.idiap.ch/~apannatier/)

deleted 2y ago
[deleted]
3abiton 2y ago
I just wonder if such models based on their method would make hallucination even worse.
- arnaudpannatier 2y ago
  Hey, I'm Arnaud, first author of the paper. The answer is a bit mixed. We actually started looking into this because of a repetition problem that appeared in a low-data regime for a sequence generation task. Basically, the left-to-right GPT was stuck repeating the same token once it sampled twice the same in a row during generation. And to mitigate that, we tried to generate the sequence in a random order and it seemed to help and we see less of this repetition issue. We initially thought when we don't have enough data, shuffling would be like data-augmentation and might actually help the model reach better performance. But this is not what we found in the experiments: apparently as learning in any order is a harder task, the model memorise the data more.
nico 2y ago
> We were surprised to see that diffusion models were able to solve path-finding task I wonder if this type of method might allow for faster solutions to the traveling salesman problem

deleted 2y ago

[deleted]

3abiton 2y ago

I just wonder if such models based on their method would make hallucination even worse.

arnaudpannatier 2y ago
Hey, I'm Arnaud, first author of the paper. The answer is a bit mixed. We actually started looking into this because of a repetition problem that appeared in a low-data regime for a sequence generation task. Basically, the left-to-right GPT was stuck repeating the same token once it sampled twice the same in a row during generation. And to mitigate that, we tried to generate the sequence in a random order and it seemed to help and we see less of this repetition issue. We initially thought when we don't have enough data, shuffling would be like data-augmentation and might actually help the model reach better performance. But this is not what we found in the experiments: apparently as learning in any order is a harder task, the model memorise the data more.

nico 2y ago

> We were surprised to see that diffusion models were able to solve path-finding task I wonder if this type of method might allow for faster solutions to the traveling salesman problem