7 ms·
This looks great. The authors randomly permute (i.e., shuffle) input tokens in training and add two positional encodings to each token: one with the token's po
by cs702 2y ago
This looks great.
The authors randomly permute (i.e., shuffle) input tokens in training and add two positional encodings to each token: one with the token's position and another with the position of the token to be predicted. Otherwise, the model is a standard autoregressive GPT. The consequences of this seemingly "simple" modification are significant:
* The authors can prompt the trained model with part of a sequence and then decode the missing tokens, all at once, in parallel, regardless of order -- i.e., the model can in-fill in parallel.
* The authors can compute conditional probability densities for every missing token in a sequence, again in parallel, i.e., densities for all missing tokens at once.
* The authors propose a rejection-sampling method for generating in-fill tokens, again in parallel. Their method seems to work well in practice.
I've added this to my reading list. Thank you for sharing it on HN.
- thomashop 2y agoI don't understand how that parallel prediction can work... Let's say I give it as input the sentence: I . . . . . . . . happily. The second word to be predicted depends on the first word.
- cs702 2y agoGive the model the tokens "happily" and "I", and add to each input token its respective position embedding and the position embedding for the token to be predicted. You can do this in parallel for all tokens to be predicted. The model has been trained so it can predict tokens in any position.
- hexomancer 2y agoYes, but is there any guarantee that the complete sentence makes sense?
- entropicdrifter 2y agoThat guarantee didn't exist with regular GPT LLMs, did it? It just came about as an emergent property of throwing more and more compute, training data, and training time at the problem
- amluto 2y agoI think it’s effectively built in to the design. The model outputs a probability distribution for the first unknown token [0]. Then some code outside the model chooses a token and runs the model again with that token provided to the model. So the second output token’s probability distribution is automatically conditioned on the first output token, etc. Sometimes people will attempt to parallelize this by using a faster model to guess a few tokens and then evaluating them in as a batch with the main model to determine whether the choices were good. [0] Usually it outputs “logits”, which become a probability distribution when combined with a “temperature” parameter.
- qeternity 2y ago> I think it’s effectively built in to the design. It isn't. There is no guarantee that successive tokens will be comprehensible. > Usually it outputs “logits”, which become a probability distribution when combined with a “temperature” parameter. The logits are the probability distribution (well technically, you would apply softmax). Temperature is a parameter for how you sample those logits in a non-greedy fashion.
- hexaga 2y ago> Temperature is a parameter for how you sample those logits in a non-greedy fashion. I think temperature is better understood as a pre-softmax pass over logits. You'd divide logits by the temp, and then their softmax becomes more/less peaky. probs = (logits / temp).softmax() Sampling is a whole different thing.
- qeternity 2y agoSure, my comment about softmax was simply about the probability distribution. But temperature is still part of sampling. If you’re greedy decoding, temperature doesn’t matter.
- toxik 2y agoThat is indeed an issue. Their sampling method rejects impossible combinations.
- KRAKRISMOTT 2y agoIsn't this bag of words all over again? Except with positional hints?
- taneq 2y agoWow, if that works that's wild (and also has that "damn, now you say it it's obvious" flavour that so many really cool discoveries share...)
- toxik 2y agoThis problem formulation has been around for a while, it’s kind of the holy grail of modeling. What is new compared to PixelCNN and related is this position embedding idea.
- deleted 2y ago[deleted]
- cs702 2y agoYes, the modification the authors propose is indeed "simple." It's "obvious" only in hindsight.
- tripplyons 2y agoThe only difference I see from XLNet is how they use it during inference.
- arnaudpannatier 2y agoHey! I'm Arnaud, first author of the paper. XLNet also shuffles the data during training, but they use a masking mechanism instead of the causal + double positional encoding. The application differs, XLNet is not AFAIK focused on generation (even if it can be used for that) and the burst-sampling idea is new.
- tripplyons 2y agoThanks for the clarification!
- RivieraKid 2y agoAre there any obvious practical application of this algorithm for existing large (10B+) text / image models? Does the rejection sampling lead to a statistically correct sample from the joint probability distribution or is that just a (possibly rough) approximation?
- arnaudpannatier 2y agoFor the application: being able to prompt anywhere in the sequence can be of interest. For what we've seen in the experiment, the rejection sampling leads to similar generation than the autoregressive one, we did not see any mode collapse or anything of that kind.
- WanderPanda 2y agoWait wasn't BERT all about non-causal masking aka predicting words in the middle?!
- nico 2y agoI know this is for tokens/text, but can the same concept be applied to images using something like a diffusion model? And then be able to upscale images arbitrarily by infilling?
- gwern 2y agoYes. See the related work section in the paper: there is a long history of models, recently like MAE and MaskGit, which predict pixels in basically arbitrary orders, and that is useful because it lets you train on subsets of each image, upscale/infill during generation, and so on. (If you know what MAE is, that might be the fastest way to summarize OP: "it's a GPT trained like a MAE".)
- DonHopkins 2y ago[flagged]
- gwern 2y agoI have no idea what you are talking about, given that you are linking to a comment a decade old which has nothing to do with Conway's death just now; nor have I commented on Conway in any way today or been on HN today until just now, where I saw your comment. (Nor do I intend to comment now that you have drawn my attention to her passing, as I generally feel that such obituary notices should try to celebrate the good about a person, as posterity will have quite enough time to consider the bad as well.) If you disagree and wish it 'kept out', my suggestion would be to... not link it? Especially in other pages which have nothing to do with Conway?
- DonHopkins 2y agoI'm sorry, I confused you with a twitter account that used the same name as yours, as I was upset by Lynn's passing. I take back what I said about you being an asshole.
- 2y ago
- RivieraKid 2y agoIf there are multiple missing tokens, what's the positional encoding for the "token to be predicted"?
- cs702 2y agoSee this thread, also on this page: https://news.ycombinator.com/item?id=40609689 https://news.ycombinator.com/item?id=40609689
- mglikesbikes 2y agoOff topic, but what do you use for your reading list?
- ofou 2y agoI use Emergent Mind[1] to keep track of new research published on ArXiv. You can bookmark articles once logged in. It's very useful for keeping track of articles, reading quick summaries, and following conversations on various social media. [1]: https://www.emergentmind.com/papers/2404.09562 https://www.emergentmind.com/papers/2404.09562
- inhumantsar 2y agohijacking for a bit of shameless self promotion: if you're an obsidian user, I recently built a plugin that simplifies web pages, parses out metadata, and saves them to obsidian as markdown files: https://github.com/inhumantsar/slurp https://github.com/inhumantsar/slurp arXiv comes through a bit ugly atm but it's high on my to-do list. I'm leveraging the library that Firefox uses for reader mode, so most sites come through quite well. A lot of my work right now is expanding their metadata support and fixing parser issues.
- cs702 2y agoold-fashioned text files
- barfbagginus 2y agoZotero is great for organizing and annotating papers, keeping notes, and building bibliographies. You can create libraries and sub libraries according to topic, and also create libraries for projects or reading lists. You can file items into multiple libraries, and you can also create shared libraries, allowing your team to share annotated papers. Finally it can archive offline copies of web pages, which makes it useful for blog articles and other online resources that might vanish. There's a learning curve, but it's worth it if you find yourself juggling dozens or hundreds of technical papers! Enjoy!
- concurrentsquar 2y agoGoogle Chrome has a built-in reading list (go open the 3-dotted menu at the top-right corner, then click on "Bookmarks and lists" -> "Reading list")