4 ms·
Transfusion: Predict the next token and diffuse images with one multimodal model
- valine 2y agoThis is such a natural extension to LLMs. I’m shocked it hasn’t been tried before. When I ask a diffusion model to generate a chessboard, I’d expect the pieces to be placed randomly. We are getting closer to image generators that not only know what chess pieces look like but also where to place them.
- ilaksh 2y agoHmm. I wonder if this is similar to Diffusion Transformers?
- darknoon 2y agothis is somewhat similar, but diffusion transformers typically use a pre-trained text model as the text conditioning whereas, in this case it's integrated and trained together multimodally.
- cosmicjedi 2y agoYou can talk to the authors directly on alphaXiv! https://www.alphaxiv.org/abs/2408.11039v1 https://www.alphaxiv.org/abs/2408.11039v1
- re 2y agoIt doesn't look like they are active there.
- BaculumMeumEst 2y agoStupid question: is their 7B model available? Is there public inference code that we could run? Or do they not usually release them along with these kinds of papers?
- K0balt 2y agoI’d also like to know this.
- HanClinto 2y agoDoesn't appear to be any weights uploaded anywhere that I can find. There are the starts of two (non-original-author) public implementations available on Github, but again -- doesn't appear to be any pretrained weights in either. * https://github.com/lucidrains/transfusion-pytorch https://github.com/lucidrains/transfusion-pytorch * https://github.com/VachanVY/Transfusion.torch https://github.com/VachanVY/Transfusion.torch
- littlestymaar 2y agoWould such a model be able to give more accurate description of images as well?
- swfsql 2y agoI think so, specially with finetuning