6 ms·
Transformer models: an introduction and catalog
- theredlancer 4y agoWhere's Cliffjumper and Ironside?
- deleted 4y ago[deleted]
- zndr 4y agoI'm glad I'm not the only one looking for a taxonomy of refugees from the great Cybertron wars
- sircastor 4y agoWhen I was younger I would often encounter mentions of electrical transformers, and be quite disappointed when it wasn't related to the toys or the series. Even in my 40s I still have a bit of disappointment about it...
- robertlagrant 4y agoDon't get me started on when we learned about the Terminator on the moon.
- swyx 4y agofigure 5 on page 10 is a ridiculously small font and unreadable. i wish there was a better way to display this kind of info on PDFs
- mdp2021 4y ago> i wish there was a better way to display this kind of info on PDFs ...Portable Document Format was /born/ to display vector (i.e. you just zoom in)... The error in the page was to embed a raster image of text!
- deleted 4y ago[deleted]
- dylan604 4y agoFrom the bottom of the page in question: "Figure 5: You can access the original table at https://docs.google.com/spreadsheets/d/ https://docs.google.com/spreadsheets/d/ 1ltyrAB6BL29cOv2fSpNQnnq2vbX8UrHl47d7FkIf6t4 for easier browsing across the different model features."
- nighthawk454 4y agoAnother tip is arXiv lets you download the original tar file from the "Other Sources" section with all the original images etc in it. https://arxiv.org/format/2302.07730 https://arxiv.org/format/2302.07730
- adamnemecek 4y agoI have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834v1 https://arxiv.org/abs/2302.01834v1 The learning mechanism of transformer models was poorly understood however it turns out that a transformer is like a circuit with a feedback. I argue that autodiff can be replaced with what I call in the paper Hopf coherence. Furthermore, if we view transformers as Hopf algebras, one can bring convolutional models, diffusion models and transformers under a single umbrella. I'm working on a next gen Hopf algebra based machine learning framework. Join my discord if you want to discuss this further https://discord.gg/mr9TAhpyBW https://discord.gg/mr9TAhpyBW
- erichocean 4y ago> Furthermore, if we view transformers as Hopf algebras, one can bring convolutional models, diffusion models and transformers under a single umbrella. Have you written any more about this?
- adamnemecek 4y agoLook into the connection between diffusion and Hopf algebras.
- mi_ainsel 4y agoApologies if I'm missing something completely obvious but I'm having some difficulties dredging up material that connects Hopf algebras and diffusion. Are there any resources that you would recommend in addition to your paper?
- amkkma 4y agoIs there any hope of understanding this with just calc and linalg knowledge?
- adamnemecek 4y agoI think so. The main idea is the idea of Hopf coherence. The transformer/Hopf algebra update their internal state in order to enforce the Hopf coherence formula (you can find that in the paper). The idea of streams (as in infinite lists) is related to this via coalgebras.
- abc20230215 4y agoDoes not even list Lundahl transformers...
- deleted 4y ago[deleted]
- peresthe 4y ago"The goal of this paper is to offer a somewhat comprehensive but simple catalog and classification of the most popular Transformer models." Yet of the 6 comments here, 2 of them are complaining about missing models and three more are arguing about the typesetting on figures.
- pama 4y agoPer the paper, here is the link to the data in fig 5: https://docs.google.com/spreadsheets/d/1ltyrAB6BL29cOv2fSpNQnnq2vbX8UrHl47d7FkIf6t4/edit#gid=0 https://docs.google.com/spreadsheets/d/1ltyrAB6BL29cOv2fSpNQ...
- deleted 4y ago[deleted]
- xamat 4y agoThanks for the feedback everyone. Here are the sources in case anyone wants to contribute (or fork): https://github.com/xamat/TransformerCatalog https://github.com/xamat/TransformerCatalog
- h_lezzaik 4y agoGood timing, I've been trying to compile a list like this myself to keep track of everything released.
- amatecha 4y agoI was so certain this was discussing Transformers like, the action figures, and have never been so confused looking at both a link and the comments section on HN before. Especially considering: https://github.com/xamat/TransformerCatalog/blob/main/02-01.jpeg https://github.com/xamat/TransformerCatalog/blob/main/02-01.... I'm just going to keep scrolling now :'D
- DerSaidin 4y agoIt is a shame the figures 5,6,7,8 break up the content of 2.5 Catalog, just to fit the figures onto pages. Are pages even needed anymore?
- visarga 4y ago> 2.5.9 ChatGPT is also more than a model since it includes extensions for Memory Store and retrieval similar to BlenderBot3 I don't think this affirmation is factual. There are people who played with this idea, but it is not part of chatGPT.
- sva_ 4y ago> 2.5.5 BERT > Extension:It can be seen as a generalization of BERT and GPT in that it combines ideas from both in the encoder and decoder I believe this is an error? Text from BART. And a space missing.
- visarga 4y agoI have a hunch they used LLM to compile the list.