5 ms·
There's a whole lot of information on this page that doesn't tell me anything about what this actually is. Can anyone spell out what the architecture is here?
by alphazard 19d ago
There's a whole lot of information on this page that doesn't tell me anything about what this actually is. Can anyone spell out what the architecture is here?
They claim it's not an LLM, which I read as "not an auto-regressive token generator". I assume they are still using a transformer, otherwise they would be talking about the thing that's not a transformer, instead of all the fluff on the linked page. But they emphasize parallel generation, so is it like a text diffusion model?
- tacoooooooo 19d agoSounds like its essentially a generalized zero-shot classifier that takes and option set at runtime and works on unstructured inputs. you pass in your "prompt" and options (described in natural language) that it can respond with, in addition to your input. it gives back that option set with a probability assigned to each one
- CompleteSkeptic 19d agoyes and can do many of those in parallel
- bigglebear 19d agoI would guess a tiny stripped down text diffusion model. It only has 32k context, and for choice mode it can only select from 10 choices.
- wesammikhail 19d ago> and for choice mode it can only select from 10 choices. Rip there goes my excitement. I have a task that something like this would be great for but the list of options is a zero or two larger than that xd