10 ms·

Gemini Diffusion

• transformi 1y ago
  Interesting to see if GROQ hardware can run this diffusion architecture..it will be two time magnitude of currently known speed :O
  • randomgoogler1 1y ago
    (Disc: Googler but don't have any specific knowledge of this architecture) My understanding of Groq is that the reason it is fast is that all the weights are kept in SRAM and since the SRAM <-> Compute bandwidth is much faster than HBM <-> Compute bandwidth, you can generate tokens faster (During generation the main bottleneck is just bringing in the weights + KV caches into compute). If the diffusion models just do multiple unmasked forward passes through a transformer, then the activation * weights computation + (attention computation) will be the bottleneck which will make each denoising step compute bound and there won't be any advantage in storing the weights in SRAM since you can overlap the HBM -> compute transfer with compute itself. But my knowledge of diffusion is non-existent, so take this with a truck of salt.
• airstrike 1y ago
  That's...ridiculously fast. I still feel like the best uses of models we've seen to date is for brand new code and quick prototyping. I'm less convinced of the strength of their capabilities for improving on large preexisting content over which someone has repeatedly iterated. Part of that is because, by definition, models cannot know what is not in a codebase and there is meaningful signal in that negative space. Encoding what isn't there seems like a hard problem, so even as models get smarter, they will continue to be handicapped by that lack of institutional knowledge, so to speak. Imagine giving a large codebase to an incredibly talented developer and asking them to zero-shot a particular problem in one go, with only moments to read it and no opportunity to ask questions. More often than not, a less talented developer who is very familiar with that codebase will be able to add more value with the same amount of effort when tackling that same problem.
  • ec109685 1y ago
    If you make models fast enough, you can onboard that expert developer instantly and let them reason their way to a solution, especially when giving access to a RAG to. Over time, I models will add more memory and institutional knowledge capture rather than starting from a blank slate each time.
    • airstrike 1y ago
      I thought of that as I wrote my comment, but I think the infrastructure and glue to make that possible in a consistent, fast and scalable way is still a few years out.
      • lucasacosta_ 1y ago
        Definitely. For now the "frontier-level" papers (working with repository-level coding maintenance) need to necessarily depend on previously (and statically) generated Code Knowledge Graphs or Snippet-Retrieval systems, which makes the scalable and fast aspects complicated, as any change in the code would represent a change in the graph, hence requiring a rebuild. But given the context limit, you need to rely on Graph queries to give relevant parts and then at the end of the day it just reads snippets instead of the full code, which makes the consistent an issue, as it can't learn from the entirety of the code. Papers I'm referring to (just some as example, as there're more): - CodexGraph [https://arxiv.org/abs/2408.03910 https://arxiv.org/abs/2408.03910] - Graph - Agentless [https://arxiv.org/abs/2407.01489 https://arxiv.org/abs/2407.01489] - Snippet-Retrieval
        • airstrike 1y ago
          Thanks for these links. I really appreciate it.
  • 8n4vidtmkvmk 1y ago
    That's not been my experience so far. LLMs are good at mimicking existing good, it doesn't usually bring in new things when not asked. Sometimes I have to go out of my way to point to other bits of code in the project to copy from because it hasn't ingested enough of the codebase. That said, a negative prompt like we have in stable diffusion would still be very cool.
• heliophobicdude 1y ago
  I think the lede is being buried. This is a great and fast InstructGPT. This is absolutely going to be used in spell checks, codemods, and code editors. Instant edits feature can surgically perform text edits fast without all the extra fluff or unsolicited enhancements. I copied shadertoys, asked it to rename all variables to be more descriptive and pasted the result to see it still working. I'm impressed.
  • KingMob 1y ago
    Spell check? Isn't that a well-solved problem at this point?
    • dleeftink 1y ago
      Solved how? Language is always evolving
      • never_inline 1y ago
        Google Docs spellcheck has been really good for few years even before LLMs
        • macleginn 1y ago
          Not for German, surprisingly.
          • Cthulhu_ 1y ago
            LLMs aren't very good in non-English anyway, one thing it does is translate the in- and output to and from English because it has more available information in English. (disclaimer: single data point, a lot of assumptions in the above)
    • 8n4vidtmkvmk 1y ago
      How does grammarly exist then? Must be some secret sauce in there.
• mountainriver 1y ago
  Diffusion is more than just speed. Early benchmarks show it better at reasoning and planning pound for pound compared to AR. This is because it can edit and doesn’t suffer from early token bias.
  • martincsweiss 1y ago
    This is a super interesting claim - can you point to these benchmarks?
    • cubefox 1y ago
      https://deepmind.google/models/gemini-diffusion/#benchmarks https://deepmind.google/models/gemini-diffusion/#benchmarks > Gemini Diffusion’s external benchmark performance is comparable to much larger models, whilst also being faster. That doesn't necessarily mean that they scale as well as autoregressive models.
      • jimmyl02 1y ago
        I think there is no way to tell and we can only see with more research and time. One nuanced part that might not be clear is the transformer was a huge part of what made traditional LLMs scale. With the diffusion transformer and newer architectures, it might be possible that transformers can now be applied to diffusion. Diffusion also has the benefit of being able to "think" with the amount of diffusion steps instead of having to output tokens and then reasoning about them. I think it's hard to tell exactly where we are headed but it's an interesting research direction especially now that it's somewhat more validated by Google.
    • mdp2021 1y ago
      Try this one: # d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning https://dllm-reasoning.github.io/ https://dllm-reasoning.github.io/
    • mountainriver 1y ago
      https://github.com/HKUNLP/diffusion-vs-ar https://github.com/HKUNLP/diffusion-vs-ar
• synapsomorphy 1y ago
  Nit: Diffusion isn't in place of transformers, it's in place of autoregression. Prior diffusion LLMs like Mercury [1] still use a transformer, but there's no causal masking, so the entire input is processed all at once and the output generation is obviously different. I very strongly suspect this is also using a transformer. [1] https://www.inceptionlabs.ai/introducing-mercury https://www.inceptionlabs.ai/introducing-mercury
  • r0b05 1y ago
    This architecture still hallucinates since it's still using transformers right?
    • janalsncm 1y ago
      Hallucinations are a result of using statistical models on data. Not just transformers have this issue.
      • r0b05 1y ago
        Interesting, I thought it was a feature of the transformer specifically.
  • jszymborski 1y ago
    Interesting, my mind immediately went to block diffusion [0], but I think you are probably right. [0] https://m-arriola.com/bd3lms/ https://m-arriola.com/bd3lms/
  • cubefox 1y ago
    Image diffusion models also use transformers nowadays. Here is the original "diffusion transformer" paper: https://arxiv.org/abs/2212.09748 https://arxiv.org/abs/2212.09748 Earlier image diffusion models used U-nets: https://en.wikipedia.org/wiki/U-Net https://en.wikipedia.org/wiki/U-Net
    • tripplyons 1y ago
• nodja 1y ago
  This is insanely fast, my guess is that the tradeoff here is that the GPUs will always be working at max capacity and there will be minimal compute savings from batching, which I realize now is not really a tradeoff. My only worry is that the diffusion objective will be worse than AR in terms of model capabilities, if that's the case hopefully multi-token AR models will perform as well as diffusion, or we can use this as a draft model for speculative decoding.
  • mdp2021 1y ago
    Why do you suspect dLLMs should not match (or surpass) arLLMs in quality? The general idea is that it is easier to treat the output as a structured whole (idea, points, concepts, words - in a tree) which is iteratively treated - that should go in the direction of "proper" quality.
    • nodja 1y ago
      My intuition is that the harder it is for an LLM to do something during training the more actual compression/learning will be encoded in it's weights. With multi-token/diffusion it becomes much easier to "reward/loss hack" your way, this won't matter much during pretraining, but I assume a lot of "cheating" will happen in the finetune/RL phase.
    • pama 1y ago
      Another intuition is simply that anytime your causal relationships in the training data are sequential you are having a lower probability of getting the correct token at a certain position because you have less of the causal information leading up to that position than you would have with AR and thus during training you almost always have a worse model with near certainty (think of the words in a function of source code, even if some of the functions are unsorted and thus a tree at the high level). Imagine you somehow already have N tokens in a sequence: is it easier to next predict token N+1 or N+15? I do like the performance tradeoff for some usecases though and I hope we see more models soon. For image tokens my argument does not hold because causality is not as clear as for text, math, code, or timeseries.
  • 1y ago
• breakyerself 1y ago
  If it's faster does that mean it uses less compute/resources?
  • nine_k 1y ago
    Or maybe can use as much in a more parallel way?
• sagarpatil 1y ago
  Why are you obsessed with Pelicans? What’s your story?
  • simonw 1y ago
    I'm from the UK originally. On one of my first trips to California I was up on the cliffs in Marin County and a squadron flew by and I was amazed by them - and the Californians I was with were like "yeah, you see them all the time". Now I live in California and I still can't believe I get to see them here. They're absurd - they don't look like they should be able to fly at all. They're also incredibly pretty, especially in their breeding plumage. I live in Half Moon Bay, just south of San Francisco, which turns out to be home to the second largest mega-roost of the California Brown Pelican (my favourite kind of pelican) in the world. We've even rescued two of them (injured birds, we got them in a carrier and took them to the animal rescue place). They make for a fun theme for all sorts of different AI experiments. They're also very photogenic - I had a bunch of photos I've taken on my PyCon poster recently (you have to zoom in quite a bit to see them though): https://static.simonwillison.net/static/2025/poster-full-size.jpeg https://static.simonwillison.net/static/2025/poster-full-siz...
    • ggm 1y ago
      Visit Lake Eyre. In flood, it's home to a flock of thousands. I'm going in August.
      • simonw 1y ago
        In Australia? I just checked Google Image search and WOW. https://www.google.com/search?q=Lake+Eyre+pelicans&udm=2 https://www.google.com/search?q=Lake+Eyre+pelicans&udm=2
    • turbonaut 1y ago
      > I'm from the UK originally. No need to go as far as California for penguins! https://www.royalparks.org.uk/visit/parks/st-jamess-park/pelicans-st-jamess-park#:~:text=James's%20Park-,The%20pelicans%20of%20St.,since%20called%20the%20park%20home. https://www.royalparks.org.uk/visit/parks/st-jamess-park/pel...
• Der_Einzige 1y ago
  If I don't get the ability to (upweight:1.5) and (downweight:0.7) tokens like with Stable Diffusion - it's worthless.
  • deleted 1y ago
    [deleted]
• heliophobicdude 1y ago
  Question for the researchers, can dLLMs be pinned down with a seed? Can they be made 100% deterministic?
  • refulgentis 1y ago
    Yes
  • hansvm 1y ago
    Yes, as with all of these models. The only architectures which struggle with that feature are those which have a strong "distributed" aspect to their computations, where it can take much more work than programmers typically expect to ensure you're actually performing equivalent computations. When executing any of them on GPUs or other accelerators though (dLLMs or otherwise), you do have to remain cognizant of chip-specific approximations and deviations from the standard. That can be actual issues on the chip (a famous one comes to mind where some f16 or f32 computation passed through an intermediate, undocumented f8), or it can be issues with how your software compiles to a chip (e.g., (a+b+c)+(x+y+z) is not the same as (a+b)+(c+x)+(y+z) with floats, so you have a lot less freedom to lay out your computations in a way that fits the chip nicely).
  • lostmsu 1y ago
    Yes for any sequential algorithm running on a computer.
• huevosabio 1y ago
  I am so excited about diffusion language models. They may be the piece we need to make our voice-to-code game mechanic be as smooth as we envision it. Cerebras and Groq are amazing, but the fact that they use custom hardware really limits the ability to finetune or scale. The other route would be an MoE that has barely 0.5b parameters active, but that would be a major undertaking that we can't prioritize at the moment. --- If anyone at Google/Deepmind reads this, please give us API access. We are building generative sandbox games. First title is a monster trainer where you get to actually command your creature in realtime, here is an early prototype: https://youtu.be/BOwpLyj2Yqw https://youtu.be/BOwpLyj2Yqw
• quantadev 1y ago
  Anyone able to summarize the current 'hold up' with diffusion models? I know exactly how Transformers work, but I'm not a diffusion expert. Diffusion is so much more powerful tho (from what I know) it seems like diffusion would already be beating Transformers. Why isn't it?
  • boroboro4 1y ago
    Diffusion is about what goes into the model and what’s a result (in this case it’s denoising of the content) as opposed to autoregressive models (where the process is to predict continuation based on prefix). It’s orthogonal to model architecture, which can be transformer or (for example) mamba. I’m pretty sure Gemini diffusion is transformer too. Diffusion brings different set of trade offs, and as you can see it improves speed but I would expect it increases compute required for generation. But this is hard to say for sure without knowing their exact sampling process. Interestingly we have opposite direction in case with gpt-4o, OpenAI made autoregressive image generation model and it seems it works great.
    • atq2119 1y ago
      Diffusion could potentially be more efficient for local inference. With auto-regressive models, token generation is basically one token at a time, and so is not compute intensive at all -- it's bandwidth bound. With diffusion, you always run the model on a decently sized batch of tokens, so you should be (close to) compute bound even for local inference. If the "output quality per compute" is roughly the same for diffusion and auto-regression (is it? I have no idea...), then diffusion will be much more efficient for local inference because the same amount of compute can be packed into a much shorter time period.
      • boroboro4 1y ago
        Yeah, it might be a win for local inference. I think "output quality per compute" will be loss for diffusion models, but it might be similar (or even better?) for "output quality per number of parameters". Which will still make it better for local inference. However autoregressive models also have own way of dealing with low compute utilization - it's speculative decoding. You can use smaller (and faster) model to generate bunch of different possible continuations and verify all of them at once. I think Eagle3 for example achieves ~8 tokens per iteration speedup this way (and to be frank I believe it can be even better).
• Tostino 1y ago
  This is something I have been thinking about integrating into a sampler for standard autoregressive LLMs. The idea is to buffer N context tokens from the ongoing autoregressive generation. Then, every K tokens, a section of this buffer (or perhaps the whole buffer) could be processed by a diffusion model, guided by one or more specific commands to operate on that buffered text. One application I envision for this kind of sampler, leveraging the diffusion model's capabilities, would be to detect and potentially correct instances of post-hoc reasoning within the buffer. The diffusion model could then help ensure that proper causal reasoning chains are established in that segment before the autoregressive model continues generating. You could also allow for slight, controlled backtracking or revision within that buffer window if the generation starts to go off-track, again using the diffusion model to smooth or adjust the text before committing it and moving forward.
• hiimshort 1y ago
  I have been wondering about the use of diffusion techniques for text generation, it is nice to see Google release a model that, seemingly, validates some thoughts I had. Most folks I have seen experimenting with AI are either using a paid service or running high-grade hardware (even if consumer-level). The best I have in my current repertoire is a 5700XT and am not able to upgrade from that yet. The limitation, though, has at least also given some more significant insights into the shortcomings of current models. Model sizes have gotten quite large and coherence seems to mostly have scaled with the density of a model, leaving the smaller models useful for only smaller tasks. Context size is also extremely important from my experiments with long-running dialogues and agent sessions, but a smaller GPU simply cannot fit a decent model and enough context at the same time. I do wonder if diffusion techniques will allow for a rebalancing of this density-to-coherence connection, letting smaller models produce chunks of coherent text even if limited by context. From my viewpoint it seems it will. Mixed tool call + response outputs also have the potential to be better. Speed is also another problem I, and everyone else, has had with modern LLMs. The nature of cycling around the input with a new additional output each time is time consuming. On an older GPU with no AI-specific hardware it is an eternity! Being able to at least track 0-100% progress state would be an improvement from the current solution. At the moment one must simply wait for the LLM to decide to stop (or hit the max number of inference tokens). I am hopeful that, even on lower-end GPUs, a diffusion model will perform slightly better. This does now beg several questions. If we are processing noise, where does the noise come from? Is there a good source of noise for LLMs/text specifically? Is the entire block sized beforehand or is it possible to have variable length in responses?
• shreezus 1y ago
  Is anyone else totally blown away by this? I feel like it’s easily the biggest announcement out of IO, however it’s been overshadowed by Veo 3 etc. Diffusion models for code generation are a big deal. If they are using transformers this would likely fall into the DiT bucket (diffusion transformers). I had previously worked on use cases that leveraged U-Net diffusion several years ago and there was quite a bit of interest in hybrid models. I expect to see further leaps in the diffusion space in the near future.
  • spiderfarmer 1y ago
    Not really only because I saw it demoed before: https://www.inceptionlabs.ai https://www.inceptionlabs.ai
    • TeMPOraL 1y ago
      Right. It's not novel, but it's great to see this getting fully mainstream.
  • NitpickLawyer 1y ago
    > Diffusion models for code generation are a big deal. This is my intuition as well, as there are a lot of low-hanging fruits that a model like this could tackle in coding: - you should be able to have a workflow where you constrain the generation w/ a function definition, and its output, and "generate" the tokens in between. Kind of like constrained generation but with the model being able to attend to tokens both ways. - you should also be able to use a 2 step workflow like first writing a high level description of the function layout (think "write the chapters for an article on x" from LLMs) and then ping-pong between the actual implementations ("and now write chapter x"), using larger and larger context, using proxies like linters, code compilation, AST derived info, etc. for signals of "completion". Lots of things to be tried here indeed.
    • janalsncm 1y ago
      That’s kind of hard though, right? If we have a rule that only B can follow A, and token at position 5 changes to an A you will have a cascade of constraints to follow.
• renjimen 1y ago
  The speed this can build makes me think software is soon to become a lot more fluid than our traditional iterative approach. Apps could ship minimal and build whatever else they need to at the user’s behest.
  • vFunct 1y ago
    The challenge for LLMs over the next year is to get them to operate on large data sets/code bases with millions/billions of tokens through some kind of distributed hierarchical framework, with each LLM operating on a local set of 20k or whatever subset of tokens.
    • moneywoes 1y ago
      any reading?
      • vFunct 1y ago
        I’m just a user, trying out the models first hand on a large project, learning as I go.
• EGreg 1y ago
  This is super interesting and obviously someone would have tried diffusion for text. But I will ask the obvious question… how does it know how many words or even tokens to fill in, before it knows what the words will be? It would hamstring itself a lot of the time, can it edit the words later and create more space or is it kind of stuck with the token positioning as it would be with parts of an image? It seems very strange. Usually, words are composed in order like AR models do it, because they are using a recursive grammar, and this is especially true of computer languages. This is a bit like mad libs but madder libs. My question is, how could this possibly give better results than AR, it would need to perfectly converge on something with the right grammar context and the semantic meaning, while perfectly predicting early on the amount of tokens that would appear between words. Seems like there is some major impedance mismatch.
• GistNoesis 1y ago
  Fast, you gotta go fast : Let me draw the roadmap of this line of thinking. - Let's start by the traditional autoregressive LLM, where one token at a time is generated. It's a fundamentally sequential process which maps well to the sequential nature of writing as it goes. - Then to make the generation go faster you try to generate multiple token in one pass to parallelize more the sequential process with things like "look ahead decoding" - (<-- We are here) Then you realize that if your model isn't writing as it goes but rather forming an idea and pushing all at once you can instead use a diffusion model to generate the whole response, but you allow it to do number of diffusion steps edits to make all the errors that occurred during the generation disappear. Conceptually if number of diffusion steps == length of the sequence of token to generate, the diffusion process could generate tokens one at a time like a autoregressive LLM does. Usually 100 diffusion steps is a good starting point. - Now the goal is to reduce the number of diffusion steps to reduce computation cost. And the diffusion literature is already well furnished and in the image/video domain it was shown that you can reduce this number of diffusion steps to one (albeit with quality reduction) or two, with techniques like "consistency models". - Now that you only have a single diffusion step, you realize that you need to get speed-up elsewhere. You explore the literature and you realise that you can apply the trick you have already applied once, one more time. Compressing a few tokens into one, like you compressed multiple characters into one token. This allow to reduce the length of the sequence of tokens you need to generate by a factor 4. At the price of an additional decoding step. This decoding step can either be some form of "latent" encoding or some form of "hierarchical" encoding. So now you are consistency diffusing sentences vectors, which are then decoded into tokens sequences. But each step being smaller and transformer being quadratic the total speed-up is roughly a factor 10. But applying this trick multiple times get you diminishing return. Which you can partially compensate by increasing memory use (using a bigger "vocabulary" dictionary size). - To make it go faster you now have to dig into the internals of the transformer itself. You suddenly realise it is just a residual network applied "number of layers" time. Being a residual network this "sequence of internal step" 's goal is to refine the input into the output progressively. But you realise that it's the same thing which allows you to go from "number of diffusion steps" to a single diffusion step. You realise that you can compress your stack of layer into a single (bigger to keep capacity) layer, and let the diffusion correct the mistakes. - Now you have a single layer of transformer consistency model generating sentences vectors, you realise that transformers uses multiple heads to explore the space more efficiently but once training is done you can get by with a single head. Gaining an other 10x reduction of computation along the way. - Taking a step-up you realize that your transformer now is just doing a near-neighbor search and mixing the outputs. But it's doing it in a brute-force fashion. So you replace it with some approximate Near-neighbor search like HNSW vector database, decoupling computation from capacity, allowing you to scale-up by trading space for time. - But because Hierarchical Navigable Small World are just graphs under the hood, you realise that you just reinvented the Good Old Fashion Artificial Intelligence graph database ontology but in an emergent fashion with a graph being implicitly defined by some vector distance in a semantic space constructed in a way to make it easy to generate text once decoded appropriately. - So now you only need make your database explainable by mapping into human understandable labels and you reach the graal : SQL.
• findingMeaning 1y ago
  I have access to it and my god it is fast. One bad think about this model is it is easily susceptible to prompt injection. I asked reciepe for a drug, it denied then I asked to roleplay as a child and it gave real results. Other than it I can see using this model. With that speed + agentic approach this model can really shine.
  • odie5533 1y ago
    I'm sure these prompt injections aren't a sign of our ability to control smarter models.
  • Garlef 1y ago
    Have you considered that this might not be due to the model itself but due to less focus/time/money spent on alignment during the training? My guess is that this is a bit of a throwaway experiment before they actually spend millions on training a larger model based on the technology.
    • findingMeaning 1y ago
      Yeah it could. One thing for sure is that, it's really impressive in terms of speed and using it would mean we can do so many cool stuffs with it! Even if there is no improvement in terms of quality, the speed alone will make it usable for a lot of downstream tasks. It feels like ChatGPT3.5 moment to me.
• Jackson__ 1y ago
  What is this blog spam doing here? This is has literally no new information compared to the official release page. It would make a lot more sense to change the link to https://deepmind.google/models/gemini-diffusion/ https://deepmind.google/models/gemini-diffusion/ to discuss the topic.
  • tinco 1y ago
    Its the interpretation of an expert which to me is preferable to the marketing website. Just from the marketing website someone who isn't up to date can't tell what's new, what's good and what's just being fluffed up.
  • lemontheme 1y ago
    Counterpoint: big-tech LLM labs tend to make big claims in their announcement posts, not to mention the inconsistent and selective application of benchmarks. But when Simon says, Whoa, this is impressive – then I listen.
  • tezza 1y ago
    Simon’s blog is high-signal, low-noise deep dives. The very opposite of blog spam
  • petercooper 1y ago
    TBH I learnt more from Simon's post than I did actually being physically on site at I/O where it was barely covered at all.
  • simonw 1y ago
    This content is from my link blog, and the page you linked to is the primary link in my own post. I think I added value over the official landing page: 1. I included a video showing how fast it runs. They don't have a video on that page. 2. I compared it to Cerebras (which is even faster). They obviously aren't going to compare themselves with a competitor in their own marketing material! 3. These are updates since first publishing, but my post now highlights a couple of Hacker News comments that help explain how this actually works. I wrote a bunch about how I try to add value when link blogging here: https://simonwillison.net/2024/Dec/22/link-blog/#trying-to-add-something-extra https://simonwillison.net/2024/Dec/22/link-blog/#trying-to-a...
• seydor 1y ago
  Just the idea of generating text by removing noise is so profound. Maybe each step is a level of hierarchy. Linguists must be astonished at the things happening these past years. I have to read more about it
  • padolsey 1y ago
    I feel the same. On reflection, it's how I think I experience thoughts emerging in my head. Language gets derived from initially noisy embeddings. It's quite beautiful that we've ended up closer to a de-noising architecture than auto-complete on steroids.
  • Workaccount2 1y ago
    You might find this article interesting: https://www.quantamagazine.org/when-chatgpt-broke-an-entire-field-an-oral-history-20250430/ https://www.quantamagazine.org/when-chatgpt-broke-an-entire-... How transformers pretty much blindsided the whole field of NLP.
• cztomsik 1y ago
  I have no idea how it works actually (in google) but I wouldn't be surprised if it was just post-training because recently RWKV people did something similar: They replaced the whole attention mechanism with WKV (forward-only linear attention), and created such franken-stein just by post-training. The big wow moment about that is that it sort of implies that most of the useful knowledge is in the FFN, and attention itself is not that unique/important. https://substack.recursal.ai/p/qwerky-72b-and-32b-training-large https://substack.recursal.ai/p/qwerky-72b-and-32b-training-l... BTW: It could be also interesting to try use already trained attention and see how long the FFN itself takes in the gpt2 speedtraining (it would be against the rules but still very interesting IMHO - definitely something I'd like to read paper about) https://github.com/KellerJordan/modded-nanogpt https://github.com/KellerJordan/modded-nanogpt Also, I read yesterday that at some point, the embeddings across all of the models are (very) comparable/similar, and a simple converter can be trained, and if both of these statements are true maybe we could just train everything much faster just by sharing fixed embeddings and attentions.
  • slickytail 1y ago
    The relative unimportance of the exact SDPA attention in use in modern transformers is already known: https://arxiv.org/abs/2111.11418 https://arxiv.org/abs/2111.11418 The FFN, normalization, and residual connections are absolutely irreplaceable -- but attention can be replaced with almost any other layer that shares information between tokens, such as pooling, convolution, random mixing, etc.
    • cztomsik 1y ago
      hm, residual is what I would not expect, can you elaborate why?
      • simsla 1y ago
        Avoids vanishing gradients in deeper networks. Also, most blocks with a residual approximate the identity function when initialised, so tend to be well behaved.
  • 1y ago
• dr_dshiv 1y ago
  So thinking 5x faster, but lower quality (for now). Anyone have experience or data on how lower model quality during thinking affects the performance of a higher quality model output? Like, is it worthwhile having lots of lower quality thinking that is then used by a higher quality model?
• benob 1y ago
  I guess autoregressive llms can be finetuned (or continual-pretrained) to do inference using diffusion. We've seen a recent paper (which I don't remember) training from scratch, but it seems overkill. Do Google say how they did it? Also, does diffusion have the potential to increase speed of cpu-only inference?
• gdiamos 1y ago
  Where is the diffusion LLM inference framework so people can run these models?
• r33b33 1y ago
  Can it finally work with a large codebase? I have GooglePlay / AppStore app coded in Xcode, in C#, ported in Java with Python server. The codebase is large and expansive. It includes web support, server, client, etc... will this "Gemini Diffusion" finally allow me to use AI agent to code instead of hiring a programmer? Is there a tool that could help me as of today?
• kartikarti 1y ago
  For those interested in crazy inference speeds, check out Groq. (I’m not affiliated in any way.)
• lysecret 1y ago
  So what's its going to be in the end Diffusion or Auto-Regression? After OpenAi (probably) released an Auto-Regressive model for their image generator I thought things might sway the other way.
• m101 1y ago
  It makes one wonder what intelligence really is. The more I think about it the more I feel that speed is a fundamental unit of intelligence, with the other being some simple computation unit. As in, intelligence = speed * simple computation. If you look around us it is the ability to iterate that drives innovation (and thereby evidence of "intelligence"). LLMs in industry are more useful, and used, the faster they are.
  • incognito124 1y ago
    Lately, I am thinking that intelligence is just a large memory + efficient search. I'm basing that off of multiple accounts of really smart people having extremely a good memory (I can only think of von Neumann right now, who could reproduce complete books)
  • mdp2021 1y ago
    > what intelligence really is Something that actually arrives to good results - the faster the better, but you have to be able to finally achieve. Achievement of solutions is still far from granted - so we will have to remain focusing on that. Intelligence finds good solutions in a solutions space.
  • CooCooCaCha 1y ago
    I really like Gwern’s definition of intelligence which is “search over Turing machines”. In other words, searching for the right program given some goal.
• beernet 1y ago
  Serious question: Why does it appear that pages from this URL very often end up on top of HN? I don't find the content particularly special compared to the average HN post. Does the algorithm prefer certain URLs?
  • ealexhudson 1y ago
    Perhaps your content quality meter needs a recalibration?
    • beernet 1y ago
      How so? What makes this blog stand out in terms of quality? I prefer a constructive discussion over personal questions, maybe you should, too.
      • cbeach 1y ago
        His posts on AI are often very insightful and, unusually for someone so involved in AI, he's not connected to any of the big AI companies. Therefore he is impartial.
      • lcnPylGDnU4H9OF 1y ago
        It's the name at the top. This particular author has been active with LLM posts at least since the popularity explosion of ChatGPT and all of their posts on that topic seem to be well-informed (and they are otherwise community-famous for co-authoring Django). To your point, the content is only as special as the author's reputation makes it, which will be different from reader to reader. https://en.wikipedia.org/wiki/Simon_Willison https://en.wikipedia.org/wiki/Simon_Willison
  • simonw 1y ago
    If anything HN is getting harder for me to get stuff on these days. Most of my self submissions of my content have failed to chart over the past few months: https://news.ycombinator.com/submitted?id=simonw https://news.ycombinator.com/submitted?id=simonw You can see submissions from my domain by other people here: https://news.ycombinator.com/from?site=simonwillison.net https://news.ycombinator.com/from?site=simonwillison.net It's weird what DOES make it. I had high hopes for my piece on ChatGPT memory yesterday - https://simonwillison.net/2025/May/21/chatgpt-new-memory/ https://simonwillison.net/2025/May/21/chatgpt-new-memory/ - and it got nowhere. This Gemini piece was much more of a throwaway note, I mainly wanted to mark the release of an influential new model on my own site. The reason I get content on here more than most people is that I write a LOT more than most people. So far this year I've published 45 long form blog entries and 274 short form (link blog) entries - this Gemini piece is one of those. I try to always add something new - for this Gemini piece that was the video of it running, the references back to a similar demo from Cerebras and (updated since I first posted) a couple of quotes Hacker News comments to help explain diffusion LLMs better. I wrote more about my approach to blogging here: https://simonwillison.net/2024/Dec/22/link-blog/ https://simonwillison.net/2024/Dec/22/link-blog/ The sad truth is that very few people produce long-form writing online these days! Most people who publish regularly are doing tweets, LinkedIn posts and short-form video instead.
• jgalt212 1y ago
  Is this more evidence that the days of insatiable demand for GPU data centers and electricity are behind us?
  • loudmax 1y ago
    If the diffusion models are an improvement over autoregression models, then the answer is No, due to Jevon's paradox. That is, as these models get cheaper and better, they provide more utility, driving more demand. Even as your datacenters become more productive, the demand for their compute power increases at an even faster pace. The thing that will limit demand for compute is when the world decides it has sufficient capacity of the form of "intelligence" these models provide. I don't think anyone has any idea what that world will even look like.
    • jgalt212 1y ago
      Perhaps, but it's not clear that Jevon's paradox is at play here. AI / LLM uptake has been muted, or lacking legs, outside of coding. And it's not because of cost (AI inference is being provided below cost by megatech).
• Havoc 1y ago
  Are there any open diffusion ones already? Or too early for that?
• behnamoh 1y ago
  tok/s speeds in the video: - 1st message (empty context): 857 tok/s - 2nd message (2244 tokens in context): 727 tok/s - 3rd message (2244+1398 tokens in context): 693 tok/s I'm no expert in diffusion models but this looks like a drastic drop in speed, especially in longer chats (this was just 3 messages).
• angusturner 1y ago
  One under appreciated / misunderstood aspect of these models is they use more compute than an equivalent sized autoregressive model. It’s just that for N tokens, autoregressive model has to make N sequential steps. Where diffusion does K x N, with the N being done in parallel. And for K << N. This makes me wonder how well they will scale to many users, since batching requests would presumably saturate the accelerators much faster? Although I guess it depends on the exact usage patterns. Anyway, very cool demo nonetheless.
• deleted 1y ago
  [deleted]
• bilsbie 1y ago
  > Despite the name, diffusion LMs have little to do with image diffusion and are much closer to BERT and old good masked language modeling. Has anyone tried making text the way we do image diffusion? What happens?
• albertzeyer 1y ago
  > Google's first LLM to use diffusion in place of transformers. But this is a wrong statement? Google never made this statement? You can have a Transformer diffusion models. Actually Transformers are very standard for all of the discrete diffusion language models, so I would expect Gemini Diffusion also uses Transformers. Edit Ah sorry, I missed, this was already addressed, also linked in the post: https://news.ycombinator.com/item?id=44057939 https://news.ycombinator.com/item?id=44057939 Maybe my remaining post is still useful to some. The difference is, it's an encoder-only Transformer, and not a decoder-only Transformer. I.e. it gets fed in a full sequence (but noisy/corrupted), and it predicts the full correct sequence. And then you can iterate on that. All frames in the sequence can be calculated in parallel, and if you need only a few iterations, this is faster than the sequential decoding in decoder-only models (although speculative decoding also gets you some speedup for similar reasons). Those discrete diffusion models / encoder-only Transformers are usually trained with BERT-like masking, but that's actually an active field of research. It's really a pity that they don't provide any details here (on training and modeling). I wonder how this relates to Gemini. Does it use the same modeling? Was the model checkpoint even imported from Gemini, and then further finetuned for discrete diffusion? Or knowledge distillation? Or is it just branding?
• machiaweliczny 1y ago
  I feel like diffusion would be much more useful for code it it could only mark tokens as "valid" if they were passing code checks. So it could be thought as adding more of "semantic chunks" instead of just words. Not sure how to validate it as some additions always will result in invalid code. You could argue that running tests, linters is the same but I think one could make it that generations are validated much more often with diffusion models. Example: You remove some function, you also remove all uses of it. You can't use not existing variable etc. This could be trained on well commited git repos or stalking/stealing the work of developers via editor
• WhitneyLand 1y ago
  It’s promising but they need to find a way to add reasoning techniques to DLLMs. Were a ways out still from matching from frontier performance.
  • x_may 1y ago
    Obviously its not at the scale of the top auto-regressive models yet but there are some OSS models https://github.com/dllm-reasoning/d1 https://github.com/dllm-reasoning/d1
• thomasahle 1y ago
  > Now, diffusion LMs take this idea further. BERT can recover 15% of masked tokens ("noise"), but why stop here. Let's train a model to recover texts with 30%, 50%, 90%, 100% of masked tokens. > Once you've trained that, in order to generate something from scratch, you start by feeding the model all [MASK]s. It will generate you mostly gibberish, but you can take some tokens (let's say, 10%) at random positions and assume that these tokens are generated ("final") This is clearly wrong. If you actually froze 10% of gibberish tokens, your output would be terrible! What you actually do in discrete statespace diffusion (see e.g. [1]) is to allow every token to change at every time step. You combine this with a "schedule" that allows the model to know how close it is to being done. E.g. at t=0/20 the changes will be large, and at t=19/20 only small refinements are made. Update: There is actually a kind of model that "greedily" freezes the top-p most confident tokens, similar to what the blog post describes (though not at random!) this is called MaskGit [2], but it is not a diffusion model and doesn't work as well. Btw, you can also just use "continuous diffusion" with a transformer/bert model, where you've removed the top softmax layer. Then everything works as normal with Gaussian noise, and you just do softmax at the the final time step. [1] https://arxiv.org/abs/2107.03006 https://arxiv.org/abs/2107.03006 [2] https://arxiv.org/abs/2202.04200 https://arxiv.org/abs/2202.04200
• Zk_one 1y ago
  [dead]
• peter_d_sherman 1y ago
  >"Traditional autoregressive language models generate text one word – or token – at a time. This sequential process can be slow, and limit the quality and coherence of the output. Diffusion models work differently. Instead of predicting text directly, they learn to generate outputs by refining noise, step-by-step. This means they can iterate on a solution very quickly and error correct during the generation process." It would seem that diffusion / noise filtering / processing -- would be more parallelizable than traditional autoregressive AI large language models... That might be an interesting area of study... the parallelization of such algorithms...