4 ms·
> We found that the model's modulation weights (~40% of the total parameters) could be pruned and replaced with a functionally equivalent lookup table, dramatic
by embedding-shape 2mo ago
> We found that the model's modulation weights (~40% of the total parameters) could be pruned and replaced with a functionally equivalent lookup table, dramatically shrinking the memory footprint with no loss in output quality.
Is this a common approach to reducing weights with "no loss in output quality", assuming this is true? Seems almost too simple to work. If this is doable, would this be applicable to LLMs as well?
Neat with native frame-to-frame generation, but wonder how easy it is to "link" together clips at the intersection, typically the models kind of lose the "momentum" across these stiches, being able to merge things with frame-to-frame between clips might help with this it feels like.
- _diyar 2mo agoAlso begs the question whether this is applicable for high-throughput applications on FPGAs, which are to my novice mind basically LUTs, right? I remember a paper which was posted on HN a few weeks ago where somebody implemented KAN networks in FPGAs, since those can readily be approximated as LUTs.
- yorwba 2mo ago"Modulation weights" here refers to weights used to adjust layer normalization depending on the task (adaLN). General-purpose LLMs generally don't have those in the first place.
- liuliu 2mo agoOne thing similar would be projecting both the head.weight and the final LLM activations into a smaller vector space, since that is basically just cosine similarity ranking step (so that would reduce the head.weight size). But again, it must be tried many times and just not working as well. LLM space is pretty saturated with tricks.
- knicholes 2mo agoWhoah, could this mean we can treat layers like a jpg, where we come up with a formula that estimates the weight values of a layer instead of storing all of the weights?
- swiftcoder 2mo agoThis is more or less what those lovely 1.5 bit quantisations are already doing
- liuliu 2mo agoIt is a well-known trick, given that the timestep is between 0 to 1, you can slicing them at any resolution (1000, or 10000, give or take), and then keep a look-up table for modulation scale / bias etc for each. It is quite different from quantization and it is indeed lossless. It is also only applicable to diffusion models as only these operates at per-timestep.
- xienze 2mo agoSo... why didn't the model ship this way to begin with? They just wanted to waste VRAM for fun?
- liuliu 2mo agoThey ship a complete checkpoint for easily management (inference & training) in their own infrastructure. Moving to a LUT would make training on these layers impossible. BTW, these are not useful for lightweight fine-tuning, but might still be useful if you do serious post-training work. Of course, these are also not an issue for things like FLUX.2 which adopts DiT-Air arch, that doesn't have this wasted space issue.
- doctorpangloss 2mo agoIt may or may not be true. The people who made this modification and the other commenters didn't do anything rigorous to verify what they did. They just eyeball it. They could very well make some other error - this has happened frequently - that developing on prod, not knowing what they are doing, has and hopefully will again solve.
- cma 2mo agoThis isn't the same with lookup tables, but in Explaining Attention with Program Synthesis they were able to replace some attention heads with Python programs: https://arxiv.org/abs/2606.19317 https://arxiv.org/abs/2606.19317