4 ms·
They tested the quantisation on 3 different models. They also show it has little to no effect relative to fp16 on these models. IMO that's enough. Comparison
by snovv_crash 2y ago
They tested the quantisation on 3 different models.
They also show it has little to no effect relative to fp16 on these models.
IMO that's enough. Comparison against smaller models is much less useful because you can't use the same random seeds. So you end up with a very objective "this is worse" based purely on aesthetic preferences of one person vs another. You already see this with Flux Schnell vs. the larger Flux models.
- djoldman 2y agoI disagree. They report that their method produces a model that is 6.5 GB from flux (22.7GB). Why wouldn't you want to know how their 6.5GB model compares to other 6.5GB models? Regarding aesthetic prefs: it's an open problem what an appropriate metric is for GenAI... LLM arena is widely regarded as a good way to measure LLMs and that's user preferences. In any case, the authors report LPIPs etc. They could do the same for other small models.
- snovv_crash 2y agoLPIPS and similar don't work if the scene is different, as happens if the random seed doesn't match. This is why they can use it to compare the quantised network, but not against networks with reduced numbers of weights.