Permuting entire columns at once should have zero overhead as long as you permute the rows of the next matrix to match. But as each entry of a column participat

Permuting entire columns at once should have zero overhead as long as you permute the rows of the next matrix to match. But as each entry of a column participates in a different scaling group, I guess swapping two columns will reduce quantization error for some while increasing it for others, making it unlikely to get a significant overall improvement in this way.