4 ms·
> Why not scale to fill the available bins, though? i.e. trunc(result * 255.999)? That’s half of the mid-riser staircase quantizer discussed in the article. (T
by klodolph 4mo ago
> Why not scale to fill the available bins, though? i.e. trunc(result * 255.999)?
That’s half of the mid-riser staircase quantizer discussed in the article. (The other half is coming up with the reverse.)
(I would implement it as min(floor(x * 256), 255).)
- dpark 4mo agoKind of. According to the article, the mid-riser vs mid-tread distinction correlates to where the 0.5 is applied in the transform. I’m proposing that there is no 0.5 applied at all. Instead of counteracting the compression of the truncation operation by adding a fixed offset, it multiplies by a scale factor. Possibly my proposal doesn’t hold up to repeated transforms and operations. It might skew toward 255 in real operations.
- klodolph 4mo agoThe 0.5 comes from debiasing the round-trip. If your conversion from high precision -> 8-bit is just multiplication by 256 and then truncation, then you’ve got the mid-riser quantizer. The +0.5 comes from interpreting a value of 0 as bucket from 0-1, just like the value of 255 is the bucket from 255-256. It’s introduced in the conversion back from 8-bit to high precision.
- dpark 4mo agoYou’re still misinterpreting what I proposed. I didn’t propose a +0.5 debias at all. I proposed that the bias is removed by scaling. In this case you divide by 255.0 but multiply by 256. But again, the likely reason no one does this is because it introduces a bias in the other direction, toward 255.
- klodolph 4mo agoIt’s unbiased, but amplifies the signal.