3 ms·

The value is encoded as a fraction out of 256. So 1 is 1/256, up to 255 being 255/256. It's still not completely correct, some values passed through that funct

• neobrain 13y ago
  "Still, shouldn't there be a round in there too?" I don't think so, since that particular code was just meant to emulate integer overflows (in contrast to the limited decimal precision of integers). If you were to emulate the precision as well, it would likely need an additional round around everything indeed, i.e. something like round(value - 2.0 * round(0.5 * value * (255.0/256.0)) * 256.0). If anything, this discussion shows that it's getting annoyingly complex to find the correct formula though, especially if all corner cases are supposed to be handled correctly. Oh right, and the real fun begins when you try to emulate 24 bit integers, for which the proposed method doesn't work at all because floats only have 23 bits of mantissa :) Chances are there are simpler ways to emulate this stuff, I really don't know. Would be interesting to hear from GPU driver developers how integers are emulated within the driver via floats if hardware does not support integers :)