3 ms·

Could any explain this section a little more? I didn’t quite follow this part. What does it mean to squish by 4 or by 2? > The only way to improve quantization

Could any explain this section a little more? I didn’t quite follow this part. What does it mean to squish by 4 or by 2? > The only way to improve quantization is through more normalization constants. A normalization constant squishes the input distribution, for example, I5, into the target distribution, for example, I3. We can increase precision, by squishing each vector only as much as is needed. For example, if you have the two vectors: > > [3, 1, 2, 3] > [0, 2, 2, 0] > > Then you can squish the first by 4 and the second by 2. This will give you twice the precision to quantize the second vector because the inputs are now spread over a broader range of the I3 data type.

thegeomaster 4y ago
It means to bring the range of the vector to between 0 and 1. In this case the operation is a simple division by a constant, and the argument is that you can decide on this constant per-vector so that you maximize the utilization of this space between 0 and 1. If you'd have squished both by 4, the second vector would be [0, 0.5, 0.5, 0], leaving essentially half the quantization space (0.5-1.0) unused, and leaving you with less precision.
- michaelbarton 4y ago
  Ah, I think I got it, thanks. The broader range is the range (0-1) as opposed to (0-0.5)? This is what the author means by "This will give you twice the precision to quantize the second vector because the inputs are now spread over a broader range of the I3 data type."?
  - thegeomaster 4y ago
    Precisely.
sigmoid10 4y ago
The wording is a bit weird, but "squishing" here just means using a constant number (the normalization constant) and multiplying all numbers in a vector by it. E.g: Multiplying [0,2,2,0] by 2 gives you [0,4,4,0], which is better distributed over the I3 distribution [0,2,4] (which goes all the way to 4). Any additional values that might get lost to rounding between 0 and 2 could be more easily restored that way. E.g. [0,1,2,2,0] would become [0,2,2,2,0] without squishing and [0,2,4,4,0] with it.
- michaelbarton 4y ago
  Ah thanks so the explanation, the second part is still a bit confusing for me. When the author writes "A normalization constant squishes the input distribution, for example, I5, into the target distribution, for example, I3." It makes me think normalisation in the range (0-2) which I3 is in. Do I have that right?
deleted 4y ago
[deleted]

thegeomaster 4y ago

It means to bring the range of the vector to between 0 and 1. In this case the operation is a simple division by a constant, and the argument is that you can decide on this constant per-vector so that you maximize the utilization of this space between 0 and 1. If you'd have squished both by 4, the second vector would be [0, 0.5, 0.5, 0], leaving essentially half the quantization space (0.5-1.0) unused, and leaving you with less precision.

michaelbarton 4y ago
Ah, I think I got it, thanks. The broader range is the range (0-1) as opposed to (0-0.5)? This is what the author means by "This will give you twice the precision to quantize the second vector because the inputs are now spread over a broader range of the I3 data type."?
- thegeomaster 4y ago
  Precisely.

sigmoid10 4y ago

The wording is a bit weird, but "squishing" here just means using a constant number (the normalization constant) and multiplying all numbers in a vector by it. E.g: Multiplying [0,2,2,0] by 2 gives you [0,4,4,0], which is better distributed over the I3 distribution [0,2,4] (which goes all the way to 4). Any additional values that might get lost to rounding between 0 and 2 could be more easily restored that way. E.g. [0,1,2,2,0] would become [0,2,2,2,0] without squishing and [0,2,4,4,0] with it.

michaelbarton 4y ago
Ah thanks so the explanation, the second part is still a bit confusing for me. When the author writes "A normalization constant squishes the input distribution, for example, I5, into the target distribution, for example, I3." It makes me think normalisation in the range (0-2) which I3 is in. Do I have that right?

deleted 4y ago

[deleted]