4 ms·

I agree: worth a read! There are a few big things I don't like about this encoding scheme, though, including: * It's not end-to-end. The input string must be

by cs702 3y ago

I agree: worth a read! There are a few big things I don't like about this encoding scheme, though, including: * It's not end-to-end. The input string must be parsed before feeding it to the LLM, for finding and substituting any numbers in the string with the special [NUM] token. Parsing is hard -- and prone to ugly edge-case failures, e.g., due to misspellings. I'll keep hoping and wishing for an end-to-end solution. * It doesn't work well with numbers written out as a combination of digits as strings, e.g., "How much is a third of their 7.123% percent ownership of that series of preferred stock?" (note the superfluous "%", which would likely contribute to messing things up). * It requires that every number be shifted and scaled, respectively, by the mean and standard deviation of all numbers in the training corpus, so it hopefully falls within the range [-5, 5], for offsetting the 'squashing' of features caused by LayerNorm before the first residual block.

vessenes 3y ago
What do you think an end-to-end solution would look like? I missed the Layernorm squashing bit, I skimmed that part when I read the paper. I'll re-read. From your description, I agree that's a problem. I wonder if closer to a floating point / exponential representation would be better, if it let you not squash - e.g. it would become a great order-of-magnitude numerist.
- cs702 3y ago
  Thank you for your thoughtful response! For me, an end-to-end solution would look like a model that can 1) read numbers in any format from the string, 2) manipulate and reason about them correctly, and 3) reliably output structured data if requested. No need for special encodings. Obviously, we're nowhere near that! In the interim, I agree with you that a representation that explicitly encodes a mantissa and an exponent seems like a better stopgap solution. We human beings already do that to some extent -- e.g., we often think in terms of tens, hundreds, thousands, and so on, as numbers get bigger, and in terms of percents, basis points, and decimal points as they get smaller. I wouldn't be surprised if large models internally will eventually learn implicitly to represent numbers as mantissas and exponents.

by cs702 3y ago

vessenes 3y ago

What do you think an end-to-end solution would look like? I missed the Layernorm squashing bit, I skimmed that part when I read the paper. I'll re-read. From your description, I agree that's a problem. I wonder if closer to a floating point / exponential representation would be better, if it let you not squash - e.g. it would become a great order-of-magnitude numerist.

cs702 3y ago
Thank you for your thoughtful response! For me, an end-to-end solution would look like a model that can 1) read numbers in any format from the string, 2) manipulate and reason about them correctly, and 3) reliably output structured data if requested. No need for special encodings. Obviously, we're nowhere near that! In the interim, I agree with you that a representation that explicitly encodes a mantissa and an exponent seems like a better stopgap solution. We human beings already do that to some extent -- e.g., we often think in terms of tens, hundreds, thousands, and so on, as numbers get bigger, and in terms of percents, basis points, and decimal points as they get smaller. I wouldn't be surprised if large models internally will eventually learn implicitly to represent numbers as mantissas and exponents.