4 ms·

xVal: A continuous number encoding for large language models

72 points by famouswaffles 3y ago

vbezhenar 3y ago
I don't understand how this is useful. People don't calculate numbers in brain, they learned to use calculator. LLM should just learn to access external API every time they need to make a calculation instead of spending precious neurones on primitive tech already solved by transistors.
- alexcannan 3y ago
  I'm personally excited about LLMs becoming more number-fluent for regression tasks, e.g. read this paper and give it a grade 0-100
- vessenes 3y ago
  Well the predicting tests they do would not be possible in this scenario, because they want to finetune a model against a set of data in order to make those predictions, and API calls aren't differentiable -> they can't train -> they can't use it to learn to predict. So, I think they see a use case here.
Someone 3y ago
So, this makes numbers in text more numbery than stringy. As a side effect, will this make it more likely that a LLM will claim C99 (https://en.wikipedia.org/wiki/C99 https://en.wikipedia.org/wiki/C99) is greater than C17 (https://en.wikipedia.org/wiki/C17_(C_standard_revision) https://en.wikipedia.org/wiki/C17_(C_standard_revision))?
dang 3y ago
Submitted title was "Single Digit Tokenization improves LLM Math abilities by up to 70x". Submitters: "Please use the original title, unless it is misleading or linkbait; don't editorialize." - https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html If you want to say what you think is important about an article, that's fine, but do it by adding a comment to the thread. Then your view will be on a level playing field with everyone else's: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&sort=byDate&type=comment&query=%22level%20playing%20field%22%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...
deleted 3y ago
[deleted]
vessenes 3y ago
So, this is a fairly interesting innovation. With what looks like really good results. The paper sort of buries the lede IMO, in that xVal looks like it could possibly be something that turns next-gen LLMs into zero-shot numeric prediction models. The big idea is to take any corpus and instead of making tokenizations for numbers that are digits (GPT-2 era) or weirdo floating point range things, instead every number goes to a single token: [NUM]. They then keep a shadow tensor/vector where the [NUM] is given an actual floating point (maybe fixed point?) number. When the model predicts a [NUM] token, there's a number prediction layer that chooses a number. This lets them train against predicting whether or not a number will be next in a generated text (the model turns out to be really good at this, not surprising). And, how cool -- their loss function can check the actual number created by the number layer, and give a high quality result as to how good the number guess was. This works really really well out of the box for a bunch of math problems up to 5 digit multiplication, like super human levels of accuracy, and vastly beats SOTA from other Foundation models. They then go on to throw fine tuning tasks of scientific data at it, and it absolutely does not shit the bed. Which is extremely profound to me, and something they do not make a big deal of. I would expect some scaled up models using this tech could be generally highly useful for a broad range of numeric / stats / prediction in the same way that GPT-3 and on have been highly useful as "text calculators". Worth a read.
- mjburgess 3y ago
  Err... interesting, maybe -- useful? I don't see why. There are a near infinite number of function approximators to choose from, and theory-building science has little use for any. Empirical, or heuristic, science already uses much faster and arbitrarily accurate ones.
- tempusalaria 3y ago
  I’ve been playing around with something similar for factual nouns, where the next token prediction is a token like [PNOUN] and then a downstream predictor chooses the actual result. One major Problem is that you need to autoregressively access the correct embedding when doing inference. And in particular in batch inference. So waiting for the downstream prediction to complete is gonna slow inference. One way I was considering getting around it was running all auxiliary classifiers in parallel with the main LLM, and just accessing those results when needed. Which then runs into the problem that those auxiliary classifiers have to be smallish relative to the main model, and then there are performance questions. A complicated thread to unravel. Their approach to num doesn’t have the same performance implications but there are some clear issues - large numbers will have blowup issues, 0 and small floats have big problems here.

72 points by famouswaffles 3y ago

I don't understand how this is useful. People don't calculate numbers in brain, they learned to use calculator. LLM should just learn to access external API every time they need to make a calculation instead of spending precious neurones on primitive tech already solved by transistors.

alexcannan 3y ago
I'm personally excited about LLMs becoming more number-fluent for regression tasks, e.g. read this paper and give it a grade 0-100
vessenes 3y ago
Well the predicting tests they do would not be possible in this scenario, because they want to finetune a model against a set of data in order to make those predictions, and API calls aren't differentiable -> they can't train -> they can't use it to learn to predict. So, I think they see a use case here.

Someone 3y ago

So, this makes numbers in text more numbery than stringy. As a side effect, will this make it more likely that a LLM will claim C99 (https://en.wikipedia.org/wiki/C99 https://en.wikipedia.org/wiki/C99) is greater than C17 (https://en.wikipedia.org/wiki/C17_(C_standard_revision) https://en.wikipedia.org/wiki/C17_(C_standard_revision))?

dang 3y ago

Submitted title was "Single Digit Tokenization improves LLM Math abilities by up to 70x". Submitters: "Please use the original title, unless it is misleading or linkbait; don't editorialize." - https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html If you want to say what you think is important about an article, that's fine, but do it by adding a comment to the thread. Then your view will be on a level playing field with everyone else's: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&sort=byDate&type=comment&query=%22level%20playing%20field%22%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...

deleted 3y ago

[deleted]

vessenes 3y ago

So, this is a fairly interesting innovation. With what looks like really good results. The paper sort of buries the lede IMO, in that xVal looks like it could possibly be something that turns next-gen LLMs into zero-shot numeric prediction models. The big idea is to take any corpus and instead of making tokenizations for numbers that are digits (GPT-2 era) or weirdo floating point range things, instead every number goes to a single token: [NUM]. They then keep a shadow tensor/vector where the [NUM] is given an actual floating point (maybe fixed point?) number. When the model predicts a [NUM] token, there's a number prediction layer that chooses a number. This lets them train against predicting whether or not a number will be next in a generated text (the model turns out to be really good at this, not surprising). And, how cool -- their loss function can check the actual number created by the number layer, and give a high quality result as to how good the number guess was. This works really really well out of the box for a bunch of math problems up to 5 digit multiplication, like super human levels of accuracy, and vastly beats SOTA from other Foundation models. They then go on to throw fine tuning tasks of scientific data at it, and it absolutely does not shit the bed. Which is extremely profound to me, and something they do not make a big deal of. I would expect some scaled up models using this tech could be generally highly useful for a broad range of numeric / stats / prediction in the same way that GPT-3 and on have been highly useful as "text calculators". Worth a read.

mjburgess 3y ago
Err... interesting, maybe -- useful? I don't see why. There are a near infinite number of function approximators to choose from, and theory-building science has little use for any. Empirical, or heuristic, science already uses much faster and arbitrarily accurate ones.
tempusalaria 3y ago
I’ve been playing around with something similar for factual nouns, where the next token prediction is a token like [PNOUN] and then a downstream predictor chooses the actual result. One major Problem is that you need to autoregressively access the correct embedding when doing inference. And in particular in batch inference. So waiting for the downstream prediction to complete is gonna slow inference. One way I was considering getting around it was running all auxiliary classifiers in parallel with the main LLM, and just accessing those results when needed. Which then runs into the problem that those auxiliary classifiers have to be smallish relative to the main model, and then there are performance questions. A complicated thread to unravel. Their approach to num doesn’t have the same performance implications but there are some clear issues - large numbers will have blowup issues, 0 and small floats have big problems here.