9 ms·

I don't understand how a "CSV file/database/model" of 70,000,000,000 (70B) "parameters" of 4-bit weights (a 4 bit value can be 1 of 16 unique numbers) gets us a

I don't understand how a "CSV file/database/model" of 70,000,000,000 (70B) "parameters" of 4-bit weights (a 4 bit value can be 1 of 16 unique numbers) gets us an interactive LLM/GPT that is near-all-knowledgable on all topics/subjects. edit: did research, the 4-bit is just a "compression method", the model ends up seeing f32? > Quantization is the process of mapping 32-bit floating-point numbers (which are the weights in the neural network) to a much smaller bit representation, like 4-bit values, for storage and memory efficiency. > Dequantization happens when the model is used (during inference or even training, if applicable). The 4-bit quantized weights are converted back into floating-point numbers that the model's computations are actually performed with. This is done using the scale and zero-point determined during the initial quantization, or through more sophisticated mapping functions that aim to preserve as much information as possible despite the reduced precision. so what is the relationship to "parameters" and "# of unique tokens the model knows about (vocabulary size)"? > At first glance, LLAMa only has a 32,000 vocabulary size and 65B parameters as compared to GPT-3, > The 65 billion parameters in a model like LLAMA (or any large language model) essentially function as a highly intricate mapping system that determines how to respond to a given input based on the learned relationships between tokens in its training data.

Filligree 3y ago
It doesn't, is the simple answer. The slightly more complicated one is that a compressed text dump of Wikipedia isn't even 70GB, and this is lossy compression of the internet.
- MuffinFlavored 3y ago
  say the average LLM these days has a unique token (vocabulary) size of ~32,000 (not its context size, # of unique tokens it can pick between in a response. English words, punctuation, math, code, etc.) the 60-70B parameters of models is basically like... just stored patterns of "if these 10 tokens in a row input, then these 10 tokens in a row output score the highest" Is that a good summary? > The model uses its learned statistical patterns to predict the probability of what comes next in a sequence of text. based on what inputs? 1. previous tokens in the sequence from immediate context 2. tokens summarizing the overall topic/subject matter from the extended context 3. scoring of learned patterns from training 4. what else?
  - numeri 3y ago
    Your suggested scheme (assuming a mapping from 10 tokens to 10 tokens, with each token taking 2 bytes to store) would take (32000 * 20) * 2 bytes = 2.3e78 TiB of storage, or about 250 MiB per atom in the observable universe (1e82), prior to compression. I think it's more likely that LLMs are actually learning and understanding concepts as well as memorizing useful facts, than that LLMs have discovered a compression method with that high of a compression ratio, haha.
    - mjburgess 3y ago
      LLMs cannot determine the physical location of any atoms. they cannot plan movement, and so on. LLMs are just completing patterns of text that have been given before, 'everthing ever written' is both a lot for any individual person to read; but also, almost nothing, in that to propertly describe a table requires more information text is itself an extremely compressed medium which lacks almost any information about the world; it succeeds in being useful to generate because we have that information and are able to map it back to it
      - numeri 3y ago
        I didn't imply that they know anything about where atoms are, I was just pointing out the sheer absurdity of that volume of data. I should make it clear that my comparison there is unfair and mostly just funny – you don't need to store every possible combination of 10 tokens, because most of them will be nonsense, so you wouldn't actually need that much storage. That being said, it's been fairly solidly proven that LLMs aren't just lookup tables/stochastic parrots.
        mjburgess 3y ago
        > fairly solidly proven that LLMs aren't just lookup tables/stochastic parrots Well i'd strongly disagree. I see no evidence of this; I'm am quite well acquainted with the literature. All empirical statistical AI is just a means of approximating an empirical distribution. The problem with NLP is that there is no empirical function from text tokens to meanings; just as there is no function from sets of 2D images to a 3D structure. We know before we start that the distributions of text tokens are only coincidentally related to the distributions of meanings. The question is just how much value that coincidence has in any given task. (Consider, eg., that if I ask, "do you like what i'm wearing?" there is no distribution of responses which is correct. I do not want you to say "yes" 99/100, or even 100/100 times. etc. what I want you to say is a word caused a mental state you have: that of (dis)liking what i'm wearing. Since no statistical AI systems generate outputs based on causal features of reality, we know a priori that almost all possible questions that can be asked cannot be answered by LLMs. They are only useful where questions have cannonical answers; and only because "cannonical" means that a text->text function is likely to be conidentally indistinguishable from a the meaning->meaning function we're interested in).
- ramses0 3y ago
  Is there some sort of "LLM-on-Wikipedia" competition? ie: given "just wikipedia" what's the best score people can get on however these models are evaluated. I know that all the commercial ventures have a voracious data-input set, but it seems like there's room for dictionary.llm + wikipedia.llm + linux-kernel.llm and some sort of judging / bake-off for their different performance capabilities. Or does the training truly _NEED_ every book every written + the entire internet + all knowledge ever known by mankind to have an effective outcome?
  - CraigJPerry 3y ago
    >> Or does the training truly _NEED_ every book every written + the entire internet + all knowledge ever known by mankind to have an effective outcome? I have the same question. Peter Norvig’s GOFAI Shakespeare generator example[1] (which is not an LLM) gets impressive results with little input data to go on. Does the leap to LLM preclude that kind of small input approach? [1] link should be here because I assumed as I wrote the above that I would just turn it up with a quick google. Alas t’was not to be. Take my word for it, somewhere on t’internet is an excellent write up by Peter Norvig on LLM vs GOFAI (good old fashioned artificial intelligence)
    - darby_eight 3y ago
      [dead]
  - bionhoward 3y ago
    Yes, that’s known as the Hutter Prize http://prize.hutter1.net/ http://prize.hutter1.net/
    - ramses0 3y ago
      Not exactly, because LLM's seem to be exhibiting value via "lossy knowledge response" vs. "exact reproduction measured in bytes", but close.
      - 3y ago
Acumen321 3y ago
Quantization in this context is the precision of each value in the vector or matrix/tensor. If the model in question has a token embedding length of 1024, even if it was a 1 bit quantization, each token has 2^1024 possible values. If the context length is 32,000 tokens, there are 32,000^2^1024 possible inputs.

Filligree 3y ago

It doesn't, is the simple answer. The slightly more complicated one is that a compressed text dump of Wikipedia isn't even 70GB, and this is lossy compression of the internet.

MuffinFlavored 3y ago
say the average LLM these days has a unique token (vocabulary) size of ~32,000 (not its context size, # of unique tokens it can pick between in a response. English words, punctuation, math, code, etc.) the 60-70B parameters of models is basically like... just stored patterns of "if these 10 tokens in a row input, then these 10 tokens in a row output score the highest" Is that a good summary? > The model uses its learned statistical patterns to predict the probability of what comes next in a sequence of text. based on what inputs? 1. previous tokens in the sequence from immediate context 2. tokens summarizing the overall topic/subject matter from the extended context 3. scoring of learned patterns from training 4. what else?
- numeri 3y ago
  Your suggested scheme (assuming a mapping from 10 tokens to 10 tokens, with each token taking 2 bytes to store) would take (32000 * 20) * 2 bytes = 2.3e78 TiB of storage, or about 250 MiB per atom in the observable universe (1e82), prior to compression. I think it's more likely that LLMs are actually learning and understanding concepts as well as memorizing useful facts, than that LLMs have discovered a compression method with that high of a compression ratio, haha.
  - mjburgess 3y ago
    LLMs cannot determine the physical location of any atoms. they cannot plan movement, and so on. LLMs are just completing patterns of text that have been given before, 'everthing ever written' is both a lot for any individual person to read; but also, almost nothing, in that to propertly describe a table requires more information text is itself an extremely compressed medium which lacks almost any information about the world; it succeeds in being useful to generate because we have that information and are able to map it back to it
    - numeri 3y ago
      I didn't imply that they know anything about where atoms are, I was just pointing out the sheer absurdity of that volume of data. I should make it clear that my comparison there is unfair and mostly just funny – you don't need to store every possible combination of 10 tokens, because most of them will be nonsense, so you wouldn't actually need that much storage. That being said, it's been fairly solidly proven that LLMs aren't just lookup tables/stochastic parrots.
      - mjburgess 3y ago
        > fairly solidly proven that LLMs aren't just lookup tables/stochastic parrots Well i'd strongly disagree. I see no evidence of this; I'm am quite well acquainted with the literature. All empirical statistical AI is just a means of approximating an empirical distribution. The problem with NLP is that there is no empirical function from text tokens to meanings; just as there is no function from sets of 2D images to a 3D structure. We know before we start that the distributions of text tokens are only coincidentally related to the distributions of meanings. The question is just how much value that coincidence has in any given task. (Consider, eg., that if I ask, "do you like what i'm wearing?" there is no distribution of responses which is correct. I do not want you to say "yes" 99/100, or even 100/100 times. etc. what I want you to say is a word caused a mental state you have: that of (dis)liking what i'm wearing. Since no statistical AI systems generate outputs based on causal features of reality, we know a priori that almost all possible questions that can be asked cannot be answered by LLMs. They are only useful where questions have cannonical answers; and only because "cannonical" means that a text->text function is likely to be conidentally indistinguishable from a the meaning->meaning function we're interested in).
ramses0 3y ago
Is there some sort of "LLM-on-Wikipedia" competition? ie: given "just wikipedia" what's the best score people can get on however these models are evaluated. I know that all the commercial ventures have a voracious data-input set, but it seems like there's room for dictionary.llm + wikipedia.llm + linux-kernel.llm and some sort of judging / bake-off for their different performance capabilities. Or does the training truly _NEED_ every book every written + the entire internet + all knowledge ever known by mankind to have an effective outcome?
- CraigJPerry 3y ago
  >> Or does the training truly _NEED_ every book every written + the entire internet + all knowledge ever known by mankind to have an effective outcome? I have the same question. Peter Norvig’s GOFAI Shakespeare generator example[1] (which is not an LLM) gets impressive results with little input data to go on. Does the leap to LLM preclude that kind of small input approach? [1] link should be here because I assumed as I wrote the above that I would just turn it up with a quick google. Alas t’was not to be. Take my word for it, somewhere on t’internet is an excellent write up by Peter Norvig on LLM vs GOFAI (good old fashioned artificial intelligence)
  - darby_eight 3y ago
    [dead]
- bionhoward 3y ago
  Yes, that’s known as the Hutter Prize http://prize.hutter1.net/ http://prize.hutter1.net/
  - ramses0 3y ago
    Not exactly, because LLM's seem to be exhibiting value via "lossy knowledge response" vs. "exact reproduction measured in bytes", but close.
    - 3y ago

Acumen321 3y ago

Quantization in this context is the precision of each value in the vector or matrix/tensor. If the model in question has a token embedding length of 1024, even if it was a 1 bit quantization, each token has 2^1024 possible values. If the context length is 32,000 tokens, there are 32,000^2^1024 possible inputs.