3 ms·

We will have M4 laptops running 400B parameter models next year. Wild times.

• visarga 2y ago
  And they will fit in the 8GB RAM with 0.02 bit quant
  • gpm 2y ago
    You can get a macbook pro with 128 GB of memory (for nearly $5000). Which still implies... a 2 bit quant?
    • freeqaz 2y ago
      There are some crazy 1/1.5 bit quants now. If you're curious I'll try to dig up the papers I was reading. 1.5bit can be done to existing models. The 1 bit (and less than 1 bit iirc) requires training a model from scratch. Still, the idea that we can have giant models running in tiny amounts of RAM is not completely far fetched at this point.
      • gpm 2y ago
        Yeah, I'm broadly aware and have seen a few of the papers, though I definitely don't try and track the state of the art here closely. My impression and experience trying low bit quants (which could easily be outdated by now) is that you are/were better off with a smaller model and a less aggressive quantization (provided you have access to said smaller model with otherwise equally good training). If that's changed I'd be interested to hear about it, but definitely don't want to make work for you digging up papers.
    • moneywoes 2y ago
      eli5 quant?
      • gpm 2y ago
        Quant is short for "quantization" here. LLMs are parameterized by a ton of weights, when we say something like 400B we mean it has 400 billion parameters. In modern LLMs those parameters are basically always 16 bit floating point numbers. It turns out you can get nearly as good results by reducing the precision of those numbers, for instance by using 4 bits per parameter instead of 16, meaning each parameter can only take on one of 16 possible values instead of one of 65536.
        • sbierwagen 2y ago
          Interestingly enough, Llama3 suffers more performance loss than Llama2 did at identical quantizations. https://arxiv.org/abs/2404.14047 https://arxiv.org/abs/2404.14047 There's some speculation that a net trained for more epochs on more data learns to pack more information into the weights, and so does worse when weight data is degraded.
        • Der_Einzige 2y ago
          Most claims of "nearly as good results" are massively overblown. Even the so called "good" quants of huge models are extremely crippled. Nothing is ever free, and even going from 16 to 8bit will massively reduce the quality of your model, no matter whatever their hacked benchmarks claim. No, it doesn't help because of "free regularization" either. Dropout and batch norm were also placebo BS that didn't actually help to back in the day when they were still being used.
      • fennecfoxy 2y ago
        Quantization is reducing the number of bits to store a parameter for a machine learning model. Put simple, a parameter is a number that determines how likely it is that something will occur, ie if the number is < 0.5 say "goodbye" otherwise say "hello". Now, if the parameter is a 32bit (unsigned) integer it can have a value of 0-4,294,967,296. If you were using this 32bit value to represent physical objects, then you could represent 4,294,967,296 objects (each object gets given its own number). However a lot of the time in machine learning, after training you can find that not quite so many different "things" need to be represented by a particular parameter, so if say you were representing types of fruit with this parameter (Google says there are over 2000 types of fruit, but let's just say there are exactly 2000). In that case 4,294,967,296/2000 means there are 2.1 million distinct values we assign each fruit, which is such a waste! Our perfect case would be that we use a number that only represents 0-2000 in the smallest way for this job. Now is where quantization comes in, where the size of the number we use to represent a parameter is reduced, saving memory size at the expense of a small performance hit of the model accuracy - it's known that many models don't really take a large accuracy hit from this, meaning that the way the parameter is used inside the model doesn't really need/take advantage of being able to represent so many values. So what we do is say, reduce that 32bit number to 16, or 8, or 4 bits. We go from being able to represent billions or millions of distinct values/states to maybe 16 (with 4bit quantization) and then we benchmark the model performance against the larger version with 32bit parameters - often finding that what training has decided to use that parameter for doesn't really need an incredibly granular value.
    • pulse7 2y ago
      Current 2 bit quant models are useless. Smaller models yield better results.