5 ms·

Surprising that the retained accuracy is so high after removing 1/2 of parameters. Does this help with being able to run inference on low-end GPUs?

• BUFU 2y ago
  I believe it definitely does. The inference cost will be much cheaper.
• int_19h 2y ago
  The main constraint on consumer GPUs is the VRAM - you can pretty much always do inference reasonably fast on any model that you can fit. And most of that VRAM is the loaded parameters, so yes, this should help with running better models locally. I wonder how much they'd be able to trim the recent QwQ-32b. That thing is actually good enough to be realistically useful, and runs decently well with 4-bit quantization, which makes it 16Gb large - small enough to fit into a 3090 or 4090, but that's about it. If it can be squeezed into more consumer hardware, we could see some interesting things.
  • concerndc1tizen 2y ago
    Does this mean that the model will be half the size? If a 32B model@4bit normally requires 16 GB VRAM, at half the size, it could be run @8bit with 16 GB VRAM? Isn't that tradeoff a great improvement? I assume the improved bit precision will more than compensate for the loss related to removal?
    • int_19h 2y ago
      There is some improvement going from 4-bit to 8-bit quantization, but if you have VRAM to spare for that, you usually see more benefit from running a 2x larger model at 4-bit. So in scenarios where an LM already fits the existing VRAM budget, I would expect larger models instead. The other thing is that VRAM is used not just for the weights, but also for prompt processing, and this last part grows proportionally as you increase the context size. For example, for the aforementioned QwQ-32, with base model size of ~18Gb at 4-bit quantization, the full context length is 32k, and you need ~10Gb extra VRAM on top of weights if you intend to use the entirety of that context. So in practice, while 30b models fit into 24Gb (= a single RTX 3090 or 4090) at 4-bit quantization, you're going to run out of VRAM once you get past 8k context. Thus the other possibility is that VRAM saved by tricks like sparse models can be used to push that further - for many tasks, context size is the limiting factor.
      • bombela 2y ago
        For readability, I recommend reserving "b" for bits, "B" for byte, "p" for parameter. I assume in your post that "30b" meant 30 billion, or in other words, 30Gp (giga-parameter). Furthermore is 24Gb of VRAM 24 gigabits (power of 10), or 24 gibibits (power of 2)?
        • int_19h 2y ago
          For readability I'm using the same convention that is generally used for these applications, where if you see "-Nb" after a model name, it always refers to the number of parameters. I have never once seen "p" for "parameter", never mind terms like "giga-parameter". Most certainly if you go searching for models on HuggingFace etc, you'll have to deal with "30b" etc terminology whether you like it or not. With VRAM, this quite obviously refers to the actual amount that high-end GPUs have, and I even specifically listed which ones I have in mind, so you can just look up their specs if you genuinely don't know the meaning in this context.
  • j-pb 2y ago
    You can run Models up to 128GB on a MacBook Pro Max. So we're already at a point where you can run all but the biggest frontier models on consumer hardware.
    • supermatt 2y ago
      > more consumer hardware
    • ben_w 2y ago
      Given the price tag, I don't think I'd call that "consumer" hardware, but rather "professional" hardware. But perhaps that's just me…
      • menaerus 2y ago
        Yeah, I also think that the ~5k price is quite hefty. It's difficult for me to imagine that running sizeable LLMs on commodity/consumer hardware will be possible without another breakthrough in the field. The prices of GPUs I wouldn't expect to fall if technology proves its worthiness.
        • ElevenLathe 2y ago
          I can imagine a world where "good enough" GPGPUs become embedded in common chipsets the same way "good enough" regular GPUs are embedded now, but we're definitely not there yet. That said, it was only a few years between the VooDoo cards coming to market and Intel integrated graphics showing up.
          • menaerus 2y ago
            We already have something similar in terms of HW accelerators for AI workloads in recent CPU designs but that's not enough. LLM inference workloads are bound by the compute power, sure, but that's not insurmountable IMO. Much bigger challenge is memory. Not even the bandwidth but just a sheer amount of RAM you need to just load the LLM weights. Specifically, even a single H100 will hardly suffice to host a mid-sized LLM such as llama3.1-70B. And H100 is ~50k. If that memory amount requirement is there to stay, and with current LLM transformer architecture it is, then what is really left as an only option for affordable consumer HW are only the smallest and least powerful LLMs. I can't imagine having a built-in GPGPU with 80G of on-die memory. IMHO.
  • jorvi 2y ago
    AMD Radeon series ≥6800 & ≥7800 have 16GB VRAM too.
    • 8jef 2y ago
      Even RX 7600 XT has 16GB
      • jorvi 2y ago
        I wonder if a 7600 XT is a cut-down 7800 XT then, because both normal and XT variants of the 6700 and 7700 only have 12GB VRAM. Nonetheless, great info. Sounds like it might be the budget inference king!
        • Numerlor 2y ago
          Completely different chips; the VRAM differences are from how GDDR can be used, with either 1 or 2 chips on a single 32bit bus, the configuration with 2 chips is called clamshell. The 7800 XT and 7600 XT have same VRAM but the 7800 XT has a 256 bit memory bus while the 7600 XT has a 128 bit memory bus. Meanwhile the 7700 XT with 12 GB is on a 192 bit memory bus. The workstation edition of GPUs usually do the clamshell configuration so they can easily double the VRAM and ramp up the price by a couple thousand
• redman25 2y ago
  I wonder if the sparse model would perform worse on out of sample test data.