4 ms·

To really be an AI PC, I'd want to be able to run something like llama 3 70B locally, and that's going to need a lot of RAM. Even running a 7B will take 14GB if

To really be an AI PC, I'd want to be able to run something like llama 3 70B locally, and that's going to need a lot of RAM. Even running a 7B will take 14GB if it's fp16. So these really just need to be able to run a few MS AI apps. A quick googling tells me things like the Gen-AI in photoshop is actually done in the cloud, so that should still work but doesn't really require a special "AI PC".

coder543 2y ago
No, 7B LLMs only need about 4GB of RAM. There is extremely little quality loss from dropping to 4-bit for LLMs, and that “extremely little” becomes “virtually unmeasurable” loss when going to 8-bit. No one should be running these models on local devices at fp16 outside of research, since fp16 makes them half as fast as q8_0 and requires twice as much RAM for no benefit. If a model is inadequate for a task at 4-bit, then there's virtually no chance it's going to be adequate at fp16. Microsoft has also been doing a lot of research into smaller models with the Phi series, and I would be surprised if Phi3 (or a hypothetical Phi4) doesn’t show up at some point under the hood.
- Tankenstein 2y ago
  The original commenter mentioned 70B, not 7B.
  - coder543 2y ago
    I had already read the comment I was responding to, and they actually mentioned both. Here's the exact quote for the 7B: "Even running a 7B will take 14GB if it's fp16." Since they called out a specific amount of memory that is entirely irrelevant to anyone actually running 7B models, I was responding to that. I'm certain that no one at Microsoft is talking about running 70B models on consumer devices. 7B models are actually a practical consideration for the hardware that exists today.
    - Tankenstein 2y ago
      Makes sense, your argument is clear now, my mistake.
    - SushiHippie 2y ago
      They said: "Even running a 7B will take 14GB if it's fp16." Which is correct, fp16 takes two bytes per weight, so it will be 7 billion * 2 bytes which is exactly 14GB. They are probably aware that you could run it with 4 bit quantization (which would use 1/4 of the RAM) but explicitly mentioned fp16.
      - coder543 2y ago
        > > Since they called out a specific amount of memory that is entirely irrelevant to anyone actually running 7B models, I was responding to that. > Which is correct, fp16 takes two bytes per weight, so it will be 7 billion * 2 bytes which is exactly 14GB. As I said, it is "entirely irrelevant", which is the exact wording I used. Nowhere did I say that the calculation was wrong for fp16. Irrelevant numbers like that can be misleading to people unfamiliar with the subject matter. No one is deploying LLMs to end users at fp16. It would be a huge waste and provide a bad experience. This discussion is about Copilot+, which is all about managed AI experiences that "just work" for the end user. Professional-grade stuff, and I believe Microsoft has good enough engineers to know better than to deploy fp16 LLMs to end users.
imiric 2y ago
You're not the intended market for this product. This is aimed at the general consumer who is interested in a fully tailored experience powered by "AI", and doesn't care whether the magic happens locally or in the cloud, not at someone who wants to run arbitrary models locally and tinker with the experience. Whether these machines can be repurposed for what you want to do, and whether the experience will be worth it, is yet to be seen.
wmf 2y ago
These laptops have up to 64 GB RAM which should be enough to run 70B models (quantized).

coder543 2y ago

No, 7B LLMs only need about 4GB of RAM. There is extremely little quality loss from dropping to 4-bit for LLMs, and that “extremely little” becomes “virtually unmeasurable” loss when going to 8-bit. No one should be running these models on local devices at fp16 outside of research, since fp16 makes them half as fast as q8_0 and requires twice as much RAM for no benefit. If a model is inadequate for a task at 4-bit, then there's virtually no chance it's going to be adequate at fp16. Microsoft has also been doing a lot of research into smaller models with the Phi series, and I would be surprised if Phi3 (or a hypothetical Phi4) doesn’t show up at some point under the hood.

Tankenstein 2y ago
The original commenter mentioned 70B, not 7B.
- coder543 2y ago
  I had already read the comment I was responding to, and they actually mentioned both. Here's the exact quote for the 7B: "Even running a 7B will take 14GB if it's fp16." Since they called out a specific amount of memory that is entirely irrelevant to anyone actually running 7B models, I was responding to that. I'm certain that no one at Microsoft is talking about running 70B models on consumer devices. 7B models are actually a practical consideration for the hardware that exists today.
  - Tankenstein 2y ago
    Makes sense, your argument is clear now, my mistake.
  - SushiHippie 2y ago
    They said: "Even running a 7B will take 14GB if it's fp16." Which is correct, fp16 takes two bytes per weight, so it will be 7 billion * 2 bytes which is exactly 14GB. They are probably aware that you could run it with 4 bit quantization (which would use 1/4 of the RAM) but explicitly mentioned fp16.
    - coder543 2y ago
      > > Since they called out a specific amount of memory that is entirely irrelevant to anyone actually running 7B models, I was responding to that. > Which is correct, fp16 takes two bytes per weight, so it will be 7 billion * 2 bytes which is exactly 14GB. As I said, it is "entirely irrelevant", which is the exact wording I used. Nowhere did I say that the calculation was wrong for fp16. Irrelevant numbers like that can be misleading to people unfamiliar with the subject matter. No one is deploying LLMs to end users at fp16. It would be a huge waste and provide a bad experience. This discussion is about Copilot+, which is all about managed AI experiences that "just work" for the end user. Professional-grade stuff, and I believe Microsoft has good enough engineers to know better than to deploy fp16 LLMs to end users.

imiric 2y ago

You're not the intended market for this product. This is aimed at the general consumer who is interested in a fully tailored experience powered by "AI", and doesn't care whether the magic happens locally or in the cloud, not at someone who wants to run arbitrary models locally and tinker with the experience. Whether these machines can be repurposed for what you want to do, and whether the experience will be worth it, is yet to be seen.

wmf 2y ago

These laptops have up to 64 GB RAM which should be enough to run 70B models (quantized).