3 ms·

Haha I'm on an Mac Studio with an M1 Ultra, 64gb ram. I bought it when it first came out, it just happens to be good for local LLMs. I have to use a smaller qua

Haha I'm on an Mac Studio with an M1 Ultra, 64gb ram. I bought it when it first came out, it just happens to be good for local LLMs. I have to use a smaller quant of Laguna S though (I think 4-bit? Not at my machine to check), as 8-bit and full size definitely don't fit in the 64gb I have.

lcnPylGDnU4H9OF 3mo ago
Yeah, a good rule of thumb is that the weights take up ~100% of the size of the model, so 100B bytes (8-bit quant) would be, well, 100GB and a 4-bit quant would be half that.
- tommica 3mo ago
  Oh, that is a useful rule to know! Thanks!
  - cyanydeez 3mo ago
    That's not exactly the math. Theres also vram needed for context. I operate several 72-128 GB machines and the larger the context the slower they go. And the context takes space +kv cache. KV cache drives usefulness as your context grows, it needs to pull the kv cache. Simplified, the context has to be run on every turn, so the KV cache supplies the processed tokens, so it just needs the new inpute.
- CyberDildonics 3mo ago
  I don't know if that's, well, a rule of thumb, it might be, well, straight multiplication.
  - lcnPylGDnU4H9OF 3mo ago
    Well, yeah, the straight multiplication, er, well, "follows" from the rule of thumb. Hope that helps!
    - CyberDildonics 3mo ago
      Saying, well a rule of thumb is, well, 100 billion bytes is a 100 gigabytes, is well, not a rule of thumb. It is, well, just the common definition.
      - lcnPylGDnU4H9OF 3mo ago
        Right. The rule of thumb is that the overhead size of the model that's not the weights is so vastly outweighed by the actual number of weights that it can be disregarded. My shorthand for that was to write "the weights take up ~100% of the size of the model". What then "follows", both in the sense that the explanation is written after the rule as well as that it logically follows, is that, well, 100 billion bytes is, you know, 100 GB. I don't see why you're, like, paying so much attention to this?
        CyberDildonics 3mo ago
        Right. That's, well, not a rule of thumb. Asking how much of a bottle of water is, well, water, and someone says "well, a good rule of thumb is that it's all water", is just an answer. You don't need an estimate when, well, there is nothing to estimate.
        lcnPylGDnU4H9OF 3mo ago
        Yeah, well, that's just, like, your opinion, man. Anyway, more seriously, I hope it's obvious by now that I don't particularly care that my means of communication is so offensive to you. I think you should, like, cry a river, build a bridge, then, well... get over it, you know? And on the, er, "topic"? A rule of thumb for me does not have to be one for you, even if it's explicitly presented as such a rule. I thought that would be obvious but, well, here we are. Anyway, how's life been treating you?
        CyberDildonics 3mo ago
        A rule of thumb is, well, an estimation. 1 = 1 isn't an estimation, it's just, well, the same number.
tommica 3mo ago
brb, going to see if 2nd hand mac studios are available!

lcnPylGDnU4H9OF 3mo ago

Yeah, a good rule of thumb is that the weights take up ~100% of the size of the model, so 100B bytes (8-bit quant) would be, well, 100GB and a 4-bit quant would be half that.

tommica 3mo ago
Oh, that is a useful rule to know! Thanks!
- cyanydeez 3mo ago
  That's not exactly the math. Theres also vram needed for context. I operate several 72-128 GB machines and the larger the context the slower they go. And the context takes space +kv cache. KV cache drives usefulness as your context grows, it needs to pull the kv cache. Simplified, the context has to be run on every turn, so the KV cache supplies the processed tokens, so it just needs the new inpute.
CyberDildonics 3mo ago
I don't know if that's, well, a rule of thumb, it might be, well, straight multiplication.
- lcnPylGDnU4H9OF 3mo ago
  Well, yeah, the straight multiplication, er, well, "follows" from the rule of thumb. Hope that helps!
  - CyberDildonics 3mo ago
    Saying, well a rule of thumb is, well, 100 billion bytes is a 100 gigabytes, is well, not a rule of thumb. It is, well, just the common definition.
    - lcnPylGDnU4H9OF 3mo ago
      Right. The rule of thumb is that the overhead size of the model that's not the weights is so vastly outweighed by the actual number of weights that it can be disregarded. My shorthand for that was to write "the weights take up ~100% of the size of the model". What then "follows", both in the sense that the explanation is written after the rule as well as that it logically follows, is that, well, 100 billion bytes is, you know, 100 GB. I don't see why you're, like, paying so much attention to this?
      - CyberDildonics 3mo ago
        Right. That's, well, not a rule of thumb. Asking how much of a bottle of water is, well, water, and someone says "well, a good rule of thumb is that it's all water", is just an answer. You don't need an estimate when, well, there is nothing to estimate.
        lcnPylGDnU4H9OF 3mo ago
        Yeah, well, that's just, like, your opinion, man. Anyway, more seriously, I hope it's obvious by now that I don't particularly care that my means of communication is so offensive to you. I think you should, like, cry a river, build a bridge, then, well... get over it, you know? And on the, er, "topic"? A rule of thumb for me does not have to be one for you, even if it's explicitly presented as such a rule. I thought that would be obvious but, well, here we are. Anyway, how's life been treating you?
        CyberDildonics 3mo ago
        A rule of thumb is, well, an estimation. 1 = 1 isn't an estimation, it's just, well, the same number.

tommica 3mo ago

brb, going to see if 2nd hand mac studios are available!