3 ms·

Please, do tell me where else can I run 32b models locally without quantization

by rolisz 2y ago

cma 2y ago
64GB of system RAM and partial offload to a laptop 3090 or 4090 will run 32b unquantized.
- hmottestad 2y ago
  What kind of performance are you seeing? The tests that I have seen are not usable for anything because they are so slow.
- rolisz 2y ago
  Will I get an answer back anytime soon? On a MBP pro it's conversational speed.
  - cma 2y ago
    Nope, on cloud it is faster than conversational where you can skim to the right place or quickly regenerate seeing it is going down the wrong path.
    - hmottestad 2y ago
      I take that to mean that it is basically so slow as to be useless and that you would in fact need a Macbook to be able to run this model locally. This whole thread has become a bit frustrating with comments about how other laptops are competitive with a Macbook. But no one can actually name a single laptop that can do all the things that a Macbook can.
      - cma 2y ago
        I just replied to someone saying it couldn't run on a Windows laptop with how it can. It's below reading rate. Both are below skimming rate that is more useful for LLMs. I would rather have a 96GB macbook than a windows laptop if my only case was local LLMs and I didn't care about all the hoops around sealed system volume, signed app troubles, etc. that are bringing Macbooks closer to ipads/chromebooks, or the many other things that are better with nvidia GPUs. For desktop I'd still rather threadripper etc. where it has more competitive CPU memory bandwidth and is upgradable and can run multiple GPUs.
adrian_b 2y ago
If you do not care about speed, pretty much everywhere. On any other computers with integrated GPU you can have more memory than on Apple computers, where memory is abnormally expensive. The main advantage of the current Apple computers is a wider interface with memory, which provides a better memory throughput for their integrated GPUs. In 2025 it is expected that others will catch up with Apple even from this point of view (e.g. AMD Strix Halo).
- hmottestad 2y ago
  Let's hope that AMD Strix Halo will allow for 128 GB of VRAM when it releases next year. That's what Apple has to offer at the moment. > On any other computers with integrated GPU you can have more memory than on Apple computers, where memory is abnormally expensive. Are there any laptops that support 256 GB of RAM that cost less than a fully speced out Macbook Pro?
  - talldayo 2y ago
    > Are there any laptops that support 256 GB of RAM that cost less than a fully speced out Macbook Pro? No, but there are absolutely laptops with more than 128gb of addressable RAM. If you did your research you might have found them, at a literal fraction of the Macbook's price: https://www.lenovo.com/us/en/p/laptops/thinkpad/thinkpadp/thinkpad-p16-gen-2-16-inch-intel/len101t0069?sortBy=Recommended https://www.lenovo.com/us/en/p/laptops/thinkpad/thinkpadp/th... https://www.dell.com/en-us/shop/cty/pdp/spd/precision-16-7680-laptop https://www.dell.com/en-us/shop/cty/pdp/spd/precision-16-768... Really, this is what I mean by wondering whether anyone even cares anymore. When you work long enough in the tech industry, you end up sitting next to people that just do not care about anything they interact with. People that use a paid GUI to write git commits, people that pay Pakastani developers to do their work for them, people that brag about computers and smartphones like jewelry. Do you actually research the domain that you claim to care about, or do you just own things and then find reasons to get defensive when people attack your insecurities online? If you have not used a non-Apple laptop since Apple Silicon released, do not pretend you're in a position of authority to make comparisons. I had an M1 Pro for work, and while it was nice for day-to-day browsing I would have never bought one for myself. It's GPU is truly one of the least appealing parts, both from a performance perspective and considering what little it actually supports.

Please, do tell me where else can I run 32b models locally without quantization

by rolisz 2y ago

Please, do tell me where else can I run 32b models locally without quantization

cma 2y ago

64GB of system RAM and partial offload to a laptop 3090 or 4090 will run 32b unquantized.

hmottestad 2y ago
What kind of performance are you seeing? The tests that I have seen are not usable for anything because they are so slow.
rolisz 2y ago
Will I get an answer back anytime soon? On a MBP pro it's conversational speed.
- cma 2y ago
  Nope, on cloud it is faster than conversational where you can skim to the right place or quickly regenerate seeing it is going down the wrong path.
  - hmottestad 2y ago
    I take that to mean that it is basically so slow as to be useless and that you would in fact need a Macbook to be able to run this model locally. This whole thread has become a bit frustrating with comments about how other laptops are competitive with a Macbook. But no one can actually name a single laptop that can do all the things that a Macbook can.
    - cma 2y ago
      I just replied to someone saying it couldn't run on a Windows laptop with how it can. It's below reading rate. Both are below skimming rate that is more useful for LLMs. I would rather have a 96GB macbook than a windows laptop if my only case was local LLMs and I didn't care about all the hoops around sealed system volume, signed app troubles, etc. that are bringing Macbooks closer to ipads/chromebooks, or the many other things that are better with nvidia GPUs. For desktop I'd still rather threadripper etc. where it has more competitive CPU memory bandwidth and is upgradable and can run multiple GPUs.

adrian_b 2y ago

If you do not care about speed, pretty much everywhere. On any other computers with integrated GPU you can have more memory than on Apple computers, where memory is abnormally expensive. The main advantage of the current Apple computers is a wider interface with memory, which provides a better memory throughput for their integrated GPUs. In 2025 it is expected that others will catch up with Apple even from this point of view (e.g. AMD Strix Halo).

hmottestad 2y ago
Let's hope that AMD Strix Halo will allow for 128 GB of VRAM when it releases next year. That's what Apple has to offer at the moment. > On any other computers with integrated GPU you can have more memory than on Apple computers, where memory is abnormally expensive. Are there any laptops that support 256 GB of RAM that cost less than a fully speced out Macbook Pro?
- talldayo 2y ago
  > Are there any laptops that support 256 GB of RAM that cost less than a fully speced out Macbook Pro? No, but there are absolutely laptops with more than 128gb of addressable RAM. If you did your research you might have found them, at a literal fraction of the Macbook's price: https://www.lenovo.com/us/en/p/laptops/thinkpad/thinkpadp/thinkpad-p16-gen-2-16-inch-intel/len101t0069?sortBy=Recommended https://www.lenovo.com/us/en/p/laptops/thinkpad/thinkpadp/th... https://www.dell.com/en-us/shop/cty/pdp/spd/precision-16-7680-laptop https://www.dell.com/en-us/shop/cty/pdp/spd/precision-16-768... Really, this is what I mean by wondering whether anyone even cares anymore. When you work long enough in the tech industry, you end up sitting next to people that just do not care about anything they interact with. People that use a paid GUI to write git commits, people that pay Pakastani developers to do their work for them, people that brag about computers and smartphones like jewelry. Do you actually research the domain that you claim to care about, or do you just own things and then find reasons to get defensive when people attack your insecurities online? If you have not used a non-Apple laptop since Apple Silicon released, do not pretend you're in a position of authority to make comparisons. I had an M1 Pro for work, and while it was nice for day-to-day browsing I would have never bought one for myself. It's GPU is truly one of the least appealing parts, both from a performance perspective and considering what little it actually supports.