10 ms·

If I were Apple I'd be thinking about the following issues with that strategy: 1. That RAM isn't empty, it's being used by apps and the OS. Fill up 64GB of RAM

If I were Apple I'd be thinking about the following issues with that strategy: 1. That RAM isn't empty, it's being used by apps and the OS. Fill up 64GB of RAM with an LLM and there's nothing left for anything else. 2. 64GB probably isn't enough for competitive LLMs anyway. 3. Inferencing is extremely energy intensive, but the MacBook / Apple Silicon brand is partly about long battery life. 4. Weights are expensive to produce and valuable IP, but hard to protect on the client unless you do a lot of work with encrypted memory. 5. Even if a high end MacBook can do local inferencing, the iPhone won't and it's the iPhone that matters. 6. You might want to fine tune models based on your personal data and history, but training is different to inference and best done in the cloud overnight (probably?). 7. Apple already has all that stuff worked out for Siri, which is a cloud service, not a local service, even though it'd be easier to run locally than an LLM. And lots more issues with doing it all locally, fun though that is to play with for developers. I hope I'm wrong, it'd be cool to have LLMs be fully local, but it's hard to see situations where the local approach beats out the cloud approach. One possibility is simply cost: if your device does it, you pay for the hardware, if a cloud does it, you have to pay for that hardware again via subscription.

LargoLasskhyfv 4y ago
1. Quadruple it. 2. see above Should be cheap, or why else are Samsung, Micron and Kioxia whining about losses? Maybe go for something like Optane memory while doing so.
- mike_hearn 4y ago
  Optane is sadly no longer being manufactured.
  - LargoLasskhyfv 4y ago
    I know. That's why I wrote something like ;-)
anentropic 4y ago
But right now what incentive have I to buy a new laptop? I got this 16GB M1 MBA two years ago and it's literally everything I need, always feels fast, silent etc 1. the idea would be that now there is a reason to buy loads more RAM, whereas currently the market for 64GB is pretty niche 2. 64GB is a big laptop today, in a few years time that will be small. And LLaMA 65B int4 quantized should fit comfortably 4. LLMs will be a commodity. There will be a free one 6. LLMs seem to avoid the need for finetuning by virtue of their size - what we see now with the largest models is you just do prompt engineering. Making use of personal data is a case of Langchain + vectorstores (or however the future of that approach pans out)
- mike_hearn 4y ago
  1. You're working backwards from a desire to buy more RAM to try and find uses for it. You don't actually need more RAM to use LLMs, ChatGPT requires no local memory, is instant and is available for free today. 2. Why would anybody be satisfied with a 64GB model when GPT-4 or 5 or 6 might even be using 1TB of RAM? 3. That may not be the case. With every day that passes, it becomes more and more clear that large LLMs are not that easy to build. Even Google has failed to make something competitive with OpenAI. It's possible that OpenAI is in fact the new Google, that they have been able to establish permanent competitive advantage, and there will no more be free commodity LLMs than there are free commodity search engines. Don't get me wrong, I would love there to be high quality local LLMs. I have at least two use cases where you can't do them or not really well with the OpenAI API and being able to run LLama locally would fix that problem. But I just don't see that being a common case and at any rate I would need server hardware to do it properly, not Mac laptop.
  - anentropic 4y ago
    1. You're working backwards from a desire to buy more RAM to try and find uses for it. I'm really not I had no desire at all until a couple of weeks ago. Even now not so much since it wouldn't be very useful to me But the current LLM business model where there are a small number of API providers, and anything built using this new tech is forced into a subscription model... I don't see it sustainable, and I think the buzz around llama.cpp is a taste of that I'm saying imagine a future where it is painless to run a ChatGPT-class LLM on your laptop (sounded crazy a year ago, to me now looks inevitable within few years), then have a look at the kind of things that can be done today with Langchain... then extrapolate
bootsmann 4y ago
> Even if a high end MacBook can do local inferencing, the iPhone won't and it's the iPhone that matters Doesn't the iPhone use the local processor for stuff like the automatic image segmentation they currently do? (Hold on any person in a recent photo you have take and iOS will segment it)
- mike_hearn 4y ago
  Yes but I'm not making a general argument about all AI, just LLMs. The L stands for Large after all. Smartphones are small.
horsawlarway 4y ago
> but it's hard to see situations where the local approach beats out the cloud approach. I think the most glaring situation where this is true is simply one of trust and privacy. Cloud solutions involve trusting 3rd parties with data. Sometimes that fine, sometimes it's really not. Personally - LLMs start to feel more like they're sitting in the confidant/peer space in many ways. I behave differently when I know I'm hitting a remote resource for LLMs in the same way that I behave differently when I know I'm on camera in person: Less genuinely. And beyond merely trusting that a company won't abuse or leak my data, there are other trust issues as well. If I use an LLM as a digital assistant - I need to know that it's looking out for me (or at least acting neutrally) and not being influenced by a 3rd party to give me responses that are weighted to benefit that 3rd party. I don't think it'll be too long before we see someone try to create an LLM that has advertising baked into it, and we have very little insight into how weights are generated and used. If I'm hitting a remote resource - the model I'm actually running can change out from underneath me at any time, jarring at best and utterly unacceptable at worst. From my end - I'd rather pay and run it locally, even if it's slower or more expensive.
- sebzim4500 4y ago
  Almost everyone is willing to trust 3rd parties with data, including enterprise and government customers. I find it hard to believe that there are enough people willing to pay a large premium to run these locally to make it worth the R&D cost.
  - horsawlarway 4y ago
    Having done a lot of Bank/Gov related work... I can tell you this > Almost everyone is willing to trust 3rd parties with data, including enterprise and government customers. Is absolutely not true. In it's most basic sense - sure... some data is trusted to some 3rd parties. Usually it's not the data that would be most useful for these models to work with. We're already getting tons of "don't put our code into chatGPT/Copilot" warnings across tech companies - I can't imagine not getting fired if I throw private financial docs for my company in there, or ask it for summaries of our high level product strategy documents.
sebzim4500 4y ago
>One possibility is simply cost: if your device does it, you pay for the hardware, if a cloud does it, you have to pay for that hardware again via subscription. Yeah but in the cloud that cost is ammortized among everyone else using the service. If you as a consumer buy a gpu in order to run LLMs for personal use, then the vast majority of the time it will just be sitting there depreciating.
- littlestymaar 4y ago
  But then again, every apple silicon user has an unused neural engine sitting around in the SoC an taking a significant amount of die space, yet people don't seem to worry too much about its depreciation.
fshbbdssbbgdd 4y ago
> 7. Apple already has all that stuff worked out for Siri, which is a cloud service, not a local service, even though it'd be easier to run locally than an LLM. iOS actually does already have an offline speech-to-text api. Some part of Siri that translates the text into intents/actions is remote. Since iOS 15, Siri will also process a limited subset of commands while offline.
anentropic 4y ago
4. Weights are expensive to produce and valuable IP, but hard to protect on the client unless you do a lot of work with encrypted memory. No, it'll be a commodity Apple wouldn't care if the weights can be extracted if you have to have a Macbook to get the sweet, futuristic, LLM-enhanced OS experience
matthewdgreen 4y ago
How much does 64GB of RAM cost, anyway? Retail it's like $200, and I'm sure it's cheaper in terms of Apple cost. Yet we treat it as an absurd luxury because Apple makes you buy the top-end 16" Macbook and pay an extra $800 beyond that. Maybe in the future they'll treat RAM as a requirement and not a luxury good.
- anentropic 4y ago
  and we know that more will be cheaper in future
  - rnk 4y ago
    With the integrated ram and cpu and gpu on apple silicon, however it's done it yields perf results. I do think that probably has higher cost than separately produced ram. And even separate from that, because they have that unified memory model unlike every other consumer device they can charge for it. So 64, 96 or 128 gb?
    - rasz 4y ago
      Its not done for perf results, Xbox doesnt have ram on package and somehow does 560 GB/s
      - rnk 4y ago
        The perf results I was referring to was the ability to run an llm locally (like llama.cpp) that uses a giant amount of ram in the gpu, like 40gig. Without this uniform memory model, you end up paging endlessly, so it's actually much faster for this application in this scenario. Unlike on a pc with a graphics card, you can use your entire ram for gpu. This isn't possible on the xbox because it doesn't have uniform memory as far as I know. So having incredible throughput still won't match not having to page. Edit - I found an example from h.n. user anentropic, pointing at https://github.com/remixer-dec/llama-mps https://github.com/remixer-dec/llama-mps . "The goal of this fork is to use GPU acceleration on Apple M1/M2 devices.... After the model is loaded, inference for max_gen_len=20 takes about 3 seconds on a 24-core M1 Max vs 12+ minutes on a CPU (running on a single core). "
highwaylights 4y ago
I think it’s quite likely that the RAM onboard these devices expands pretty massively, pretty quickly as a direct result of LLMs. Google had already done some very convincing demos in the last few years well before ChatGPT and GPT-4 captured the popular imagination. Microsoft’s OpenAI deal I would assume will lead to a “Cortana 2.0” (obviously rebranded, probably “Bing for Windows”, “Windows Copilot” or something similar). Google Assistant has been far ahead of Siri for many years longer than that, and they have extensive experience with LLMs. Apple surely realises the position their platforms are in and the risk of being left behind. I’m also not sure the barrier on iPhone is as great as you suggest - it’s obviously constrained in terms of what it can support now but if the RAM on the device doubles a few times over the next few years I can see this being less of an issue. Multiple models (like the Alpaca sets) could be used for devices with different RAM/performance profiles and this could be sold as another metric to upgrade (i.e. iPhone 16 runs Siri-2.0-7b while iPhone 17 runs Siri-2.0-30b - “More than 3x smarter than iPhone 16. The smartest iPhone we’ve ever made.” etc).
gregbander 4y ago
Chips have a 5-7 year lead time. Apple has been shipping neural chips for years while everyone else is still designing their v1. Apple is ahead of the game for a change getting their chips in line as the software exits alpha and goes mainstream.
- rnk 4y ago
  But they haven't exposed them to use. They are missing a tremendous opportunity. They have that unique unified memory model on the m1/m2 arms so they have something no other consumer devices have. If they exposed their neural chips they'd solidify their lead. They could sell a lot more hardware.
  - gleenn 4y ago
    They are though. Apple released a library to use Apple Silicon for training via PyTorch recently, and has libraries to leverage the NE in CoreML.
    - anentropic 4y ago
      > Apple Silicon for training via PyTorch recently This is just allowing PyTorch to make use of the Apple GPU, assuming the models you want to train aren't written with hard-coded CUDA calls (I've seen many that are like that, since for a long time that was the only game in town) PyTorch can't use the Neural Engine at all currently AFAIK Neural Engine is only usable for inference, and only via CoreML (coremltools in Python)
    - rnk 4y ago
      Thank you! I wasn't aware of that. Let me research that. May 2022 announcement. Is this suitable for the the apps like llama.cpp since it's a Python library? It appears to be a library but they didn't document how to use the underlying hardware - but I welcome more info.
gleenn 4y ago
iPhones have similar Neural Engine capabilities, obviously far more limited but still quite powerful. You can run some pretty cool DNNs for image generation using e.g. Draw Things app quite quickly: https://apps.apple.com/us/app/draw-things-ai-generation/id6444050820 https://apps.apple.com/us/app/draw-things-ai-generation/id64...
rnk 4y ago
I've been looking into buy a mac for llm experimentation - 64, 96 or 128gb of ram? I'm trying to decide if 64gb is enough, or should I go to 96gb or even 128gb. But it's really expensive - even for an overpaid software engineer. Then there's the 1 or 2 tb storage question. Apple list price is another $400 for that second tb of storage. For 64gb of ram, you can get an m2 pro, or get 96gb which requires the upgraded cpu on the pro. The studio does 64gb or 128gb. But the 128 requires you to spend 5k. I can't decide between 64 or 96 on m2 pro, and 128 on the studio. Probably go for 96gb. Also what's the impact of the extra gpu cores on the various options? And there are still some "m1" 64gb pros & studios out there. What's the perf difference for m1 vs m2? This area needs serious perf benchmarking. If anyone wants to work with me, maybe I would try my hand. But I'm not spending 15k just to get 3 pieces of hardware. List prices: 64gb/2tb m2 12cpu/30gpu 14" pro $3900 96gb/2tb m2 max 12/38 14" pro $4500 128gb/2tb m2 max 28/48 studio $5200
- bugglebeetle 4y ago
  I’m pretty sure you can get a purpose-built pc tower in that range. Why would you favor a Mac over that? A lot of this stuff only has limited support for MacOS.
  - selectodude 4y ago
    The unified GPU/CPU memory structure on ARM Macs is very, very helpful for running these LLMs locally.
    - EntrePrescott 4y ago
      Is there a big difference in principle between that and the "shared video memory" that has long existed on cheap x86 machines? …or is it just that the latter had a way too weak iGPU and not enough RAM for AI purposes, whereas the bigger ARM MACs have more GPU power and enough RAM (more than most affordable discrete graphic cards) so that they are usable for some AI models?
  - evilduck 4y ago
    You can't get that much VRAM on a PC for a comparible price.
nerpderp82 4y ago
Running models locally is the future for most inferencing cycles. There is a lot of accuracy that could be improved in your numbered list trying to dissuade people. > 64GB probably isn't enough for competitive LLMs anyway I am trying to charitable, but this is pretty not true. And the hedging in your statement only telegraphs your experience.

LargoLasskhyfv 4y ago

1. Quadruple it. 2. see above Should be cheap, or why else are Samsung, Micron and Kioxia whining about losses? Maybe go for something like Optane memory while doing so.

mike_hearn 4y ago
Optane is sadly no longer being manufactured.
- LargoLasskhyfv 4y ago
  I know. That's why I wrote something like ;-)

anentropic 4y ago

But right now what incentive have I to buy a new laptop? I got this 16GB M1 MBA two years ago and it's literally everything I need, always feels fast, silent etc 1. the idea would be that now there is a reason to buy loads more RAM, whereas currently the market for 64GB is pretty niche 2. 64GB is a big laptop today, in a few years time that will be small. And LLaMA 65B int4 quantized should fit comfortably 4. LLMs will be a commodity. There will be a free one 6. LLMs seem to avoid the need for finetuning by virtue of their size - what we see now with the largest models is you just do prompt engineering. Making use of personal data is a case of Langchain + vectorstores (or however the future of that approach pans out)

mike_hearn 4y ago
1. You're working backwards from a desire to buy more RAM to try and find uses for it. You don't actually need more RAM to use LLMs, ChatGPT requires no local memory, is instant and is available for free today. 2. Why would anybody be satisfied with a 64GB model when GPT-4 or 5 or 6 might even be using 1TB of RAM? 3. That may not be the case. With every day that passes, it becomes more and more clear that large LLMs are not that easy to build. Even Google has failed to make something competitive with OpenAI. It's possible that OpenAI is in fact the new Google, that they have been able to establish permanent competitive advantage, and there will no more be free commodity LLMs than there are free commodity search engines. Don't get me wrong, I would love there to be high quality local LLMs. I have at least two use cases where you can't do them or not really well with the OpenAI API and being able to run LLama locally would fix that problem. But I just don't see that being a common case and at any rate I would need server hardware to do it properly, not Mac laptop.
- anentropic 4y ago
  1. You're working backwards from a desire to buy more RAM to try and find uses for it. I'm really not I had no desire at all until a couple of weeks ago. Even now not so much since it wouldn't be very useful to me But the current LLM business model where there are a small number of API providers, and anything built using this new tech is forced into a subscription model... I don't see it sustainable, and I think the buzz around llama.cpp is a taste of that I'm saying imagine a future where it is painless to run a ChatGPT-class LLM on your laptop (sounded crazy a year ago, to me now looks inevitable within few years), then have a look at the kind of things that can be done today with Langchain... then extrapolate

bootsmann 4y ago

> Even if a high end MacBook can do local inferencing, the iPhone won't and it's the iPhone that matters Doesn't the iPhone use the local processor for stuff like the automatic image segmentation they currently do? (Hold on any person in a recent photo you have take and iOS will segment it)

mike_hearn 4y ago
Yes but I'm not making a general argument about all AI, just LLMs. The L stands for Large after all. Smartphones are small.

horsawlarway 4y ago

> but it's hard to see situations where the local approach beats out the cloud approach. I think the most glaring situation where this is true is simply one of trust and privacy. Cloud solutions involve trusting 3rd parties with data. Sometimes that fine, sometimes it's really not. Personally - LLMs start to feel more like they're sitting in the confidant/peer space in many ways. I behave differently when I know I'm hitting a remote resource for LLMs in the same way that I behave differently when I know I'm on camera in person: Less genuinely. And beyond merely trusting that a company won't abuse or leak my data, there are other trust issues as well. If I use an LLM as a digital assistant - I need to know that it's looking out for me (or at least acting neutrally) and not being influenced by a 3rd party to give me responses that are weighted to benefit that 3rd party. I don't think it'll be too long before we see someone try to create an LLM that has advertising baked into it, and we have very little insight into how weights are generated and used. If I'm hitting a remote resource - the model I'm actually running can change out from underneath me at any time, jarring at best and utterly unacceptable at worst. From my end - I'd rather pay and run it locally, even if it's slower or more expensive.

sebzim4500 4y ago
Almost everyone is willing to trust 3rd parties with data, including enterprise and government customers. I find it hard to believe that there are enough people willing to pay a large premium to run these locally to make it worth the R&D cost.
- horsawlarway 4y ago
  Having done a lot of Bank/Gov related work... I can tell you this > Almost everyone is willing to trust 3rd parties with data, including enterprise and government customers. Is absolutely not true. In it's most basic sense - sure... some data is trusted to some 3rd parties. Usually it's not the data that would be most useful for these models to work with. We're already getting tons of "don't put our code into chatGPT/Copilot" warnings across tech companies - I can't imagine not getting fired if I throw private financial docs for my company in there, or ask it for summaries of our high level product strategy documents.

sebzim4500 4y ago

>One possibility is simply cost: if your device does it, you pay for the hardware, if a cloud does it, you have to pay for that hardware again via subscription. Yeah but in the cloud that cost is ammortized among everyone else using the service. If you as a consumer buy a gpu in order to run LLMs for personal use, then the vast majority of the time it will just be sitting there depreciating.

littlestymaar 4y ago
But then again, every apple silicon user has an unused neural engine sitting around in the SoC an taking a significant amount of die space, yet people don't seem to worry too much about its depreciation.

fshbbdssbbgdd 4y ago

> 7. Apple already has all that stuff worked out for Siri, which is a cloud service, not a local service, even though it'd be easier to run locally than an LLM. iOS actually does already have an offline speech-to-text api. Some part of Siri that translates the text into intents/actions is remote. Since iOS 15, Siri will also process a limited subset of commands while offline.

anentropic 4y ago

4. Weights are expensive to produce and valuable IP, but hard to protect on the client unless you do a lot of work with encrypted memory. No, it'll be a commodity Apple wouldn't care if the weights can be extracted if you have to have a Macbook to get the sweet, futuristic, LLM-enhanced OS experience

matthewdgreen 4y ago

How much does 64GB of RAM cost, anyway? Retail it's like $200, and I'm sure it's cheaper in terms of Apple cost. Yet we treat it as an absurd luxury because Apple makes you buy the top-end 16" Macbook and pay an extra $800 beyond that. Maybe in the future they'll treat RAM as a requirement and not a luxury good.

anentropic 4y ago
and we know that more will be cheaper in future
- rnk 4y ago
  With the integrated ram and cpu and gpu on apple silicon, however it's done it yields perf results. I do think that probably has higher cost than separately produced ram. And even separate from that, because they have that unified memory model unlike every other consumer device they can charge for it. So 64, 96 or 128 gb?
  - rasz 4y ago
    Its not done for perf results, Xbox doesnt have ram on package and somehow does 560 GB/s
    - rnk 4y ago
      The perf results I was referring to was the ability to run an llm locally (like llama.cpp) that uses a giant amount of ram in the gpu, like 40gig. Without this uniform memory model, you end up paging endlessly, so it's actually much faster for this application in this scenario. Unlike on a pc with a graphics card, you can use your entire ram for gpu. This isn't possible on the xbox because it doesn't have uniform memory as far as I know. So having incredible throughput still won't match not having to page. Edit - I found an example from h.n. user anentropic, pointing at https://github.com/remixer-dec/llama-mps https://github.com/remixer-dec/llama-mps . "The goal of this fork is to use GPU acceleration on Apple M1/M2 devices.... After the model is loaded, inference for max_gen_len=20 takes about 3 seconds on a 24-core M1 Max vs 12+ minutes on a CPU (running on a single core). "

highwaylights 4y ago

I think it’s quite likely that the RAM onboard these devices expands pretty massively, pretty quickly as a direct result of LLMs. Google had already done some very convincing demos in the last few years well before ChatGPT and GPT-4 captured the popular imagination. Microsoft’s OpenAI deal I would assume will lead to a “Cortana 2.0” (obviously rebranded, probably “Bing for Windows”, “Windows Copilot” or something similar). Google Assistant has been far ahead of Siri for many years longer than that, and they have extensive experience with LLMs. Apple surely realises the position their platforms are in and the risk of being left behind. I’m also not sure the barrier on iPhone is as great as you suggest - it’s obviously constrained in terms of what it can support now but if the RAM on the device doubles a few times over the next few years I can see this being less of an issue. Multiple models (like the Alpaca sets) could be used for devices with different RAM/performance profiles and this could be sold as another metric to upgrade (i.e. iPhone 16 runs Siri-2.0-7b while iPhone 17 runs Siri-2.0-30b - “More than 3x smarter than iPhone 16. The smartest iPhone we’ve ever made.” etc).

gregbander 4y ago

Chips have a 5-7 year lead time. Apple has been shipping neural chips for years while everyone else is still designing their v1. Apple is ahead of the game for a change getting their chips in line as the software exits alpha and goes mainstream.

rnk 4y ago
But they haven't exposed them to use. They are missing a tremendous opportunity. They have that unique unified memory model on the m1/m2 arms so they have something no other consumer devices have. If they exposed their neural chips they'd solidify their lead. They could sell a lot more hardware.
- gleenn 4y ago
  They are though. Apple released a library to use Apple Silicon for training via PyTorch recently, and has libraries to leverage the NE in CoreML.
  - anentropic 4y ago
    > Apple Silicon for training via PyTorch recently This is just allowing PyTorch to make use of the Apple GPU, assuming the models you want to train aren't written with hard-coded CUDA calls (I've seen many that are like that, since for a long time that was the only game in town) PyTorch can't use the Neural Engine at all currently AFAIK Neural Engine is only usable for inference, and only via CoreML (coremltools in Python)
  - rnk 4y ago
    Thank you! I wasn't aware of that. Let me research that. May 2022 announcement. Is this suitable for the the apps like llama.cpp since it's a Python library? It appears to be a library but they didn't document how to use the underlying hardware - but I welcome more info.

gleenn 4y ago

iPhones have similar Neural Engine capabilities, obviously far more limited but still quite powerful. You can run some pretty cool DNNs for image generation using e.g. Draw Things app quite quickly: https://apps.apple.com/us/app/draw-things-ai-generation/id6444050820 https://apps.apple.com/us/app/draw-things-ai-generation/id64...

rnk 4y ago

I've been looking into buy a mac for llm experimentation - 64, 96 or 128gb of ram? I'm trying to decide if 64gb is enough, or should I go to 96gb or even 128gb. But it's really expensive - even for an overpaid software engineer. Then there's the 1 or 2 tb storage question. Apple list price is another $400 for that second tb of storage. For 64gb of ram, you can get an m2 pro, or get 96gb which requires the upgraded cpu on the pro. The studio does 64gb or 128gb. But the 128 requires you to spend 5k. I can't decide between 64 or 96 on m2 pro, and 128 on the studio. Probably go for 96gb. Also what's the impact of the extra gpu cores on the various options? And there are still some "m1" 64gb pros & studios out there. What's the perf difference for m1 vs m2? This area needs serious perf benchmarking. If anyone wants to work with me, maybe I would try my hand. But I'm not spending 15k just to get 3 pieces of hardware. List prices: 64gb/2tb m2 12cpu/30gpu 14" pro $3900 96gb/2tb m2 max 12/38 14" pro $4500 128gb/2tb m2 max 28/48 studio $5200

bugglebeetle 4y ago
I’m pretty sure you can get a purpose-built pc tower in that range. Why would you favor a Mac over that? A lot of this stuff only has limited support for MacOS.
- selectodude 4y ago
  The unified GPU/CPU memory structure on ARM Macs is very, very helpful for running these LLMs locally.
  - EntrePrescott 4y ago
    Is there a big difference in principle between that and the "shared video memory" that has long existed on cheap x86 machines? …or is it just that the latter had a way too weak iGPU and not enough RAM for AI purposes, whereas the bigger ARM MACs have more GPU power and enough RAM (more than most affordable discrete graphic cards) so that they are usable for some AI models?
- evilduck 4y ago
  You can't get that much VRAM on a PC for a comparible price.

nerpderp82 4y ago

Running models locally is the future for most inferencing cycles. There is a lot of accuracy that could be improved in your numbered list trying to dissuade people. > 64GB probably isn't enough for competitive LLMs anyway I am trying to charitable, but this is pretty not true. And the hedging in your statement only telegraphs your experience.