5 ms·

Apple's take on GPUs is quite different and very interesting IMO. Shared memory architecture with absolutely massive RAM (and hence VRAM) support, e.g. the new

Apple's take on GPUs is quite different and very interesting IMO. Shared memory architecture with absolutely massive RAM (and hence VRAM) support, e.g. the new Mac Studio having 192GB RAM/VRAM which can run pretty massive models, much more so than is easily consumer-accessible even at the high end with Nvidia 4090s. It's not as fast as Nvidia, but it's not horribly far off in the latest chips. As LLM adoption grows, I wonder whether Apple's approach will start to make more sense for consumer adoption, so that you can run models on your machine without needing to pay large subscription costs for AI-powered apps (since OpenAI et al have fairly high fees). The high cost of using the APIs in my opinion is a drag on certain types of adoption in consumer apps. Llama.cpp actually first started as a way to run LLMs on Macs! At first CPU-only, but then later the first GPU driver backend added was Metal, not anything from Nvidia.

mliker 3y ago
In addition to the cost, OpenAI censors their models, and while that seems protective at first, if you think about it, if the model is the knowledge graph, then censor data in the graph is censoring free speech.
- Tostino 3y ago
  I don't personally like model censorship, but you need to come up with a better argument than...whatever that is. It is entirely unconvincing, and seems like you misunderstand what free speech is.
  - mliker 3y ago
    Freedom of speech is a principle that supports the right of an individual or a community to articulate their opinions and ideas without fear of retaliation, censorship, or legal sanction. Since the models are not suppressed to answer certain questions and indeed have been demonstrated to be biased toward one end of the political spectrum, and if you treat the model’s output as a knowledge graph, as proposed by John Schulman, one of the cofounders of OpenAI, then yes, I would say the suppression of freedom of speech is a valid argument to make. Otherwise why would there be a set of “uncensored” models that exist in the open source world? I would suggest you read about these models and think about the implications. Perhaps that’ll lead to you to reconsider your stance
    - svnt 3y ago
      If we ignore the corporate structure details where your analogy breaks down, in the simplest case, choosing to self-inhibit is not violation of the freedom of speech. OpenAI is voluntarily self-censoring their output. This isn’t an ideological battle for them, it’s a business.
    - smoldesu 3y ago
      > I would suggest you read about these models and think about the implications. I have done that, and come to the conclusion that "that's business, baby!" Do you have any more meaningful objections to this "censorship" of a free thing you agreed to license?
bottled_poe 3y ago
I agree, though it’s still vendor lock-in. I think a lot of devs aren’t aware of how tightly integrated these frameworks are with the hardware. It’s not trivial to separate the two and the companies driving this tech have no motive to do so, while also being amongst a very small number of hardware designers.
anikom15 3y ago
I think criticizing Apple for not supporting Nvidia is a classic missing-the-forest-for-the-trees. Unifying memory is a next logical step in general purpose computing. The flexibility that comes from it has unseen potential.
- ttflee 3y ago
  Thinking what Apple sells is a GPU bundled with a large amount of VRAM and a free ARM CPU makes things easier to accept. NVIDIA never sold a GPU with expandable memory, either.
- jsnell 3y ago
  It's a next step that AMD has been trying to make happen for like 10 years, but not managing to make it commercially successful in anything but consoles, ending up permanently stuck in the lowest low end of the market. In both the PC and datacenter market spaces it has ended up with basically the opposite of product market fit. Nobody actually wants the generic CPU compute tied to the GPU compute. The PC gamers really want the GPU component to be separately upgradable from the CPU. Non-gamer PC users don't care about the GPU performance, just cost. The datacenter folks want to be able to use a single $1k CPU to host $100k worth of GPUs. It's plausible that AI accelerators follow a different path for the consumers. It's harder to see it happening for the datacenter market.
  - anikom15 3y ago
    You’re missing the forest for the trees. Being able to switch from an optimized CPU-centric workload to an optimized GPU-centric workload without any hardware changes sounds useful to me. You could even do unified memory with upgradeable separate CPU and GPU. You won’t get the benefits of having them on the same chip, but there’s nothing intrinsic about the separation requiring separate memory space.
eurekin 3y ago
Is that RAM as fast as A100/H100 VRAM? AFAIR, gpus push ~ 1 TB/s ish
- jlokier 3y ago
  The Apple M2 Ultra memory bandwidth is 800GB/s so it's not a long way off 1TB/s. A100 goes up to 2TB/s in the largest configuration, and H100 claims 3TB/s per GPU node. (These figures keep changing with new variants.) But you can buy several Mac Studios for the price :-) The real use of H100 is for training, as it can pool many GPUs together with a dedicated high bandwidth network. You can't do that with Mac Studios.
  - eurekin 3y ago
    That figures are really impressive. A single node is still useful for finetuning smaller models. Apple could move the needle in this market a bit

mliker 3y ago

In addition to the cost, OpenAI censors their models, and while that seems protective at first, if you think about it, if the model is the knowledge graph, then censor data in the graph is censoring free speech.

Tostino 3y ago
I don't personally like model censorship, but you need to come up with a better argument than...whatever that is. It is entirely unconvincing, and seems like you misunderstand what free speech is.
- mliker 3y ago
  Freedom of speech is a principle that supports the right of an individual or a community to articulate their opinions and ideas without fear of retaliation, censorship, or legal sanction. Since the models are not suppressed to answer certain questions and indeed have been demonstrated to be biased toward one end of the political spectrum, and if you treat the model’s output as a knowledge graph, as proposed by John Schulman, one of the cofounders of OpenAI, then yes, I would say the suppression of freedom of speech is a valid argument to make. Otherwise why would there be a set of “uncensored” models that exist in the open source world? I would suggest you read about these models and think about the implications. Perhaps that’ll lead to you to reconsider your stance
  - svnt 3y ago
    If we ignore the corporate structure details where your analogy breaks down, in the simplest case, choosing to self-inhibit is not violation of the freedom of speech. OpenAI is voluntarily self-censoring their output. This isn’t an ideological battle for them, it’s a business.
  - smoldesu 3y ago
    > I would suggest you read about these models and think about the implications. I have done that, and come to the conclusion that "that's business, baby!" Do you have any more meaningful objections to this "censorship" of a free thing you agreed to license?

bottled_poe 3y ago

I agree, though it’s still vendor lock-in. I think a lot of devs aren’t aware of how tightly integrated these frameworks are with the hardware. It’s not trivial to separate the two and the companies driving this tech have no motive to do so, while also being amongst a very small number of hardware designers.

anikom15 3y ago

I think criticizing Apple for not supporting Nvidia is a classic missing-the-forest-for-the-trees. Unifying memory is a next logical step in general purpose computing. The flexibility that comes from it has unseen potential.

ttflee 3y ago
Thinking what Apple sells is a GPU bundled with a large amount of VRAM and a free ARM CPU makes things easier to accept. NVIDIA never sold a GPU with expandable memory, either.
jsnell 3y ago
It's a next step that AMD has been trying to make happen for like 10 years, but not managing to make it commercially successful in anything but consoles, ending up permanently stuck in the lowest low end of the market. In both the PC and datacenter market spaces it has ended up with basically the opposite of product market fit. Nobody actually wants the generic CPU compute tied to the GPU compute. The PC gamers really want the GPU component to be separately upgradable from the CPU. Non-gamer PC users don't care about the GPU performance, just cost. The datacenter folks want to be able to use a single $1k CPU to host $100k worth of GPUs. It's plausible that AI accelerators follow a different path for the consumers. It's harder to see it happening for the datacenter market.
- anikom15 3y ago
  You’re missing the forest for the trees. Being able to switch from an optimized CPU-centric workload to an optimized GPU-centric workload without any hardware changes sounds useful to me. You could even do unified memory with upgradeable separate CPU and GPU. You won’t get the benefits of having them on the same chip, but there’s nothing intrinsic about the separation requiring separate memory space.

eurekin 3y ago

Is that RAM as fast as A100/H100 VRAM? AFAIR, gpus push ~ 1 TB/s ish

jlokier 3y ago
The Apple M2 Ultra memory bandwidth is 800GB/s so it's not a long way off 1TB/s. A100 goes up to 2TB/s in the largest configuration, and H100 claims 3TB/s per GPU node. (These figures keep changing with new variants.) But you can buy several Mac Studios for the price :-) The real use of H100 is for training, as it can pool many GPUs together with a dedicated high bandwidth network. You can't do that with Mac Studios.
- eurekin 3y ago
  That figures are really impressive. A single node is still useful for finetuning smaller models. Apple could move the needle in this market a bit