4 ms·

Woah, this is exciting. I'm traveling but I have a 5090 lying around at home. I'm eager to give it a go. Docs are here: https://docs.tinygrad.org/tinygpu/ https

by arjie 7mo ago

Woah, this is exciting. I'm traveling but I have a 5090 lying around at home. I'm eager to give it a go. Docs are here: https://docs.tinygrad.org/tinygpu/ https://docs.tinygrad.org/tinygpu/ I hope it'll work on an M4 Mac Mini. Does anyone know what hardware to get? You'll need a full ATX PSU to supply power, right? And then tinygrad can do LLM inference on it?

manmal 7mo ago
Maybe I’m lacking imagination. But how will a GPU with small-ish but fast VRAM and great compute, augment a Mac with large but slow VRAM and weak compute? The interconnect isn’t powerful enough to change layers on the GPU rapidly, I guess?
- arjie 7mo ago
  My Mini is actually the smallest model so it actually has "small but slow VRAM" (haha!) so the reason I want the GPU for are the smaller Gemmas or Qwens. Realistically, I'll probably run on an RTX 6000 Pro but this might be fun for home.
- zozbot234 7mo ago
  > But how will a GPU with small-ish but fast VRAM and great compute, augment a Mac with large but slow VRAM and weak compute? It would work just like a discrete GPU when doing CPU+GPU inference: you'd run a few shared layers on the discrete GPU and place the rest in unified memory. You'd want to minimize CPU/GPU transfers even more than usual, since a Thunderbolt connection only gives you equivalent throughput to PCIe 4.0 x4.
  - manmal 7mo ago
    But isn’t the Mac Mini the weak link in that scenario?
    - zozbot234 7mo ago
      It has way more unified memory than your typical dGPU.
      - manmal 7mo ago
        Yes obviously. That VRAM is also slower and has weak compute attached. Loading to the external GPU will slow things down too much.
- GeekyBear 7mo ago
  We've seen many recent projects to stream models direct from SSD to a discrete GPU's limited VRAM on PCs. How big a bottleneck is Thunderbolt 5 compared to an SSD? Is the 120 Gbps mode only available when linked to a monitor?
  - manmal 7mo ago
    That’s what, 14GB/s? The GPU‘s VRAM can do 100x that.
    - GeekyBear 7mo ago
      A discrete consumer GPU card doesn't have enough fast RAM to run a very large model that hasn't been quanitized to hell. That's why all the projects streaming models into the GPU from an SSD popped up recently.
      - manmal 7mo ago
        Yes. There’s just no way to get above 1t/s that way with a large model.
lowbloodsugar 7mo ago
“Lying around”. I’ve got an unopened 5090 in a box that I know will suffer the same fate, so I’m sending it back. So privileged to have the money to impulse buy a 5090 and yet no time to actually do anything with it.
- c-c-c-c-c 7mo ago
  You should see his ferrari.
999900000999 7mo ago
You can buy a cheap GPU enclosure for about 100$ off ali express. Takes a standard PSU. However, Mac Minis don't have occulink. So you might be a bit limited by whatever USB C can do. Now if Intel can get there Arc drivers in order we'll see some real budget fun. https://www.newegg.com/intel-arc-pro-b70-32gb-graphics-card/p/N82E16814883008?item=N82E16814883008 https://www.newegg.com/intel-arc-pro-b70-32gb-graphics-card/... 32 GB of VRAM for 1000$. Plus a 500$ Mac Mini.
- Fnoord 7mo ago
  Those $100 ones don't come with a cage. If you do want a cage, you'll end up with $180 in total, with zero warranty. Article mentions: "Apple finally approved our driver for both AMD and NVIDIA" Does not mention Intel (GPUs). Select AMD GPUs work on macOS, but... Macs (both Intel and ARM) support TB, but eGPU only work on Intel Macs, and basically only with AMD. Good news is for medium end gaming choices are solid, and CUDA works on AMD these days.
  - 999900000999 7mo ago
    Fortune favors the bold my friend. I own one of these, the cage is just a piece of plastic. Anyway, I don't think 80$ is that big of a difference here. I can't really afford a 4k Nvidia GPU. Intel is my only hope.
    - Fnoord 7mo ago
      Almost twice the price and simply more accurate info regarding price and features. Brand is TH3P4G3. Egpu.io has decent eGPU comparisons. I wouldn't want all that dust in my GPU fans, prefer that near my case fans. I also don't like it given I got cats and want to store/box hw. I do use the eGPU in the fuse box. If I had a larger house, I'd use a server rack. I was recently in the market for an eGPU but for a different niche (not eGPU/eNPU/eTPU but getting a HBA via TB to connect a LTO-6 drive via SAS). I went for a Sonnet instead, very low profile and small. I also bought an Asus one. Slightly bigger, came with more fans but TB4 instead of TB3 on the Sonnet. The cages are aluminium. Those eGPU were second hand (also without warranty but quicker S&H than Chinese New Year) but came with PSU. As you also gotta buy a PSU for it which came with the eGPUs I mentioned. For me no biggie, as I got a decent PSU lying around.
rldjbpin 7mo ago
based on your card, it should be using a decent bit of the 600w or so passed through the new 16-pin connector. goes without saying, a proper PSU (doesn't have to be ATX, but at least 750W to be on the safer side) is a must. for thunderbolt enclosures, consider going through the list - https://egpu.io/best-egpu-buyers-guide/#tb3-enclosures https://egpu.io/best-egpu-buyers-guide/#tb3-enclosures zero idea about mac support so YMMV.

by arjie 7mo ago

manmal 7mo ago

Maybe I’m lacking imagination. But how will a GPU with small-ish but fast VRAM and great compute, augment a Mac with large but slow VRAM and weak compute? The interconnect isn’t powerful enough to change layers on the GPU rapidly, I guess?

arjie 7mo ago
My Mini is actually the smallest model so it actually has "small but slow VRAM" (haha!) so the reason I want the GPU for are the smaller Gemmas or Qwens. Realistically, I'll probably run on an RTX 6000 Pro but this might be fun for home.
zozbot234 7mo ago
> But how will a GPU with small-ish but fast VRAM and great compute, augment a Mac with large but slow VRAM and weak compute? It would work just like a discrete GPU when doing CPU+GPU inference: you'd run a few shared layers on the discrete GPU and place the rest in unified memory. You'd want to minimize CPU/GPU transfers even more than usual, since a Thunderbolt connection only gives you equivalent throughput to PCIe 4.0 x4.
- manmal 7mo ago
  But isn’t the Mac Mini the weak link in that scenario?
  - zozbot234 7mo ago
    It has way more unified memory than your typical dGPU.
    - manmal 7mo ago
      Yes obviously. That VRAM is also slower and has weak compute attached. Loading to the external GPU will slow things down too much.
GeekyBear 7mo ago
We've seen many recent projects to stream models direct from SSD to a discrete GPU's limited VRAM on PCs. How big a bottleneck is Thunderbolt 5 compared to an SSD? Is the 120 Gbps mode only available when linked to a monitor?
- manmal 7mo ago
  That’s what, 14GB/s? The GPU‘s VRAM can do 100x that.
  - GeekyBear 7mo ago
    A discrete consumer GPU card doesn't have enough fast RAM to run a very large model that hasn't been quanitized to hell. That's why all the projects streaming models into the GPU from an SSD popped up recently.
    - manmal 7mo ago
      Yes. There’s just no way to get above 1t/s that way with a large model.

lowbloodsugar 7mo ago

“Lying around”. I’ve got an unopened 5090 in a box that I know will suffer the same fate, so I’m sending it back. So privileged to have the money to impulse buy a 5090 and yet no time to actually do anything with it.

c-c-c-c-c 7mo ago
You should see his ferrari.

999900000999 7mo ago

You can buy a cheap GPU enclosure for about 100$ off ali express. Takes a standard PSU. However, Mac Minis don't have occulink. So you might be a bit limited by whatever USB C can do. Now if Intel can get there Arc drivers in order we'll see some real budget fun. https://www.newegg.com/intel-arc-pro-b70-32gb-graphics-card/p/N82E16814883008?item=N82E16814883008 https://www.newegg.com/intel-arc-pro-b70-32gb-graphics-card/... 32 GB of VRAM for 1000$. Plus a 500$ Mac Mini.

Fnoord 7mo ago
Those $100 ones don't come with a cage. If you do want a cage, you'll end up with $180 in total, with zero warranty. Article mentions: "Apple finally approved our driver for both AMD and NVIDIA" Does not mention Intel (GPUs). Select AMD GPUs work on macOS, but... Macs (both Intel and ARM) support TB, but eGPU only work on Intel Macs, and basically only with AMD. Good news is for medium end gaming choices are solid, and CUDA works on AMD these days.
- 999900000999 7mo ago
  Fortune favors the bold my friend. I own one of these, the cage is just a piece of plastic. Anyway, I don't think 80$ is that big of a difference here. I can't really afford a 4k Nvidia GPU. Intel is my only hope.
  - Fnoord 7mo ago
    Almost twice the price and simply more accurate info regarding price and features. Brand is TH3P4G3. Egpu.io has decent eGPU comparisons. I wouldn't want all that dust in my GPU fans, prefer that near my case fans. I also don't like it given I got cats and want to store/box hw. I do use the eGPU in the fuse box. If I had a larger house, I'd use a server rack. I was recently in the market for an eGPU but for a different niche (not eGPU/eNPU/eTPU but getting a HBA via TB to connect a LTO-6 drive via SAS). I went for a Sonnet instead, very low profile and small. I also bought an Asus one. Slightly bigger, came with more fans but TB4 instead of TB3 on the Sonnet. The cages are aluminium. Those eGPU were second hand (also without warranty but quicker S&H than Chinese New Year) but came with PSU. As you also gotta buy a PSU for it which came with the eGPUs I mentioned. For me no biggie, as I got a decent PSU lying around.

rldjbpin 7mo ago

based on your card, it should be using a decent bit of the 600w or so passed through the new 16-pin connector. goes without saying, a proper PSU (doesn't have to be ATX, but at least 750W to be on the safer side) is a must. for thunderbolt enclosures, consider going through the list - https://egpu.io/best-egpu-buyers-guide/#tb3-enclosures https://egpu.io/best-egpu-buyers-guide/#tb3-enclosures zero idea about mac support so YMMV.