9 ms·

Tried Nvidia’s GTX 1080 – still no external GPU on a Pi

127 points by geerlingguy 5y ago

ComputerGuru 5y ago
TLDR: it didn’t work, no real clue why, more updates to come.
marcan_42 5y ago
The Raspberry Pi does not support PCIe I/O BAR space. That's not a hard requirement to run a modern GPU as far as I know, but drivers do kind of assume it's available. In particular, VBIOS stuff often needs it. The message from the Radeon driver is not a hard error, but presumably other things break as a result. The Broadcom SoCs used on Pis still suck, and although this can probably be made to work, someone familiar with this codebase needs to care enough to fix the drivers to make it happen. It's probably quite doable for the open source drivers, but I wouldn't expect Nvidia's blob to work any time soon unless they take the time to fix it. Not all PCIe root ports are equal. Some are, shall we say, less than amenable to devices more complicated than a USB controller. Looking through the discussion here, there is some rather concerning stuff about DMA mappings (e.g. the thing with 64bit accesses not being supported). It's possible the Pi4 PCIe implementation is actually too broken to support these GPUs, though I haven't read through the whole thread in detail. It certainly looks pretty messy. https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/4 https://github.com/geerlingguy/raspberry-pi-pcie-devices/iss...
- geerlingguy 5y ago
  Thanks for taking a glance. There's a groundswell of people commenting "get marcan to take a look". And I would ask, but you're already doing some really awesome work over on the M1 side and I've learned enough to fill up two of my brains already from that. At this point I kinda have to wave the white flag on the driver issues because I'm not deep enough on the kernel / PCI spec side to be productive figuring out the holes on the Pi's 'messy' bus :)
  - monocasa 5y ago
    I spent a night looking through the radeon driver and register documentation, and it seems to fall back on sane defaults without being able to access IO space, it just prints a nasty gram. All of the expected IO space is mirrored in the MMIO space and the VBIOS code will just use that as a fallback pathway just fine. One sort of cheap way to get a sense of what's actually going on (at least way cheaper than a PCIe protocol analyzer) might be to put your hands on one of the USB3382 cards (the poor man's PCIe protocol analyzer for those who aren't that comfortable with FPGAs). Those are really neat soft PCIe devices that bridge the PCIe transactions over USB3 to another host computer and can do things like emulate other devices or send arbitrary TLPs. There's some hacker con talks about using them to attack systems by using their DMA TLPs as a read/write primitive. Using that to either bridge to a card on the host computer and tap the PCIe bus, or send your own TLPs to verify that stuff like 64 bit transactions are actually a problem and not a red herring. You can use a FPGA for the same thing pretty easily, but a USB3382 might be a lot closer to the skill set that you feel comfortable with.
mrspeaker 5y ago
"10% chance of releasing the magic smoke" made me laugh out loud. Off topic-ish, but maybe someone here would know... I have a 2013 macbook pro. It's still great for coding but recently I got into Blender, and now I understand why you might want an external GPU. Is that a possibility for such an old laptop? I couldn't find any definitive information on how to go about buying one, and setting it up.
- Wowfunhappy 5y ago
  I looked into getting an eGPU for a Macbook in 2013. Thunderbolt 2 has the bandwidth to make it possible, but it's not ideal and will cost performance. More critically, at least in 2013, the docks you needed cost thousands of dollars—and made more sense to just get a separate desktop PC. I don't know if the situation has changed since then—there are more eGPUs out there but it's all for Thunderbolt 3. Thunderbolt 2 is more niche and thus may be more expensive.
  - deleted 5y ago
    [deleted]
  - teruakohatu 5y ago
    The other issue is drivers. Nvidia is definitely out since the drivers have not been updated in many years.
  - 1_player 5y ago
    Thunderbolt 2, if supported natively by the hardware and using an external monitor, might actually be enough in practice. I used to game on rMBP 2015 with a GTX 1080 in a TB2-connected eGPU and it didn't even saturate the link, so when I upgraded to an iMac with TB3 the performance was pretty much identical. The rule of thumb is: use an external monitor and the highest resolution possible. Sending control commands one way consumes much less bandwidth if you don't need to send whole images back down the link, and the high resolution and quality is to make the GPU the bottleneck, not the TB2 link.
  - salamandersauce 5y ago
    It's possible to use at least some TB3 eGPUs with TB2 and adapters, it's not officially supported by Apple on macOS though. They are still insanely expensive in my opinion for what's basically a case, small PSU and PCI-E to TB3. The cheapest ones that can take most cards are around $300 and usually have some downside. I considered it for my Mac Mini but in the end it just made more sense to sell the Mini and buy a desktop PC with space for a real GPU, more storage, faster processor, etc. I mean my case, power supply and motherboard was less than the price of a Razer Core X...
onli 5y ago
> But probably not AAA gaming, since gaming on Linux is a tough proposition even in the best of circumstances. Just in case that's unknown to you or to other readers: It's really not true. Have a look at https://www.protondb.com/ https://www.protondb.com/ - Linux is excellent for gaming, even AAA gaming. The PI processor just would be too weak either way, and not being x86 would prevent the games from running directly. Maybe that incompatibility and not having proprietary games compiled for ARM available is what you meant? That would need something like https://github.com/ptitSeb/box86 https://github.com/ptitSeb/box86, which actually seems to work way better than expected.
- geerlingguy 5y ago
  Eh... the problem is even with Proton it's extra work and sometimes not pleasant try to keep games running natively in Linux. I had a nice Kubuntu Focus 2 laptop with a fast Nvidia RTX mobile chip, and all the bells and whistles, and I could only get 2 of my 5 favorite Steam games to run through Proton, and one of those required some janky tweaks to make it run correctly. When they did run they ran great (testament to the hardware and the decent driver support), but that's what I mean when it's 'tough'. Not impossible but it's not like Windows, where you just click buy, wait for the game to download, and start playing.
  - onli 5y ago
    With 1 out of 5 you were extremely unlucky. For the most part proton works exactly like described: You click on play and the game does run. In the other cases where there are fixes to be applied that's most of the time just setting a simple start parameter directly in Steam. You also should consider that on Windows in practice there are many issues, ranging from drivers and OS updates critically breaking performance (happened just now again), middleware stopping to work, and many older games that run perfectly fine on Linux not running at all on Windows anymore. It always depends on your game selection of course, but I for example also removed my dual boot Windows installation to exclusively game on Linux a few years ago now because it got way too annoying dealing with all the breakage and incompatibility on that platform.
deleted 5y ago
[deleted]
bullen 5y ago
Just buy a Jetson Nano 2GB, they are cheaper than a Raspberry 2GB now anyway (atleast in Sweden where the Raspberry is sold out and Arrow has free shipping of the Jetson). And that little 10W machine is 1/2 a Nintendo Switch, it can play my MMO engine with 300 characters on screen without world or physics (probably 100 players when the game is complete): http://talk.binarytask.com/task?id=5959519327505901449 http://talk.binarytask.com/task?id=5959519327505901449
- kanwisher 5y ago
  If you follow his channel he just wants to push the Pi to the limits, and part of that is documenting all pci/e devices that work on the pi. The nano doesn't have Pci/e
  - my123 5y ago
    Jetson Nano has PCIe, exposed through the M.2 slot. Far more bandwidth than an RPi too.
    - papercrane 5y ago
      The suggested 2GB Nano doesn't have the M.2 slot, instead it has 802.11ac wireless.
      - my123 5y ago
        If I remember well, it’s just a regular M2 Wi-Fi card included inside.
        numpad0 5y ago
        Looks like the redesigned carrier board for 2GB is missing M.2 Key E connector
- geerlingguy 5y ago
  I'm more interested in trying to get my hands on a SolidRun Honeycomb board, or heck, if the stars aligned, an Ampere server... I figure it might give a good reference for a better/full-featured ARM PCI Express implementation (especially the Ampere as it implements the latest ARM arch). That goes against some of my "get things done the cheapest way possible" philosophy, but it would be nice to be able to test things on hardware where it is known to _likely_ work.
vimsee 5y ago
Just wanted to say thank you for the content you put out. I was watching your stream yesterday and found it very interesting despite my lack of knowledge when it comes to Linux kernel stuff. Hoping to learn more as time moves forward so I can start to grasp more of the content in the videos of this topic. Looking forward to the next one, Jeff!
pjc50 5y ago
How many more orders of magnitude processing power than the Pi does the 1080 have?
lmeyerov 5y ago
anyone have experiences w ~modern Nvidia eGPUs for os x CUDA dev? (arm / pre-arm?) Currently traveling w 2 systems just because of this , which is nuts..
Decabytes 5y ago
I been playing around with an RPI4 and have been getting interested in graphics programming on Wayland. Unfortunately efforts to get Wayland running under the RPI have stalled due to the difficulty in working with the Broadcom chip. So I’m pretty much left looking for another SBC if I want to experiment.
- floatboth 5y ago
  Huh? I've used KWin-Wayland on Manjaro ARM on an RPi4, it Just Worked. And why wouldn't it — V3D is a normal DRM+Mesa thing.
spamizbad 5y ago
Seems odd that even RISC-V has external GPU support but ARM boards don't.
- floatboth 5y ago
  The ISA of the CPU cores doesn't matter. The PCIe controller does. And Broadcom being Broadcom, they have a weird one with interesting bugs like 64-bit I/O not even working. On something like a MACCHIATObin or Honeycomb LX2K, AMD GPUs just work, even with pre-boot display in UEFI (thanks to EDK2 optionally including QEMU for this, it can just emulate x86 for the display driver it reads from the card). Well, the MCbin has a fun quirk with device enumeration, but actual operation of the device is perfect.
  - geerlingguy 5y ago
    ^ This. I'm working around the edges of the SoC in the Pi, the BCM2711, which has some implementation quirks.
georgeecollins 5y ago
I opened this article right after working on a rust program on my Jetson Nanon. Jetson Nanos are like $50 on Newegg. They are very cool. HN community should be all over this. I have used raspberry pis for robotics, but now I am seeing if I can switch and it seems to work. A Jetson Nano has (I believe) 128 cuda cores. If you want more, get a TX1 or a bigger board. Seriously, trying to add an Nvidia care to a Pi is a dumb choice IMO. Show the Nano some love!
- wang_li 5y ago
  Nano has 128 Maxwell cores, which is the GeForce 900 generation. The Xavier MX, for 4x the cost, has 384 Volta cores. Dhewm3 on the Nano pulls 34 FPS at 1280x720. On the Xavier NX it runs at 88 FPS. On the pi4 it runs at 7.5 FPS. On a pi3 at 640x480 it takes several seconds per frame. Mostly a shortage of memory though. CPU speed is ordered pi3, nano, pi4, Xavier no. Memory bandwidth on the jetsons is much higher than on the pis.
  - georgeecollins 5y ago
    Right, but the nano is cheap and you can get the TX1, TX is you need more power. There is no world where that is not a cheaper and more versatile solution than hooking up Pi to like a 1080 graphics card. And plus.. really I want to get people using these things so we can share code :)

127 points by geerlingguy 5y ago

TLDR: it didn’t work, no real clue why, more updates to come.

The Raspberry Pi does not support PCIe I/O BAR space. That's not a hard requirement to run a modern GPU as far as I know, but drivers do kind of assume it's available. In particular, VBIOS stuff often needs it. The message from the Radeon driver is not a hard error, but presumably other things break as a result. The Broadcom SoCs used on Pis still suck, and although this can probably be made to work, someone familiar with this codebase needs to care enough to fix the drivers to make it happen. It's probably quite doable for the open source drivers, but I wouldn't expect Nvidia's blob to work any time soon unless they take the time to fix it. Not all PCIe root ports are equal. Some are, shall we say, less than amenable to devices more complicated than a USB controller. Looking through the discussion here, there is some rather concerning stuff about DMA mappings (e.g. the thing with 64bit accesses not being supported). It's possible the Pi4 PCIe implementation is actually too broken to support these GPUs, though I haven't read through the whole thread in detail. It certainly looks pretty messy. https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/4 https://github.com/geerlingguy/raspberry-pi-pcie-devices/iss...

geerlingguy 5y ago
Thanks for taking a glance. There's a groundswell of people commenting "get marcan to take a look". And I would ask, but you're already doing some really awesome work over on the M1 side and I've learned enough to fill up two of my brains already from that. At this point I kinda have to wave the white flag on the driver issues because I'm not deep enough on the kernel / PCI spec side to be productive figuring out the holes on the Pi's 'messy' bus :)
- monocasa 5y ago
  I spent a night looking through the radeon driver and register documentation, and it seems to fall back on sane defaults without being able to access IO space, it just prints a nasty gram. All of the expected IO space is mirrored in the MMIO space and the VBIOS code will just use that as a fallback pathway just fine. One sort of cheap way to get a sense of what's actually going on (at least way cheaper than a PCIe protocol analyzer) might be to put your hands on one of the USB3382 cards (the poor man's PCIe protocol analyzer for those who aren't that comfortable with FPGAs). Those are really neat soft PCIe devices that bridge the PCIe transactions over USB3 to another host computer and can do things like emulate other devices or send arbitrary TLPs. There's some hacker con talks about using them to attack systems by using their DMA TLPs as a read/write primitive. Using that to either bridge to a card on the host computer and tap the PCIe bus, or send your own TLPs to verify that stuff like 64 bit transactions are actually a problem and not a red herring. You can use a FPGA for the same thing pretty easily, but a USB3382 might be a lot closer to the skill set that you feel comfortable with.

mrspeaker 5y ago

"10% chance of releasing the magic smoke" made me laugh out loud. Off topic-ish, but maybe someone here would know... I have a 2013 macbook pro. It's still great for coding but recently I got into Blender, and now I understand why you might want an external GPU. Is that a possibility for such an old laptop? I couldn't find any definitive information on how to go about buying one, and setting it up.

Wowfunhappy 5y ago
I looked into getting an eGPU for a Macbook in 2013. Thunderbolt 2 has the bandwidth to make it possible, but it's not ideal and will cost performance. More critically, at least in 2013, the docks you needed cost thousands of dollars—and made more sense to just get a separate desktop PC. I don't know if the situation has changed since then—there are more eGPUs out there but it's all for Thunderbolt 3. Thunderbolt 2 is more niche and thus may be more expensive.
- deleted 5y ago
  [deleted]
- teruakohatu 5y ago
  The other issue is drivers. Nvidia is definitely out since the drivers have not been updated in many years.
- 1_player 5y ago
  Thunderbolt 2, if supported natively by the hardware and using an external monitor, might actually be enough in practice. I used to game on rMBP 2015 with a GTX 1080 in a TB2-connected eGPU and it didn't even saturate the link, so when I upgraded to an iMac with TB3 the performance was pretty much identical. The rule of thumb is: use an external monitor and the highest resolution possible. Sending control commands one way consumes much less bandwidth if you don't need to send whole images back down the link, and the high resolution and quality is to make the GPU the bottleneck, not the TB2 link.
- salamandersauce 5y ago
  It's possible to use at least some TB3 eGPUs with TB2 and adapters, it's not officially supported by Apple on macOS though. They are still insanely expensive in my opinion for what's basically a case, small PSU and PCI-E to TB3. The cheapest ones that can take most cards are around $300 and usually have some downside. I considered it for my Mac Mini but in the end it just made more sense to sell the Mini and buy a desktop PC with space for a real GPU, more storage, faster processor, etc. I mean my case, power supply and motherboard was less than the price of a Razer Core X...

onli 5y ago

> But probably not AAA gaming, since gaming on Linux is a tough proposition even in the best of circumstances. Just in case that's unknown to you or to other readers: It's really not true. Have a look at https://www.protondb.com/ https://www.protondb.com/ - Linux is excellent for gaming, even AAA gaming. The PI processor just would be too weak either way, and not being x86 would prevent the games from running directly. Maybe that incompatibility and not having proprietary games compiled for ARM available is what you meant? That would need something like https://github.com/ptitSeb/box86 https://github.com/ptitSeb/box86, which actually seems to work way better than expected.

geerlingguy 5y ago
Eh... the problem is even with Proton it's extra work and sometimes not pleasant try to keep games running natively in Linux. I had a nice Kubuntu Focus 2 laptop with a fast Nvidia RTX mobile chip, and all the bells and whistles, and I could only get 2 of my 5 favorite Steam games to run through Proton, and one of those required some janky tweaks to make it run correctly. When they did run they ran great (testament to the hardware and the decent driver support), but that's what I mean when it's 'tough'. Not impossible but it's not like Windows, where you just click buy, wait for the game to download, and start playing.
- onli 5y ago
  With 1 out of 5 you were extremely unlucky. For the most part proton works exactly like described: You click on play and the game does run. In the other cases where there are fixes to be applied that's most of the time just setting a simple start parameter directly in Steam. You also should consider that on Windows in practice there are many issues, ranging from drivers and OS updates critically breaking performance (happened just now again), middleware stopping to work, and many older games that run perfectly fine on Linux not running at all on Windows anymore. It always depends on your game selection of course, but I for example also removed my dual boot Windows installation to exclusively game on Linux a few years ago now because it got way too annoying dealing with all the breakage and incompatibility on that platform.

deleted 5y ago

[deleted]

bullen 5y ago

Just buy a Jetson Nano 2GB, they are cheaper than a Raspberry 2GB now anyway (atleast in Sweden where the Raspberry is sold out and Arrow has free shipping of the Jetson). And that little 10W machine is 1/2 a Nintendo Switch, it can play my MMO engine with 300 characters on screen without world or physics (probably 100 players when the game is complete): http://talk.binarytask.com/task?id=5959519327505901449 http://talk.binarytask.com/task?id=5959519327505901449

kanwisher 5y ago
If you follow his channel he just wants to push the Pi to the limits, and part of that is documenting all pci/e devices that work on the pi. The nano doesn't have Pci/e
- my123 5y ago
  Jetson Nano has PCIe, exposed through the M.2 slot. Far more bandwidth than an RPi too.
  - papercrane 5y ago
    The suggested 2GB Nano doesn't have the M.2 slot, instead it has 802.11ac wireless.
    - my123 5y ago
      If I remember well, it’s just a regular M2 Wi-Fi card included inside.
      - numpad0 5y ago
        Looks like the redesigned carrier board for 2GB is missing M.2 Key E connector
geerlingguy 5y ago
I'm more interested in trying to get my hands on a SolidRun Honeycomb board, or heck, if the stars aligned, an Ampere server... I figure it might give a good reference for a better/full-featured ARM PCI Express implementation (especially the Ampere as it implements the latest ARM arch). That goes against some of my "get things done the cheapest way possible" philosophy, but it would be nice to be able to test things on hardware where it is known to _likely_ work.

vimsee 5y ago

Just wanted to say thank you for the content you put out. I was watching your stream yesterday and found it very interesting despite my lack of knowledge when it comes to Linux kernel stuff. Hoping to learn more as time moves forward so I can start to grasp more of the content in the videos of this topic. Looking forward to the next one, Jeff!

pjc50 5y ago

How many more orders of magnitude processing power than the Pi does the 1080 have?

lmeyerov 5y ago

anyone have experiences w ~modern Nvidia eGPUs for os x CUDA dev? (arm / pre-arm?) Currently traveling w 2 systems just because of this , which is nuts..

Decabytes 5y ago

I been playing around with an RPI4 and have been getting interested in graphics programming on Wayland. Unfortunately efforts to get Wayland running under the RPI have stalled due to the difficulty in working with the Broadcom chip. So I’m pretty much left looking for another SBC if I want to experiment.

floatboth 5y ago
Huh? I've used KWin-Wayland on Manjaro ARM on an RPi4, it Just Worked. And why wouldn't it — V3D is a normal DRM+Mesa thing.

spamizbad 5y ago

Seems odd that even RISC-V has external GPU support but ARM boards don't.

floatboth 5y ago
The ISA of the CPU cores doesn't matter. The PCIe controller does. And Broadcom being Broadcom, they have a weird one with interesting bugs like 64-bit I/O not even working. On something like a MACCHIATObin or Honeycomb LX2K, AMD GPUs just work, even with pre-boot display in UEFI (thanks to EDK2 optionally including QEMU for this, it can just emulate x86 for the display driver it reads from the card). Well, the MCbin has a fun quirk with device enumeration, but actual operation of the device is perfect.
- geerlingguy 5y ago
  ^ This. I'm working around the edges of the SoC in the Pi, the BCM2711, which has some implementation quirks.

georgeecollins 5y ago

I opened this article right after working on a rust program on my Jetson Nanon. Jetson Nanos are like $50 on Newegg. They are very cool. HN community should be all over this. I have used raspberry pis for robotics, but now I am seeing if I can switch and it seems to work. A Jetson Nano has (I believe) 128 cuda cores. If you want more, get a TX1 or a bigger board. Seriously, trying to add an Nvidia care to a Pi is a dumb choice IMO. Show the Nano some love!

wang_li 5y ago
Nano has 128 Maxwell cores, which is the GeForce 900 generation. The Xavier MX, for 4x the cost, has 384 Volta cores. Dhewm3 on the Nano pulls 34 FPS at 1280x720. On the Xavier NX it runs at 88 FPS. On the pi4 it runs at 7.5 FPS. On a pi3 at 640x480 it takes several seconds per frame. Mostly a shortage of memory though. CPU speed is ordered pi3, nano, pi4, Xavier no. Memory bandwidth on the jetsons is much higher than on the pis.
- georgeecollins 5y ago
  Right, but the nano is cheap and you can get the TX1, TX is you need more power. There is no world where that is not a cheaper and more versatile solution than hooking up Pi to like a 1080 graphics card. And plus.. really I want to get people using these things so we can share code :)