7 ms·

I built a special PCIe card to test GPUs on the Pi

151 points by gravitate 3y ago

vxNsr 3y ago
I’m always impressed when YouTubers who mostly to me appear to be reviewers go down a technical rabbit hole and make something that requires a lot of research and technical expertise.
- dcow 3y ago
  Did this guy start as a reviewer, or is it more that YT generally feels product review heavy on the software tech side so you assumed this guy was cut from the same stuff? (Honestly don’t know I’ve only seen a few of his videos and they were all deeply technical.)
  - r2_pilot 3y ago
    Jeff Geerling is majorly involved with Ansible, so he's not really reviewing for the sake of reviewing. It's more that he's sharing his knowledge about his off-work pursuits, I believe.
    - sureglymop 3y ago
      Iirc he was also heavily involved with Linux phones like the Pinephone and maybe even wrote firmware for it. I think it's more that the guy is a professional who happens to also have a YouTube account, not the other way around.
    - vxNsr 3y ago
      Ah ok. Yea I only know him from when he appears in the sidebar next to other technical videos, there he’s mostly reviewing docker setups or various self hosted options. I didn’t realize how technical he is.
  - michaelt 3y ago
    The thumbnail is triggering people's youtube bullshit detectors. That big shot of the reviewer's face. The giant red arrow. The bright background. The breathless title with ALL CAPS. That's like 4/5 for clickbait videos. All it's missing is the face showing some ridiculously exaggerated emotion.
    - loudmax 3y ago
      Jeff Geerling is working within the YouTube algorithm and he does post some clickbaity thumbnails. Don't let those thumbnails mislead you. geerlingguy is an active member of the open source community and he produces some absolute top rate instructional and informative videos.
      - geerlingguy 3y ago
        He's also here on HN ;) And regarding clickbait, I wrote a bit on that topic over in the Ampere Dev Community site[1], but in summary: > I decided to focus the intro bit on ‘Could this be the next Mac Pro’? (needs to be slightly clickbaity to get people to actually click. All of us engineering types SAY that we don’t like clickbait, but for 99% of us, if you gave us the clickbait version and the non-clickbait version in your YouTube feed… we’d click the clickbait version 100% of the time [1] https://community.amperecomputing.com/t/review-of-the-ampere-developer-platform-by-jeff-geerling/277/2?u=geerlingguy https://community.amperecomputing.com/t/review-of-the-ampere...
        ddalex 3y ago
        Dude, keep up the good work, you're an inspiration !
        zamadatix 3y ago
        Clickbait is absolutely effective, I don't think anyone is doubting so when they say they don't like clickbait. It's in the same boat as ads. Few like ads but even fewer actually avoid ad funded content (particularly when it comes to sponsors) so ads are effective monetization.
      - 3y ago
- krabizzwainch 3y ago
  He does also run a company (some kind of web solutions business?) and is heavily involved in Ansible (having written multiple books). The youtube channel came after that stuff.
rektide 3y ago
I'll give $20 to the first person who gets one of these cards & sets up a host with Non-Transparent Bridging (NTB, effectively making that PCIe slot look like a device), and connects & talks to the Pi.
- jacquesm 3y ago
  You're cheap ;)
- geerlingguy 3y ago
  Supposedly the folks at Timebeat are working on a similar card[1], but one that appears as a PCIe device with two serial ports—one through to a Raspberry Pi, and one through to a U-blox module for timekeeping. [1] https://store.timebeat.app/products/ocp-tap-timecard-mini-platinum-edition https://store.timebeat.app/products/ocp-tap-timecard-mini-pl...
  - rektide 3y ago
    Woof $1200! But for a very neat application, certainly, that certainly vendors would have been charging a lot more for in the past. And it includes the $100 LEA-M8F GPS module... And the $150 SiTIME SiT5721 oxco over controlled oscillator. Excellent parts. I was going to ask if the Pi had decent ptp support to actually be useful speaking ethernet, but you've already answered that (yes), https://www.jeffgeerling.com/blog/2022/ptp-and-ieee-1588-hardware-timestamping-on-raspberry-pi-cm4 https://www.jeffgeerling.com/blog/2022/ptp-and-ieee-1588-har... I do kind of wish it had a PCIe ethernet controller onboard for the host too, also with ptp tied to the host. Sending that fine timing data across serial & PCIe bused to the host & to the hosts card feels unideal, like a source of some jitter, but maybe it's not really an issue.
    - namibj 3y ago
      PCIe has a continuous 100MHz clock if you don't utilize link parking power be saving mechanisms. The device should be able to lock to the host's base clock that way, and phase offsets should be something you can resolve by measuring. Ideally computers would support consuming a PLL reference via 1GBASE-T, as that already has a need for a locked clock for TX and RX near-side cross-talk compensation, and CERN demonstrated usage of this for clock carrier distribution in their GPS-shielding tunnels. OFC integrated with PTP to accurately sync time; AFAIK they did for ns-accurate event time stamp assignment in some distributed sensor networks.
dcow 3y ago
Often overlooked value of open source: free labor. I bet you’d see a patched driver within a week if the sources were available to enthusiasts.
- faefox 3y ago
  Overlooked? It's the foundation of the modern tech industry. :) Hell, we've all been brainwashed to the point where OSS maintainers having the audacity to ask for a modicum of support for their efforts is met with scorn and ridicule.
- koito17 3y ago
  > Often overlooked Why do you think corporations and everyone here push the MIT license so hard and in general dislike copyleft licenses, even weak ones like the EPL? It always boils down to "I want to use and modify your work for free at my workplace", though people will usually try to say this very indirectly, usually with some rant on "commercialization".
  - dcow 3y ago
    I was referring to the reverse but wasn't clear: Nvidia would get better driver compatibility for free if they open-sourced it fully. Point was not that #corporation gets free labor by using open source, it’s that their product would be better for free if open sourced.
rustybolt 3y ago
> there's no desire to either maintain a fork of their drivers or spend time hacking their driver to work on the buggy PCIe bus on the Pi That seems like the right choice. Specs exist for a reason. If you don't follow them, there is no reason your device should work. To adapt the world to work around your mistakes is insanity.
- ChuckMcM 3y ago
  Implicit assumption that the Pi's PCIe doesn't follow the spec? There is a lot of room between "it works in our target market" and "it follows the spec". I only mention that because I have come across a number of situations where the specifications related to "downsizing" the interface when insufficient resources are detected are often ignored when the target platform is invested in providing only the maximum amount of resources. Think USB2.0 high speed devices failing to down throttle in a USB 1.1 port and still work as an example. That said, there are often latent bugs in these situations that this sort of research/experimentation can uncover. Manufacturers would do well to invest the time in understanding why it doesn't work because those reasons can often lead to problems that are hard to catch when it is working.
  - manv1 3y ago
    FTA there seem to be at least 2 problems: "the Pi has problems with cache coherency on the PCI Express bus beyond 32-bits. And many (well, all nowadays) of the drivers expect that to function." Yeah, that's a problem. "there were issues with the BAR (Base Address Register) space allocated on the Pi's OS." From the GitHub link: # The default BAR address space available on the CM4 may be too small to allow # some devices to initialize correctly. To avoid 'failed to assign memory' These don't look like implementation quirks, they are just flat-out errors.
    - simcop2387 3y ago
      I believe from previous articles/videos from Jeff that the BAR one is at least addressable via a custom kernel or device tree that sets up a large enough BAR space for those devices. That said I would be surprised if there's a way to easily fix the first problem in software aside from forcing constant cache flushes when dealing with MMIO registers which would be a huge undertaking and probably tank performance in ways that nobody would want since this should be handled by the controller on an as needed basis instead.
dvdbloc 3y ago
Sooo where can I buy one?
- geerlingguy 3y ago
  Right now we aren't selling any prebuilt cards—but the hardware is under an OSHW license and the designs are available here: https://github.com/geerlingguy/pi4gpu https://github.com/geerlingguy/pi4gpu
codedokode 3y ago
Interesting, I thought that to initialize a PCI card one needs to run Video BIOS located on the card in a x86 emulator. Is that wrong?
- geerlingguy 3y ago
  It might be the case on some of the older cards, but most modern(ish) cards that don't have VGA anymore don't require that.
  - monocasa 3y ago
    There's also some cute hacks to emulate in different parts of the software stack for the cases where there's still value in running the onboard boot time firmware. For instance https://github.com/openSUSE/X86EmulatorPkg https://github.com/openSUSE/X86EmulatorPkg and https://cgit.freedesktop.org/xorg/xserver/tree/hw/xfree86/int10/xf86x86emu.c https://cgit.freedesktop.org/xorg/xserver/tree/hw/xfree86/in...
- deleted 3y ago
  [deleted]
b33j0r 3y ago
You can know that someone has built a brand when I didn't have to check that this was Jeff! However, I also saw his youtube video about it over a week ago. Given his progress and dedication, I fully expect a stack of A100's "graciously sent by _someone_" by this time in 2024 ;) To compare with a 32xRP2040 UHD VGA cluster in a separate post. Or something! This Dude is killing it in this niche.
villgax 3y ago
Apple Silicon really dropped the ball on Thunderbolt 4 with no support for eGPUs of any kind....

151 points by gravitate 3y ago

I’m always impressed when YouTubers who mostly to me appear to be reviewers go down a technical rabbit hole and make something that requires a lot of research and technical expertise.

dcow 3y ago
Did this guy start as a reviewer, or is it more that YT generally feels product review heavy on the software tech side so you assumed this guy was cut from the same stuff? (Honestly don’t know I’ve only seen a few of his videos and they were all deeply technical.)
- r2_pilot 3y ago
  Jeff Geerling is majorly involved with Ansible, so he's not really reviewing for the sake of reviewing. It's more that he's sharing his knowledge about his off-work pursuits, I believe.
  - sureglymop 3y ago
    Iirc he was also heavily involved with Linux phones like the Pinephone and maybe even wrote firmware for it. I think it's more that the guy is a professional who happens to also have a YouTube account, not the other way around.
  - vxNsr 3y ago
    Ah ok. Yea I only know him from when he appears in the sidebar next to other technical videos, there he’s mostly reviewing docker setups or various self hosted options. I didn’t realize how technical he is.
- michaelt 3y ago
  The thumbnail is triggering people's youtube bullshit detectors. That big shot of the reviewer's face. The giant red arrow. The bright background. The breathless title with ALL CAPS. That's like 4/5 for clickbait videos. All it's missing is the face showing some ridiculously exaggerated emotion.
  - loudmax 3y ago
    Jeff Geerling is working within the YouTube algorithm and he does post some clickbaity thumbnails. Don't let those thumbnails mislead you. geerlingguy is an active member of the open source community and he produces some absolute top rate instructional and informative videos.
    - geerlingguy 3y ago
      He's also here on HN ;) And regarding clickbait, I wrote a bit on that topic over in the Ampere Dev Community site[1], but in summary: > I decided to focus the intro bit on ‘Could this be the next Mac Pro’? (needs to be slightly clickbaity to get people to actually click. All of us engineering types SAY that we don’t like clickbait, but for 99% of us, if you gave us the clickbait version and the non-clickbait version in your YouTube feed… we’d click the clickbait version 100% of the time [1] https://community.amperecomputing.com/t/review-of-the-ampere-developer-platform-by-jeff-geerling/277/2?u=geerlingguy https://community.amperecomputing.com/t/review-of-the-ampere...
      - ddalex 3y ago
        Dude, keep up the good work, you're an inspiration !
      - zamadatix 3y ago
        Clickbait is absolutely effective, I don't think anyone is doubting so when they say they don't like clickbait. It's in the same boat as ads. Few like ads but even fewer actually avoid ad funded content (particularly when it comes to sponsors) so ads are effective monetization.
    - 3y ago
krabizzwainch 3y ago
He does also run a company (some kind of web solutions business?) and is heavily involved in Ansible (having written multiple books). The youtube channel came after that stuff.

rektide 3y ago

I'll give $20 to the first person who gets one of these cards & sets up a host with Non-Transparent Bridging (NTB, effectively making that PCIe slot look like a device), and connects & talks to the Pi.

jacquesm 3y ago
You're cheap ;)
geerlingguy 3y ago
Supposedly the folks at Timebeat are working on a similar card[1], but one that appears as a PCIe device with two serial ports—one through to a Raspberry Pi, and one through to a U-blox module for timekeeping. [1] https://store.timebeat.app/products/ocp-tap-timecard-mini-platinum-edition https://store.timebeat.app/products/ocp-tap-timecard-mini-pl...
- rektide 3y ago
  Woof $1200! But for a very neat application, certainly, that certainly vendors would have been charging a lot more for in the past. And it includes the $100 LEA-M8F GPS module... And the $150 SiTIME SiT5721 oxco over controlled oscillator. Excellent parts. I was going to ask if the Pi had decent ptp support to actually be useful speaking ethernet, but you've already answered that (yes), https://www.jeffgeerling.com/blog/2022/ptp-and-ieee-1588-hardware-timestamping-on-raspberry-pi-cm4 https://www.jeffgeerling.com/blog/2022/ptp-and-ieee-1588-har... I do kind of wish it had a PCIe ethernet controller onboard for the host too, also with ptp tied to the host. Sending that fine timing data across serial & PCIe bused to the host & to the hosts card feels unideal, like a source of some jitter, but maybe it's not really an issue.
  - namibj 3y ago
    PCIe has a continuous 100MHz clock if you don't utilize link parking power be saving mechanisms. The device should be able to lock to the host's base clock that way, and phase offsets should be something you can resolve by measuring. Ideally computers would support consuming a PLL reference via 1GBASE-T, as that already has a need for a locked clock for TX and RX near-side cross-talk compensation, and CERN demonstrated usage of this for clock carrier distribution in their GPS-shielding tunnels. OFC integrated with PTP to accurately sync time; AFAIK they did for ns-accurate event time stamp assignment in some distributed sensor networks.

dcow 3y ago

Often overlooked value of open source: free labor. I bet you’d see a patched driver within a week if the sources were available to enthusiasts.

faefox 3y ago
Overlooked? It's the foundation of the modern tech industry. :) Hell, we've all been brainwashed to the point where OSS maintainers having the audacity to ask for a modicum of support for their efforts is met with scorn and ridicule.
koito17 3y ago
> Often overlooked Why do you think corporations and everyone here push the MIT license so hard and in general dislike copyleft licenses, even weak ones like the EPL? It always boils down to "I want to use and modify your work for free at my workplace", though people will usually try to say this very indirectly, usually with some rant on "commercialization".
- dcow 3y ago
  I was referring to the reverse but wasn't clear: Nvidia would get better driver compatibility for free if they open-sourced it fully. Point was not that #corporation gets free labor by using open source, it’s that their product would be better for free if open sourced.

rustybolt 3y ago

> there's no desire to either maintain a fork of their drivers or spend time hacking their driver to work on the buggy PCIe bus on the Pi That seems like the right choice. Specs exist for a reason. If you don't follow them, there is no reason your device should work. To adapt the world to work around your mistakes is insanity.

ChuckMcM 3y ago
Implicit assumption that the Pi's PCIe doesn't follow the spec? There is a lot of room between "it works in our target market" and "it follows the spec". I only mention that because I have come across a number of situations where the specifications related to "downsizing" the interface when insufficient resources are detected are often ignored when the target platform is invested in providing only the maximum amount of resources. Think USB2.0 high speed devices failing to down throttle in a USB 1.1 port and still work as an example. That said, there are often latent bugs in these situations that this sort of research/experimentation can uncover. Manufacturers would do well to invest the time in understanding why it doesn't work because those reasons can often lead to problems that are hard to catch when it is working.
- manv1 3y ago
  FTA there seem to be at least 2 problems: "the Pi has problems with cache coherency on the PCI Express bus beyond 32-bits. And many (well, all nowadays) of the drivers expect that to function." Yeah, that's a problem. "there were issues with the BAR (Base Address Register) space allocated on the Pi's OS." From the GitHub link: # The default BAR address space available on the CM4 may be too small to allow # some devices to initialize correctly. To avoid 'failed to assign memory' These don't look like implementation quirks, they are just flat-out errors.
  - simcop2387 3y ago
    I believe from previous articles/videos from Jeff that the BAR one is at least addressable via a custom kernel or device tree that sets up a large enough BAR space for those devices. That said I would be surprised if there's a way to easily fix the first problem in software aside from forcing constant cache flushes when dealing with MMIO registers which would be a huge undertaking and probably tank performance in ways that nobody would want since this should be handled by the controller on an as needed basis instead.

dvdbloc 3y ago

Sooo where can I buy one?

geerlingguy 3y ago
Right now we aren't selling any prebuilt cards—but the hardware is under an OSHW license and the designs are available here: https://github.com/geerlingguy/pi4gpu https://github.com/geerlingguy/pi4gpu

codedokode 3y ago

Interesting, I thought that to initialize a PCI card one needs to run Video BIOS located on the card in a x86 emulator. Is that wrong?

geerlingguy 3y ago
It might be the case on some of the older cards, but most modern(ish) cards that don't have VGA anymore don't require that.
- monocasa 3y ago
  There's also some cute hacks to emulate in different parts of the software stack for the cases where there's still value in running the onboard boot time firmware. For instance https://github.com/openSUSE/X86EmulatorPkg https://github.com/openSUSE/X86EmulatorPkg and https://cgit.freedesktop.org/xorg/xserver/tree/hw/xfree86/int10/xf86x86emu.c https://cgit.freedesktop.org/xorg/xserver/tree/hw/xfree86/in...
deleted 3y ago
[deleted]

b33j0r 3y ago

You can know that someone has built a brand when I didn't have to check that this was Jeff! However, I also saw his youtube video about it over a week ago. Given his progress and dedication, I fully expect a stack of A100's "graciously sent by _someone_" by this time in 2024 ;) To compare with a 32xRP2040 UHD VGA cluster in a separate post. Or something! This Dude is killing it in this niche.

villgax 3y ago

Apple Silicon really dropped the ball on Thunderbolt 4 with no support for eGPUs of any kind....