7 ms·

Uptime Lab's CM4 Blade Adds NVMe, TPM 2.0 to Raspberry Pi

152 points by geerlingguy 5y ago

geerlingguy 5y ago
To be clear, the blade adds TPM 2.0 via an Infineon chip, and you can interact with the chip, but features like secure boot require bootloader-level integration, and at this time, since no other Pi hardware has TPM, the Pi's closed-source bootloader doesn't support it. Also something interesting to noodle on: technically the Pi 4/CM4/Pi 400 have ECC RAM [1] — but some ECC functionality may require CPU-level integration... which I'm guessing isn't present on the Pi's SoC (but who knows? I posted this forum topic asking for clarification: [2]). [1] Product brief mentions LPDDR4 with on-die ECC: https://datasheets.raspberrypi.org/rpi4/raspberry-pi-4-product-brief.pdf https://datasheets.raspberrypi.org/rpi4/raspberry-pi-4-produ... [2] https://www.raspberrypi.org/forums/viewtopic.php?t=315415 https://www.raspberrypi.org/forums/viewtopic.php?t=315415
- fakesheriff 5y ago
  Without bootloader integration, what's the difference between adding a TPM vs an HSM like [0]? Does TPM just have a more standardized interface? [0] https://www.zymbit.com/2020/11/10/blog-security-module-raspberry-pi/ https://www.zymbit.com/2020/11/10/blog-security-module-raspb...
  - geerlingguy 5y ago
    You can actually add both—there's a partial GPIO header for the Zymkey 4i on the board. But yeah, I think the idea is TPM is a bit more standardized across hardware, so some software that uses it would not need any tweaks to run on the Pi with a TPM built-in.
    - gruez 5y ago
      >But yeah, I think the idea is TPM is a bit more standardized across hardware But there are USB HSMs, along with smart cards (which are also HSMs). Aren't those pretty standard?
      - tadfisher 5y ago
        Sort of; the communication protocol is standard (CCID), but the actual HSM interface varies. Yubikeys implement the OpenPGP smartcard interface, for example, as well as PKCS #11. The TPM specification has its own crypto interface that is standard across all hardware, so you can do things like generate a key and perform crypto operations without requiring the hardware implement a particular interface beyond whatever TPM version you require. There are advantages and disadvantages to both approaches. On Linux, TPMs are implemented in the kernel, and CCID is handled by userspace drivers.
  - pmorici 5y ago
    The Zymbit is trying to make up for the lack of secureboot/tpm integration on the Pi by securing the enclosure and monitoring that perimeter.
- Stampo00 5y ago
  The inclusion of TPM is definitely the most interesting aspect of this to me. I was initially turned off by the idea because I tend to associate them with nasty enterprise-y DRM types of stuff. But I wonder why it's included here. Maybe for secure storage of private keys? I'm guessing that's the desired purpose for TPM in a server setting. Obviously I'm not super familiar with TPM. I wonder if it can be used for things like better random number generation or hardware-accelerated hashing. I'm guessing if it supported hashing from userspace, the cryptocurrency crowd would have been all over it already.
  - simcop2387 5y ago
    I believe the 2.0 standard does support rng, but I;m not sure of the specifics. there might be some hashing functionality but the interface is going to be too slow for large data. I think it;s there to do integrity checks for bootloaders and such
  - megous 5y ago
    TPMs are just SPI connected devices. You can add them to any SBC. Hardware accelerated hashing would be severely limited by the SPI bus speed.
  - duskwuff 5y ago
    > I wonder if it can be used for things like better random number generation... Too slow to be useful, and the BCM283x SoC already has an internal RNG: https://github.com/torvalds/linux/blob/master/drivers/char/hw_random/bcm2835-rng.c https://github.com/torvalds/linux/blob/master/drivers/char/h... > ... or hardware-accelerated hashing The TPM is on a really slow bus (33 MHz, 4 bits, tons of wait states). Even without any hardware acceleration, hashing on the CPU is much faster.
    - staticassertion 5y ago
      You wouldn't hit a TPM every time you wanted a random number. Your OS would just use it when it wanted to seed its entropy pool. This is particularly useful at boot when entropy is very low.
  - 5y ago
- formerly_proven 5y ago
  On-die ECC means that the DRAM array in the chip stores ECC and the chip itself handles that transparently. The interface does not use ECC in this case. One of the reasons this is done is higher cell densities (just like in storage) and allowing longer refresh intervals (refreshing requires power regardless of usage).
mbreese 5y ago
At this level, I don’t understand the PoE requirement. If you’re already making “blades”, why not also make a proper backplane that includes power and network connections as part of the backplane. You’d probably be more power efficient and remove the need for so many network ports. This would also increase the costs for the chassis (because you’d have to add a network switch), but you could probably also pack in more blades… But even if you kept the ethernet PHY ports, you’d probably be more power efficient with even just power on a backplane.
- geerlingguy 5y ago
  I think it's mostly convention and convenience—in the Pi ecosystem, most people using these things headless or as small servers are used to powering with PoE already. Networking gear is often powered by PoE, but most servers have higher power requirements so a custom backplane or separate power method would be required. Boards like the Turing Pi v2 (so far just prototypes and a marketing page on a website) would have a built-in network switch and power backplane (direct to each CM4), and I'm guessing another board or two like it will appear someday. I hope.
  - h2odragon 5y ago
    Is there somewhere after the POE that power could be tapped in? Even just a pair of pins for 5v sounds like it might make for significant power savings.
  - mbreese 5y ago
    No, I get that... and there is a different level between hobbyist projects and larger production projects. Using the existing PoE "infrastructure" makes a lot of sense in the prototyping stages. But if you're already going through the process to build custom boards for a cluster, then a backplane makes much more sense. We're not talking about a small 4-5 RPi cluster where everything can work off of a single PoE gigabit switch with a rat's nest of cat6. This project is talking about 16 of these blades in a 1U chassis. If you're going to go that far, a backplane isn't a big leap and would save power, space, and make cabling much easier.
bserge 5y ago
Wait, since when is using several separate machines an acceptable way of computing? And especially counting total RAM as if all of it can be used when needed? I've been doing that for a decade now (for cheapness mostly) and people said it was stupid and worse than a single computer with the same number of cores and RAM. Also would those chips need heatsinks and cooling?
- toast0 5y ago
  Using several separate machines is what you have to do when your computing needs don't fit in a single computer. Of course, the total computing power here would fit in a single x86 computer, but it can be fun to play with clustering and this might be a lot less expensive than an x86 cluster.
- 908B64B197 5y ago
  > Wait, since when is using several separate machines an acceptable way of computing? And especially counting total RAM as if all of it can be used when needed? It brings joy to Pi owners. There's more to the Pi than just the hardware.
- gh02t 5y ago
  Supercomputers have been built like that for a long time (decades) now, albeit with ultra-fast networking and storage to back them. There is a whole lot of research and money poured into specialized algorithms and software optimized for distributed machines. And that's far the only form of distributed computing in use.
jdoss 5y ago
I know this the Pi home lab hardware ecosystem is not the datacenter ecosystem, but what I really want on a Pi is not TPM but IPMI for remote management.
- geerlingguy 5y ago
  Ping @merocle on Twitter—the board's design is not 100% final, and there may be a few other features he could cram in (I've already thought of moving at least the serial UART connection to the front...).
- mbreese 5y ago
  That’s funny. Because one of the more legitimate uses for a Pi in a data center is as a remote management KVM. https://pikvm.org/ https://pikvm.org/
anoonmoose 5y ago
Getting kind of sick of OP reviewing products that we can't buy and, imo, probably won't ever be able to buy. Rather, sick of it making it to the HN front page. https://news.ycombinator.com/item?id=27460885 https://news.ycombinator.com/item?id=27460885
- geerlingguy 5y ago
  Most of these products are able to be built by hobbyists (in fact, almost all of them are produced by hobbyists). The reason most fail to launch to production is not only the extreme difficulty of scaling from prototype to manufactured production quantities of product, but also from 2020 onwards, the chip shortages. For each product I've been able to actually touch, there are ten more I've heard about and wanted to see happen, but they just couldn't be made because certain specialty ICs are just not available to small-time hobbyists/makers. I don't tend to review these boards because I want people to go out and buy them. I review them because I am inspired to try out new ideas and try my hand at new things (circuit designs, PCB design, etc.), and I figure maybe some other people can get inspired to try too. I try to feature as much open source hardware as possible, because it actually is possible for people to get custom PCBs and put together their own versions of them.
- jimmies 5y ago
  Being off-the-shelf is an added bonus, but not a requirement. This website is called Hacker news, not Bestbuy news. As long as someone else can replicate the results from what is given in the post, I do think the post has enough merit.
- rsync 5y ago
  Ugh, what ? I love seeing the Jeff Geerling posts and his blog is great!
- wila 5y ago
  You don't have to read his posts, the base URL is right next to the title. It is very easy to skip an article if it is not your cup of tea. Personally I love the posts of Jeff as I think they are very well written and well researched.
diarmuidc 5y ago
"The board gets fairly warm, likely due to the overhead of the PoE+ power conversion (which consumes 6-7W on its own!" 7 W of losses? That's either wrong or insane for DC/DC regulation. Rough figures, pi4CM (5W) + NVMe (2W) = 7W. So regulation of 50% efficiency?
- geerlingguy 5y ago
  I found the same to be the case on the official Pi 4 PoE+ HAT (the new 2021 version)[1]. [1] https://www.jeffgeerling.com/blog/2021/review-raspberry-pis-poe-hat-june-2021 https://www.jeffgeerling.com/blog/2021/review-raspberry-pis-...
- shadowpho 5y ago
  PoE can't do "purely" DC/DC regulation. They require (in any reasonable setup) a transformer for isolation on both ends, which lowers efficiency, especially at lower end. Now with that said you can definitely get decent efficiency with PoE, it's just that most people don't care and so it's not as big of a focus.
karmicthreat 5y ago
What I want to do for a funsies project is have a bunch of Pi4 CMs plugged into a PCIE switch and a multi-host 10gige adapter. I don't think this is possible though, I think I read somewhere that the Pi4 PCIE Root doesn't support host<->host comms. If anyone knows different, speak up!
rsync 5y ago
Can we talk about the 10" rack please ? It appears that the rack posts are just chopped/hacked from some other rack ... but where are people sourcing these little 10" wide rack shelves ? Also, the bottom component - is that a patch panel or a switch ?
- youngtaff 5y ago
  Bottom component looks like a patch panel. Plenty of 10” kit about e.g. https://datacabinetsdirect.co.uk/soho-10-inch-data-network-rack-cabinets/10-inch-soho-data-cabinet-accessories.html https://datacabinetsdirect.co.uk/soho-10-inch-data-network-r... 10” soho rack appears to be magic search phrase I've see a rack similar to the in the article before and can't find it now, but one way to recreate would be to buy soemthing like this a use a 10” blanking plate instead of the 19” one https://www.amazon.com/Procraft-Desktop-System-Washers-DTR-10/dp/B01CR74Q1C https://www.amazon.com/Procraft-Desktop-System-Washers-DTR-1... Lots of music gear shops sell desktop racks BTW
- Merocle 5y ago
  I used this one: https://www.rack-magic.com/Mini-Rack-4826mm-19-Rack-Stand-6HE-9HE-10-Zoll https://www.rack-magic.com/Mini-Rack-4826mm-19-Rack-Stand-6H...
ajklsdhfniuwehf 5y ago
the only good TPM can bring to this world is to entice people to finally hack a open source bootloader for the pi :)
dheera 5y ago
> Integrated PoE+ with Gigabit Ethernet Are there any good PoE hats for Pi that won't melt everything around it, and have spare power for USB?
exabrial 5y ago
Noob here... Would it be possible to do some sort of provisioned storage over PCI-E? So rather than each module having its own M2 drive, there is a PCIe connection on the backplane back to an SSD Raid.
- jlawer 5y ago
  For geek cred you could put in another Network or Storage adapter and do ISCSI or FC. SAS would allow 2 to share storage, however I don't think SAS supports more then 2 initiators. Directly connecting every host to a pool of Flash chips would likely require a custom controller for the flash and a way to present it to each Raspberry Pi This will almost certainly not make sense economically, but it seems a lot around RPi clustering doesn't make sense from a performance/$ perspective.

152 points by geerlingguy 5y ago

To be clear, the blade adds TPM 2.0 via an Infineon chip, and you can interact with the chip, but features like secure boot require bootloader-level integration, and at this time, since no other Pi hardware has TPM, the Pi's closed-source bootloader doesn't support it. Also something interesting to noodle on: technically the Pi 4/CM4/Pi 400 have ECC RAM [1] — but some ECC functionality may require CPU-level integration... which I'm guessing isn't present on the Pi's SoC (but who knows? I posted this forum topic asking for clarification: [2]). [1] Product brief mentions LPDDR4 with on-die ECC: https://datasheets.raspberrypi.org/rpi4/raspberry-pi-4-product-brief.pdf https://datasheets.raspberrypi.org/rpi4/raspberry-pi-4-produ... [2] https://www.raspberrypi.org/forums/viewtopic.php?t=315415 https://www.raspberrypi.org/forums/viewtopic.php?t=315415

fakesheriff 5y ago
Without bootloader integration, what's the difference between adding a TPM vs an HSM like [0]? Does TPM just have a more standardized interface? [0] https://www.zymbit.com/2020/11/10/blog-security-module-raspberry-pi/ https://www.zymbit.com/2020/11/10/blog-security-module-raspb...
- geerlingguy 5y ago
  You can actually add both—there's a partial GPIO header for the Zymkey 4i on the board. But yeah, I think the idea is TPM is a bit more standardized across hardware, so some software that uses it would not need any tweaks to run on the Pi with a TPM built-in.
  - gruez 5y ago
    >But yeah, I think the idea is TPM is a bit more standardized across hardware But there are USB HSMs, along with smart cards (which are also HSMs). Aren't those pretty standard?
    - tadfisher 5y ago
      Sort of; the communication protocol is standard (CCID), but the actual HSM interface varies. Yubikeys implement the OpenPGP smartcard interface, for example, as well as PKCS #11. The TPM specification has its own crypto interface that is standard across all hardware, so you can do things like generate a key and perform crypto operations without requiring the hardware implement a particular interface beyond whatever TPM version you require. There are advantages and disadvantages to both approaches. On Linux, TPMs are implemented in the kernel, and CCID is handled by userspace drivers.
- pmorici 5y ago
  The Zymbit is trying to make up for the lack of secureboot/tpm integration on the Pi by securing the enclosure and monitoring that perimeter.
Stampo00 5y ago
The inclusion of TPM is definitely the most interesting aspect of this to me. I was initially turned off by the idea because I tend to associate them with nasty enterprise-y DRM types of stuff. But I wonder why it's included here. Maybe for secure storage of private keys? I'm guessing that's the desired purpose for TPM in a server setting. Obviously I'm not super familiar with TPM. I wonder if it can be used for things like better random number generation or hardware-accelerated hashing. I'm guessing if it supported hashing from userspace, the cryptocurrency crowd would have been all over it already.
- simcop2387 5y ago
  I believe the 2.0 standard does support rng, but I;m not sure of the specifics. there might be some hashing functionality but the interface is going to be too slow for large data. I think it;s there to do integrity checks for bootloaders and such
- megous 5y ago
  TPMs are just SPI connected devices. You can add them to any SBC. Hardware accelerated hashing would be severely limited by the SPI bus speed.
- duskwuff 5y ago
  > I wonder if it can be used for things like better random number generation... Too slow to be useful, and the BCM283x SoC already has an internal RNG: https://github.com/torvalds/linux/blob/master/drivers/char/hw_random/bcm2835-rng.c https://github.com/torvalds/linux/blob/master/drivers/char/h... > ... or hardware-accelerated hashing The TPM is on a really slow bus (33 MHz, 4 bits, tons of wait states). Even without any hardware acceleration, hashing on the CPU is much faster.
  - staticassertion 5y ago
    You wouldn't hit a TPM every time you wanted a random number. Your OS would just use it when it wanted to seed its entropy pool. This is particularly useful at boot when entropy is very low.
- 5y ago
formerly_proven 5y ago
On-die ECC means that the DRAM array in the chip stores ECC and the chip itself handles that transparently. The interface does not use ECC in this case. One of the reasons this is done is higher cell densities (just like in storage) and allowing longer refresh intervals (refreshing requires power regardless of usage).

mbreese 5y ago

At this level, I don’t understand the PoE requirement. If you’re already making “blades”, why not also make a proper backplane that includes power and network connections as part of the backplane. You’d probably be more power efficient and remove the need for so many network ports. This would also increase the costs for the chassis (because you’d have to add a network switch), but you could probably also pack in more blades… But even if you kept the ethernet PHY ports, you’d probably be more power efficient with even just power on a backplane.

geerlingguy 5y ago
I think it's mostly convention and convenience—in the Pi ecosystem, most people using these things headless or as small servers are used to powering with PoE already. Networking gear is often powered by PoE, but most servers have higher power requirements so a custom backplane or separate power method would be required. Boards like the Turing Pi v2 (so far just prototypes and a marketing page on a website) would have a built-in network switch and power backplane (direct to each CM4), and I'm guessing another board or two like it will appear someday. I hope.
- h2odragon 5y ago
  Is there somewhere after the POE that power could be tapped in? Even just a pair of pins for 5v sounds like it might make for significant power savings.
- mbreese 5y ago
  No, I get that... and there is a different level between hobbyist projects and larger production projects. Using the existing PoE "infrastructure" makes a lot of sense in the prototyping stages. But if you're already going through the process to build custom boards for a cluster, then a backplane makes much more sense. We're not talking about a small 4-5 RPi cluster where everything can work off of a single PoE gigabit switch with a rat's nest of cat6. This project is talking about 16 of these blades in a 1U chassis. If you're going to go that far, a backplane isn't a big leap and would save power, space, and make cabling much easier.

bserge 5y ago

Wait, since when is using several separate machines an acceptable way of computing? And especially counting total RAM as if all of it can be used when needed? I've been doing that for a decade now (for cheapness mostly) and people said it was stupid and worse than a single computer with the same number of cores and RAM. Also would those chips need heatsinks and cooling?

toast0 5y ago
Using several separate machines is what you have to do when your computing needs don't fit in a single computer. Of course, the total computing power here would fit in a single x86 computer, but it can be fun to play with clustering and this might be a lot less expensive than an x86 cluster.
908B64B197 5y ago
> Wait, since when is using several separate machines an acceptable way of computing? And especially counting total RAM as if all of it can be used when needed? It brings joy to Pi owners. There's more to the Pi than just the hardware.
gh02t 5y ago
Supercomputers have been built like that for a long time (decades) now, albeit with ultra-fast networking and storage to back them. There is a whole lot of research and money poured into specialized algorithms and software optimized for distributed machines. And that's far the only form of distributed computing in use.

jdoss 5y ago

I know this the Pi home lab hardware ecosystem is not the datacenter ecosystem, but what I really want on a Pi is not TPM but IPMI for remote management.

geerlingguy 5y ago
Ping @merocle on Twitter—the board's design is not 100% final, and there may be a few other features he could cram in (I've already thought of moving at least the serial UART connection to the front...).
mbreese 5y ago
That’s funny. Because one of the more legitimate uses for a Pi in a data center is as a remote management KVM. https://pikvm.org/ https://pikvm.org/

anoonmoose 5y ago

Getting kind of sick of OP reviewing products that we can't buy and, imo, probably won't ever be able to buy. Rather, sick of it making it to the HN front page. https://news.ycombinator.com/item?id=27460885 https://news.ycombinator.com/item?id=27460885

geerlingguy 5y ago
Most of these products are able to be built by hobbyists (in fact, almost all of them are produced by hobbyists). The reason most fail to launch to production is not only the extreme difficulty of scaling from prototype to manufactured production quantities of product, but also from 2020 onwards, the chip shortages. For each product I've been able to actually touch, there are ten more I've heard about and wanted to see happen, but they just couldn't be made because certain specialty ICs are just not available to small-time hobbyists/makers. I don't tend to review these boards because I want people to go out and buy them. I review them because I am inspired to try out new ideas and try my hand at new things (circuit designs, PCB design, etc.), and I figure maybe some other people can get inspired to try too. I try to feature as much open source hardware as possible, because it actually is possible for people to get custom PCBs and put together their own versions of them.
jimmies 5y ago
Being off-the-shelf is an added bonus, but not a requirement. This website is called Hacker news, not Bestbuy news. As long as someone else can replicate the results from what is given in the post, I do think the post has enough merit.
rsync 5y ago
Ugh, what ? I love seeing the Jeff Geerling posts and his blog is great!
wila 5y ago
You don't have to read his posts, the base URL is right next to the title. It is very easy to skip an article if it is not your cup of tea. Personally I love the posts of Jeff as I think they are very well written and well researched.

diarmuidc 5y ago

"The board gets fairly warm, likely due to the overhead of the PoE+ power conversion (which consumes 6-7W on its own!" 7 W of losses? That's either wrong or insane for DC/DC regulation. Rough figures, pi4CM (5W) + NVMe (2W) = 7W. So regulation of 50% efficiency?

geerlingguy 5y ago
I found the same to be the case on the official Pi 4 PoE+ HAT (the new 2021 version)[1]. [1] https://www.jeffgeerling.com/blog/2021/review-raspberry-pis-poe-hat-june-2021 https://www.jeffgeerling.com/blog/2021/review-raspberry-pis-...
shadowpho 5y ago
PoE can't do "purely" DC/DC regulation. They require (in any reasonable setup) a transformer for isolation on both ends, which lowers efficiency, especially at lower end. Now with that said you can definitely get decent efficiency with PoE, it's just that most people don't care and so it's not as big of a focus.

karmicthreat 5y ago

What I want to do for a funsies project is have a bunch of Pi4 CMs plugged into a PCIE switch and a multi-host 10gige adapter. I don't think this is possible though, I think I read somewhere that the Pi4 PCIE Root doesn't support host<->host comms. If anyone knows different, speak up!

rsync 5y ago

Can we talk about the 10" rack please ? It appears that the rack posts are just chopped/hacked from some other rack ... but where are people sourcing these little 10" wide rack shelves ? Also, the bottom component - is that a patch panel or a switch ?

youngtaff 5y ago
Bottom component looks like a patch panel. Plenty of 10” kit about e.g. https://datacabinetsdirect.co.uk/soho-10-inch-data-network-rack-cabinets/10-inch-soho-data-cabinet-accessories.html https://datacabinetsdirect.co.uk/soho-10-inch-data-network-r... 10” soho rack appears to be magic search phrase I've see a rack similar to the in the article before and can't find it now, but one way to recreate would be to buy soemthing like this a use a 10” blanking plate instead of the 19” one https://www.amazon.com/Procraft-Desktop-System-Washers-DTR-10/dp/B01CR74Q1C https://www.amazon.com/Procraft-Desktop-System-Washers-DTR-1... Lots of music gear shops sell desktop racks BTW
Merocle 5y ago
I used this one: https://www.rack-magic.com/Mini-Rack-4826mm-19-Rack-Stand-6HE-9HE-10-Zoll https://www.rack-magic.com/Mini-Rack-4826mm-19-Rack-Stand-6H...

ajklsdhfniuwehf 5y ago

the only good TPM can bring to this world is to entice people to finally hack a open source bootloader for the pi :)

dheera 5y ago

> Integrated PoE+ with Gigabit Ethernet Are there any good PoE hats for Pi that won't melt everything around it, and have spare power for USB?

exabrial 5y ago

Noob here... Would it be possible to do some sort of provisioned storage over PCI-E? So rather than each module having its own M2 drive, there is a PCIe connection on the backplane back to an SSD Raid.

jlawer 5y ago
For geek cred you could put in another Network or Storage adapter and do ISCSI or FC. SAS would allow 2 to share storage, however I don't think SAS supports more then 2 initiators. Directly connecting every host to a pool of Flash chips would likely require a custom controller for the flash and a way to present it to each Raspberry Pi This will almost certainly not make sense economically, but it seems a lot around RPi clustering doesn't make sense from a performance/$ perspective.