4 ms·

Factor in a proper PSU, storage, case and cooling, amd64 boxes are cheaper than Pis now. Going to replace a pi 3 server with one of those, not having to deal wi

by lvass 3y ago

Factor in a proper PSU, storage, case and cooling, amd64 boxes are cheaper than Pis now. Going to replace a pi 3 server with one of those, not having to deal with ARM is a huge plus.

kkielhofner 3y ago
For most people with self-hosting tasks amd64 is back as the way to go. As you say, there are a ton of "minipcs" on the market that directly compete with the Raspberry Pi on cost and power usage. They're typically slightly larger but the expansion options (bring your own RAM/storage) plus real I/O (with real PCIe), disk, etc IMO significantly outweighs this. They're also typically more performant and while aarch64 platform support is increasing dramatically there are still the occasions where there's a project, docker container, etc that doesn't support it. Taking it a step further, there are a TON of decommissioned/recycled corporate/enterprise SFF desktops on the market. They don't compete in terms of size (13" x 15" or so) but they can actually get close in power usage. Many of them have multiple SATA ports, real NVMe, multiple real half-height PCIe slots, significantly better USB and PCIe bandwidth, etc. With my project Willow and Willow Inference Server[0] we're trying to drive this approach in the self-hosting community with an initial emphasis on Home Assistant. They're generally sick of Raspberry PI supply shortages, very limited performance, poor I/O, flaky SD cards, etc. The Raspberry Pi is still pretty popular for "my first Home Assistant" but generally once people get bitten by the self-hosting bug they end up looking more like homelab very quickly. For Willow particularly we emphasize use of GPUs because a voice assistant can't be waiting > 10 seconds to do speech recognition and speech synthesis. There are approaches out there trying to kind of get something working using Whisper tiny but in our ample internal testing and community feedback we feel that Whisper small is the bare minimum for voice assistant tasks, with many users going all out and using Whisper large-v2 at beam size 5. With GPU it's still so fast it doesn't really matter. The Raspberry Pi is especially poorly suited for this use case (and even amd64). We have some benchmarks here[1]. TLDR a ~seven year old Tesla P4 (single slot, slot power only, half-height, used for $70) does speech recognition 87x faster, with the multiple increasing for more complex models and longer speech segments. A 3.8 second voice command takes 586ms on the Tesla P4 and 51 seconds on the Raspberry Pi 4. Even with the Pi 5 being twice as fast that's still 25 seconds, which is completely unusable. Not fair to compare GPU to Raspberry Pi but consider the economics and practicality... You can get an SFF desktop and Tesla P4 from eBay for $200 shipped to your door. It will idle (with GPU and models loaded) at ~30 watts. The CPU, RAM, disk (NVMe), I/O, etc will walk all over a Raspberry Pi anything. Add the GPU and obviously it's not even close - you end up with a machine that can easily do 10x-100x what a Raspberry Pi can do for 2x the cost and power usage. You can even throw a 2.5gb Ethernet card in another slot for $20 and replace your router if you want to go really dense. Even factoring in power usage (10-15w vs 30, 2-3x) the cost difference comes down to nearly nothing and for many users this configuration is essentially future-proof to anything they may want to do for many years (my system with everything running maxes out around 50% of one core). Many also gradually grew their self-hosted situation over the years with people ending up with three or more Raspberry Pis for different tasks (PiHole, Home Assistant, Plex, etc). At this point the SFF configuration starts to pull far head in every way including power usage. Users were initially very skeptical to GPU use, likely from taking their experience in the desktop market and assuming things like "300 watt power usage with a huge > $500 card". Now they love having a GPU around for Willow and miscellaneous other CUDA tasks like encoding/decoding/transcoding with Plex/Jellyfin, accelerated Frigate, and all kinds of other applications. Willow Inference Server (depending on configuration) uses somewhere between 1-4GB of VRAM so with an 8GB VRAM card that leaves for plenty of additional tasks. We even have users who started with the Tesla P4 and then got the LLM bug and figured out how to get an RTX 3090 working with their setup which also of course leads to absurd performance with Willow - my local RTX 3090 goes from end of speech to command completion in HA to TTS feedback in ~250ms. It's "speak, blink, done" fast. [0] - https://heywillow.io/ https://heywillow.io/ [1] - https://heywillow.io/components/willow-inference-server/#benchmarks https://heywillow.io/components/willow-inference-server/#ben...
- squarefoot 3y ago
  The lack of GPIOs problem in Mini PCs could be solved with a cheap external USB->GPIO adapter such as this one: https://www.hardkernel.com/shop/usb-io-board/ https://www.hardkernel.com/shop/usb-io-board/ The above board was intended neither for the Raspberries nor for Mini PCs, but the code is Open Source and shouldn't be too hard to adapt.
  - kkielhofner 3y ago
    In the Home Assistant community most GPIO and other duties have migrated to ESP devices and the excellent esphome[0]. I have at least 10 devices around my home and it's fantastic, I haven't wired up GPIO to a Pi or anything other than a $5 ESP8266/ESP32 for years. Another instance of using the right tool for the job. [0] - https://esphome.io/index.html https://esphome.io/index.html
    - squarefoot 3y ago
      Yes, that would likely be an even better and cheaper solution if one doesn't need to drive GPIOs from a complex OS such as Linux which would require a fatter board than an ESP one.
      - kkielhofner 3y ago
        If you haven't tried it you may be surprised by what HA + esphome offers with the 8266/32. That said, the link you provided is potentially a great add-on for someone who wants/needs the logic available on a real Linux host!
- philsnow 3y ago
  Any leads on the ESP32-S3-BOX-3 SKU (the replacement for the ESP32-S3-BOX that Willow was developed for, IIUC)? I saw someplace say the only place to get it is Espressif's Aliexpress page for it, but they show out of stock for me. Thanks!
  - 3y ago
wingworks 3y ago
^ This. I moved away from my Pi years ago to an old 2nd hand Mac mini I got cheap online. Dealing with ARM has a huge pain for many things I did. Especially back then when docker and arm support was pretty limited. But there were other times where ARM were an issue too. I just don't want to (and sometimes can't) compile things.
comboy 3y ago
Any recommendations for these amd64 boxes?
- briHass 3y ago
  It's hard to go wrong with the Optiplex i5-6500 boxes that seem to have flooded the market recently. Amazon has them for $100 with 16GB RAM and a 256 SATA SSD. They have a NVME slot and PCIx16. They seem to be cheap, because that chip won't technically support Windows 11.
  - pcdoodle 3y ago
    These optiplex also idle around 5W and have more than enough horsepower for pretty much anything. I do like the pi for the long term form factor as bad as I think it is.
  - simfree 3y ago
    The nonstandard, low wattage power supplies combined with lack of room for full size dual slot (single PCIe x16 slot) cards limits the utility of these boxes :{
    - briHass 3y ago
      Sure, you can't fit a huge graphics card in there (power or space), but I'd argue that if running a full size graphics card and/or multiple 3.5" drives is your goal, a SFF isn't the right case. Pretty sure you can find an equivalent full-tower machine as well, though you do have the Dell proprietary PSU. I had mine running for a bit with long SATA cables snaked out of the case to a couple 3.5 drives in a makeshift enclosure, but SATA/NVMe drives have gotten so cheap, it calls into question the need for all the power.

by lvass 3y ago

Factor in a proper PSU, storage, case and cooling, amd64 boxes are cheaper than Pis now. Going to replace a pi 3 server with one of those, not having to deal with ARM is a huge plus.

kkielhofner 3y ago

For most people with self-hosting tasks amd64 is back as the way to go. As you say, there are a ton of "minipcs" on the market that directly compete with the Raspberry Pi on cost and power usage. They're typically slightly larger but the expansion options (bring your own RAM/storage) plus real I/O (with real PCIe), disk, etc IMO significantly outweighs this. They're also typically more performant and while aarch64 platform support is increasing dramatically there are still the occasions where there's a project, docker container, etc that doesn't support it. Taking it a step further, there are a TON of decommissioned/recycled corporate/enterprise SFF desktops on the market. They don't compete in terms of size (13" x 15" or so) but they can actually get close in power usage. Many of them have multiple SATA ports, real NVMe, multiple real half-height PCIe slots, significantly better USB and PCIe bandwidth, etc. With my project Willow and Willow Inference Server[0] we're trying to drive this approach in the self-hosting community with an initial emphasis on Home Assistant. They're generally sick of Raspberry PI supply shortages, very limited performance, poor I/O, flaky SD cards, etc. The Raspberry Pi is still pretty popular for "my first Home Assistant" but generally once people get bitten by the self-hosting bug they end up looking more like homelab very quickly. For Willow particularly we emphasize use of GPUs because a voice assistant can't be waiting > 10 seconds to do speech recognition and speech synthesis. There are approaches out there trying to kind of get something working using Whisper tiny but in our ample internal testing and community feedback we feel that Whisper small is the bare minimum for voice assistant tasks, with many users going all out and using Whisper large-v2 at beam size 5. With GPU it's still so fast it doesn't really matter. The Raspberry Pi is especially poorly suited for this use case (and even amd64). We have some benchmarks here[1]. TLDR a ~seven year old Tesla P4 (single slot, slot power only, half-height, used for $70) does speech recognition 87x faster, with the multiple increasing for more complex models and longer speech segments. A 3.8 second voice command takes 586ms on the Tesla P4 and 51 seconds on the Raspberry Pi 4. Even with the Pi 5 being twice as fast that's still 25 seconds, which is completely unusable. Not fair to compare GPU to Raspberry Pi but consider the economics and practicality... You can get an SFF desktop and Tesla P4 from eBay for $200 shipped to your door. It will idle (with GPU and models loaded) at ~30 watts. The CPU, RAM, disk (NVMe), I/O, etc will walk all over a Raspberry Pi anything. Add the GPU and obviously it's not even close - you end up with a machine that can easily do 10x-100x what a Raspberry Pi can do for 2x the cost and power usage. You can even throw a 2.5gb Ethernet card in another slot for $20 and replace your router if you want to go really dense. Even factoring in power usage (10-15w vs 30, 2-3x) the cost difference comes down to nearly nothing and for many users this configuration is essentially future-proof to anything they may want to do for many years (my system with everything running maxes out around 50% of one core). Many also gradually grew their self-hosted situation over the years with people ending up with three or more Raspberry Pis for different tasks (PiHole, Home Assistant, Plex, etc). At this point the SFF configuration starts to pull far head in every way including power usage. Users were initially very skeptical to GPU use, likely from taking their experience in the desktop market and assuming things like "300 watt power usage with a huge > $500 card". Now they love having a GPU around for Willow and miscellaneous other CUDA tasks like encoding/decoding/transcoding with Plex/Jellyfin, accelerated Frigate, and all kinds of other applications. Willow Inference Server (depending on configuration) uses somewhere between 1-4GB of VRAM so with an 8GB VRAM card that leaves for plenty of additional tasks. We even have users who started with the Tesla P4 and then got the LLM bug and figured out how to get an RTX 3090 working with their setup which also of course leads to absurd performance with Willow - my local RTX 3090 goes from end of speech to command completion in HA to TTS feedback in ~250ms. It's "speak, blink, done" fast. [0] - https://heywillow.io/ https://heywillow.io/ [1] - https://heywillow.io/components/willow-inference-server/#benchmarks https://heywillow.io/components/willow-inference-server/#ben...

squarefoot 3y ago
The lack of GPIOs problem in Mini PCs could be solved with a cheap external USB->GPIO adapter such as this one: https://www.hardkernel.com/shop/usb-io-board/ https://www.hardkernel.com/shop/usb-io-board/ The above board was intended neither for the Raspberries nor for Mini PCs, but the code is Open Source and shouldn't be too hard to adapt.
- kkielhofner 3y ago
  In the Home Assistant community most GPIO and other duties have migrated to ESP devices and the excellent esphome[0]. I have at least 10 devices around my home and it's fantastic, I haven't wired up GPIO to a Pi or anything other than a $5 ESP8266/ESP32 for years. Another instance of using the right tool for the job. [0] - https://esphome.io/index.html https://esphome.io/index.html
  - squarefoot 3y ago
    Yes, that would likely be an even better and cheaper solution if one doesn't need to drive GPIOs from a complex OS such as Linux which would require a fatter board than an ESP one.
    - kkielhofner 3y ago
      If you haven't tried it you may be surprised by what HA + esphome offers with the 8266/32. That said, the link you provided is potentially a great add-on for someone who wants/needs the logic available on a real Linux host!
philsnow 3y ago
Any leads on the ESP32-S3-BOX-3 SKU (the replacement for the ESP32-S3-BOX that Willow was developed for, IIUC)? I saw someplace say the only place to get it is Espressif's Aliexpress page for it, but they show out of stock for me. Thanks!
- 3y ago

wingworks 3y ago

^ This. I moved away from my Pi years ago to an old 2nd hand Mac mini I got cheap online. Dealing with ARM has a huge pain for many things I did. Especially back then when docker and arm support was pretty limited. But there were other times where ARM were an issue too. I just don't want to (and sometimes can't) compile things.

comboy 3y ago

Any recommendations for these amd64 boxes?

briHass 3y ago
It's hard to go wrong with the Optiplex i5-6500 boxes that seem to have flooded the market recently. Amazon has them for $100 with 16GB RAM and a 256 SATA SSD. They have a NVME slot and PCIx16. They seem to be cheap, because that chip won't technically support Windows 11.
- pcdoodle 3y ago
  These optiplex also idle around 5W and have more than enough horsepower for pretty much anything. I do like the pi for the long term form factor as bad as I think it is.
- simfree 3y ago
  The nonstandard, low wattage power supplies combined with lack of room for full size dual slot (single PCIe x16 slot) cards limits the utility of these boxes :{
  - briHass 3y ago
    Sure, you can't fit a huge graphics card in there (power or space), but I'd argue that if running a full size graphics card and/or multiple 3.5" drives is your goal, a SFF isn't the right case. Pretty sure you can find an equivalent full-tower machine as well, though you do have the Dell proprietary PSU. I had mine running for a bit with long SATA cables snaked out of the case to a couple 3.5 drives in a makeshift enclosure, but SATA/NVMe drives have gotten so cheap, it calls into question the need for all the power.