27 ms·

Building an ARM64 home server the hard way

258 points by jforberg 4y ago

prmoustache 4y ago
What's with arm sbc users not setting up any disk encryption? I have yet to see a guide / tutorial / experience reports that do not totally skip the subject.
- EnigmaCurry 4y ago
  Check this out, even has SSH unlock - https://gist.github.com/devgioele/e897c341b8d1c18d58b44ffe21d72cf6 https://gist.github.com/devgioele/e897c341b8d1c18d58b44ffe21...
  - prmoustache 4y ago
    I had setup mine with tiny-ssh because last time I checked dropbear only supported rsa keys. My point was not that I was looking for a guide, but I have the feeling that subject is totally ignored by most of the sbc-arm community. I take the risk of being victim of burglary as a when, not an if. I totally don't want anyone to have easy access to my data. If an SBC is used for server purpose and not for embedded/domotic use, it will likely contain data and/or secrets.
daneel_w 4y ago
For the Pine family of SBCs I highly recommend installing Tow-Boot - https://tow-boot.org/ https://tow-boot.org/ - on the SPI flash memory to allow yourself much better boot options, including booting directly from NVMe so you don't need to keep the MicroSD card plugged-in.
- Timon3 4y ago
  Does this have a mechanism for automatic redundant OS upgrades? I just built a Yocto-based distribution for a board based on rk3399, but the currently integrated U-Boot is not in the best state. This could be a great alternative if it really is a bit easier to integrate/build upon.
  - daneel_w 4y ago
    It's just a bootloader, with a few tricks up its sleeve. All it does is letting you boot from any storage medium you want (most notably NVMe) instead of being restricted to the hardcoded sequence of the boot ROM.
    - rjsw 4y ago
      It is a bootloader bundled with the ARM64 firmware. That means you don't need to add the firmware to each OS that you might want to use so it is easier to switch between them.
    - Timon3 4y ago
      I understand that part. What I'm talking about specifically is part of a bootloader, see for example this page in the documentation of SWUpdate: https://sbabic.github.io/swupdate/bootloader_interface.html https://sbabic.github.io/swupdate/bootloader_interface.html By adding communication between the OS and the bootloader it's possible to implement redundant updates for whole partitions (specifically A/B-updates with a boot counter). U-Boot supports this (depending on the state of the vendor-provided fork better or worse), and Tow-Boot seems to be based on U-Boot.
      - irusensei 4y ago
        The tow-boot software devs goes out of the way to say they are offering a boring PC boot loader experience so I wouldn’t expect any advanced features other than booting from devices.
        yencabulator 4y ago
        PC boot loaders have been able to fall back to previous configurations on boot failure ("automatic redundant OS upgrades") for a long time[1], so that's not a valid excuse. https://systemd.io/AUTOMATIC_BOOT_ASSESSMENT/ https://systemd.io/AUTOMATIC_BOOT_ASSESSMENT/ https://www.gnu.org/software/grub/manual/grub/html_node/fallback.html https://www.gnu.org/software/grub/manual/grub/html_node/fall... https://www.gnu.org/software/grub/manual/legacy/Booting-fallback-systems.html https://www.gnu.org/software/grub/manual/legacy/Booting-fall... [1]: Minimum about 23 years, from personal use in creating a product with A/B-root partitions.
- jforberg 4y ago
  Yes, I considered that and agree that it would have been nicer! I didn't pursue it for this project because my jury-rigged SD boot was working fine and I wanted to move on to other parts of the system.
  - throwaway173738 4y ago
    SD cards are pretty unreliable so it might be good to take it out of the loop at some point. Most field failures I’ve seen for products I develop have been related to SD cards dying during power loss or falling out due to vibration.
- throwaway173738 4y ago
  Does the U-boot in the SPI-NOR not support booting from NVMe? It might also be possible to patch that in from mainline if it exists. You can also often provide a “boot script” in the vfat partition that overrides the boot config in non volatile memory. This was something Freescale did with the i.MX6 that became a relatively standard thing for vendor-supplied U-boot.
  - daneel_w 4y ago
    The default setup does not support it for any Rockchip SoC up to and including the RK3399, which is why everyone goes for tow-boot on SPI for their Pine devices. The SoCs have a hardcoded boot sequence which includes SPI, SD, USB and eMMC, but not PCIe. It is however available since the RK3588 - e.g. the Radxa Rock5B boots from NVMe right off the bat.
- fellowmartian 4y ago
  Alternatively one could go for Barebox, it’s neat and standardized like Tow-boot, but it’s not phone-focused and works on many more boards.
rcarmo 4y ago
I set up Proxmox on one of my Pi 4s (using an external SSD) and am quite happy with it. Runs four different LXC containers (one of which is a public-facing ActivityPub server for testing) and gives me zero headaches, so am currently looking for a beefier alternative that has a proper M.2 slot and at least 16GB of RAM... I do wish that alternative boards had better OS support (especially the Rockchip ones, which tend to have weird kernel builds, etc),
- lostlogin 4y ago
  It’s a bit of a sledgehammer approach, but a Nuc with Proxmox is pretty excellent. You can even use 10gbe via thunderbolt (or the pci slot on the larger Nucs).
  - doublepg23 4y ago
    This is the way. An x86 NUC/Mini PC/Thin client smokes the ARM SBC market.
    - radiator 4y ago
      Are there any boxes without a graphics card or at least with a primitive one? So that they would be more economical in case you only need them for the role of a server?
      - Saris 4y ago
        Pretty much all of the small boxes just use the iGPU.
        lostlogin 4y ago
        And that iGPU is great for a Plex server for transcoding, especially if it has QuickSync for hardware acceleration.
      - TheCoelacanth 4y ago
        None of them have a dedicated GPU, just a low-powered one built into the CPU. Using a CPU without an integrated GPU would only shave a few dollars off the cost, so I don't that is a common choice.
kristianpaul 4y ago
Get a 16G rockb5b board
- jforberg 4y ago
  I probably would have if it had been available at the time. This project was actually done last spring/summer.
andai 4y ago
Somewhat tangential but is Arch really suitable for servers? Most Arch users I know still prefer Debian for servers. Yet I know at least one company that uses them for servers, which surprised me. I know the Arch breaking meme is overblown but for a server I'd still want something with less moving parts.
- jforberg 4y ago
  If this was going to be used for some "big and serious" application, maybe different choices would have been made. Hopefully it was clear from the post that my goals here were the exact opposite! In my own anecdotal experience of running a hobby server on Arch for several years, I haven't experienced anything to make me think the distro is unsuitable for server work.
  - joshuakogut 4y ago
    I've used debian for almost twenty years and arch for over half that, despite being comfortable with Arch, I would not sleep well at night if anything mission critical depended on it. You install a rig with Debian on it and touch nothing, it will last longer than you.
    - megous 4y ago
      No, it will not last longer than you. If you want security updates, you have to do major ditro upgrades when the support for previous Debian release is over. And these are somewhat tricky. You'll just have your upgrade dance clustered into a single event once every N years, instead of spread over randomly, based on major releases of SW your server depends on.
- bionade24 4y ago
  That big release upgrades provide more hassle than benifits was also observed by Google, hence they switched to rolling release. The reason Debian breaks more at release changes though is probably more due to them patching and modifying software, which they have sometimes have to change/drop with a new SW release. Or you have a hard time deploying a newer SW version on top of the old binaries. Arch follows upstream very close, which maybe increases the times things could/have to be reconfigured, but it still mostly means running vimdiff against config.conf and config.conf.pac{new,save}. Sure Debian is more reliable if you want do want to really change deployed systems, but if your company strategy is to keep up with upstream, Arch may work better than it's reputation. And if you'd need stability once, you could just set Archive on a specific day as package mirror on your cache server.
sylware 4y ago
Got myself a Pi, a plastic box, a memory card, a big usb key, wrote my own SMTP server in super lean no-libc C (c89 with benign bit of c99/c11), put a devuan GNU/linux (NOT debian with its toxic trashy bloat and kludge of systemd). I did the same thing with a nanomimal http server to serve static content and maybe dynamic in the future: a noscript/basic (x)html http server for maps (which uses openstreet map tiles), which does provide proper map display in links2, with a font not too big, and with harmless html tables. Configured the "server" to restart everything if something is detected missing (you know, cron with SH scripts and certainly not bash scripts). It has been running for years. I never had to modify the code of my smtp server, yet (and I run IPv4 and native IPv6 provided by default to millions of clients by my ISP, I think it has been the case for more than a decade, may be wrong about this one though). I am kind of surprise it was not already pown by some trashy hackers. The main issue: spamhaus block lists, they are hostile to all self-hosted people and they don't provide a irc server, or a non blocked email to be removed from their lists (which are unfortunately used by too many open source related companies/project, which is a mistake). Basically, they force ppl to use one of google/apple super heavy javascripted web engine (no better than the default security checks from cloudflare). Yes, those ppl are seriously worse than spam itself, hope they will fix that (they are a shaddy swiss-andoran company...). Did you know you cannot send an email to redhat(IBM now) people using an ipv6 smtp? yeah... And it is coming: I'll move everything to a similar RISC-V mini-computer because I am aware of the super toxic IP tied to arm64 ISA (same for x86_64), that will be the first step, the 2nd step will be to hand compile (=assembly programming with near Zero-SDK) all of them and forget this C syntax too complex and those horribly massive and complex compilers, not stable on the long run (thanks ISO, gcc extensions and c++). And with all that, I would not be surprise to port to 64bits RISC-V assembly a minimal IPv6 stack... and maybe more.
- traceroute66 4y ago
  > The main issue: spamhaus block lists, they are hostile to all self-hosted people Allow me to correct that for you. There is nothing wrong with spamhaus. They provide one of the best anti-spam options amongst all the commercial providers. Spamhaus have many lists, I suspect the one you are referring to is the PBL, in their words "DNSBL database of end-user IP address ranges which should not be delivering unauthenticated SMTP email to any Internet mail server except those provided for specifically by an ISP for that customer's use.". We are in 2023, I think it is beyond any sort of doubt by now that a significant proportion of spam and phishing mails originates from home internet connections because people can't be bothered to keep their computers up to date and virus free, so they become part of a botnet. So the fact of the matter is that even if Spamhaus PBL did not exist, someone else (or the MX operators themselves) would very soon fill their place by blocking the very same ranges. Added to which, most home ISPs don't even provide reverse DNS ... so again, even if Spamhaus PBL did not exist, you would likely STILL find yourself being blocked by other measures that most sensible sysadmins implement on their servers. Hell, many home ISPs just block outbound port 25 these days anyway !
jraph 4y ago
I've been hosting some websites on a RockPro64 board running armbian for 2 years. I'm quite happy with it. I recommend using an SSD and not an SD card though.
- rbanffy 4y ago
  I had issues with SD card that may be related to durability. A lot of it was mitigated by moving the /var/log folder to a tmpfs (if you don't care much for the logs, or are using something to ship them to another machine, you really don't mind them not being written to durable storage).
megous 4y ago
The hard way? Copy bootloader from somewhere, partition, extract readymade rootfs, setup bootloader, reboot. Sounds more like the Arch way. :) The only ARM specific thing here is probably the need to use a DTB. This just shows that manual Linux installation on random ARM board is not more complex than on x86_64. Perhaps even simpler, since you're just extracting a pre-made rootfs instead of using a package manager during installation.
- m463 4y ago
  Experts sometimes forget being beginners.
  - megous 4y ago
    "Hard" is relative, sure. But regular Arch Linux installation is harder than this in a sense, and the author is Arch Linux user on multiple devices already, so not a beginner. If I go through the article and list arm64 SBC specific pieces, it's just copying the booltloader from the manjaro image for rockpro64 and maybe different names of files in /dev for some block devices. Even configuring U-Boot is done the same way you do it on regular Arch, if you use extlinux on x86_64, which many do/did.
- anthomtb 4y ago
  Right? With that article title you figure the author had written his own bootloader in Typescript then transpiled to Rust, ultimately cross compiling to his Arm64 target from a homebrewed x86 CPU fabricated in his garage. For real though, what the author did is much harder than downloading and booting an official OS image from Pine. The article also documents all the successful steps and skips any missteps or debugging, making the process look very simple (not a criticism, I thought it was an excellent read). Maybe those missteps took place during previous projects, but suffice to say that you don’t make booting a non-standard image look easy without expending significant effort, at some time.
  - megous 4y ago
    Writing your own bootloader is an option if you want to play hard level, sure. I did follow that path, at one time https://xnux.eu/p-boot/ https://xnux.eu/p-boot/ for another Pine64 device. ;) Anyway, I didn't say the things you're criticizing my reply for.
nubinetwork 4y ago
That uboot is pretty old, but not as old as the one on my armada 8040 boards... they can't even boot a modern kernel properly without having to compile uboot and ATF and upgrading the firmware.
- megous 4y ago
  It's possible to just build the uptodate one. The support for rockpro64-rk3399 is mainline in U-Boot. The same for TF-A.
znpy 4y ago
> The total cost comes to around €350 A few weeks ago I bought an used intel nuc7 with a 7th gen core i5… for 150€. It came with a 120gb ssd, 4gb ram and a power brick. I still don’t see the value in this SBCs used as home servers.
- tyingq 4y ago
  The Lenovo "Tiny" machines are also cheap on eBay, even one with a Ryzen 5 and 8Gb RAM for ~$140USD. Though there are reasons to specifically want an ARM64 machine for builds, etc.
  - candiddevmike 4y ago
    I just want something small with ECC RAM
    - tyingq 4y ago
      Not cheap, but does support ECC: https://morefine.com/products/morefine-s500-mini-pc https://morefine.com/products/morefine-s500-mini-pc The FAQ question reads a bit funny: "Q19:Support ECC RAM？ YES. S500+ Support ECC RAM， But compatibility requirements are higher, and cannot be used casually." Also, these AsRock products: https://www.asrockind.com/en-gb/4X4%20BOX-V1000M https://www.asrockind.com/en-gb/4X4%20BOX-V1000M https://www.asrockind.com/en-gb/4X4%20BOX-R1000M https://www.asrockind.com/en-gb/4X4%20BOX-R1000M
      - candiddevmike 4y ago
        Thank you for the links, that morefine one seems really nice, would need to get 2.5Gb networking gear... I currently have a single 16 core/64 GB ECC Ryzen tower with ZFS that I'd like to break into a ZFS ARM ECC NFS server with a separate small form factor Ryzen or ARM compute cluster.
        malteg 4y ago
        Honeycomb? https://www.solid-run.com/arm-servers-networking-platforms/honeycomb-servers-workstation/#overview https://www.solid-run.com/arm-servers-networking-platforms/h... I think they also have a lite version supporting ECC…
tbrock 4y ago
You could do this for cheaper (and with more ram) using a raspberry pi 4 with a usb nvme ssd, it’s got gigabit ethernet and is arm64. Sure you have two less cores than this solution but it’s more likely to be supported over time and once you get the SD card out of the mix the I/O is solid. I’ve been surprised by how much the SD card throughput was limiting the experience. I run arch Linux arm on mine and it’s a fantastic little device. I wonder if these boards are way faster or just more of the same. I guess the pcie expansion makes this more extensible.
- detrites 4y ago
  > "Looking at the available offerings [...]" The slight problem with the deservedly often-recommended RP4 is that for most people it's so hard to come by it effectively doesn't exist.
- johnklos 4y ago
  > cheaper No, you can't, unless you know of some source of retail priced Raspberry Pi 4s. In some basic tests (compiling, ffmpeg), the Pi 4 and the Rock Pro 64 are within a small percentage of difference in performance.
  - sliken 4y ago
    Indeed, but the much nicer/faster/better PCIe and faster CPUs in the RK3588 for another $25, with the option to get double the ram of the top RPi 4.
- megous 4y ago
  RK3399 is completely FOSS, from bootloader, to firmware, to Linux drivers and mainlined. It will be supported as long as someone wants to run software on it.
  - jamesy0ung 4y ago
    Raspberry Pi, despite not being fully FOSS, has a much better community supporting it. I’d bet it will last longer than the pine boards. Watched Jeff Geerling’s videos on it and he had trouble getting the pine board to work, whereas the Raspberry Pi worked first try. The only images for it were some images a guy made and put on the pine wiki.
_joel 4y ago
How is btrfs nowadays? I have PTSD from it and stick to ZFS now.
- graton 4y ago
  Synology and Fedora use btrfs by default. I've been using it for years and have had zero issues.
- prmoustache 4y ago
  Still as shitty as it ever was. I experienced some data loss recently on a btrfs fedora install.
johnklos 4y ago
I was hoping for something more akin to what the word "building" usually implies - something a bit more physical. If nothing else, the author made a box for it ;) The RockPro64 is a good board with lots of expandability. I run NetBSD/aarch64eb on one to build all of NetBSD's pkgsrc packages (26,000). It performs well with an m.2 NVMe, and has been rock solid. Of course, for anyone using the RockPro64, if you plan to do lots of processor intensive work like compiling, you'll either need a very large heat sink (no Flirc cases for these, unfortunately) or you'll need active cooling. Without good cooling, it'll throttle. https://klos.com/~john/rockpro64.jpeg https://klos.com/~john/rockpro64.jpeg
- e40 4y ago
  Yes, I was hoping to learn about inexpensive aarch64 server hardware solutions (ready made).
  - galangalalgol 4y ago
    I was thinking maybe they found a salvage sale for old oracle or aws aarch64 servers. I'd love one of those. A second best would be a jetson agx probably.
NelsonMinar 4y ago
Thank you for sharing these notes. I've been impatient for ARM64 hardware to become easy to use for home servers. I've got an RPi 4 (at retail prices, no less!) and it is quite good but I want more options. The previous stories I've read of RockPro64 have been much worse, it was nice to read this went relatively easily.
swtyshinytimmy 4y ago
hi, im looking towards a mesh network. is there a way to do it with these boards?
swtyshinytimmy 4y ago
any recommendation to build a mesh network? is it worthy to worry about?
older 4y ago
Don't know how you missed that but Pine64 has official EU store: https://pine64eu.com/ https://pine64eu.com/
- vincnetas 4y ago
  I had to check, but ALL products in that store are "out of stock". This does not look like useful store at the moment.
  - older 4y ago
    Sure, I just wanted to mention that it exists, not making any claims about its usefulness. Yes, they are running out of stock on a regular basis, but they usually have re-stocking dates, so you know what to expect. Also their main store often is out of stock on many items as well. That's just how they operate.
- jforberg 4y ago
  Maybe the store was recently opened? This project is from last year, I just finished the write-up now.
MobiusHorizons 4y ago
I have this same board running freebsd as a NAS. It has been a pretty great experience overall. My main gripe has been that I would really like to be able to run an nvme drive for a cache at the same time as SATA ports, and I haven't found any cost effective PCIE2.0 x4 switches that I could use for that purpose. There is an x1 switch for use with RP4, but it is a shame to lose all the bandwidth. I'm looking forward to someone making a NAS board on the new RK3588 since that has enough connectivity for everything I want.
- irusensei 4y ago
  Issue with FreeBSD on RockPro64 and similar boards is that the scheduler doesnt support big.little configurations therefore the OS doesnt make distinctions between the 2 A72 cores and the weaker A55s. https://wiki.freebsd.org/arm/RockChip#Known_issues https://wiki.freebsd.org/arm/RockChip#Known_issues
  - MobiusHorizons 4y ago
    It’s true, but performance has honestly been great. It’s still quite responsive over ssh under load.
drdaeman 4y ago
This seems to be low-power entry-level stuff. I'm curious, is there anything more serious - but less serious than some proper rack server hardware? Currently I'm running a home server on EPYC 3251 mini-ITX board, which I use to route 1GbE WAN and 10GbE LAN, serve as a NAS, and run a bunch of services all without it breaking a sweat, and leaving plenty of headroom shall I want to run more stuff there. It sits on my desk in a small-ish cubic Supermicro chassis and barely makes any noise beyond the normal HDD screeching. And it's an entry-level server-oriented board so I have proper LOM without having to throw in an IPKVM. I would fancy an ARMv8 machine - just for fun of it (and possibly better performance per watt) - but I think I can't get anything comparable from a RPi-level hardware. But the next "step" I see when searching for ARM servers are those fan-screaming behemoths you put in a rack in a proper server room, which is something I dread for a homelab, as I don't have a dedicated room for it. I've had a pleasure of WfH involving setting up some PowerEdges in my living room, was fun but extremely noisy. So I wonder, where are the middle grounds?
- jrockway 4y ago
  I'm interested too and have pretty much come to the same conclusion as you. There are workstations like this: https://store.avantek.co.uk/ampere-emag-64bit-arm-workstation.html https://store.avantek.co.uk/ampere-emag-64bit-arm-workstatio... It's a middle ground between Raspberry Pi and a 128 core they sell to cloud providers. For the money, you can probably get more work done with an amd64 workstation, unless you're paying someone to generate your electricity by riding a bicycle or something. (Cooling and power matter to cloud-scale datacenters, but not really for one computer in a room that you use to generate your income.)
  - drdaeman 4y ago
    Goodness gracious, five grand, only 3 drive bays, and that's without drives? My whole AMD machine (motherboard, RAM, PSU, NVMe SSD, case - basically everything except for the HDDs for the bulk storage) costed me slightly less than $2k, and I've surely overprovisioned.
- flyinghamster 4y ago
FullyFunctional 4y ago
It's a nice write up. but as much as I love whole-drive filesystems, in this case I would have used a partition table and used a fixed partition for the swap space. Not only is it (slightly) more efficient, it's also simpler than using a btrfs subvolume and remembering to +C the swapfile. I think the general approach is very good and could probably be used for the VisionFive 2 RISC-V SBC as well.
rubatuga 4y ago
Unfortunately it doesn't have ECC, and uses a buggy file system BTRFS.
- sgt 4y ago
  BTRFS is buggy?
  - M95D 4y ago
    Yes, especially for external drives. See: https://lore.kernel.org/all/20200326013007.GS15123@merlins.org/T/ https://lore.kernel.org/all/20200326013007.GS15123@merlins.o... ... and many other problems.
    - sgt 4y ago
      I was literally recommended to use BTRFS because it has been battle tested and proven. I should probably have done more research but given its maturity (we're in 2023 after all), I feel a bit let down.
      - M95D 4y ago
        BTRFS is far better designed than other recent linux filesystems. It's just that linux always was and always will be very very buggy. If you want less bugs, use the oldest possible FS that is still usable. That's probably ext2. Most problems with btrfs is that it fails to mount in many situations and there's no automated fsck to fix it, requiring manual intervention; in fact running fsck on btrfs is considered a very bad thing. This makes it OK for desktop, but not at all suitable for headless servers and other unsupervised machines. Speaking of bad design, F2FS for example, a filesystem designed for flash drives, keeps both primary and backup superblocks in the same flash erase block. If that block gets corrupted the entire fs is lost.
        sgt 4y ago
        That's interesting. I've always thought ext3fs would at least be safer than ext2 seeing that ext2 isn't a journaling filesystem. What happens if you abruptly shut it down e.g. a 1000 times.
jbirer 4y ago
This article was a joy to read, thank you.
voidmain0001 4y ago
Do it the easy way with Oracle cloud’s free tier and get Arm Ampere A1 CPUs, 24GB RAM, and 10TB egress with the hard part being creating an account which requires a credit card. https://www.oracle.com/cloud/free/ https://www.oracle.com/cloud/free/
- lockhouse 4y ago
  This sounds awesome but the idea of giving Oracle my credit card is terrifying.
M95D 4y ago
I tried the really hard way with a RK3288 (Asus TinkerBoard). After about a month I had a barely working uboot built from unpatched official sources. After two months I still didn't have a bootable kernel built from unpatched official sources. - with power regulator drivers, the board powers itself off while booting - without power regulator drivers, it boots the kernel, but there's no power to usb, ethernet and wifi. What I learned: To stay away from Rockchip.
disdi89 4y ago
check this out - https://www.kickstarter.com/projects/uptimelab/compute-blade https://www.kickstarter.com/projects/uptimelab/compute-blade I am waiting for this to you with my RPI Compute Module 4

258 points by jforberg 4y ago

What's with arm sbc users not setting up any disk encryption? I have yet to see a guide / tutorial / experience reports that do not totally skip the subject.

EnigmaCurry 4y ago
Check this out, even has SSH unlock - https://gist.github.com/devgioele/e897c341b8d1c18d58b44ffe21d72cf6 https://gist.github.com/devgioele/e897c341b8d1c18d58b44ffe21...
- prmoustache 4y ago
  I had setup mine with tiny-ssh because last time I checked dropbear only supported rsa keys. My point was not that I was looking for a guide, but I have the feeling that subject is totally ignored by most of the sbc-arm community. I take the risk of being victim of burglary as a when, not an if. I totally don't want anyone to have easy access to my data. If an SBC is used for server purpose and not for embedded/domotic use, it will likely contain data and/or secrets.

daneel_w 4y ago

For the Pine family of SBCs I highly recommend installing Tow-Boot - https://tow-boot.org/ https://tow-boot.org/ - on the SPI flash memory to allow yourself much better boot options, including booting directly from NVMe so you don't need to keep the MicroSD card plugged-in.

Timon3 4y ago
Does this have a mechanism for automatic redundant OS upgrades? I just built a Yocto-based distribution for a board based on rk3399, but the currently integrated U-Boot is not in the best state. This could be a great alternative if it really is a bit easier to integrate/build upon.
- daneel_w 4y ago
  It's just a bootloader, with a few tricks up its sleeve. All it does is letting you boot from any storage medium you want (most notably NVMe) instead of being restricted to the hardcoded sequence of the boot ROM.
  - rjsw 4y ago
    It is a bootloader bundled with the ARM64 firmware. That means you don't need to add the firmware to each OS that you might want to use so it is easier to switch between them.
  - Timon3 4y ago
    I understand that part. What I'm talking about specifically is part of a bootloader, see for example this page in the documentation of SWUpdate: https://sbabic.github.io/swupdate/bootloader_interface.html https://sbabic.github.io/swupdate/bootloader_interface.html By adding communication between the OS and the bootloader it's possible to implement redundant updates for whole partitions (specifically A/B-updates with a boot counter). U-Boot supports this (depending on the state of the vendor-provided fork better or worse), and Tow-Boot seems to be based on U-Boot.
    - irusensei 4y ago
      The tow-boot software devs goes out of the way to say they are offering a boring PC boot loader experience so I wouldn’t expect any advanced features other than booting from devices.
      - yencabulator 4y ago
        PC boot loaders have been able to fall back to previous configurations on boot failure ("automatic redundant OS upgrades") for a long time[1], so that's not a valid excuse. https://systemd.io/AUTOMATIC_BOOT_ASSESSMENT/ https://systemd.io/AUTOMATIC_BOOT_ASSESSMENT/ https://www.gnu.org/software/grub/manual/grub/html_node/fallback.html https://www.gnu.org/software/grub/manual/grub/html_node/fall... https://www.gnu.org/software/grub/manual/legacy/Booting-fallback-systems.html https://www.gnu.org/software/grub/manual/legacy/Booting-fall... [1]: Minimum about 23 years, from personal use in creating a product with A/B-root partitions.
jforberg 4y ago
Yes, I considered that and agree that it would have been nicer! I didn't pursue it for this project because my jury-rigged SD boot was working fine and I wanted to move on to other parts of the system.
- throwaway173738 4y ago
  SD cards are pretty unreliable so it might be good to take it out of the loop at some point. Most field failures I’ve seen for products I develop have been related to SD cards dying during power loss or falling out due to vibration.
throwaway173738 4y ago
Does the U-boot in the SPI-NOR not support booting from NVMe? It might also be possible to patch that in from mainline if it exists. You can also often provide a “boot script” in the vfat partition that overrides the boot config in non volatile memory. This was something Freescale did with the i.MX6 that became a relatively standard thing for vendor-supplied U-boot.
- daneel_w 4y ago
  The default setup does not support it for any Rockchip SoC up to and including the RK3399, which is why everyone goes for tow-boot on SPI for their Pine devices. The SoCs have a hardcoded boot sequence which includes SPI, SD, USB and eMMC, but not PCIe. It is however available since the RK3588 - e.g. the Radxa Rock5B boots from NVMe right off the bat.
fellowmartian 4y ago
Alternatively one could go for Barebox, it’s neat and standardized like Tow-boot, but it’s not phone-focused and works on many more boards.

rcarmo 4y ago

I set up Proxmox on one of my Pi 4s (using an external SSD) and am quite happy with it. Runs four different LXC containers (one of which is a public-facing ActivityPub server for testing) and gives me zero headaches, so am currently looking for a beefier alternative that has a proper M.2 slot and at least 16GB of RAM... I do wish that alternative boards had better OS support (especially the Rockchip ones, which tend to have weird kernel builds, etc),

lostlogin 4y ago
It’s a bit of a sledgehammer approach, but a Nuc with Proxmox is pretty excellent. You can even use 10gbe via thunderbolt (or the pci slot on the larger Nucs).
- doublepg23 4y ago
  This is the way. An x86 NUC/Mini PC/Thin client smokes the ARM SBC market.
  - radiator 4y ago
    Are there any boxes without a graphics card or at least with a primitive one? So that they would be more economical in case you only need them for the role of a server?
    - Saris 4y ago
      Pretty much all of the small boxes just use the iGPU.
      - lostlogin 4y ago
        And that iGPU is great for a Plex server for transcoding, especially if it has QuickSync for hardware acceleration.
    - TheCoelacanth 4y ago
      None of them have a dedicated GPU, just a low-powered one built into the CPU. Using a CPU without an integrated GPU would only shave a few dollars off the cost, so I don't that is a common choice.

kristianpaul 4y ago

Get a 16G rockb5b board

jforberg 4y ago
I probably would have if it had been available at the time. This project was actually done last spring/summer.

andai 4y ago

Somewhat tangential but is Arch really suitable for servers? Most Arch users I know still prefer Debian for servers. Yet I know at least one company that uses them for servers, which surprised me. I know the Arch breaking meme is overblown but for a server I'd still want something with less moving parts.

jforberg 4y ago
If this was going to be used for some "big and serious" application, maybe different choices would have been made. Hopefully it was clear from the post that my goals here were the exact opposite! In my own anecdotal experience of running a hobby server on Arch for several years, I haven't experienced anything to make me think the distro is unsuitable for server work.
- joshuakogut 4y ago
  I've used debian for almost twenty years and arch for over half that, despite being comfortable with Arch, I would not sleep well at night if anything mission critical depended on it. You install a rig with Debian on it and touch nothing, it will last longer than you.
  - megous 4y ago
    No, it will not last longer than you. If you want security updates, you have to do major ditro upgrades when the support for previous Debian release is over. And these are somewhat tricky. You'll just have your upgrade dance clustered into a single event once every N years, instead of spread over randomly, based on major releases of SW your server depends on.
bionade24 4y ago
That big release upgrades provide more hassle than benifits was also observed by Google, hence they switched to rolling release. The reason Debian breaks more at release changes though is probably more due to them patching and modifying software, which they have sometimes have to change/drop with a new SW release. Or you have a hard time deploying a newer SW version on top of the old binaries. Arch follows upstream very close, which maybe increases the times things could/have to be reconfigured, but it still mostly means running vimdiff against config.conf and config.conf.pac{new,save}. Sure Debian is more reliable if you want do want to really change deployed systems, but if your company strategy is to keep up with upstream, Arch may work better than it's reputation. And if you'd need stability once, you could just set Archive on a specific day as package mirror on your cache server.

sylware 4y ago

Got myself a Pi, a plastic box, a memory card, a big usb key, wrote my own SMTP server in super lean no-libc C (c89 with benign bit of c99/c11), put a devuan GNU/linux (NOT debian with its toxic trashy bloat and kludge of systemd). I did the same thing with a nanomimal http server to serve static content and maybe dynamic in the future: a noscript/basic (x)html http server for maps (which uses openstreet map tiles), which does provide proper map display in links2, with a font not too big, and with harmless html tables. Configured the "server" to restart everything if something is detected missing (you know, cron with SH scripts and certainly not bash scripts). It has been running for years. I never had to modify the code of my smtp server, yet (and I run IPv4 and native IPv6 provided by default to millions of clients by my ISP, I think it has been the case for more than a decade, may be wrong about this one though). I am kind of surprise it was not already pown by some trashy hackers. The main issue: spamhaus block lists, they are hostile to all self-hosted people and they don't provide a irc server, or a non blocked email to be removed from their lists (which are unfortunately used by too many open source related companies/project, which is a mistake). Basically, they force ppl to use one of google/apple super heavy javascripted web engine (no better than the default security checks from cloudflare). Yes, those ppl are seriously worse than spam itself, hope they will fix that (they are a shaddy swiss-andoran company...). Did you know you cannot send an email to redhat(IBM now) people using an ipv6 smtp? yeah... And it is coming: I'll move everything to a similar RISC-V mini-computer because I am aware of the super toxic IP tied to arm64 ISA (same for x86_64), that will be the first step, the 2nd step will be to hand compile (=assembly programming with near Zero-SDK) all of them and forget this C syntax too complex and those horribly massive and complex compilers, not stable on the long run (thanks ISO, gcc extensions and c++). And with all that, I would not be surprise to port to 64bits RISC-V assembly a minimal IPv6 stack... and maybe more.

traceroute66 4y ago
> The main issue: spamhaus block lists, they are hostile to all self-hosted people Allow me to correct that for you. There is nothing wrong with spamhaus. They provide one of the best anti-spam options amongst all the commercial providers. Spamhaus have many lists, I suspect the one you are referring to is the PBL, in their words "DNSBL database of end-user IP address ranges which should not be delivering unauthenticated SMTP email to any Internet mail server except those provided for specifically by an ISP for that customer's use.". We are in 2023, I think it is beyond any sort of doubt by now that a significant proportion of spam and phishing mails originates from home internet connections because people can't be bothered to keep their computers up to date and virus free, so they become part of a botnet. So the fact of the matter is that even if Spamhaus PBL did not exist, someone else (or the MX operators themselves) would very soon fill their place by blocking the very same ranges. Added to which, most home ISPs don't even provide reverse DNS ... so again, even if Spamhaus PBL did not exist, you would likely STILL find yourself being blocked by other measures that most sensible sysadmins implement on their servers. Hell, many home ISPs just block outbound port 25 these days anyway !

jraph 4y ago

I've been hosting some websites on a RockPro64 board running armbian for 2 years. I'm quite happy with it. I recommend using an SSD and not an SD card though.

rbanffy 4y ago
I had issues with SD card that may be related to durability. A lot of it was mitigated by moving the /var/log folder to a tmpfs (if you don't care much for the logs, or are using something to ship them to another machine, you really don't mind them not being written to durable storage).

megous 4y ago

The hard way? Copy bootloader from somewhere, partition, extract readymade rootfs, setup bootloader, reboot. Sounds more like the Arch way. :) The only ARM specific thing here is probably the need to use a DTB. This just shows that manual Linux installation on random ARM board is not more complex than on x86_64. Perhaps even simpler, since you're just extracting a pre-made rootfs instead of using a package manager during installation.

m463 4y ago
Experts sometimes forget being beginners.
- megous 4y ago
  "Hard" is relative, sure. But regular Arch Linux installation is harder than this in a sense, and the author is Arch Linux user on multiple devices already, so not a beginner. If I go through the article and list arm64 SBC specific pieces, it's just copying the booltloader from the manjaro image for rockpro64 and maybe different names of files in /dev for some block devices. Even configuring U-Boot is done the same way you do it on regular Arch, if you use extlinux on x86_64, which many do/did.
anthomtb 4y ago
Right? With that article title you figure the author had written his own bootloader in Typescript then transpiled to Rust, ultimately cross compiling to his Arm64 target from a homebrewed x86 CPU fabricated in his garage. For real though, what the author did is much harder than downloading and booting an official OS image from Pine. The article also documents all the successful steps and skips any missteps or debugging, making the process look very simple (not a criticism, I thought it was an excellent read). Maybe those missteps took place during previous projects, but suffice to say that you don’t make booting a non-standard image look easy without expending significant effort, at some time.
- megous 4y ago
  Writing your own bootloader is an option if you want to play hard level, sure. I did follow that path, at one time https://xnux.eu/p-boot/ https://xnux.eu/p-boot/ for another Pine64 device. ;) Anyway, I didn't say the things you're criticizing my reply for.

nubinetwork 4y ago

That uboot is pretty old, but not as old as the one on my armada 8040 boards... they can't even boot a modern kernel properly without having to compile uboot and ATF and upgrading the firmware.

megous 4y ago
It's possible to just build the uptodate one. The support for rockpro64-rk3399 is mainline in U-Boot. The same for TF-A.

znpy 4y ago

> The total cost comes to around €350 A few weeks ago I bought an used intel nuc7 with a 7th gen core i5… for 150€. It came with a 120gb ssd, 4gb ram and a power brick. I still don’t see the value in this SBCs used as home servers.

tyingq 4y ago
The Lenovo "Tiny" machines are also cheap on eBay, even one with a Ryzen 5 and 8Gb RAM for ~$140USD. Though there are reasons to specifically want an ARM64 machine for builds, etc.
- candiddevmike 4y ago
  I just want something small with ECC RAM
  - tyingq 4y ago
    Not cheap, but does support ECC: https://morefine.com/products/morefine-s500-mini-pc https://morefine.com/products/morefine-s500-mini-pc The FAQ question reads a bit funny: "Q19:Support ECC RAM？ YES. S500+ Support ECC RAM， But compatibility requirements are higher, and cannot be used casually." Also, these AsRock products: https://www.asrockind.com/en-gb/4X4%20BOX-V1000M https://www.asrockind.com/en-gb/4X4%20BOX-V1000M https://www.asrockind.com/en-gb/4X4%20BOX-R1000M https://www.asrockind.com/en-gb/4X4%20BOX-R1000M
    - candiddevmike 4y ago
      Thank you for the links, that morefine one seems really nice, would need to get 2.5Gb networking gear... I currently have a single 16 core/64 GB ECC Ryzen tower with ZFS that I'd like to break into a ZFS ARM ECC NFS server with a separate small form factor Ryzen or ARM compute cluster.
      - malteg 4y ago
        Honeycomb? https://www.solid-run.com/arm-servers-networking-platforms/honeycomb-servers-workstation/#overview https://www.solid-run.com/arm-servers-networking-platforms/h... I think they also have a lite version supporting ECC…

tbrock 4y ago

You could do this for cheaper (and with more ram) using a raspberry pi 4 with a usb nvme ssd, it’s got gigabit ethernet and is arm64. Sure you have two less cores than this solution but it’s more likely to be supported over time and once you get the SD card out of the mix the I/O is solid. I’ve been surprised by how much the SD card throughput was limiting the experience. I run arch Linux arm on mine and it’s a fantastic little device. I wonder if these boards are way faster or just more of the same. I guess the pcie expansion makes this more extensible.

detrites 4y ago
> "Looking at the available offerings [...]" The slight problem with the deservedly often-recommended RP4 is that for most people it's so hard to come by it effectively doesn't exist.
johnklos 4y ago
> cheaper No, you can't, unless you know of some source of retail priced Raspberry Pi 4s. In some basic tests (compiling, ffmpeg), the Pi 4 and the Rock Pro 64 are within a small percentage of difference in performance.
- sliken 4y ago
  Indeed, but the much nicer/faster/better PCIe and faster CPUs in the RK3588 for another $25, with the option to get double the ram of the top RPi 4.
megous 4y ago
RK3399 is completely FOSS, from bootloader, to firmware, to Linux drivers and mainlined. It will be supported as long as someone wants to run software on it.
- jamesy0ung 4y ago
  Raspberry Pi, despite not being fully FOSS, has a much better community supporting it. I’d bet it will last longer than the pine boards. Watched Jeff Geerling’s videos on it and he had trouble getting the pine board to work, whereas the Raspberry Pi worked first try. The only images for it were some images a guy made and put on the pine wiki.

_joel 4y ago

How is btrfs nowadays? I have PTSD from it and stick to ZFS now.

graton 4y ago
Synology and Fedora use btrfs by default. I've been using it for years and have had zero issues.
prmoustache 4y ago
Still as shitty as it ever was. I experienced some data loss recently on a btrfs fedora install.

johnklos 4y ago

I was hoping for something more akin to what the word "building" usually implies - something a bit more physical. If nothing else, the author made a box for it ;) The RockPro64 is a good board with lots of expandability. I run NetBSD/aarch64eb on one to build all of NetBSD's pkgsrc packages (26,000). It performs well with an m.2 NVMe, and has been rock solid. Of course, for anyone using the RockPro64, if you plan to do lots of processor intensive work like compiling, you'll either need a very large heat sink (no Flirc cases for these, unfortunately) or you'll need active cooling. Without good cooling, it'll throttle. https://klos.com/~john/rockpro64.jpeg https://klos.com/~john/rockpro64.jpeg

e40 4y ago
Yes, I was hoping to learn about inexpensive aarch64 server hardware solutions (ready made).
- galangalalgol 4y ago
  I was thinking maybe they found a salvage sale for old oracle or aws aarch64 servers. I'd love one of those. A second best would be a jetson agx probably.

NelsonMinar 4y ago

Thank you for sharing these notes. I've been impatient for ARM64 hardware to become easy to use for home servers. I've got an RPi 4 (at retail prices, no less!) and it is quite good but I want more options. The previous stories I've read of RockPro64 have been much worse, it was nice to read this went relatively easily.

swtyshinytimmy 4y ago

hi, im looking towards a mesh network. is there a way to do it with these boards?

swtyshinytimmy 4y ago

any recommendation to build a mesh network? is it worthy to worry about?

older 4y ago

Don't know how you missed that but Pine64 has official EU store: https://pine64eu.com/ https://pine64eu.com/

vincnetas 4y ago
I had to check, but ALL products in that store are "out of stock". This does not look like useful store at the moment.
- older 4y ago
  Sure, I just wanted to mention that it exists, not making any claims about its usefulness. Yes, they are running out of stock on a regular basis, but they usually have re-stocking dates, so you know what to expect. Also their main store often is out of stock on many items as well. That's just how they operate.
jforberg 4y ago
Maybe the store was recently opened? This project is from last year, I just finished the write-up now.

MobiusHorizons 4y ago

I have this same board running freebsd as a NAS. It has been a pretty great experience overall. My main gripe has been that I would really like to be able to run an nvme drive for a cache at the same time as SATA ports, and I haven't found any cost effective PCIE2.0 x4 switches that I could use for that purpose. There is an x1 switch for use with RP4, but it is a shame to lose all the bandwidth. I'm looking forward to someone making a NAS board on the new RK3588 since that has enough connectivity for everything I want.

irusensei 4y ago
Issue with FreeBSD on RockPro64 and similar boards is that the scheduler doesnt support big.little configurations therefore the OS doesnt make distinctions between the 2 A72 cores and the weaker A55s. https://wiki.freebsd.org/arm/RockChip#Known_issues https://wiki.freebsd.org/arm/RockChip#Known_issues
- MobiusHorizons 4y ago
  It’s true, but performance has honestly been great. It’s still quite responsive over ssh under load.

drdaeman 4y ago

This seems to be low-power entry-level stuff. I'm curious, is there anything more serious - but less serious than some proper rack server hardware? Currently I'm running a home server on EPYC 3251 mini-ITX board, which I use to route 1GbE WAN and 10GbE LAN, serve as a NAS, and run a bunch of services all without it breaking a sweat, and leaving plenty of headroom shall I want to run more stuff there. It sits on my desk in a small-ish cubic Supermicro chassis and barely makes any noise beyond the normal HDD screeching. And it's an entry-level server-oriented board so I have proper LOM without having to throw in an IPKVM. I would fancy an ARMv8 machine - just for fun of it (and possibly better performance per watt) - but I think I can't get anything comparable from a RPi-level hardware. But the next "step" I see when searching for ARM servers are those fan-screaming behemoths you put in a rack in a proper server room, which is something I dread for a homelab, as I don't have a dedicated room for it. I've had a pleasure of WfH involving setting up some PowerEdges in my living room, was fun but extremely noisy. So I wonder, where are the middle grounds?

jrockway 4y ago
I'm interested too and have pretty much come to the same conclusion as you. There are workstations like this: https://store.avantek.co.uk/ampere-emag-64bit-arm-workstation.html https://store.avantek.co.uk/ampere-emag-64bit-arm-workstatio... It's a middle ground between Raspberry Pi and a 128 core they sell to cloud providers. For the money, you can probably get more work done with an amd64 workstation, unless you're paying someone to generate your electricity by riding a bicycle or something. (Cooling and power matter to cloud-scale datacenters, but not really for one computer in a room that you use to generate your income.)
- drdaeman 4y ago
  Goodness gracious, five grand, only 3 drive bays, and that's without drives? My whole AMD machine (motherboard, RAM, PSU, NVMe SSD, case - basically everything except for the HDDs for the bulk storage) costed me slightly less than $2k, and I've surely overprovisioned.
flyinghamster 4y ago

FullyFunctional 4y ago

It's a nice write up. but as much as I love whole-drive filesystems, in this case I would have used a partition table and used a fixed partition for the swap space. Not only is it (slightly) more efficient, it's also simpler than using a btrfs subvolume and remembering to +C the swapfile. I think the general approach is very good and could probably be used for the VisionFive 2 RISC-V SBC as well.

rubatuga 4y ago

Unfortunately it doesn't have ECC, and uses a buggy file system BTRFS.

sgt 4y ago
BTRFS is buggy?
- M95D 4y ago
  Yes, especially for external drives. See: https://lore.kernel.org/all/20200326013007.GS15123@merlins.org/T/ https://lore.kernel.org/all/20200326013007.GS15123@merlins.o... ... and many other problems.
  - sgt 4y ago
    I was literally recommended to use BTRFS because it has been battle tested and proven. I should probably have done more research but given its maturity (we're in 2023 after all), I feel a bit let down.
    - M95D 4y ago
      BTRFS is far better designed than other recent linux filesystems. It's just that linux always was and always will be very very buggy. If you want less bugs, use the oldest possible FS that is still usable. That's probably ext2. Most problems with btrfs is that it fails to mount in many situations and there's no automated fsck to fix it, requiring manual intervention; in fact running fsck on btrfs is considered a very bad thing. This makes it OK for desktop, but not at all suitable for headless servers and other unsupervised machines. Speaking of bad design, F2FS for example, a filesystem designed for flash drives, keeps both primary and backup superblocks in the same flash erase block. If that block gets corrupted the entire fs is lost.
      - sgt 4y ago
        That's interesting. I've always thought ext3fs would at least be safer than ext2 seeing that ext2 isn't a journaling filesystem. What happens if you abruptly shut it down e.g. a 1000 times.

jbirer 4y ago

This article was a joy to read, thank you.

voidmain0001 4y ago

Do it the easy way with Oracle cloud’s free tier and get Arm Ampere A1 CPUs, 24GB RAM, and 10TB egress with the hard part being creating an account which requires a credit card. https://www.oracle.com/cloud/free/ https://www.oracle.com/cloud/free/

lockhouse 4y ago
This sounds awesome but the idea of giving Oracle my credit card is terrifying.

M95D 4y ago

I tried the really hard way with a RK3288 (Asus TinkerBoard). After about a month I had a barely working uboot built from unpatched official sources. After two months I still didn't have a bootable kernel built from unpatched official sources. - with power regulator drivers, the board powers itself off while booting - without power regulator drivers, it boots the kernel, but there's no power to usb, ethernet and wifi. What I learned: To stay away from Rockchip.

disdi89 4y ago

check this out - https://www.kickstarter.com/projects/uptimelab/compute-blade https://www.kickstarter.com/projects/uptimelab/compute-blade I am waiting for this to you with my RPI Compute Module 4