7 ms·

NixOS on ARM/Raspberry Pi

116 points by botayhard 6y ago

Luker88 6y ago
Tried it (rpi4), working. Even managed to encrypt the SD. Can't get nixops to work from my x86_64 notebook to deploy on the rpi though, I gave up on that. Nixos is really nice, But I find that the declarative conf still has a lot to iron out. Currently I only use it as a base system with declarative podman containers for pihole, freeipa (manually docker-built from the fedora image), postgres, nextcloud. Only nginx installed via nixos conf on the base system. I use podman containers since I could not find a way to have user-namespaced containers, or even just containers that don't share the whole /nix, which I find as a huge security risk. I really don't care about disk space. Duplicate everything but let me isolate those containers as much as possible. Podman containers give me read-only filesystem, too. Maybe nixops might have helped, but it would have prevented me from going read-only filesystem I tried using the declarative part of nixos to configure nginx, but I ended up dumping all the nginx conf on the single "extra" conf variable since proper declarative nginx vhosting with more than one location just does not let you do what you want (and is broken: do NOT reader my locations!!). Manual dump of the conf also for the cni network management of podman. wpa_supplicant most of the times starts too early and I didn't manage to make it run later, crashed. I am using ethernet without problems. So for me Big Kudos to the Nix(,Os,Ops) teams for the work done, I believe that is the future we should work towards. But we are still a far way off on many things. Documentation is also really scarce (and at rare times wrong), which increase the already steep learning curve. So sorry for the 'bad' review, but know that I still do not lose faith and will keep learning and trying. The basic idea is really that good for a sysadmin like me.
- aidenn0 6y ago
  I talked in the nixos aarch64 channel and cross-compiling is very hit-or-miss which is why nixops won't deploy well from an x86_64 machine. You can run a qemu aarch64 VM to delpoy if you need to.
- lloeki 6y ago
  I tried to love aarch64 NixOS very much for my pi 3 but, latest kernel is broken and doesn’t boot at all, previous (5.4) needs a firmware adjustment otherwise has HDMI issues, and 4.19 has no HDMI issue but is lacking on hardware support (no cpufreq scaling, sound output broken). This means the install process is painful as it requires a serial tty or typing blind commands to enable ssh. Also, once you get it running, if you ever screw up, booting the previous gen requires editing the bootloader files as it seems uboot is unable to initialise some usb keyboards properly (although they are listed). And finally, there seems to be no X support. I don’t want to sound harsh, I know it’s tough work, so I’m definitely extending my thanks to anyone working on that port.
  - aidenn0 6y ago
    There is definitely X support on Aarch64 nixos. I've run X on an Odroid C2 and a rockpro64. It does, however require putting the right drivers in your config, as the default is only sane on the pc platform
    - lloeki 6y ago
      Ah, when I tried to install it, nix loudly warned though a terminal message that it was entirely unsupported and bailed out. At that point I had enough issues that I bailed out at it's not practical yet for my pet projects.
bfrog 6y ago
I dream of a day nix can replace yocto
- foolmeonce 6y ago
  Mobile nixOS supports external device definitions so if enough examples of non-phones are made, I think a repo/project will begin to emerge.
amelius 6y ago
> Especially with the newer power-hungry Raspberry Pi families (3, 4), it is important to have a sufficient enough power supply or weirdness may happen. Weirdness may include: Lightning bolt on HDMI output "breaking" the display. Seriously?
- progval 6y ago
  They mean the lightning bolt icon, not an actual one. Image here: https://raspberrypi.stackexchange.com/questions/57963/what-does-the-lightning-bolt-mean https://raspberrypi.stackexchange.com/questions/57963/what-d...
novok 6y ago
What is a good rpi alternative with good software / distro support that supports on board mSATA or M.2 boards vs. SD cards?
- foolmeonce 6y ago
  The rockpro 64? But that is limited to 4gb, it looks like it's going to be another year† before pine64 is ready with newer SoCs that would be likely to support more RAM and they will again be missing a lot of kernel support. († well actually they will have prototypes shipping to some developers in the next months.)
  - deleted 6y ago
    [deleted]
- tra3 6y ago
  I’m curious about this too. I suspect the answer is a cheap x86 box. I haven’t seen anything with comparable support..
  - novok 6y ago
    I'm starting to suspect that too. The 2 or 3 that can do it are ~$50-$80, which is almost getting to x86 territory anyway.
    - tra3 6y ago
      This (https://www.aliexpress.com/item/4001316233680.html?srcSns=sns_Copy&spreadType=socialShare&bizType=ProductDetail&social_params=20007073225&tt=MG&aff_platform=default&sk=_mOHoT7X&aff_trace_key=c03d7be3b26349b5991a20d3d46aa536-1609028595646-03068-_mOHoT7X&shareId=20007073225&businessType=ProductDetail&platform=AE&terminal_id=1a396f3472c14afa8bd0f67f5afdd520 https://www.aliexpress.com/item/4001316233680.html?srcSns=sn...) is what my friend recommended. About double the price of a Pi but imminently more useful potentially.
- geerlingguy 6y ago
psanford 6y ago
I really want to use nixos on my RPis, but its not yet out of the "really painful" stage. There's no official arm package cache so you have to cross compile everything. Unfortunately, with nix this means running an ARM compiler inside an emulator, so it is very slow. When I finally got an image built the performance was much worse than rasbian or ubuntu. Kodi was basically unusable. I hope these issues eventually get fixed. Being able to build an sdcard image from a nix config would be a great way to manage a bunch of Raspberry Pis.
- Gaelan 6y ago
  If you're using Pi 3 and 4s, then there is an official aarch64 package cache. But yeah, older Pis (and the 0) are a PITA. Also, cross-compiling almost works now - I have a Pi 4 cross-compiling NixOS to armv6l for a Pi 0. But I'm using a clone of nixpkgs with several PRs merged, and I've taken care to avoid packages where cross doesn't work yet - definitely not plug-and-play. I imagine things will be a lot better in a year or so.
jnxx 6y ago
Has anybody tried Guix on top of Raspbian / Debian? What are the experiences?

116 points by botayhard 6y ago

Tried it (rpi4), working. Even managed to encrypt the SD. Can't get nixops to work from my x86_64 notebook to deploy on the rpi though, I gave up on that. Nixos is really nice, But I find that the declarative conf still has a lot to iron out. Currently I only use it as a base system with declarative podman containers for pihole, freeipa (manually docker-built from the fedora image), postgres, nextcloud. Only nginx installed via nixos conf on the base system. I use podman containers since I could not find a way to have user-namespaced containers, or even just containers that don't share the whole /nix, which I find as a huge security risk. I really don't care about disk space. Duplicate everything but let me isolate those containers as much as possible. Podman containers give me read-only filesystem, too. Maybe nixops might have helped, but it would have prevented me from going read-only filesystem I tried using the declarative part of nixos to configure nginx, but I ended up dumping all the nginx conf on the single "extra" conf variable since proper declarative nginx vhosting with more than one location just does not let you do what you want (and is broken: do NOT reader my locations!!). Manual dump of the conf also for the cni network management of podman. wpa_supplicant most of the times starts too early and I didn't manage to make it run later, crashed. I am using ethernet without problems. So for me Big Kudos to the Nix(,Os,Ops) teams for the work done, I believe that is the future we should work towards. But we are still a far way off on many things. Documentation is also really scarce (and at rare times wrong), which increase the already steep learning curve. So sorry for the 'bad' review, but know that I still do not lose faith and will keep learning and trying. The basic idea is really that good for a sysadmin like me.

aidenn0 6y ago
I talked in the nixos aarch64 channel and cross-compiling is very hit-or-miss which is why nixops won't deploy well from an x86_64 machine. You can run a qemu aarch64 VM to delpoy if you need to.
lloeki 6y ago
I tried to love aarch64 NixOS very much for my pi 3 but, latest kernel is broken and doesn’t boot at all, previous (5.4) needs a firmware adjustment otherwise has HDMI issues, and 4.19 has no HDMI issue but is lacking on hardware support (no cpufreq scaling, sound output broken). This means the install process is painful as it requires a serial tty or typing blind commands to enable ssh. Also, once you get it running, if you ever screw up, booting the previous gen requires editing the bootloader files as it seems uboot is unable to initialise some usb keyboards properly (although they are listed). And finally, there seems to be no X support. I don’t want to sound harsh, I know it’s tough work, so I’m definitely extending my thanks to anyone working on that port.
- aidenn0 6y ago
  There is definitely X support on Aarch64 nixos. I've run X on an Odroid C2 and a rockpro64. It does, however require putting the right drivers in your config, as the default is only sane on the pc platform
  - lloeki 6y ago
    Ah, when I tried to install it, nix loudly warned though a terminal message that it was entirely unsupported and bailed out. At that point I had enough issues that I bailed out at it's not practical yet for my pet projects.

bfrog 6y ago

I dream of a day nix can replace yocto

foolmeonce 6y ago
Mobile nixOS supports external device definitions so if enough examples of non-phones are made, I think a repo/project will begin to emerge.

amelius 6y ago

> Especially with the newer power-hungry Raspberry Pi families (3, 4), it is important to have a sufficient enough power supply or weirdness may happen. Weirdness may include: Lightning bolt on HDMI output "breaking" the display. Seriously?

progval 6y ago
They mean the lightning bolt icon, not an actual one. Image here: https://raspberrypi.stackexchange.com/questions/57963/what-does-the-lightning-bolt-mean https://raspberrypi.stackexchange.com/questions/57963/what-d...

novok 6y ago

What is a good rpi alternative with good software / distro support that supports on board mSATA or M.2 boards vs. SD cards?

foolmeonce 6y ago
The rockpro 64? But that is limited to 4gb, it looks like it's going to be another year† before pine64 is ready with newer SoCs that would be likely to support more RAM and they will again be missing a lot of kernel support. († well actually they will have prototypes shipping to some developers in the next months.)
- deleted 6y ago
  [deleted]
tra3 6y ago
I’m curious about this too. I suspect the answer is a cheap x86 box. I haven’t seen anything with comparable support..
- novok 6y ago
  I'm starting to suspect that too. The 2 or 3 that can do it are ~$50-$80, which is almost getting to x86 territory anyway.
  - tra3 6y ago
    This (https://www.aliexpress.com/item/4001316233680.html?srcSns=sns_Copy&spreadType=socialShare&bizType=ProductDetail&social_params=20007073225&tt=MG&aff_platform=default&sk=_mOHoT7X&aff_trace_key=c03d7be3b26349b5991a20d3d46aa536-1609028595646-03068-_mOHoT7X&shareId=20007073225&businessType=ProductDetail&platform=AE&terminal_id=1a396f3472c14afa8bd0f67f5afdd520 https://www.aliexpress.com/item/4001316233680.html?srcSns=sn...) is what my friend recommended. About double the price of a Pi but imminently more useful potentially.
geerlingguy 6y ago

psanford 6y ago

I really want to use nixos on my RPis, but its not yet out of the "really painful" stage. There's no official arm package cache so you have to cross compile everything. Unfortunately, with nix this means running an ARM compiler inside an emulator, so it is very slow. When I finally got an image built the performance was much worse than rasbian or ubuntu. Kodi was basically unusable. I hope these issues eventually get fixed. Being able to build an sdcard image from a nix config would be a great way to manage a bunch of Raspberry Pis.

Gaelan 6y ago
If you're using Pi 3 and 4s, then there is an official aarch64 package cache. But yeah, older Pis (and the 0) are a PITA. Also, cross-compiling almost works now - I have a Pi 4 cross-compiling NixOS to armv6l for a Pi 0. But I'm using a clone of nixpkgs with several PRs merged, and I've taken care to avoid packages where cross doesn't work yet - definitely not plug-and-play. I imagine things will be a lot better in a year or so.

jnxx 6y ago

Has anybody tried Guix on top of Raspbian / Debian? What are the experiences?