5 ms·

If you want to be able to reboot remotely, and non-interactively (i.e. while you sleep), I (and one other person) created Mandos for this purpose: <https://www.

by teddyh 8mo ago

If you want to be able to reboot remotely, and non-interactively (i.e. while you sleep), I (and one other person) created Mandos for this purpose: <https://www.recompile.se/mandos https://www.recompile.se/mandos>. It gets the password over the network. If this seems insecure to you, read the FAQ: <https://www.recompile.se/mandos/man/intro.8mandos https://www.recompile.se/mandos/man/intro.8mandos> Mandos works with initramfs images created by both initramfs-tools and Dracut, and is present in Debian since 2011, so no need to use a third-party package.

kro 8mo ago
Good FAQ, clearly stating the weak point of physical access. For a server that threatmodel can work, for a fleet of edge/iot devices in unsecured locations without permanent uptime there is no real solution to be expected without custom silicon logic (like in smartcards) on the soc.
akdev1l 8mo ago
How does this compare to: https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/security_hardening/configuring-automated-unlocking-of-encrypted-volumes-using-policy-based-decryption_security-hardening https://docs.redhat.com/en/documentation/red_hat_enterprise_...
- belorn 8mo ago
  (I am the other Mandos developer) I think FOSDEM had a talk a few years ago about that solution and their tang server. It is very similar in concept. They use McCallum-Relyea exchange, and we use PGP encryption, but the basics are the same in that you need information stored at the server and information stored in the unencrypted initramfs in order to be combined and turned into the encryption key. If my understanding of the McCallum-Relyea exchange is correct, they also combine the key creation and perfect forward security of the transaction into a single protocol, while we use TLS and client keys for the perfect forward security aspect and PGP for data at rest. With tang you need to verify the hash of the tang server in order to prevent a MITM threat from extracting the server side part during the key creation process, while with Mandos you configure it manually with configuration files. Tang uses REST and JSON. Mandos sends a single protocol number and then sends the data. Both operate primarily through local LAN, through Mandos client also support as an option to use global reachable ip addresses. Outside of those design differences, there are some packaging aspects. Tang was designed for Redhat/Dracut, while Mandos was designed for Debian and initramfs-tools (Today Mandos also support Dracut, but it is not packaged for Redhat). Redhat packaging has been requested multiple times for Mandos, but neither of us two developer are Redhat users. To add some historical context, I recall a Debconf BoF by the developers of initramfs-tools about if they were going to continue developing initramfs-tools or giving up and port everything to Dracut, and the silence was fairly deafening. People did not want to give up on initramfs-tools, but everyone recognized the massive duplication that those two project are. Similarly, the reason why Redhat had chosen initially to develop Dracut rather than just port initramfs-tools is also fairly big mystery, and is generally considered a Not Invented Here syndrome. Today there are however some distinct difference in design between the two systems.
  - rlpb 8mo ago
    clevis and tang do currently work seamlessly on Debian and Ubuntu using initramfs-tools. So while the initramfs-tools/dracut discussion is valid, it seems mostly orthogonal to this topic.
    - belorn 8mo ago
      I was unaware that they no longer depended on Dracut and now support initramfs-tools, which also seem to be the earliest clevis version that got packaged in Debian. That makes the initramfs-tools/dracut distinction a historical aspect of the project.
- teddyh 8mo ago
  Despite the “tang/clevis” system being too clever by half (and even more complex), it is essentially very similar. Mandos, though, is much simpler; you can actually understand it, since it uses standard components and protocols like DNS-SD¹, OpenPGP and TLS. (Mandos is documented in man pages, in contrast to page you linked, which is both huge, and – for me at least – keeps reloading itself and replacing the entire page with a 500 server error.) Mandos is also for Debian (and all Debian-based systems), and not Red Hat (although there is nothing preventing a port, since Mandos now also supports initramfs images created by Dracut). Finally, Mandos was initially created in in 2007, many years before tang/clevis, and literally by a person in a dorm room, not by whatever academics/scientists who seem to have dreamed up the cryptography/protocol tangle that is tang/clevis. Mandos has changed some since then, from initially using broadcast UDP and X.509 certificates, to using DNS-SD and raw public keys², but has otherwise remained very similar to its initial design. 1. <https://www.dns-sd.org/ https://www.dns-sd.org/> 2. <https://www.rfc-editor.org/rfc/rfc7250 https://www.rfc-editor.org/rfc/rfc7250>
koolba 8mo ago
Why is this needed at all? As the decrypted key is in memory before the reboot, can’t it just be written to a know location in memory and have kexec be instructed to read it early on?
- teddyh 8mo ago
  You’re assuming a controlled reboot. Mandos was initially created to deal with intermittent power failures. It’s also good for kernel panics.
  - koolba 8mo ago
    Oh for sure something is needed for a full start from zero. But the much more common case for a computer with backup power is regular restarts after applying patches that require a reboot. Would be much more pleasant for that to work out of the box with no manual interaction at all.
    - PunchyHamster 8mo ago
      There is plenty of cases of cold boot in datacenter. Also most distros don't support using kexec for kernel upgrades anyway.
- sillystuff 8mo ago
  > As the decrypted key is in memory before the reboot, can’t it just be written to a know location in memory and have kexec be instructed to read it early on? I set up what you are suggesting (sort of anyway[1]) on a personal VPS to reboot after updates, that require one. I just generate an initrd in tmpfs that contains a keyfile[2] and kexec with that ephemeral initrd; The newest kernel can be found by looking at what the /boot/vmlinuz symlink points to. Been running this for years. It is 100% reliable, and simple. And, for the purposes of this box, secure enough. For remote unlocks from initial power on, Debian has had that since forever using keyscripts and dropbear in the initrd. [1] You could pull the key from memory, and use that to unlock the disk from within the generated initrd, but it would be more work than just setting up a keyfile in advance. It was my first thought as well. [2] Easiest way was to use a mount namespace to use a diff crypttab file that points to the keyfile, since cannot specify crypttab location when creating the initrd. E.g., unshare --mount sh -c "mount --bind $CRYPTTAB_KEXEC $CRYPTTAB; mkinitramfs -o $kexec_initramfs $kernel" (mkinitramfs is usually wrapped by update-initramfs, but calling it directly allows specifying a location)
YorickPeterse 8mo ago
I haven't looked deeply into either, but how does this compare to the combination of Clevis and Tang that e.g. Red Hat/Fedora seems to favor?
neocron 8mo ago
Which server today doesnt have Raid? Just pull one hdd out, extract what you need or change the image. Then you turn off the server, and just start a vm with the captured init and capture the key. Now you can decrypt the server offline with all the time in the world.
- teddyh 8mo ago
  > Just pull one hdd out, That only works with RAID 1. If the server uses RAID 5 or RAID 6, this won’t work. > extract what you need Well, yes. This is addressed in the FAQ. > or change the image. > Then you turn off the server, and just start a vm with the captured init and capture the key. Well, as explained in the FAQ, an attacker will have to do so quickly, before the Mandos server decides that the Mandos client has been offline for too long, and disables that client. The default value is five minutes, but is configurable per client.
  - neocron 8mo ago
    Why wouldn't this work with Raid5? One of my raid5 hdds cought fire and the server was still running. Talking about mdadm in this case. 5 minutes is plenty to boot initrd from a vm... what's that gonna take? 10 seconds?
    - deleted 8mo ago
      [deleted]
    - Terr_ 8mo ago
      I assume they're saying it won't work because the suggestion is to "pull one" HDD out. With RAID 5, pulling 1/3 gives you only partial data, and pulling 2/3 removes too much so the system can't run.
      - neocron 8mo ago
        Ok, that is assuming /boot is ON the raid which I wouldn't want to rate for probability But even if it is, you could just pull one after the other and wait for the resilver before pulling the next one (you will hear if it resilvers automatically)
snthpy 8mo ago
Sounds great, I'm going to give this a try. Is there a plug-in for passkeys/webauthn for the mandos-server unlock?
- teddyh 8mo ago
  The whole architecure of Mandos is very plugin-based, it will therefore likely not be hard to add. But I am not sure what you are asking for? The Mandos server will, by default, unlock all clients, without asking. There is support for not unlocking immediately and instead wait for external approval for clients explicitly so configured, but what is the scenario? Is the client the one supplying the passkeys/webauthn? Or are you providing that manually on a web page somewhere? The latter is possible; the web page server process would then, when a passkey/webauthn has been verified, send a D-Bus “Approve” message to the Mandos server process, which would then send the client its password.
amelius 8mo ago
The kind of tool that I'm waiting for would allow me to log into a machine remotely, then reboot it once into a provided image (which could then run Linux in a ramdisk). I.e., take over the entire machine until it is restarted again. Does something like this exist? I want something like "kexec" but taking over the entire hardware at the lowest possible level. Edit, found this: https://github.com/marcan/takeover.sh https://github.com/marcan/takeover.sh But it's not as low level as I hoped, though it keeps the network running which is nice :)

by teddyh 8mo ago

kro 8mo ago

Good FAQ, clearly stating the weak point of physical access. For a server that threatmodel can work, for a fleet of edge/iot devices in unsecured locations without permanent uptime there is no real solution to be expected without custom silicon logic (like in smartcards) on the soc.

akdev1l 8mo ago

How does this compare to: https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/security_hardening/configuring-automated-unlocking-of-encrypted-volumes-using-policy-based-decryption_security-hardening https://docs.redhat.com/en/documentation/red_hat_enterprise_...

belorn 8mo ago
(I am the other Mandos developer) I think FOSDEM had a talk a few years ago about that solution and their tang server. It is very similar in concept. They use McCallum-Relyea exchange, and we use PGP encryption, but the basics are the same in that you need information stored at the server and information stored in the unencrypted initramfs in order to be combined and turned into the encryption key. If my understanding of the McCallum-Relyea exchange is correct, they also combine the key creation and perfect forward security of the transaction into a single protocol, while we use TLS and client keys for the perfect forward security aspect and PGP for data at rest. With tang you need to verify the hash of the tang server in order to prevent a MITM threat from extracting the server side part during the key creation process, while with Mandos you configure it manually with configuration files. Tang uses REST and JSON. Mandos sends a single protocol number and then sends the data. Both operate primarily through local LAN, through Mandos client also support as an option to use global reachable ip addresses. Outside of those design differences, there are some packaging aspects. Tang was designed for Redhat/Dracut, while Mandos was designed for Debian and initramfs-tools (Today Mandos also support Dracut, but it is not packaged for Redhat). Redhat packaging has been requested multiple times for Mandos, but neither of us two developer are Redhat users. To add some historical context, I recall a Debconf BoF by the developers of initramfs-tools about if they were going to continue developing initramfs-tools or giving up and port everything to Dracut, and the silence was fairly deafening. People did not want to give up on initramfs-tools, but everyone recognized the massive duplication that those two project are. Similarly, the reason why Redhat had chosen initially to develop Dracut rather than just port initramfs-tools is also fairly big mystery, and is generally considered a Not Invented Here syndrome. Today there are however some distinct difference in design between the two systems.
- rlpb 8mo ago
  clevis and tang do currently work seamlessly on Debian and Ubuntu using initramfs-tools. So while the initramfs-tools/dracut discussion is valid, it seems mostly orthogonal to this topic.
  - belorn 8mo ago
    I was unaware that they no longer depended on Dracut and now support initramfs-tools, which also seem to be the earliest clevis version that got packaged in Debian. That makes the initramfs-tools/dracut distinction a historical aspect of the project.
teddyh 8mo ago
Despite the “tang/clevis” system being too clever by half (and even more complex), it is essentially very similar. Mandos, though, is much simpler; you can actually understand it, since it uses standard components and protocols like DNS-SD¹, OpenPGP and TLS. (Mandos is documented in man pages, in contrast to page you linked, which is both huge, and – for me at least – keeps reloading itself and replacing the entire page with a 500 server error.) Mandos is also for Debian (and all Debian-based systems), and not Red Hat (although there is nothing preventing a port, since Mandos now also supports initramfs images created by Dracut). Finally, Mandos was initially created in in 2007, many years before tang/clevis, and literally by a person in a dorm room, not by whatever academics/scientists who seem to have dreamed up the cryptography/protocol tangle that is tang/clevis. Mandos has changed some since then, from initially using broadcast UDP and X.509 certificates, to using DNS-SD and raw public keys², but has otherwise remained very similar to its initial design. 1. <https://www.dns-sd.org/ https://www.dns-sd.org/> 2. <https://www.rfc-editor.org/rfc/rfc7250 https://www.rfc-editor.org/rfc/rfc7250>

koolba 8mo ago

Why is this needed at all? As the decrypted key is in memory before the reboot, can’t it just be written to a know location in memory and have kexec be instructed to read it early on?

teddyh 8mo ago
You’re assuming a controlled reboot. Mandos was initially created to deal with intermittent power failures. It’s also good for kernel panics.
- koolba 8mo ago
  Oh for sure something is needed for a full start from zero. But the much more common case for a computer with backup power is regular restarts after applying patches that require a reboot. Would be much more pleasant for that to work out of the box with no manual interaction at all.
  - PunchyHamster 8mo ago
    There is plenty of cases of cold boot in datacenter. Also most distros don't support using kexec for kernel upgrades anyway.
sillystuff 8mo ago
> As the decrypted key is in memory before the reboot, can’t it just be written to a know location in memory and have kexec be instructed to read it early on? I set up what you are suggesting (sort of anyway[1]) on a personal VPS to reboot after updates, that require one. I just generate an initrd in tmpfs that contains a keyfile[2] and kexec with that ephemeral initrd; The newest kernel can be found by looking at what the /boot/vmlinuz symlink points to. Been running this for years. It is 100% reliable, and simple. And, for the purposes of this box, secure enough. For remote unlocks from initial power on, Debian has had that since forever using keyscripts and dropbear in the initrd. [1] You could pull the key from memory, and use that to unlock the disk from within the generated initrd, but it would be more work than just setting up a keyfile in advance. It was my first thought as well. [2] Easiest way was to use a mount namespace to use a diff crypttab file that points to the keyfile, since cannot specify crypttab location when creating the initrd. E.g., unshare --mount sh -c "mount --bind $CRYPTTAB_KEXEC $CRYPTTAB; mkinitramfs -o $kexec_initramfs $kernel" (mkinitramfs is usually wrapped by update-initramfs, but calling it directly allows specifying a location)

YorickPeterse 8mo ago

I haven't looked deeply into either, but how does this compare to the combination of Clevis and Tang that e.g. Red Hat/Fedora seems to favor?

neocron 8mo ago

Which server today doesnt have Raid? Just pull one hdd out, extract what you need or change the image. Then you turn off the server, and just start a vm with the captured init and capture the key. Now you can decrypt the server offline with all the time in the world.

teddyh 8mo ago
> Just pull one hdd out, That only works with RAID 1. If the server uses RAID 5 or RAID 6, this won’t work. > extract what you need Well, yes. This is addressed in the FAQ. > or change the image. > Then you turn off the server, and just start a vm with the captured init and capture the key. Well, as explained in the FAQ, an attacker will have to do so quickly, before the Mandos server decides that the Mandos client has been offline for too long, and disables that client. The default value is five minutes, but is configurable per client.
- neocron 8mo ago
  Why wouldn't this work with Raid5? One of my raid5 hdds cought fire and the server was still running. Talking about mdadm in this case. 5 minutes is plenty to boot initrd from a vm... what's that gonna take? 10 seconds?
  - deleted 8mo ago
    [deleted]
  - Terr_ 8mo ago
    I assume they're saying it won't work because the suggestion is to "pull one" HDD out. With RAID 5, pulling 1/3 gives you only partial data, and pulling 2/3 removes too much so the system can't run.
    - neocron 8mo ago
      Ok, that is assuming /boot is ON the raid which I wouldn't want to rate for probability But even if it is, you could just pull one after the other and wait for the resilver before pulling the next one (you will hear if it resilvers automatically)

snthpy 8mo ago

Sounds great, I'm going to give this a try. Is there a plug-in for passkeys/webauthn for the mandos-server unlock?

teddyh 8mo ago
The whole architecure of Mandos is very plugin-based, it will therefore likely not be hard to add. But I am not sure what you are asking for? The Mandos server will, by default, unlock all clients, without asking. There is support for not unlocking immediately and instead wait for external approval for clients explicitly so configured, but what is the scenario? Is the client the one supplying the passkeys/webauthn? Or are you providing that manually on a web page somewhere? The latter is possible; the web page server process would then, when a passkey/webauthn has been verified, send a D-Bus “Approve” message to the Mandos server process, which would then send the client its password.

amelius 8mo ago

The kind of tool that I'm waiting for would allow me to log into a machine remotely, then reboot it once into a provided image (which could then run Linux in a ramdisk). I.e., take over the entire machine until it is restarted again. Does something like this exist? I want something like "kexec" but taking over the entire hardware at the lowest possible level. Edit, found this: https://github.com/marcan/takeover.sh https://github.com/marcan/takeover.sh But it's not as low level as I hoped, though it keeps the network running which is nice :)