5 ms·

I'm one of the Tailscale engineers who built node state encryption initially (@awly on Github), and who made the call to turn it off by default in 1.92.5. Anot

by cronos 9mo ago

I'm one of the Tailscale engineers who built node state encryption initially (@awly on Github), and who made the call to turn it off by default in 1.92.5. Another comment in this thread guessed right - this feature is too support intensive. Our original thinking was that a TPM being reset or replaced is always sign of tampering and should result in the client refusing to start or connect. But turns out there are many situations where TPMs are not reliable for non-malicious reasons. Some examples: * https://github.com/tailscale/tailscale/issues/17654 https://github.com/tailscale/tailscale/issues/17654 * https://github.com/tailscale/tailscale/issues/18288 https://github.com/tailscale/tailscale/issues/18288 * https://github.com/tailscale/tailscale/issues/18302 https://github.com/tailscale/tailscale/issues/18302 * plus a number of support tickets TPMs are a great tool for organizations that have good control of their devices. But the very heterogeneous fleet of devices that Tailscale users have is very difficult to support out of the box. So for now we leave it to security-conscious users and admins to enable, while avoiding unexpected breakage for the broader user base. We should've provided more of this context in the changelog, apologies!

Thaxll 9mo ago
Did you rely on the Google go tpm lib for that?
- cronos 9mo ago
  Yes, we use github.com/google/go-tpm/tpm2
sydbarrett74 9mo ago
That's an eminently reasonable and logical policy. Thanks for the context.
traceroute66 9mo ago
@cronos Question: You link to https://github.com/tailscale/tailscale/issues/17654 https://github.com/tailscale/tailscale/issues/17654 where a user states[1]: "Previous workaround from some comments (TS_ENCRYPT_STATE=false, FLAGS="--encrypt-state=false") didn't help on this problematic Debian 13 host" And the same user states "I confirm this issue is NOT found anymore with tailscale version 1.92.1". Could you provide a little extra context to clarify those types of comments which seem to suggest it wasn't state encryption after all ? [1] https://github.com/tailscale/tailscale/issues/17654#issuecomment-3652622607 https://github.com/tailscale/tailscale/issues/17654#issuecom...
- cronos 9mo ago
  There are two new-ish features in Tailscale that use TPMs: node state encryption (https://tailscale.com/kb/1596/secure-node-state-storage https://tailscale.com/kb/1596/secure-node-state-storage) and hardware attestation keys. Hardware key attestation is a yet-unfinished feature that we're building. The idea is to generate a signing key inside of the TPM and use it to send signatures to our control plane and other nodes, proving that it's the same node still. (The difference from node state encryption is that an attacker can still steal the node credentials from memory while they are decrypted at runtime). We started by always generating hardware attestation keys on first start or loading them from the TPM if they were already generated (which seemed safe enough to do by default). That loading part was causing startup failures in some cases. To be honest, I didn't get to the bottom of all the reports in that github issue, but this is likely why for some users setting `--encrypt-state=false` didn't help.
  - traceroute66 9mo ago
    Also I assume "off by default" also affects macOS, iOS and Android users who don't rely on TPM at all ?
    - cronos 9mo ago
      Nope, only Windows/Linux where TPMs exist.
snailmailman 9mo ago
Those issues are a surprising read. I would expect issues with TPM on old or niche devices, but not Dell XPS laptops, or a variety of VMs. But I guess I'm not entirely sure how my vms handle TPM state, or if they even can. I'm running nearly all of my personal tailscale instances in containers and VMs. Looking now at the dashboard, it appears this feature really only encrypted things on my primary linux and windows pc, my iphone, and my main linux server's host. None of the VMs+containers i use were able to take advantage of this, nor was my laptop. Although my laptop might be too old.
- Macha 9mo ago
  I had a Ryzen 3900x on a gigabyte motherboard and the fTPM was just totally unreliable for a pretty mainstream combination. Not fully sure which was to blame there. At least it was fixed in the 5900x (and _different_ gigabyte motherboard, but from the same lineup) that replaced it.
  - Marsymars 9mo ago
    This jumped out to me because I had a TPM problem on an FM2 Gigabyte mobo in ~2015. (Back when a TPM on desktop mobos required a plug-in module.) It took me months of hassling Gigabyte to get them to issue me a beta BIOS that fixed the bug, and the fix never did make it to a non-beta BIOS.
- evanjrowley 9mo ago
  My eyes have opened up to the pitfalls of TPM recently while upgrading CPUs and BIOS/UEFI versions on various hardware in my home. VMs typically do not use TPMs, so it is not surprising that the feature was not being used there. One common exception is VMware, which can provide the host's TPM to the VM for a better Windows 11 experience. One caveat is this doesn't work on most Ryzen systems because they implement a CPU-based fTPM that VMware does not accept.
  - bdavbdav 9mo ago
    AIUI most hypervisors offer vTPM - it’s disabled by default often, but most solutions have it (including Proxmox / KVM (using swtpm)
pja 9mo ago
A BIOS update to my PC reset the TPM only this week. I did get a warning that Bitlocker keys would be wiped as a result before acting at least. (I believe this was because it was fixing an AMD TPM exploit - presumably updating the TPM code wipes the TPM storage either deliberately or as an inevitable side effect.)
- plagiarist 9mo ago
  TPMs are basically storing the hashes of various pieces of software, then deterministically generating a key from those. Since the BIOS software changed, that hash changed, and the key it generates is completely new. If someone had messed with your BIOS maliciously, that's desirable. Unfortunately you messing with your BIOS intentionally also makes the original key pretty much unrecoverable.
  - cronos 9mo ago
    IIUC, it's a bit more nuanced: TPM stores hashes of various things like firmware in PCRs, and when creating keys in the TPM you can optionally bind the key to specific PCR values. But you also don't have to (and Tailscale doesn't), in which case keys survive firmware updates for example.
dietr1ch 9mo ago
I too thought that the TPM was something to be trusted with a secret until a BIOS upgrade just wiped mine. I'm not relying on TPM again.
- johncolanduoni 9mo ago
  It was designed mostly for mechanisms where in the event of certain changes (BIOS upgrades, certain other firmware changes, some OS changes) there is a fallback mechanism to unlock the system and reset the key. This is why Windows BitLocker is so insistent about you saving your key somewhere else - if you do a BIOS update and it can’t decrypt, it’ll require your copy of the key and then reset the TPM-encrypted copy with the new BIOS accounted for. A TPM’s primary function works by hashing things during the boot process, and then telling the TPM to only allow a certain operation if hashes X & Z don’t change. Depending on how the OS/software uses it, a whole host of things that go into that hash can change: BIOS updates being a common one. A hostile BIOS update can compromise the boot process, so some systems will not permit automatic decryption of the boot drive (or similar things) until the user can confirm that they have the key.
dist-epoch 9mo ago
Your suspicion is correct. I have an AMD AM5 motherboard, and everytime I update it's BIOS it warns me that the fTPM will be reset, and I know it does so because afterwards Bitlocker prompts me to introduce the recovery key since it can't unlock the drive anymore.
AceJohnny2 9mo ago
Thanks! In your change https://github.com/tailscale/tailscale/pull/18336 https://github.com/tailscale/tailscale/pull/18336 you mention: > There's also tailscaled-on-macOS, but it won't have a TPM or Keychain bindings anyway. Do you mean that on macOS, tailscaled does not and has never leveraged equivalent hardware-attestation functionality from the SEP? (Assuming such functionality is available)
- cronos 9mo ago
  On macOS we have 3 ways to run Tailscale: https://tailscale.com/kb/1065/macos-variants https://tailscale.com/kb/1065/macos-variants Two of them have a GUI component and use the Keychain to store their state. The third one is just the open-source tailscaled binary that you have to compile yourself, and it doesn't talk to the Keychain. It stores a plaintext file on disk like the Linux variant without state encryption. Unlike the GUI variants, this one is not a Swift program that can easily talk to the Keychain API.
  - cyberax 9mo ago
    You don't need Swift to use the Keychain API. It's doable from pure C.
    - johncolanduoni 9mo ago
      In fact, SecurityFramework doesn’t have a real Swift/Obj-C API. The relevant functions are all direct bindings to C ABIs (just with wrappers around the CoreFoundation types).
      - lloeki 9mo ago
        > The third one is just the open-source tailscaled binary that you have to compile yourself, and it doesn't talk to the Keychain. I use this one (via nix-darwin) because it has the nice property of starting as a systemwide daemon outside of any user context, which in turn means that it has no (user) keychain to access (there are some conundrums between accessing such keychains and "GUI" i.e user login being needed, irrespective of C vs Swift or whatever). Maybe it _could_ store things in the system keychain? But I'm not entirely sure what the gain would be when the intent is to have tailscale access through fully unattended reboots.
keepamovin 9mo ago
Does this mean TS is not FIPS 140-3 now?
- tatersolid 9mo ago
  It never was FIPS-approved and likely will never be. The wireguard protocol used by Tailscale uses ChaCha20 for encryption which is not FIPS approved.
  - keepamovin 9mo ago
    Interesting. What is the FIPS version of wireguard?
    - cronos 9mo ago
      There are some forks that are not compatible with regular wireguard, for example from wolfssl. Or just classic mTLS.
    - tatersolid 9mo ago
      > What is the FIPS version of wireguard? IPsec or TLS-based overlays which use AES encryption and NIST-approved ECC curves or (gasp) RSA for key exchange and authentication. They generally suck in comparison with wireguard, which is a clean-sheet modern cryptographic protocol.
jkaplowitz 9mo ago
Thank you for explaining that context!
zdware 9mo ago
i just started using tailscale and responses like this make me believe in the product. awesome!
nathanlied 9mo ago
Thank you for your openness here - and yes, it would be nice to see this kind of reasoning in the changelog, even if it's tucked a little out of the way! Those of us who care will read it. Also very welcome is to separate it into a small blogpost providing details, if the situation warrants a longer, more detailed format.
lloeki 9mo ago
Coincidentally this was a feature unknown to me until I performed a SSD migration from one server to another and Tailscale failed to connect because ("of course!" in hindsight) it failed to decrypt whatever. So not a TPM failure but certainly a gotcha! moment; luckily I had a fallback method to connect to the machine, otherwise in the particular situation I was in I would have been very sorry. The "whoever needs this will enable it" + support angle makes total sense.
miki123211 9mo ago
So this is only disabled on platforms that use a TPM, e.g. Linux and Windows? What about Mac OS?
- cronos 9mo ago
  The macOS client uses the keychain by default, that's not changed here .

by cronos 9mo ago

Thaxll 9mo ago

Did you rely on the Google go tpm lib for that?

cronos 9mo ago
Yes, we use github.com/google/go-tpm/tpm2

sydbarrett74 9mo ago

That's an eminently reasonable and logical policy. Thanks for the context.

traceroute66 9mo ago

@cronos Question: You link to https://github.com/tailscale/tailscale/issues/17654 https://github.com/tailscale/tailscale/issues/17654 where a user states[1]: "Previous workaround from some comments (TS_ENCRYPT_STATE=false, FLAGS="--encrypt-state=false") didn't help on this problematic Debian 13 host" And the same user states "I confirm this issue is NOT found anymore with tailscale version 1.92.1". Could you provide a little extra context to clarify those types of comments which seem to suggest it wasn't state encryption after all ? [1] https://github.com/tailscale/tailscale/issues/17654#issuecomment-3652622607 https://github.com/tailscale/tailscale/issues/17654#issuecom...

cronos 9mo ago
There are two new-ish features in Tailscale that use TPMs: node state encryption (https://tailscale.com/kb/1596/secure-node-state-storage https://tailscale.com/kb/1596/secure-node-state-storage) and hardware attestation keys. Hardware key attestation is a yet-unfinished feature that we're building. The idea is to generate a signing key inside of the TPM and use it to send signatures to our control plane and other nodes, proving that it's the same node still. (The difference from node state encryption is that an attacker can still steal the node credentials from memory while they are decrypted at runtime). We started by always generating hardware attestation keys on first start or loading them from the TPM if they were already generated (which seemed safe enough to do by default). That loading part was causing startup failures in some cases. To be honest, I didn't get to the bottom of all the reports in that github issue, but this is likely why for some users setting `--encrypt-state=false` didn't help.
- traceroute66 9mo ago
  Also I assume "off by default" also affects macOS, iOS and Android users who don't rely on TPM at all ?
  - cronos 9mo ago
    Nope, only Windows/Linux where TPMs exist.

snailmailman 9mo ago

Those issues are a surprising read. I would expect issues with TPM on old or niche devices, but not Dell XPS laptops, or a variety of VMs. But I guess I'm not entirely sure how my vms handle TPM state, or if they even can. I'm running nearly all of my personal tailscale instances in containers and VMs. Looking now at the dashboard, it appears this feature really only encrypted things on my primary linux and windows pc, my iphone, and my main linux server's host. None of the VMs+containers i use were able to take advantage of this, nor was my laptop. Although my laptop might be too old.

Macha 9mo ago
I had a Ryzen 3900x on a gigabyte motherboard and the fTPM was just totally unreliable for a pretty mainstream combination. Not fully sure which was to blame there. At least it was fixed in the 5900x (and _different_ gigabyte motherboard, but from the same lineup) that replaced it.
- Marsymars 9mo ago
  This jumped out to me because I had a TPM problem on an FM2 Gigabyte mobo in ~2015. (Back when a TPM on desktop mobos required a plug-in module.) It took me months of hassling Gigabyte to get them to issue me a beta BIOS that fixed the bug, and the fix never did make it to a non-beta BIOS.
evanjrowley 9mo ago
My eyes have opened up to the pitfalls of TPM recently while upgrading CPUs and BIOS/UEFI versions on various hardware in my home. VMs typically do not use TPMs, so it is not surprising that the feature was not being used there. One common exception is VMware, which can provide the host's TPM to the VM for a better Windows 11 experience. One caveat is this doesn't work on most Ryzen systems because they implement a CPU-based fTPM that VMware does not accept.
- bdavbdav 9mo ago
  AIUI most hypervisors offer vTPM - it’s disabled by default often, but most solutions have it (including Proxmox / KVM (using swtpm)

pja 9mo ago

A BIOS update to my PC reset the TPM only this week. I did get a warning that Bitlocker keys would be wiped as a result before acting at least. (I believe this was because it was fixing an AMD TPM exploit - presumably updating the TPM code wipes the TPM storage either deliberately or as an inevitable side effect.)

plagiarist 9mo ago
TPMs are basically storing the hashes of various pieces of software, then deterministically generating a key from those. Since the BIOS software changed, that hash changed, and the key it generates is completely new. If someone had messed with your BIOS maliciously, that's desirable. Unfortunately you messing with your BIOS intentionally also makes the original key pretty much unrecoverable.
- cronos 9mo ago
  IIUC, it's a bit more nuanced: TPM stores hashes of various things like firmware in PCRs, and when creating keys in the TPM you can optionally bind the key to specific PCR values. But you also don't have to (and Tailscale doesn't), in which case keys survive firmware updates for example.

dietr1ch 9mo ago

I too thought that the TPM was something to be trusted with a secret until a BIOS upgrade just wiped mine. I'm not relying on TPM again.

johncolanduoni 9mo ago
It was designed mostly for mechanisms where in the event of certain changes (BIOS upgrades, certain other firmware changes, some OS changes) there is a fallback mechanism to unlock the system and reset the key. This is why Windows BitLocker is so insistent about you saving your key somewhere else - if you do a BIOS update and it can’t decrypt, it’ll require your copy of the key and then reset the TPM-encrypted copy with the new BIOS accounted for. A TPM’s primary function works by hashing things during the boot process, and then telling the TPM to only allow a certain operation if hashes X & Z don’t change. Depending on how the OS/software uses it, a whole host of things that go into that hash can change: BIOS updates being a common one. A hostile BIOS update can compromise the boot process, so some systems will not permit automatic decryption of the boot drive (or similar things) until the user can confirm that they have the key.

dist-epoch 9mo ago

Your suspicion is correct. I have an AMD AM5 motherboard, and everytime I update it's BIOS it warns me that the fTPM will be reset, and I know it does so because afterwards Bitlocker prompts me to introduce the recovery key since it can't unlock the drive anymore.

AceJohnny2 9mo ago

Thanks! In your change https://github.com/tailscale/tailscale/pull/18336 https://github.com/tailscale/tailscale/pull/18336 you mention: > There's also tailscaled-on-macOS, but it won't have a TPM or Keychain bindings anyway. Do you mean that on macOS, tailscaled does not and has never leveraged equivalent hardware-attestation functionality from the SEP? (Assuming such functionality is available)

cronos 9mo ago
On macOS we have 3 ways to run Tailscale: https://tailscale.com/kb/1065/macos-variants https://tailscale.com/kb/1065/macos-variants Two of them have a GUI component and use the Keychain to store their state. The third one is just the open-source tailscaled binary that you have to compile yourself, and it doesn't talk to the Keychain. It stores a plaintext file on disk like the Linux variant without state encryption. Unlike the GUI variants, this one is not a Swift program that can easily talk to the Keychain API.
- cyberax 9mo ago
  You don't need Swift to use the Keychain API. It's doable from pure C.
  - johncolanduoni 9mo ago
    In fact, SecurityFramework doesn’t have a real Swift/Obj-C API. The relevant functions are all direct bindings to C ABIs (just with wrappers around the CoreFoundation types).
    - lloeki 9mo ago
      > The third one is just the open-source tailscaled binary that you have to compile yourself, and it doesn't talk to the Keychain. I use this one (via nix-darwin) because it has the nice property of starting as a systemwide daemon outside of any user context, which in turn means that it has no (user) keychain to access (there are some conundrums between accessing such keychains and "GUI" i.e user login being needed, irrespective of C vs Swift or whatever). Maybe it _could_ store things in the system keychain? But I'm not entirely sure what the gain would be when the intent is to have tailscale access through fully unattended reboots.

keepamovin 9mo ago

Does this mean TS is not FIPS 140-3 now?

tatersolid 9mo ago
It never was FIPS-approved and likely will never be. The wireguard protocol used by Tailscale uses ChaCha20 for encryption which is not FIPS approved.
- keepamovin 9mo ago
  Interesting. What is the FIPS version of wireguard?
  - cronos 9mo ago
    There are some forks that are not compatible with regular wireguard, for example from wolfssl. Or just classic mTLS.
  - tatersolid 9mo ago
    > What is the FIPS version of wireguard? IPsec or TLS-based overlays which use AES encryption and NIST-approved ECC curves or (gasp) RSA for key exchange and authentication. They generally suck in comparison with wireguard, which is a clean-sheet modern cryptographic protocol.

jkaplowitz 9mo ago

Thank you for explaining that context!

zdware 9mo ago

i just started using tailscale and responses like this make me believe in the product. awesome!

nathanlied 9mo ago

Thank you for your openness here - and yes, it would be nice to see this kind of reasoning in the changelog, even if it's tucked a little out of the way! Those of us who care will read it. Also very welcome is to separate it into a small blogpost providing details, if the situation warrants a longer, more detailed format.

lloeki 9mo ago

Coincidentally this was a feature unknown to me until I performed a SSD migration from one server to another and Tailscale failed to connect because ("of course!" in hindsight) it failed to decrypt whatever. So not a TPM failure but certainly a gotcha! moment; luckily I had a fallback method to connect to the machine, otherwise in the particular situation I was in I would have been very sorry. The "whoever needs this will enable it" + support angle makes total sense.

miki123211 9mo ago

So this is only disabled on platforms that use a TPM, e.g. Linux and Windows? What about Mac OS?

cronos 9mo ago
The macOS client uses the keychain by default, that's not changed here .