5 ms·

This never should have been on by default. The end user (read: administrator) needs to know they want to use the TPM. This is a huge foot gun for many devices.

This never should have been on by default. The end user (read: administrator) needs to know they want to use the TPM. This is a huge foot gun for many devices. The accompanying changelog note hints at why: > Failure to load hardware attestation keys no longer prevents the client from starting. This could happen when the TPM device is reset or replaced. This is unfortunate as for many, many deployments, you absolutely want this on. But because it's a time bomb for certain device/OS combinations that Tailscale can't control or predict, and folks install Tailscale on pretty much everything, then the incidence of borked installs can only rise.

candiddevmike 10mo ago
As someone with a passing interest in using TPM for crypto things, everytime I think deeply about the implementation details like this, I come back to needing some kind of recovery password/backup key setup that entirely negates the point of the TPM in the first place. They seem really neat, but I struggle to see the benefit they have for doing crypto when a tiny slip up means your users' keys are poof, gone. And the tiny slip up may not even be with your software, but some edge case in the OS/TPM stack.
- XorNot 10mo ago
  The benefit is that you don't enter the recovery password most of the time. And when you do it should be rare and lead to a password reset.
- nottorp 10mo ago
  Same with passkeys actually.
  - SchemaLoad 10mo ago
    Passkeys get synced between your devices so they aren't any more fragile than passwords in a password manager.
    - lsowen 10mo ago
      Passkeys _may_ be synced, but that isn't guaranteed. For example a "device bound passkey" isn't synced.
      - tadfisher 10mo ago
        There is a project under way to specify how to "sync" device-bound keys between authenticators: https://fidoalliance.org/specs/cx/cxp-v1.0-wd-20241003.html https://fidoalliance.org/specs/cx/cxp-v1.0-wd-20241003.html Ideally this should have been hashed out before deploying passkeys everywhere, but I guess you can always register multiple passkeys for the sites that allow you to.
        nottorp 10mo ago
        Iirc the original idea was that passkeys should be device specific. Of course that's impractical so now they're morphing to a long password that a human can't process. In a few years someone will post "how about a long human retainable passphrase?" as a new and improved discovery.
        SchemaLoad 10mo ago
        They are still different to a password in that the service you are logging in to never gets the private key. So in the case the database gets compromised, if the service provider ensures no edits were made / restores a backup, there is no need to change your passkey since it was never exposed.
        yencabulator 9mo ago
        That was possible before passkeys. https://www.rfc-editor.org/rfc/rfc9807.html https://www.rfc-editor.org/rfc/rfc9807.html https://en.wikipedia.org/wiki/Password-authenticated_key_agreement https://en.wikipedia.org/wiki/Password-authenticated_key_agr...
    - 0cf8612b2e1e 10mo ago
- belorn 10mo ago
  The primary argument in favor of TPM's is the desire to assert against tampering to the boot system, and as a secondary effect it can be one of the solutions to reduce the need for users to type in passwords. You can still use crypto without a TPM, including with full disk encryption, and for LUKS specifically you can use multiple passwords and mechanisms to unlock the system. Different solutions will give different benefits and drawbacks. Me and a friend wrote a remote password provider for Debian called Mandos which uses machines on the local network as a way for unattended boots. It does not address the issue of tampering with the bios/boot loader, but for the primary purpose of protecting against someone stealing the server/disks it serves the purpose of allowing us to use encrypted disk without drawbacks of typing in passwords, and the backup server, itself with encrypted disks, handles the risk of needing recovery passwords. At most one needs to have an additional backup key installed for the backup server.
- johncolanduoni 10mo ago
  The TPM was never designed to be the only holder of a key that cannot be reset. The idea was that it prevents you from typing in a password or reseting an attestation signature in a database for 99% of boots, but if certain things in the boot process change (as determined by the firmware, the CPU, the OS, and the application using the TPM) it’s designed to lock you out so those things cannot change without anyone’s notice. For that purpose they’re pretty good, though there are advantages to a more signature-oriented boot security option like Apple’s Secure Enclave. But that only works so well because Apple simply doesn’t permit altering parts of the macOS boot process. For Windows/Linux, you have a variety of hardware, firmware, and OS vendors all in the mix and agreeing on escrow of keys for all of them is hard.
  - paulddraper 10mo ago
    Whether by design or accident, this is correct. You backup a key or key creation mechanism or whatever elsewhere somewhere very safe. Then almost never touch it, as the TPM authenticates.
  - themafia 10mo ago
    The presumption is that the contents being secured are /so/ valuable that locking my device is preferable to any leak of them whatsoever. This is military level security and just isn't appropriate for most consumers. Particularly around something so rarely exercised and utilized by users as the boot process. A simple warning with a long timeout would have sufficed. Aside from that you have a hardware vendor, sourced into an integrated product from another vendor, sold to a user, with various third party software interacting with it. This was always going to result in questionable experiences for end users.
    - johncolanduoni 10mo ago
      A warning doesn’t help at all. The main threat model for FDE is that someone steals your device and dumps the disk. If you don’t protect the boot process somehow, then you’re just storing the encryption key next to the data. If you don’t care about that (which is not “military level security”, laptop thieves stealing creds is a thing), just don’t use FDE or use it with an on-boot password every time. No point in the theater.
- briHass 10mo ago
  TPM keys are great for things like SSH keys or Passkeys, which surprisingly works well even in Windows. The private key is safe from any exfiltration, and usage only requires a short PIN instead of a long passphrase. The TPM ensures you're physically typing that PIN at the machine not a remote desktop window or other redirection that could be hacked. Obviously, this is problematic/annoying for scripts and things that can't share the SSH session, because you need to PIN with every authentication. Also, for encryption, you want to use something where you can backup the private key before stashing it in the TPM. Windows allows you to do this with certificates that are exported for backup prior to encrypting the private key with an unexportable TPM key in Hello.
  - johncolanduoni 10mo ago
    An easy solution to having to put your PIN in too often for SSH is to use the `ControlPersist` option in your SSH client config. This lets you only create a new SSH connection every 30s (or whatever you put), even if you’re doing separate operations. With a low timeout, there’s no realistic security risk (what’s the chance an attacker will only have control of your machine for 30s?). I do this for GitHub in particular, because of tools that connect to the remote multiple times. Works with anything that uses the actual ssh executable under the hood.
abtinf 10mo ago
>> This could happen when the TPM device is reset or replaced. Isn’t that exactly the desired behavior to defend against physical attacks?
- horsawlarway 10mo ago
  Sure, but most users probably don't actually want this level of defense. For the same reason that most folks don't use bank vault doors on their house. Ex - even reasonably technical people hit this footgun in lots of edge cases... like updating their bios, changing the host of a vm running the tool, or having a k8s pod get scheduled on a different node. I'm surprised this was "default on" at all.
- SchemaLoad 10mo ago
  Yes, but it turns out the TPM gets reset quite often on shitty hardware.
traceroute66 10mo ago
But e.g. Windows uses a TPM by default now ? If TPMs were such a major issue then there would be millions of Windows users with TPM problems, no ? I have no inside info, but this strikes me more as a bit of a "sledgehammer to crack a nut". Tailscale turning off important functionality due to small-but-vocal number of TPM edge cases ? It is also very unfortunate they did not manage to find any middle ground between the hard-binary all-on or all-off.
- cronos 10mo ago
  Windows uses TPM for Bitlocker. A very common scenario where TPMs get reset is BIOS updates (when a TPM is implemented in firmware). AFAIK, Windows cheats here because it also manages BIOS updates. When an update happens, it takes extra steps to preserve the Bitlocker encryption key in plaintext, and re-seals it to the TPM after the update completes. Apart from Windows, there are many setups that fail in fun ways: Kubernetes pods that migrate from one VM with a TPM to another one, hypervisors that mount a virtual TPM to VMs, containers or VM images that do Tailscale registration on one machine and then get replicated to others, etc. Tailscale already did some attempts at cleverness when deciding whether to enable features using a TPM (e.g. probing for TPM health/version on startup, disabling node state encryption on Kubernetes pods), but there was still a long tail of edge cases.
  - kozziollek 10mo ago
    > Windows cheats here because it also manages BIOS updates Is this (relatively) new? I don't use TPM and I rarely update BIOS unless I really need to, but I thought there was an option on my BIOS/UEFI to use USB drive to update it. How would Windows know about it?
    - toast0 10mo ago
      Window can get BIOS updates through windows update, if the OEM participates and packages them. I haven't seen BIOS updates through windows update on my systems where I built it from components, I've only seen it on integrated systems from major builders (HP, Lenovo, etc). The BIOS update instructions for my retail packaged motherboard indicate to turn off BitLocker before doing upgrades to prevent loss of TPM turning into a loss of access, but it'd be easier if it were automated.

candiddevmike 10mo ago

As someone with a passing interest in using TPM for crypto things, everytime I think deeply about the implementation details like this, I come back to needing some kind of recovery password/backup key setup that entirely negates the point of the TPM in the first place. They seem really neat, but I struggle to see the benefit they have for doing crypto when a tiny slip up means your users' keys are poof, gone. And the tiny slip up may not even be with your software, but some edge case in the OS/TPM stack.

XorNot 10mo ago
The benefit is that you don't enter the recovery password most of the time. And when you do it should be rare and lead to a password reset.
nottorp 10mo ago
Same with passkeys actually.
- SchemaLoad 10mo ago
  Passkeys get synced between your devices so they aren't any more fragile than passwords in a password manager.
  - lsowen 10mo ago
    Passkeys _may_ be synced, but that isn't guaranteed. For example a "device bound passkey" isn't synced.
    - tadfisher 10mo ago
      There is a project under way to specify how to "sync" device-bound keys between authenticators: https://fidoalliance.org/specs/cx/cxp-v1.0-wd-20241003.html https://fidoalliance.org/specs/cx/cxp-v1.0-wd-20241003.html Ideally this should have been hashed out before deploying passkeys everywhere, but I guess you can always register multiple passkeys for the sites that allow you to.
      - nottorp 10mo ago
        Iirc the original idea was that passkeys should be device specific. Of course that's impractical so now they're morphing to a long password that a human can't process. In a few years someone will post "how about a long human retainable passphrase?" as a new and improved discovery.
        SchemaLoad 10mo ago
        They are still different to a password in that the service you are logging in to never gets the private key. So in the case the database gets compromised, if the service provider ensures no edits were made / restores a backup, there is no need to change your passkey since it was never exposed.
        yencabulator 9mo ago
        That was possible before passkeys. https://www.rfc-editor.org/rfc/rfc9807.html https://www.rfc-editor.org/rfc/rfc9807.html https://en.wikipedia.org/wiki/Password-authenticated_key_agreement https://en.wikipedia.org/wiki/Password-authenticated_key_agr...
  - 0cf8612b2e1e 10mo ago
belorn 10mo ago
The primary argument in favor of TPM's is the desire to assert against tampering to the boot system, and as a secondary effect it can be one of the solutions to reduce the need for users to type in passwords. You can still use crypto without a TPM, including with full disk encryption, and for LUKS specifically you can use multiple passwords and mechanisms to unlock the system. Different solutions will give different benefits and drawbacks. Me and a friend wrote a remote password provider for Debian called Mandos which uses machines on the local network as a way for unattended boots. It does not address the issue of tampering with the bios/boot loader, but for the primary purpose of protecting against someone stealing the server/disks it serves the purpose of allowing us to use encrypted disk without drawbacks of typing in passwords, and the backup server, itself with encrypted disks, handles the risk of needing recovery passwords. At most one needs to have an additional backup key installed for the backup server.
johncolanduoni 10mo ago
The TPM was never designed to be the only holder of a key that cannot be reset. The idea was that it prevents you from typing in a password or reseting an attestation signature in a database for 99% of boots, but if certain things in the boot process change (as determined by the firmware, the CPU, the OS, and the application using the TPM) it’s designed to lock you out so those things cannot change without anyone’s notice. For that purpose they’re pretty good, though there are advantages to a more signature-oriented boot security option like Apple’s Secure Enclave. But that only works so well because Apple simply doesn’t permit altering parts of the macOS boot process. For Windows/Linux, you have a variety of hardware, firmware, and OS vendors all in the mix and agreeing on escrow of keys for all of them is hard.
- paulddraper 10mo ago
  Whether by design or accident, this is correct. You backup a key or key creation mechanism or whatever elsewhere somewhere very safe. Then almost never touch it, as the TPM authenticates.
- themafia 10mo ago
  The presumption is that the contents being secured are /so/ valuable that locking my device is preferable to any leak of them whatsoever. This is military level security and just isn't appropriate for most consumers. Particularly around something so rarely exercised and utilized by users as the boot process. A simple warning with a long timeout would have sufficed. Aside from that you have a hardware vendor, sourced into an integrated product from another vendor, sold to a user, with various third party software interacting with it. This was always going to result in questionable experiences for end users.
  - johncolanduoni 10mo ago
    A warning doesn’t help at all. The main threat model for FDE is that someone steals your device and dumps the disk. If you don’t protect the boot process somehow, then you’re just storing the encryption key next to the data. If you don’t care about that (which is not “military level security”, laptop thieves stealing creds is a thing), just don’t use FDE or use it with an on-boot password every time. No point in the theater.
briHass 10mo ago
TPM keys are great for things like SSH keys or Passkeys, which surprisingly works well even in Windows. The private key is safe from any exfiltration, and usage only requires a short PIN instead of a long passphrase. The TPM ensures you're physically typing that PIN at the machine not a remote desktop window or other redirection that could be hacked. Obviously, this is problematic/annoying for scripts and things that can't share the SSH session, because you need to PIN with every authentication. Also, for encryption, you want to use something where you can backup the private key before stashing it in the TPM. Windows allows you to do this with certificates that are exported for backup prior to encrypting the private key with an unexportable TPM key in Hello.
- johncolanduoni 10mo ago
  An easy solution to having to put your PIN in too often for SSH is to use the `ControlPersist` option in your SSH client config. This lets you only create a new SSH connection every 30s (or whatever you put), even if you’re doing separate operations. With a low timeout, there’s no realistic security risk (what’s the chance an attacker will only have control of your machine for 30s?). I do this for GitHub in particular, because of tools that connect to the remote multiple times. Works with anything that uses the actual ssh executable under the hood.

abtinf 10mo ago

>> This could happen when the TPM device is reset or replaced. Isn’t that exactly the desired behavior to defend against physical attacks?

horsawlarway 10mo ago
Sure, but most users probably don't actually want this level of defense. For the same reason that most folks don't use bank vault doors on their house. Ex - even reasonably technical people hit this footgun in lots of edge cases... like updating their bios, changing the host of a vm running the tool, or having a k8s pod get scheduled on a different node. I'm surprised this was "default on" at all.
SchemaLoad 10mo ago
Yes, but it turns out the TPM gets reset quite often on shitty hardware.

traceroute66 10mo ago

But e.g. Windows uses a TPM by default now ? If TPMs were such a major issue then there would be millions of Windows users with TPM problems, no ? I have no inside info, but this strikes me more as a bit of a "sledgehammer to crack a nut". Tailscale turning off important functionality due to small-but-vocal number of TPM edge cases ? It is also very unfortunate they did not manage to find any middle ground between the hard-binary all-on or all-off.

cronos 10mo ago
Windows uses TPM for Bitlocker. A very common scenario where TPMs get reset is BIOS updates (when a TPM is implemented in firmware). AFAIK, Windows cheats here because it also manages BIOS updates. When an update happens, it takes extra steps to preserve the Bitlocker encryption key in plaintext, and re-seals it to the TPM after the update completes. Apart from Windows, there are many setups that fail in fun ways: Kubernetes pods that migrate from one VM with a TPM to another one, hypervisors that mount a virtual TPM to VMs, containers or VM images that do Tailscale registration on one machine and then get replicated to others, etc. Tailscale already did some attempts at cleverness when deciding whether to enable features using a TPM (e.g. probing for TPM health/version on startup, disabling node state encryption on Kubernetes pods), but there was still a long tail of edge cases.
- kozziollek 10mo ago
  > Windows cheats here because it also manages BIOS updates Is this (relatively) new? I don't use TPM and I rarely update BIOS unless I really need to, but I thought there was an option on my BIOS/UEFI to use USB drive to update it. How would Windows know about it?
  - toast0 10mo ago
    Window can get BIOS updates through windows update, if the OEM participates and packages them. I haven't seen BIOS updates through windows update on my systems where I built it from components, I've only seen it on integrated systems from major builders (HP, Lenovo, etc). The BIOS update instructions for my retail packaged motherboard indicate to turn off BitLocker before doing upgrades to prevent loss of TPM turning into a loss of access, but it'd be easier if it were automated.