4 ms·

Wow, this article is super smart marketing by tailscale. Not only do they list all the nice and expensive features, that can help in such a situation but they a

Wow, this article is super smart marketing by tailscale. Not only do they list all the nice and expensive features, that can help in such a situation but they also show that someone at huggingface made a very stupid thing by writing a reusable auth key in an env file. Everyone using mesh VPNs like tailscale, netbird etc. knows that this is like leaving the keys right at the door.

inigyou 2mo ago
And yet everyone seems to do it anyway. Fine for medium security, but maybe the product needs a high security mode that enforces inconvenient decisions?
- moontear 2mo ago
  Not sure that everyone does this anyway. There are some good security postures you can take with Tailscale as well and leaving an auth key lying around is not one of them. Tailscale lock should have been enabled. For CI/CD purposes the auth key could have set specific tags, which would result in specific ACLs that limit blast radius. The auth key could have a short validity. You could actually use the Tailscale API to generate alerts when a new device joins your Tailnet and ping your phone or something. You could have a complete separate Tailnet for your CI/CD workers. And all of that was only the free features.
  - AbstractH24 2mo ago
    I see a tailscale product that is basically “middleware for agents” using whatever buzzword you feel (maybe “orchestration”) coming soon
- deleted 2mo ago
  [deleted]
- stingraycharles 2mo ago
  I don’t think everyone does this at all. Maybe very small scale companies, but I would have expected a company like Huggingface to have better security standards. As soon as you’re big enough to have dedicated devops people, you shouldn’t be doing this anymore.
  - inigyou 2mo ago
    You have just been shown an example of "everyone does this". You have gotten moralistic and said they shouldn't do this, but how does this refute the fact that they do?
    - stingraycharles 2mo ago
      One example from a root cause analysis of a security incident like this is not evidence for “everyone does this”. I believe that Huggingface, with 250+ employees, is an outlier.
minraws 2mo ago
Wait are people really do this so often this isn't even the default tailscale setup method. I am quite surprised people consider this to be business as usual. Huggingface for a serious service has never felt truly serious to me for reasons like these.
lukeschlather 2mo ago
> they also show that someone at huggingface made a very stupid thing by writing a reusable auth key in an env file I don't see where it says that. The Tailscale key specifically it says was stored in the kubernetes secret manager, and obtained once the attacker already had root on the k8s cluster, so would've had full access to all the secrets stored in a sensible fashion. They did get root by dumping the environment for a process, but "don't store secrets in environment variables" while it is a valid bit of hardening advice, I wouldn't call it stupid to store a secret in an environment variable.
- itopaloglu83 2mo ago
  > How did you get past security? His fortress is impenetrable. > > Door was unlocked. Once you’re a root at a system that has the ability to add and remove nodes to a network, it’s pretty much over, at least for being able to add a Tailscale node.
Rafert 2mo ago
And in case people are looking for an open source alternative, Fly.io's tokenizer is a credential-injecting proxy: https://fly.io/blog/tokenized-tokens/ https://fly.io/blog/tokenized-tokens/

inigyou 2mo ago

And yet everyone seems to do it anyway. Fine for medium security, but maybe the product needs a high security mode that enforces inconvenient decisions?

moontear 2mo ago
Not sure that everyone does this anyway. There are some good security postures you can take with Tailscale as well and leaving an auth key lying around is not one of them. Tailscale lock should have been enabled. For CI/CD purposes the auth key could have set specific tags, which would result in specific ACLs that limit blast radius. The auth key could have a short validity. You could actually use the Tailscale API to generate alerts when a new device joins your Tailnet and ping your phone or something. You could have a complete separate Tailnet for your CI/CD workers. And all of that was only the free features.
- AbstractH24 2mo ago
  I see a tailscale product that is basically “middleware for agents” using whatever buzzword you feel (maybe “orchestration”) coming soon
deleted 2mo ago
[deleted]
stingraycharles 2mo ago
I don’t think everyone does this at all. Maybe very small scale companies, but I would have expected a company like Huggingface to have better security standards. As soon as you’re big enough to have dedicated devops people, you shouldn’t be doing this anymore.
- inigyou 2mo ago
  You have just been shown an example of "everyone does this". You have gotten moralistic and said they shouldn't do this, but how does this refute the fact that they do?
  - stingraycharles 2mo ago
    One example from a root cause analysis of a security incident like this is not evidence for “everyone does this”. I believe that Huggingface, with 250+ employees, is an outlier.

minraws 2mo ago

Wait are people really do this so often this isn't even the default tailscale setup method. I am quite surprised people consider this to be business as usual. Huggingface for a serious service has never felt truly serious to me for reasons like these.

lukeschlather 2mo ago

> they also show that someone at huggingface made a very stupid thing by writing a reusable auth key in an env file I don't see where it says that. The Tailscale key specifically it says was stored in the kubernetes secret manager, and obtained once the attacker already had root on the k8s cluster, so would've had full access to all the secrets stored in a sensible fashion. They did get root by dumping the environment for a process, but "don't store secrets in environment variables" while it is a valid bit of hardening advice, I wouldn't call it stupid to store a secret in an environment variable.

itopaloglu83 2mo ago
> How did you get past security? His fortress is impenetrable. > > Door was unlocked. Once you’re a root at a system that has the ability to add and remove nodes to a network, it’s pretty much over, at least for being able to add a Tailscale node.

Rafert 2mo ago

And in case people are looking for an open source alternative, Fly.io's tokenizer is a credential-injecting proxy: https://fly.io/blog/tokenized-tokens/ https://fly.io/blog/tokenized-tokens/