5 ms·

Zerotier – multiple vulnerabilities lead to private network access

135 points by miles 5y ago

tiernano 5y ago
> Zerotier promptly addressed the root server identity verification issues. This attack is still possible (as of 20/09/2021); however, an attacker is required to invest significant compute time (many many years with a single RTX2070) to generate a valid identity that collides with an intended target. Zerotier client’s still accept any identity learned via WHOIS packet as implicitly valid. It also says below that the main vuln is fixed... Can't now figure out if I should be worried or not...
- andag 5y ago
  > This attack is still possible (as of 20/09/2021); however, an attacker is required to invest significant compute time (many many years with a single RTX2070) to generate a valid identity So it's entirely feasible if you have access to any cloud vendor then.
  - Uberphallus 5y ago
    > So it's entirely feasible if you have access to any cloud vendor then. At around $0.50 per hour, and assuming "many many years" means 10 years, then the attack needs to be worth those $43800.
    - m_eiman 5y ago
      Many many means 512 in the article. But that's with lots of room for optimizations.
    - fragmede 5y ago
      That's... less than I expected? Certainly more expensive that a normal person could afford, but with the levels of income disparity in the US, that's frighteningly affordable for a large corporation and within reach for highly motivated rich individual. If you own a cloud (aka Amazon wants to break into your VPN), they get major discounts on time on their fleet. If, instead, it costs $0.01/hr due to using unused capacity during quiet times of day, that's works out to be like $1000 to crack it. As to what makes it worth it. $50k is too expensive to attack all users of this particular VPN software, but there's way more than $50k worth of information companies want to keep secret (call it competitive intelligence) being protected by VPN software the world over.
  - m_eiman 5y ago
    In addition to the compute power, you also need som other things: "An attacker would need to hijack the address for a peer that has been offline for 30 days or communicate only with other peers that do not have a cached identity for the hijacked address."
    - _hyn3 5y ago
      If you have access to the victim's ZeroTier dashboard (or API calls), that's trivial to get. Even if there are currently no disclosed vulns in the dashboard, that doesn't mean that none exist.
- mistaken 5y ago
  What worries me is that 2 months have passed since the vulnerability was fixed and yet there is no new version released which contains the patch... So you're exposed unless you build ZT from source.
  - m4lvin 5y ago
    If the problem is in the ZT root servers, then the clients do not need to be patched, I guess?
  - api 5y ago
    The patch was on the roots and was applied within 6 hours of learning of the vulnerability. A new release wasn't needed since the issue was not in there.
  - mistaken 5y ago
    A new packaged version is now available. zerotier-one (1.6.6) unstable; urgency=medium * Backport endpoint mitigation against address collision attack. -- Adam Ierymenko <adam.ierymenko@zerotier.com> Tue, 21 Sep 2021 01:00:00 -0700
- Griffinsauce 5y ago
  > many many years with a single RTX2070 Unless they're lucky.
  - midasuni 5y ago
    If you’re lucky you can break any encryption in under a second by pulling the key from /dev/urandom. It’s a meaningless statement without defining how lucky.
    - fragmede 5y ago
      If it takes 2 years to exhaustively search the entire keyspace, you'd expect to be 50% lucky a year in, assuming the keys are evenly distributed (which is a huge assumption). If you have a lot of keys to crack and it's a feasibly short time, mean time to brute-force key recovery becomes far more interesting to you than the worst case scenario.
      - midasuni 5y ago
        Which is why saying “if your lucky” isn’t helpful, but quoting time to exhaust keyspace (knowing that half the time you’ll get it in half the time) is more useful
- _kbh_ 5y ago
  unless the number of years that compute time is measured in has a lot of digits (like 3 or more) I wouldn't be comfortable. a RTX2070 isn't even the fastest card available now, let alone in the next generation or two. Compute performance for video cards is still rapidly scaling and that means that if the "many many years" isn't large enough it may not even be years soon enough.
- downrightmike 5y ago
  I kind of wish that the metric would be X hours on MSFT or Google's largest available cluster.
idorosen 5y ago
Zerotier seems a bit too clever about auto-discovery/peering. For comparison: WireGuard, by design, is a fairly simple protocol that requires some other software or out of band configuration to provide the peer pubkey list (among other things). As a result, there are multiple mesh configuration tools built on top of it, some of which support dynamic (auto-discovered) peers/tunnel establishment. This separation of concerns for the well-designed mesh configuration layers on top of WG make it a little safer than integrated options like ZeroTier in my opinion, because the peer config/establishment and tunneling protocol are separate, independently auditable pieces. I am wary of mesh VPNs that don’t require you to statically enumerate peers, especially when I don’t fully understand the underlying authorization/trust scheme. I’m extra wary of those that don’t have a well staffed team to respond to threats, and that don’t exhibit the appropriate amount of humility when making security claims. (This is not directed at ZeroTier specifically.) On an unrelated note, there is a really fun infosec podcast named Darknet Diaries, worth a listen.
- numpad0 5y ago
  I think the intended use case for ZeroTier is six Android phones and two Windows laptops connected to a same L2 with ~500kbps UL/DL, that has to be bulletproof, dumbproof and as close as possible to zero setup and nannying, even with Android phones constantly roaming between a Wi-Fi router in a car and Starbucks Wi-Fi and its own metered WWAN. It's a very specialized solution, not a generic VPN. They do seem to care about security but they also probably prioritize zero setup/maintenance/service restart over it. Like, you can post that two apks and that zip and a ZeroTier key on Discord, let that trusted pseudonymous guy to approve connection requests on ZT, tell people to clear cache and retry if it's not working, and that's all you have to do. WireGuard would take more effort to deal with. dc: No, I've never been involved in that kind of activity, but came across ZT from a different direction it is useful for, and got what the app is about.
  - op00to 5y ago
    What is the actual intended use case, not the implementation details?
- 5y ago
aborsy 5y ago
Cryptographic part of ZeroTier protocol was audited last year https://www.zerotier.com/wp-content/uploads/2020/10/ZeroTier-Protocol-Review-Summary.pdf https://www.zerotier.com/wp-content/uploads/2020/10/ZeroTier... The code (not the protocol) has not been audited yet apparently. I wonder if the public key handling was audited above! Overall, I was surprised of such vulnerability in ZeroTier!
- api 5y ago
  It's existed since 2014 (first alpha release). This is the first time we've had anyone find anything.
api 5y ago
> Zerotier promptly addressed the root server identity verification issues. This attack is still possible (as of 20/09/2021); however, an attacker is required to invest significant compute time (many many years with a single RTX2070) to generate a valid identity that collides with an intended target. Zerotier client’s still accept any identity learned via WHOIS packet as implicitly valid. ZeroTier founder here: Clients accept identities learned from WHOIS via root servers, but not from regular nodes. As mentioned in the writeup, this was reported and fixed rapidly. I don't think there is any remaining issue here, but I am doing a re-read and will publish a blog post in a bit. It's also important to note that the original (fixed) issue also required a node to be offline for >30 days. So the attacker would have to know your network and the identity of a node that was has not been online for a long time and invest >$500K worth of GPU compute power (at current cloud rates) per identity target. The certificate thing mentioned is well known and hasn't been considered a vulnerability because certificates of membership are not secrets. They are akin to the SSL server and client certificates presented on SSL connection. That being said, it wouldn't be a bad idea (defense in depth) to make it harder to get these to make enumeration more difficult. We're looking at some mechanisms for this in new versions of ZeroTier.
- _hyn3 5y ago
  It's a good point that the cost could (would always?) be high, because you could probably just park a van outside the target's wifi network, given the sad state of wireless crypto. However, ZeroTier is designed to provide encryption on top of an existing network, so by targeting ZT, you gain access to everything within that network, even in other locations. Is the ZeroTier root code open source? Can someone run their own root servers?
  - api 5y ago
    The root code is open source. That's how they found the issue. We always encourage our users not to (just) trust ZeroTier, not because we don't think it's safe but because you should never trust just one thing. Never, never, never. One layer of security is no security. There are people who will tell you their software could contain bugs. Then there are people who will lie, or who are delusional. Anything you care about should be gated by at least two security systems, preferably ones with different code bases from different vendors. Bonus points if they use different mechanisms or algorithms.
tyingq 5y ago
I wonder how many corporate networks have end users using things like Zerotier or Tailscale without the admins being aware. Especially running self-hosted where you can use well-known, allowed ports for the control pieces.
- tempfs 5y ago
  Security teams have way more visibility than you seem to be aware of. My advice to you and others is to always ask yourself, "Is this really worth losing my job over?", when considering any sort of security control circumvention.
  - tyingq 5y ago
    >Security teams have way more visibility than you seem to be aware of. I imagine in the real world, this varies greatly. I've worked in many large, non-tech, companies, and saw a lot of false bravado. They watched some things, but not much. Big companies with zero "deep packet inspection", for example, do exist. >My advice to you and others is to always ask yourself, "Is this really worth losing my job over?", when considering any sort of security control circumvention. Good advice, but I wasn't wondering for the sake of trying it. I was wondering in the context of this vulnerability.
    - tempfs 5y ago
      Don't really need DPI to detect services and protocol patterns, just a decent perimeter device which almost everyone has. Ex. Tons of UDP streaming from an endpoint to a cloud box sticks out like a sore thumb using little to no special detection magic.
jlark77777 5y ago
For those who are interested, this is the official ZeroTier response. https://www.zerotier.com/2021/09/21/incident-response-to-september-20th-2021/ https://www.zerotier.com/2021/09/21/incident-response-to-sep...

135 points by miles 5y ago

andag 5y ago
> This attack is still possible (as of 20/09/2021); however, an attacker is required to invest significant compute time (many many years with a single RTX2070) to generate a valid identity So it's entirely feasible if you have access to any cloud vendor then.
- Uberphallus 5y ago
  > So it's entirely feasible if you have access to any cloud vendor then. At around $0.50 per hour, and assuming "many many years" means 10 years, then the attack needs to be worth those $43800.
  - m_eiman 5y ago
    Many many means 512 in the article. But that's with lots of room for optimizations.
  - fragmede 5y ago
    That's... less than I expected? Certainly more expensive that a normal person could afford, but with the levels of income disparity in the US, that's frighteningly affordable for a large corporation and within reach for highly motivated rich individual. If you own a cloud (aka Amazon wants to break into your VPN), they get major discounts on time on their fleet. If, instead, it costs $0.01/hr due to using unused capacity during quiet times of day, that's works out to be like $1000 to crack it. As to what makes it worth it. $50k is too expensive to attack all users of this particular VPN software, but there's way more than $50k worth of information companies want to keep secret (call it competitive intelligence) being protected by VPN software the world over.
- m_eiman 5y ago
  In addition to the compute power, you also need som other things: "An attacker would need to hijack the address for a peer that has been offline for 30 days or communicate only with other peers that do not have a cached identity for the hijacked address."
  - _hyn3 5y ago
    If you have access to the victim's ZeroTier dashboard (or API calls), that's trivial to get. Even if there are currently no disclosed vulns in the dashboard, that doesn't mean that none exist.
mistaken 5y ago
What worries me is that 2 months have passed since the vulnerability was fixed and yet there is no new version released which contains the patch... So you're exposed unless you build ZT from source.
- m4lvin 5y ago
  If the problem is in the ZT root servers, then the clients do not need to be patched, I guess?
- api 5y ago
  The patch was on the roots and was applied within 6 hours of learning of the vulnerability. A new release wasn't needed since the issue was not in there.
- mistaken 5y ago
  A new packaged version is now available. zerotier-one (1.6.6) unstable; urgency=medium * Backport endpoint mitigation against address collision attack. -- Adam Ierymenko <adam.ierymenko@zerotier.com> Tue, 21 Sep 2021 01:00:00 -0700
Griffinsauce 5y ago
> many many years with a single RTX2070 Unless they're lucky.
- midasuni 5y ago
  If you’re lucky you can break any encryption in under a second by pulling the key from /dev/urandom. It’s a meaningless statement without defining how lucky.
  - fragmede 5y ago
    If it takes 2 years to exhaustively search the entire keyspace, you'd expect to be 50% lucky a year in, assuming the keys are evenly distributed (which is a huge assumption). If you have a lot of keys to crack and it's a feasibly short time, mean time to brute-force key recovery becomes far more interesting to you than the worst case scenario.
    - midasuni 5y ago
      Which is why saying “if your lucky” isn’t helpful, but quoting time to exhaust keyspace (knowing that half the time you’ll get it in half the time) is more useful
_kbh_ 5y ago
unless the number of years that compute time is measured in has a lot of digits (like 3 or more) I wouldn't be comfortable. a RTX2070 isn't even the fastest card available now, let alone in the next generation or two. Compute performance for video cards is still rapidly scaling and that means that if the "many many years" isn't large enough it may not even be years soon enough.
downrightmike 5y ago
I kind of wish that the metric would be X hours on MSFT or Google's largest available cluster.

Zerotier seems a bit too clever about auto-discovery/peering. For comparison: WireGuard, by design, is a fairly simple protocol that requires some other software or out of band configuration to provide the peer pubkey list (among other things). As a result, there are multiple mesh configuration tools built on top of it, some of which support dynamic (auto-discovered) peers/tunnel establishment. This separation of concerns for the well-designed mesh configuration layers on top of WG make it a little safer than integrated options like ZeroTier in my opinion, because the peer config/establishment and tunneling protocol are separate, independently auditable pieces. I am wary of mesh VPNs that don’t require you to statically enumerate peers, especially when I don’t fully understand the underlying authorization/trust scheme. I’m extra wary of those that don’t have a well staffed team to respond to threats, and that don’t exhibit the appropriate amount of humility when making security claims. (This is not directed at ZeroTier specifically.) On an unrelated note, there is a really fun infosec podcast named Darknet Diaries, worth a listen.

numpad0 5y ago
I think the intended use case for ZeroTier is six Android phones and two Windows laptops connected to a same L2 with ~500kbps UL/DL, that has to be bulletproof, dumbproof and as close as possible to zero setup and nannying, even with Android phones constantly roaming between a Wi-Fi router in a car and Starbucks Wi-Fi and its own metered WWAN. It's a very specialized solution, not a generic VPN. They do seem to care about security but they also probably prioritize zero setup/maintenance/service restart over it. Like, you can post that two apks and that zip and a ZeroTier key on Discord, let that trusted pseudonymous guy to approve connection requests on ZT, tell people to clear cache and retry if it's not working, and that's all you have to do. WireGuard would take more effort to deal with. dc: No, I've never been involved in that kind of activity, but came across ZT from a different direction it is useful for, and got what the app is about.
- op00to 5y ago
  What is the actual intended use case, not the implementation details?
5y ago

aborsy 5y ago

Cryptographic part of ZeroTier protocol was audited last year https://www.zerotier.com/wp-content/uploads/2020/10/ZeroTier-Protocol-Review-Summary.pdf https://www.zerotier.com/wp-content/uploads/2020/10/ZeroTier... The code (not the protocol) has not been audited yet apparently. I wonder if the public key handling was audited above! Overall, I was surprised of such vulnerability in ZeroTier!

api 5y ago
It's existed since 2014 (first alpha release). This is the first time we've had anyone find anything.

api 5y ago

> Zerotier promptly addressed the root server identity verification issues. This attack is still possible (as of 20/09/2021); however, an attacker is required to invest significant compute time (many many years with a single RTX2070) to generate a valid identity that collides with an intended target. Zerotier client’s still accept any identity learned via WHOIS packet as implicitly valid. ZeroTier founder here: Clients accept identities learned from WHOIS via root servers, but not from regular nodes. As mentioned in the writeup, this was reported and fixed rapidly. I don't think there is any remaining issue here, but I am doing a re-read and will publish a blog post in a bit. It's also important to note that the original (fixed) issue also required a node to be offline for >30 days. So the attacker would have to know your network and the identity of a node that was has not been online for a long time and invest >$500K worth of GPU compute power (at current cloud rates) per identity target. The certificate thing mentioned is well known and hasn't been considered a vulnerability because certificates of membership are not secrets. They are akin to the SSL server and client certificates presented on SSL connection. That being said, it wouldn't be a bad idea (defense in depth) to make it harder to get these to make enumeration more difficult. We're looking at some mechanisms for this in new versions of ZeroTier.

_hyn3 5y ago
It's a good point that the cost could (would always?) be high, because you could probably just park a van outside the target's wifi network, given the sad state of wireless crypto. However, ZeroTier is designed to provide encryption on top of an existing network, so by targeting ZT, you gain access to everything within that network, even in other locations. Is the ZeroTier root code open source? Can someone run their own root servers?
- api 5y ago
  The root code is open source. That's how they found the issue. We always encourage our users not to (just) trust ZeroTier, not because we don't think it's safe but because you should never trust just one thing. Never, never, never. One layer of security is no security. There are people who will tell you their software could contain bugs. Then there are people who will lie, or who are delusional. Anything you care about should be gated by at least two security systems, preferably ones with different code bases from different vendors. Bonus points if they use different mechanisms or algorithms.

tyingq 5y ago

I wonder how many corporate networks have end users using things like Zerotier or Tailscale without the admins being aware. Especially running self-hosted where you can use well-known, allowed ports for the control pieces.

tempfs 5y ago
Security teams have way more visibility than you seem to be aware of. My advice to you and others is to always ask yourself, "Is this really worth losing my job over?", when considering any sort of security control circumvention.
- tyingq 5y ago
  >Security teams have way more visibility than you seem to be aware of. I imagine in the real world, this varies greatly. I've worked in many large, non-tech, companies, and saw a lot of false bravado. They watched some things, but not much. Big companies with zero "deep packet inspection", for example, do exist. >My advice to you and others is to always ask yourself, "Is this really worth losing my job over?", when considering any sort of security control circumvention. Good advice, but I wasn't wondering for the sake of trying it. I was wondering in the context of this vulnerability.
  - tempfs 5y ago
    Don't really need DPI to detect services and protocol patterns, just a decent perimeter device which almost everyone has. Ex. Tons of UDP streaming from an endpoint to a cloud box sticks out like a sore thumb using little to no special detection magic.

jlark77777 5y ago

For those who are interested, this is the official ZeroTier response. https://www.zerotier.com/2021/09/21/incident-response-to-september-20th-2021/ https://www.zerotier.com/2021/09/21/incident-response-to-sep...