8 ms·

This seems pretty bad. I was pretty leery of Tailscale having too much control over my network when I tested it out and now seeing this security notification re

by jdoss 4y ago

This seems pretty bad. I was pretty leery of Tailscale having too much control over my network when I tested it out and now seeing this security notification reinforces my decision to use vanilla WireGuard and Nebula for my home and datacenter use cases.

Arnavion 4y ago
Every time it gets mentioned here I wonder the same thing. Eg in the discussion of their last vulnerability they said they knew the vulnerability had not been exploited maliciously because they did not detect it on their end when their software phones home to them. https://github.com/tailscale/tailscale/security/advisories/GHSA-vqp6-rc3h-83cp https://github.com/tailscale/tailscale/security/advisories/G... >A vulnerability in the Tailscale Windows client [...] Reviewing all logs confirms this vulnerability was not triggered or exploited. Here too we have: >This vulnerability was not triggered or exploited. Analysis of tailnet logs shows that [...] Why on Earth would I want to use a security product that phones home... Regular WireGuard works perfectly fine for me.
- serf 4y ago
  >Why on Earth would I want to use a security product that phones home.. I share your opinion, I don't want my security products to phone home, but the question was answered earlier : 'you' want a product that phones home so that the central data collection group can make statements like "vulnerability was not triggered or exploited." -- otherwise the onus is on the data holder to make similar assessments. in other words , the proprietors of wireguard cannot make statements regarding their entire userbase. (This may or may not be a good thing.)
  - deleted 4y ago
    [deleted]
  - djbusby 4y ago
    With Vanilla WG you'd have to guess my SecretKey. With TailScale you guess some ID in their system.
- lostmsu 4y ago
  WireGuard is a pain in the ass to setup though. Also, their Windows client is no longer supported and has showstopper bugs (tunnels disappear until UI is launched and they are manually reactivated there).
  - ehPReth 4y ago
    do you have source for windows not being supported? I can’t find one after a quick search
    - lostmsu 4y ago
      They may never said so, but in practice the bug I mentioned above is over 6 months old, and is critical to the core functionality.
  - alibert 4y ago
    Wireguard is a small team (I think) and they are actually focusing on other platform right now but they will get back to Windows. https://lists.zx2c4.com/pipermail/wireguard/2022-September/007815.html https://lists.zx2c4.com/pipermail/wireguard/2022-September/0...
    - lostmsu 4y ago
      That talks about future state. The current state is that Windows is not supported.
      - djbusby 4y ago
        Nope, Jason says > But right now? Is there something pending?
        lostmsu 4y ago
        I mean, if they had proper bug tracker instead of a mailing list, we could see the answer to that right away. But I am referring to https://lore.kernel.org/wireguard/CAHmME9pbY0Cgbj5JbTVvsxpkdpvJnZ1ZBLQruuUrumgnJ3U73A@mail.gmail.com/T/#ma78c548cb15c192291bfa7ffacd56c539d75bde2 https://lore.kernel.org/wireguard/CAHmME9pbY0Cgbj5JbTVvsxpkd... which I believe just made me travel to a remote location to restore connectivity.
- lilyball 4y ago
  It's not phoning home. These vulnerabilities involve the coordination server, which is run by Tailscale, and therefore they have logs.
  - Operyl 4y ago
    It does phone home actually. The clients are constantly sending their logs for ingestion. I think this is a good thing but ymmv.
  - Arnavion 4y ago
    This one involves a Tailscale-run coordination server. The one I linked involves their clients uploading logs to home.
    - lilyball 4y ago
      Ah, I skimmed it too fast I saw the description reference the coordination server, but didn't look closely.
- phpisthebest 4y ago
  >>Why on Earth would I want to use a security product that phones home.. that ship sailed a long time ago in enterprise networking. Almost all major vendors now have Cloud Control, and all of them phone home for various things Cisco, Aruba, Juniper, etc. All of them
- Thaxll 4y ago
  How do you connect back to your home from the outside? - your ISP IP change all the time - You need to open port - You need to forward port to something else where the IP does not change ( wireguard server) - You need to setup wireguard - You need to manage auth / authz Lot of things can go wrong and it's a lot of things to setup / maintain properly.
  - wmf 4y ago
    Stop introducing facts; you're killing the outrage.
  - Arnavion 4y ago
    >How do you connect back to your home from the outside? >- your ISP IP change all the time I have a static v6 IP on the WG server (home router, running OPNsense). Even if I did have a changing IP, I have a programmable DNS server that points to the WG server's IP. >- You need to open port One time setup on the server. >- You need to setup wireguard Installed just like any other distro package, plus one-time setup to generate the key and import it and the server's public key into systemd-networkd / NetworkManager. >- You need to manage auth / authz You generate a key on each device and register the public key with the server. There's literally nothing else to it. >Lot of things can go wrong and it's a lot of things to setup / maintain properly. I set it up about two years ago and it's been working unchanged since.
    - yardstick 4y ago
      > Even if I did have a changing IP, I have a programmable DNS server that points to the WG server's IP. I believe that WG resolves the domain only once, upon config load. Not much help if the server IP changes after starting the WG client. Up to the user to realise and stop/start- not so convenient for eg embedded WG clients in travel routers
      - greyface- 4y ago
        If the client is online when the server changes IPs, WireGuard's built-in roaming will handle it transparently: https://www.wireguard.com/#built-in-roaming https://www.wireguard.com/#built-in-roaming
- voidwtf 4y ago
  > Why on Earth would I want to use a security product that phones home... Regular WireGuard works perfectly fine for me. Then you should continue using regular WireGuard. However, modern industry is plagued by an endless war with vulnerabilities, exploits, and malicious insiders. Even with adequate staffing, it's like a dam you're constantly patching to prevent leakage. We have to log everything, often offsite, and often into immutable storage. When the dam does eventually leak, we have to know how much and how it started. The logging is a feature to me. Tailscale is building a service that doesn't require me to run and maintain a centrally connectable server, one that ties into a single-sign-on solution, one that logs activity, one that's introduced a system in which I don't even have to trust their control plane exclusively (Tailnet Lock). Just the seemless integration with Azure AD has saved maintenance time over NPS+Radius+ADConnect+OpenVPN. Wireguard is great, I'm using it for all my site-to-site still (and it blows OpenVPN out of the water). But Tailscale has replaced all my client vpns for good reason.
  - ilyt 4y ago
    > When the dam does eventually leak, we have to know how much and how it started. The logging is a feature to me. Logging is a feature. Logging to some random 3rd party is not. Sure, if the 3rd party provides the service itself they need the logs to make it better but if stuff stays within your own infrastructure it should not phone home. And VPN service for enterprise certainly shouldn't have controller hosted on outside of company's own infrastructure
    - unethical_ban 4y ago
      Hoo boy, don't look up "Prisma Access" or "the cloud".
    - ithkuil 4y ago
      Well it's not a "random" 3rd party, it's a company you decide to rely on for your security. You pay, they offer you a service. If you don't trust them, that's fine, the product+service isn't for you. But it's not a "random" 3rd party.
  - etc-hosts 4y ago
- j16sdiz 4y ago
  > Why on Earth would I want to use a security product that phones home... Same as outsourcing security: When you want a team of professional security engineers take care of it. (In that case, you may want to wish they are ethical, not overworked and actually care about your servers…)
  - fps_doug 4y ago
    Did you mean: Same as outsourcing security: When shit hits the fan, you have somebody else to point your finger at. No inconvenient questions to ask withing your org, potentially having to blame someone you like on a personal level etc. Worst case, you have to ask the guy who green-lit the outsourcing some inconvenient questions, but if the company you outsourced to is big enough, you can easily take the "nobody ever got fired for buying IBM" escape route.
- unethical_ban 4y ago
  So they can do analysis on the logs and proactively fix vulnerabilities before they are exploited? I'm sitting here in shock wondering if you don't realize what you typed, or your implicitly saying you don't want a software company to have useful analytics to fix their product. The second option is understandable, but at least acknowledge the utility of logging. Also, I'm pretty sure you'll find detailed discussion of why people use Tailscale when there are options like vanilla Wireguard and OpenVPN if you google it or check their homepage. It fits many peoples' usecases.
  - Arnavion 4y ago
    >I'm sitting here in shock wondering if you don't realize what you typed After you're done being shocked, maybe you should read my comment more carefully. I didn't say "Tailscale shouldn't have logging". I didn't even ask "Why does Tailscale have logging?" I asked "Why would I want to use a security product that phones home?" I don't make decisions for Tailscale. If they want to have logging, analytics or free puppies, they're welcome to. I make decisions for myself, and I've decided that I wouldn't use any security product that phones home.
    - unethical_ban 4y ago
      "Why would I want to use a security product that phones home?" For all the reasons I listed, then. If you can operate more manual, less user-friendly, and difficult-to-troubleshoot software, then by all means.
sneak 4y ago
Nebula is amazing and I'm so happy it exists. I use it everywhere now.
nixpulvis 4y ago
People mostly seem to pay for these products so that they can blame a third party if things go wrong.
- jay-barronville 4y ago
  Ding, ding, ding! A lot of security/compliance folks like to pay money for scapegoats. It certainly helps with job security. Worst case scenario? Find another vendor.
- api 4y ago
  You just described the reason a huge chunk of the enterprise security industry exists.
- wmf 4y ago
  Or because SaaS is easier to use and requires little or no maintenance.
  - trog 4y ago
    > Or because SaaS is easier to use and requires little or no maintenance. YUP We recently deployed Tailscale in our organisation; I had looked at Wireguard very closely but decided that the ease of use of Tailscale meant we could get up and running much more quickly and easily and at relatively low cost (small team). We'll re-evaluate as we move forward & probably need to deploy it to more users, but it was the difference between a few minutes setup to get all users into it versus several hours of figuring out the ins and outs of Wireguard & then getting the team onboarded. While any security incident is frustrating, they are inevitable, and the only way to really judge an organisation is by its response. Tailscale's response here gives me more confidence, not less.
    - rstupek 4y ago
      Which auth mechanism did you end up using?
      - 4y ago
femiagbabiaka 4y ago
How does using Nebula instead of Tailscale protect against bugs of this variety?
- wmf 4y ago
  IMO it's more about agency. With SaaS people think "they had a bug and there's nothing I could have done to prevent it or expedite the fix" but with on-prem software they think "once I discover a bug I can whip my people to have it fixed within an hour". This is not true of course.
  - femiagbabiaka 4y ago
    Agreed! Was wondering if I'd missed something. On-prem is useful, but building competency with a technology can be painful and takes time.
  - jounker 4y ago
    And then there is the realists take with running your own: OH F*K. Another bug I have to deal with.
- linsomniac 4y ago
  Nebula doesn't really have a control server of this sort, it largely uses a CA to do the node authentication and a coordination server that helps nodes get introduced and NAT bust, more like the DERP server for tailscale. The Nebula equivalent of this would be the Defined Networking folks, who do run a control server more akin to Tailscale. They say they are moving slow to focus on security, and I haven't heard of vulnerabilities like Tailscale, but also I think Defined Networks is much, much smaller in terms of users, so it may be a time will tell situation. They both seem to have pretty smart folks.
  - femiagbabiaka 4y ago
    Nice thanks! Time to check out Nebula/DN.
- ikiris 4y ago
kissgyorgy 4y ago
I'm so glad you 1. read the article 2. precisely understood the vulnerability 3. carefully analyzed the situation 4. came up with a high quality analytical comment

by jdoss 4y ago

Arnavion 4y ago

Every time it gets mentioned here I wonder the same thing. Eg in the discussion of their last vulnerability they said they knew the vulnerability had not been exploited maliciously because they did not detect it on their end when their software phones home to them. https://github.com/tailscale/tailscale/security/advisories/GHSA-vqp6-rc3h-83cp https://github.com/tailscale/tailscale/security/advisories/G... >A vulnerability in the Tailscale Windows client [...] Reviewing all logs confirms this vulnerability was not triggered or exploited. Here too we have: >This vulnerability was not triggered or exploited. Analysis of tailnet logs shows that [...] Why on Earth would I want to use a security product that phones home... Regular WireGuard works perfectly fine for me.

serf 4y ago
>Why on Earth would I want to use a security product that phones home.. I share your opinion, I don't want my security products to phone home, but the question was answered earlier : 'you' want a product that phones home so that the central data collection group can make statements like "vulnerability was not triggered or exploited." -- otherwise the onus is on the data holder to make similar assessments. in other words , the proprietors of wireguard cannot make statements regarding their entire userbase. (This may or may not be a good thing.)
- deleted 4y ago
  [deleted]
- djbusby 4y ago
  With Vanilla WG you'd have to guess my SecretKey. With TailScale you guess some ID in their system.
lostmsu 4y ago
WireGuard is a pain in the ass to setup though. Also, their Windows client is no longer supported and has showstopper bugs (tunnels disappear until UI is launched and they are manually reactivated there).
- ehPReth 4y ago
  do you have source for windows not being supported? I can’t find one after a quick search
  - lostmsu 4y ago
    They may never said so, but in practice the bug I mentioned above is over 6 months old, and is critical to the core functionality.
- alibert 4y ago
  Wireguard is a small team (I think) and they are actually focusing on other platform right now but they will get back to Windows. https://lists.zx2c4.com/pipermail/wireguard/2022-September/007815.html https://lists.zx2c4.com/pipermail/wireguard/2022-September/0...
  - lostmsu 4y ago
    That talks about future state. The current state is that Windows is not supported.
    - djbusby 4y ago
      Nope, Jason says > But right now? Is there something pending?
      - lostmsu 4y ago
        I mean, if they had proper bug tracker instead of a mailing list, we could see the answer to that right away. But I am referring to https://lore.kernel.org/wireguard/CAHmME9pbY0Cgbj5JbTVvsxpkdpvJnZ1ZBLQruuUrumgnJ3U73A@mail.gmail.com/T/#ma78c548cb15c192291bfa7ffacd56c539d75bde2 https://lore.kernel.org/wireguard/CAHmME9pbY0Cgbj5JbTVvsxpkd... which I believe just made me travel to a remote location to restore connectivity.
lilyball 4y ago
It's not phoning home. These vulnerabilities involve the coordination server, which is run by Tailscale, and therefore they have logs.
- Operyl 4y ago
  It does phone home actually. The clients are constantly sending their logs for ingestion. I think this is a good thing but ymmv.
- Arnavion 4y ago
  This one involves a Tailscale-run coordination server. The one I linked involves their clients uploading logs to home.
  - lilyball 4y ago
    Ah, I skimmed it too fast I saw the description reference the coordination server, but didn't look closely.
phpisthebest 4y ago
>>Why on Earth would I want to use a security product that phones home.. that ship sailed a long time ago in enterprise networking. Almost all major vendors now have Cloud Control, and all of them phone home for various things Cisco, Aruba, Juniper, etc. All of them
Thaxll 4y ago
How do you connect back to your home from the outside? - your ISP IP change all the time - You need to open port - You need to forward port to something else where the IP does not change ( wireguard server) - You need to setup wireguard - You need to manage auth / authz Lot of things can go wrong and it's a lot of things to setup / maintain properly.
- wmf 4y ago
  Stop introducing facts; you're killing the outrage.
- Arnavion 4y ago
  >How do you connect back to your home from the outside? >- your ISP IP change all the time I have a static v6 IP on the WG server (home router, running OPNsense). Even if I did have a changing IP, I have a programmable DNS server that points to the WG server's IP. >- You need to open port One time setup on the server. >- You need to setup wireguard Installed just like any other distro package, plus one-time setup to generate the key and import it and the server's public key into systemd-networkd / NetworkManager. >- You need to manage auth / authz You generate a key on each device and register the public key with the server. There's literally nothing else to it. >Lot of things can go wrong and it's a lot of things to setup / maintain properly. I set it up about two years ago and it's been working unchanged since.
  - yardstick 4y ago
    > Even if I did have a changing IP, I have a programmable DNS server that points to the WG server's IP. I believe that WG resolves the domain only once, upon config load. Not much help if the server IP changes after starting the WG client. Up to the user to realise and stop/start- not so convenient for eg embedded WG clients in travel routers
    - greyface- 4y ago
      If the client is online when the server changes IPs, WireGuard's built-in roaming will handle it transparently: https://www.wireguard.com/#built-in-roaming https://www.wireguard.com/#built-in-roaming
voidwtf 4y ago
> Why on Earth would I want to use a security product that phones home... Regular WireGuard works perfectly fine for me. Then you should continue using regular WireGuard. However, modern industry is plagued by an endless war with vulnerabilities, exploits, and malicious insiders. Even with adequate staffing, it's like a dam you're constantly patching to prevent leakage. We have to log everything, often offsite, and often into immutable storage. When the dam does eventually leak, we have to know how much and how it started. The logging is a feature to me. Tailscale is building a service that doesn't require me to run and maintain a centrally connectable server, one that ties into a single-sign-on solution, one that logs activity, one that's introduced a system in which I don't even have to trust their control plane exclusively (Tailnet Lock). Just the seemless integration with Azure AD has saved maintenance time over NPS+Radius+ADConnect+OpenVPN. Wireguard is great, I'm using it for all my site-to-site still (and it blows OpenVPN out of the water). But Tailscale has replaced all my client vpns for good reason.
- ilyt 4y ago
  > When the dam does eventually leak, we have to know how much and how it started. The logging is a feature to me. Logging is a feature. Logging to some random 3rd party is not. Sure, if the 3rd party provides the service itself they need the logs to make it better but if stuff stays within your own infrastructure it should not phone home. And VPN service for enterprise certainly shouldn't have controller hosted on outside of company's own infrastructure
  - unethical_ban 4y ago
    Hoo boy, don't look up "Prisma Access" or "the cloud".
  - ithkuil 4y ago
    Well it's not a "random" 3rd party, it's a company you decide to rely on for your security. You pay, they offer you a service. If you don't trust them, that's fine, the product+service isn't for you. But it's not a "random" 3rd party.
- etc-hosts 4y ago
j16sdiz 4y ago
> Why on Earth would I want to use a security product that phones home... Same as outsourcing security: When you want a team of professional security engineers take care of it. (In that case, you may want to wish they are ethical, not overworked and actually care about your servers…)
- fps_doug 4y ago
  Did you mean: Same as outsourcing security: When shit hits the fan, you have somebody else to point your finger at. No inconvenient questions to ask withing your org, potentially having to blame someone you like on a personal level etc. Worst case, you have to ask the guy who green-lit the outsourcing some inconvenient questions, but if the company you outsourced to is big enough, you can easily take the "nobody ever got fired for buying IBM" escape route.
unethical_ban 4y ago
So they can do analysis on the logs and proactively fix vulnerabilities before they are exploited? I'm sitting here in shock wondering if you don't realize what you typed, or your implicitly saying you don't want a software company to have useful analytics to fix their product. The second option is understandable, but at least acknowledge the utility of logging. Also, I'm pretty sure you'll find detailed discussion of why people use Tailscale when there are options like vanilla Wireguard and OpenVPN if you google it or check their homepage. It fits many peoples' usecases.
- Arnavion 4y ago
  >I'm sitting here in shock wondering if you don't realize what you typed After you're done being shocked, maybe you should read my comment more carefully. I didn't say "Tailscale shouldn't have logging". I didn't even ask "Why does Tailscale have logging?" I asked "Why would I want to use a security product that phones home?" I don't make decisions for Tailscale. If they want to have logging, analytics or free puppies, they're welcome to. I make decisions for myself, and I've decided that I wouldn't use any security product that phones home.
  - unethical_ban 4y ago
    "Why would I want to use a security product that phones home?" For all the reasons I listed, then. If you can operate more manual, less user-friendly, and difficult-to-troubleshoot software, then by all means.

sneak 4y ago

Nebula is amazing and I'm so happy it exists. I use it everywhere now.

nixpulvis 4y ago

People mostly seem to pay for these products so that they can blame a third party if things go wrong.

jay-barronville 4y ago
Ding, ding, ding! A lot of security/compliance folks like to pay money for scapegoats. It certainly helps with job security. Worst case scenario? Find another vendor.
api 4y ago
You just described the reason a huge chunk of the enterprise security industry exists.
wmf 4y ago
Or because SaaS is easier to use and requires little or no maintenance.
- trog 4y ago
  > Or because SaaS is easier to use and requires little or no maintenance. YUP We recently deployed Tailscale in our organisation; I had looked at Wireguard very closely but decided that the ease of use of Tailscale meant we could get up and running much more quickly and easily and at relatively low cost (small team). We'll re-evaluate as we move forward & probably need to deploy it to more users, but it was the difference between a few minutes setup to get all users into it versus several hours of figuring out the ins and outs of Wireguard & then getting the team onboarded. While any security incident is frustrating, they are inevitable, and the only way to really judge an organisation is by its response. Tailscale's response here gives me more confidence, not less.
  - rstupek 4y ago
    Which auth mechanism did you end up using?
    - 4y ago

femiagbabiaka 4y ago

How does using Nebula instead of Tailscale protect against bugs of this variety?

wmf 4y ago
IMO it's more about agency. With SaaS people think "they had a bug and there's nothing I could have done to prevent it or expedite the fix" but with on-prem software they think "once I discover a bug I can whip my people to have it fixed within an hour". This is not true of course.
- femiagbabiaka 4y ago
  Agreed! Was wondering if I'd missed something. On-prem is useful, but building competency with a technology can be painful and takes time.
- jounker 4y ago
  And then there is the realists take with running your own: OH F*K. Another bug I have to deal with.
linsomniac 4y ago
Nebula doesn't really have a control server of this sort, it largely uses a CA to do the node authentication and a coordination server that helps nodes get introduced and NAT bust, more like the DERP server for tailscale. The Nebula equivalent of this would be the Defined Networking folks, who do run a control server more akin to Tailscale. They say they are moving slow to focus on security, and I haven't heard of vulnerabilities like Tailscale, but also I think Defined Networks is much, much smaller in terms of users, so it may be a time will tell situation. They both seem to have pretty smart folks.
- femiagbabiaka 4y ago
  Nice thanks! Time to check out Nebula/DN.
ikiris 4y ago

kissgyorgy 4y ago

I'm so glad you 1. read the article 2. precisely understood the vulnerability 3. carefully analyzed the situation 4. came up with a high quality analytical comment