6 ms·

Windows Kills SMB Speeds When Using Tailscale

107 points by salmon 2y ago

accrual 2y ago
It was nice to see PowerShell could change the interface metric when the adapter GUI refused due to the empty IP field. I bet that check has been there since the 90s. It makes me a little happy when a new CLI is able to do something the old GUI cannot!
- ygra 2y ago
  It's not just the new CLI. I guess you could have done the same with netsh for ages as well.
  - accrual 2y ago
    Nice, you're right, looks like `netsh` can do it: netsh interface ipv4 set interface "<if>" metric=<metric>
muststopmyths 2y ago
So, a virtual adapter advertises 100Gbps link speed, but is not capable of delivering that and the takeaway is "Windows kills..." ? How do other OSes handle the situation of having two interfaces with identical routes to a given destination ? I don't see a better solution than using link speed, but I haven't thought about it too deeply.
- windexh8er 2y ago
  Tailscale shouldn't advertise a route that is local to the machine. This is a routing loop. The way SR route distribution works in Tailscale is that you accept all routes or nothing. Routing platforms have the concept of route filters to prevent accepting an advertised route that would create a loop. There are hacky ways around this without having to deal with metrics (just advertise a /23 instead of a /24 and the /24 will be selected by default). But if you've got contiguous subnets you may not be able to clobber the additional address space just to avoid the route.
  - RockRobotRock 2y ago
    I really thought Tailscale would automagically figure this out. If this were true in all cases, my internet would not work at all since it would try to reach my router through the Tailscale interface. It's odd.
    - readyplayeremma 2y ago
      You don’t have any specific routes to random internet addresses though. And Tailscale would not either. Unless your Windows server is running BGP, all your Internet traffic is hitting the default route.
    - windexh8er 2y ago
      The SR can't route for your gateway or else Tailscale itself would break it's own connection. A gateway IP isn't treated the same as a subnet route.
hk1337 2y ago
I don’t think this is exclusive to Windows. SMB is a crappy service for anything outside local LAN. I am not too familiar with Tailscale but from what I understand, it’s basically akin to a VPN.
- mixdup 2y ago
  Did you read the linked post? It actually has nothing to do with SMB
  - hk1337 2y ago
    [flagged]
- not_a_bot_4sho 2y ago
  Article spoiler: the issue is Tailscale. SMB and Windows are red herrings.
  - EVa5I7bHFq9mnYK 2y ago
    True. Have exactly same speed issues with SCP as with SMB. It also depends on exit node used - some exit nodes give 10MB/s speed, same give 1MB/s. Doesn't work without exit nodes at all - cross-border blocking issues.
- dawnerd 2y ago
  Sounds like ya it’s a different issue here but I can confirm using WireGuard absolutely destroys smb performance. It’s not as bad on windows but on Mac it’s basically unusable.
- karlgkk 2y ago
  “I don’t know what I’m talking about but here’s my opinion.” Thanks for your contribution
luma 2y ago
This is framed as a problem with windows, when it’s clearly a problem with tailscale misreporting its capabilities to the OS. If I have a 100gbit and a 1gbit interface, it’s perfectly reasonable for the OS to auto assign route metrics to prefer the much faster interface. This is the OS working as designed, switching to Linux won’t help. Tailscale needs to do a better job reporting link characteristics.
- windexh8er 2y ago
  This actually would have nothing to do with the problem (advertised link speed) if Tailscale had more fine grained control over how SR routes are distributed. Currently it's all or nothing and there's no way to ignore specific routes if you're local to that network. Route length trumps network metrics, so it would be better handled in that manner. In almost every OS I've seen interface metrics will only be used for equal cost route lengths.
- deleted 2y ago
  [deleted]
- cosmotic 2y ago
  Using actual throughput would be a nice improvement over using link speed. In this case, just detecting the current throughput is a tiny fraction of the link speed and falling back to another link would be a huge improvement.
windexh8er 2y ago
This isn't a Windows problem. The OP would experience the same problem on Linux. I've run into this with SRs. I believe I may have even opened an issue with Tailscale to detect when a client is local to an exit and/or provide more fine grained route ingestion depending on where the client is with respect to the SR. But... Again, not a Windows problem. It is easy to fix by just advertising a longer route. But that implies you won't clobber other things. By default a more specific route will be chosen so a longer route advertised on the TS interface won't be selected.
caconym_ 2y ago
If Tailscale is being used for remote access to the author's LAN, why is it running on a desktop that's always physically connected to the LAN? I have a similar setup for remote access but using Wireguard instead; my main router (pfSense VM running on Proxmox like the author's thing) handles the tunnels and routing for the remote subnet(s), and it all Just Works. Only the devices that actually get used remotely need to be set up as Wireguard peers, and they're configured to disconnect from the tunnel when they're on my home wifi. IIUC Wireguard automatically does the setup/teardown of routes on those peers when it's toggled on/off.
- stego-tech 2y ago
  Because, for whatever reason I’ve yet to grasp, homelab folks like to implement Tailscale as some sort of “secure virtual network” abstraction layer - think something similar to zScaler ZPA - on top of their local LAN. To be fair, I didn’t think Tailscale did a good job explaining why this isn’t a great idea last time I tinkered with it in 2022. If you can juggle SSH keys and forward ports on your firewall, you can just run plain old Wireguard. Don’t use Tailscale as a network abstractor unless you know what and why you’re using it that way for.
  - deleted 2y ago
    [deleted]
  - code_biologist 2y ago
    I needed access to my home NAS and linux GPU box while visiting family last year over the holidays. I was in a rush. I spent 45 minutes trying to get Wireguard configured and working, then tried Tailscale and had the network I was looking for in 15 minutes. I'm not a homelabber. I hate network admin. Is Just Works™ / being moron-resistant, with good first-party client apps, a bad reason to pick Tailscale?
    - stego-tech 2y ago
      Of course not - if it works, it works, and I won’t fault folks for using Tailscale (heck, I like Tailscale, but I just got Wireguard working suitably for my needs first). My gripe was more that folks use it for a virtual network on their home LAN without seemingly grasping the implications of such abstraction - kind of like how the trend during the pandemic was “everything in Kubernetes” even though VMs might have been a better fit for their given problem. If you’re willing to put in the effort to make it work, then go for it, but I just caution folks to understand there might be better solutions to consider - and that especially when talking about abstraction layers, you absolutely need to understand the implications of said layers before deployment.
magicalhippo 2y ago
I have my desktop PC connected to my TrueNAS box via both regular 1GbE via switch and a direct 10GbE link. I experienced similar issues where sometimes Windows would pick the sub-optimal interface. I decided to brute force it, by editing my hosts file on Windows and adding a custom entry for the static IP assigned to the 10GbE adapter in TrueNAS. So if my NAS was named "mynas" I'd add a "mynas10" entry in hosts file.
bGl2YW5j 2y ago
Thanks to the author for this! What oddly coincidental timing ... I finished setup of Tailscale just yesterday and ran into this exact issue when testing it. I didn't think too much of it and blamed the USB connection I'm using to connect my external drive.
insaneirish 2y ago
I feel like this whole thing buries the lede a bit. Yes, turns out running overlay/VPN type things disrupts traffic patterns. This is a non-story. But we're talking about using wireguard on a local network, so the actual interesting question is: why does it cause the performance to plummet? Is it an implementation issue or something more fundamental? I expect some performance impact. I don't expect a three orders of magnitude impact (which is what 355 KB/s imputes).
- EVa5I7bHFq9mnYK 2y ago
  I check the "Allow local network access" in Exit Nodes, then it transfers at max speed over local Ethernet.
  - windexh8er 2y ago
    This doesn't actually affect anything if you're accepting Tailscale SRs. The conflict the article states is accepting a route advertised by Tailscale for their local network (the SR route) while on the local network (same network as the SR route). This forces all traffic through the wireguard interface, then it's routed to the SR and then back out because the interface metric is better than the hardware because of the link speed advertised. This is the root of the bandwidth issue. The "Allow local network access" is an IP filter that's put into place or not.
- thowawatp302 2y ago
  It’s TCP, so bandwidth-delay product, if the hairpin that gets the traffic back to the local lan does anything non-trivial.
Animats 2y ago
Ah, non-transparent middlebox trouble.
tetrisgm 2y ago
I’ve been curious as to why SMB seems to get little attention, and NFS even less. I had to go through hoops to even get NFS working at all on Windows. I treated myself to 10GbE a while ago, and it feels like the protocol side of this is something that just gets overlooked. Unclear why. Maybe people just assume once it works, it works?
wtcactus 2y ago
Does this also happen in Zerotier? Don't take me wrong, I think tailscale is absolutely great, I'm just interested in trying Zerotier for a while since it has integration with OPNSense (in the GUI, I know tailscale works fine if you install the package and configure it manually).

107 points by salmon 2y ago

It was nice to see PowerShell could change the interface metric when the adapter GUI refused due to the empty IP field. I bet that check has been there since the 90s. It makes me a little happy when a new CLI is able to do something the old GUI cannot!

ygra 2y ago
It's not just the new CLI. I guess you could have done the same with netsh for ages as well.
- accrual 2y ago
  Nice, you're right, looks like `netsh` can do it: netsh interface ipv4 set interface "<if>" metric=<metric>

muststopmyths 2y ago

So, a virtual adapter advertises 100Gbps link speed, but is not capable of delivering that and the takeaway is "Windows kills..." ? How do other OSes handle the situation of having two interfaces with identical routes to a given destination ? I don't see a better solution than using link speed, but I haven't thought about it too deeply.

windexh8er 2y ago
Tailscale shouldn't advertise a route that is local to the machine. This is a routing loop. The way SR route distribution works in Tailscale is that you accept all routes or nothing. Routing platforms have the concept of route filters to prevent accepting an advertised route that would create a loop. There are hacky ways around this without having to deal with metrics (just advertise a /23 instead of a /24 and the /24 will be selected by default). But if you've got contiguous subnets you may not be able to clobber the additional address space just to avoid the route.
- RockRobotRock 2y ago
  I really thought Tailscale would automagically figure this out. If this were true in all cases, my internet would not work at all since it would try to reach my router through the Tailscale interface. It's odd.
  - readyplayeremma 2y ago
    You don’t have any specific routes to random internet addresses though. And Tailscale would not either. Unless your Windows server is running BGP, all your Internet traffic is hitting the default route.
  - windexh8er 2y ago
    The SR can't route for your gateway or else Tailscale itself would break it's own connection. A gateway IP isn't treated the same as a subnet route.

hk1337 2y ago

I don’t think this is exclusive to Windows. SMB is a crappy service for anything outside local LAN. I am not too familiar with Tailscale but from what I understand, it’s basically akin to a VPN.

mixdup 2y ago
Did you read the linked post? It actually has nothing to do with SMB
- hk1337 2y ago
  [flagged]
not_a_bot_4sho 2y ago
Article spoiler: the issue is Tailscale. SMB and Windows are red herrings.
- EVa5I7bHFq9mnYK 2y ago
  True. Have exactly same speed issues with SCP as with SMB. It also depends on exit node used - some exit nodes give 10MB/s speed, same give 1MB/s. Doesn't work without exit nodes at all - cross-border blocking issues.
dawnerd 2y ago
Sounds like ya it’s a different issue here but I can confirm using WireGuard absolutely destroys smb performance. It’s not as bad on windows but on Mac it’s basically unusable.
karlgkk 2y ago
“I don’t know what I’m talking about but here’s my opinion.” Thanks for your contribution

luma 2y ago

This is framed as a problem with windows, when it’s clearly a problem with tailscale misreporting its capabilities to the OS. If I have a 100gbit and a 1gbit interface, it’s perfectly reasonable for the OS to auto assign route metrics to prefer the much faster interface. This is the OS working as designed, switching to Linux won’t help. Tailscale needs to do a better job reporting link characteristics.

windexh8er 2y ago
This actually would have nothing to do with the problem (advertised link speed) if Tailscale had more fine grained control over how SR routes are distributed. Currently it's all or nothing and there's no way to ignore specific routes if you're local to that network. Route length trumps network metrics, so it would be better handled in that manner. In almost every OS I've seen interface metrics will only be used for equal cost route lengths.
deleted 2y ago
[deleted]
cosmotic 2y ago
Using actual throughput would be a nice improvement over using link speed. In this case, just detecting the current throughput is a tiny fraction of the link speed and falling back to another link would be a huge improvement.

windexh8er 2y ago

This isn't a Windows problem. The OP would experience the same problem on Linux. I've run into this with SRs. I believe I may have even opened an issue with Tailscale to detect when a client is local to an exit and/or provide more fine grained route ingestion depending on where the client is with respect to the SR. But... Again, not a Windows problem. It is easy to fix by just advertising a longer route. But that implies you won't clobber other things. By default a more specific route will be chosen so a longer route advertised on the TS interface won't be selected.

caconym_ 2y ago

If Tailscale is being used for remote access to the author's LAN, why is it running on a desktop that's always physically connected to the LAN? I have a similar setup for remote access but using Wireguard instead; my main router (pfSense VM running on Proxmox like the author's thing) handles the tunnels and routing for the remote subnet(s), and it all Just Works. Only the devices that actually get used remotely need to be set up as Wireguard peers, and they're configured to disconnect from the tunnel when they're on my home wifi. IIUC Wireguard automatically does the setup/teardown of routes on those peers when it's toggled on/off.

stego-tech 2y ago
Because, for whatever reason I’ve yet to grasp, homelab folks like to implement Tailscale as some sort of “secure virtual network” abstraction layer - think something similar to zScaler ZPA - on top of their local LAN. To be fair, I didn’t think Tailscale did a good job explaining why this isn’t a great idea last time I tinkered with it in 2022. If you can juggle SSH keys and forward ports on your firewall, you can just run plain old Wireguard. Don’t use Tailscale as a network abstractor unless you know what and why you’re using it that way for.
- deleted 2y ago
  [deleted]
- code_biologist 2y ago
  I needed access to my home NAS and linux GPU box while visiting family last year over the holidays. I was in a rush. I spent 45 minutes trying to get Wireguard configured and working, then tried Tailscale and had the network I was looking for in 15 minutes. I'm not a homelabber. I hate network admin. Is Just Works™ / being moron-resistant, with good first-party client apps, a bad reason to pick Tailscale?
  - stego-tech 2y ago
    Of course not - if it works, it works, and I won’t fault folks for using Tailscale (heck, I like Tailscale, but I just got Wireguard working suitably for my needs first). My gripe was more that folks use it for a virtual network on their home LAN without seemingly grasping the implications of such abstraction - kind of like how the trend during the pandemic was “everything in Kubernetes” even though VMs might have been a better fit for their given problem. If you’re willing to put in the effort to make it work, then go for it, but I just caution folks to understand there might be better solutions to consider - and that especially when talking about abstraction layers, you absolutely need to understand the implications of said layers before deployment.

magicalhippo 2y ago

I have my desktop PC connected to my TrueNAS box via both regular 1GbE via switch and a direct 10GbE link. I experienced similar issues where sometimes Windows would pick the sub-optimal interface. I decided to brute force it, by editing my hosts file on Windows and adding a custom entry for the static IP assigned to the 10GbE adapter in TrueNAS. So if my NAS was named "mynas" I'd add a "mynas10" entry in hosts file.

bGl2YW5j 2y ago

Thanks to the author for this! What oddly coincidental timing ... I finished setup of Tailscale just yesterday and ran into this exact issue when testing it. I didn't think too much of it and blamed the USB connection I'm using to connect my external drive.

insaneirish 2y ago

I feel like this whole thing buries the lede a bit. Yes, turns out running overlay/VPN type things disrupts traffic patterns. This is a non-story. But we're talking about using wireguard on a local network, so the actual interesting question is: why does it cause the performance to plummet? Is it an implementation issue or something more fundamental? I expect some performance impact. I don't expect a three orders of magnitude impact (which is what 355 KB/s imputes).

EVa5I7bHFq9mnYK 2y ago
I check the "Allow local network access" in Exit Nodes, then it transfers at max speed over local Ethernet.
- windexh8er 2y ago
  This doesn't actually affect anything if you're accepting Tailscale SRs. The conflict the article states is accepting a route advertised by Tailscale for their local network (the SR route) while on the local network (same network as the SR route). This forces all traffic through the wireguard interface, then it's routed to the SR and then back out because the interface metric is better than the hardware because of the link speed advertised. This is the root of the bandwidth issue. The "Allow local network access" is an IP filter that's put into place or not.
thowawatp302 2y ago
It’s TCP, so bandwidth-delay product, if the hairpin that gets the traffic back to the local lan does anything non-trivial.

Animats 2y ago

Ah, non-transparent middlebox trouble.

tetrisgm 2y ago

I’ve been curious as to why SMB seems to get little attention, and NFS even less. I had to go through hoops to even get NFS working at all on Windows. I treated myself to 10GbE a while ago, and it feels like the protocol side of this is something that just gets overlooked. Unclear why. Maybe people just assume once it works, it works?

wtcactus 2y ago

Does this also happen in Zerotier? Don't take me wrong, I think tailscale is absolutely great, I'm just interested in trying Zerotier for a while since it has integration with OPNSense (in the GUI, I know tailscale works fine if you install the package and configure it manually).