12 ms·

Inferring and hijacking VPN-tunneled TCP connections

320 points by jedisct1 7y ago

colmmacc 7y ago
Disclaimer: I work at AWS, on Amazon Linux and our VPN products; those aren't impacted by this issue. The attack that the researchers describe is very impressive, and using traffic analysis and error messages to find the details of an open TCP connection is extremely clever. Unfortunately a similar approach can be used even more practically to target DNS on the VPN: https://www.openwall.com/lists/oss-security/2019/12/05/3 https://www.openwall.com/lists/oss-security/2019/12/05/3 Encrypted DNS queries and replies can be profiled by traffic analysis, and the reply "paused", making it easier to ensure that a DNS spoofing attempt will succeed. This is a good reminder that cryptographic protections are best done end to end; DNSSEC does not help with this attack, because it does not protect traffic between the stub resolver and the resolver. It's also a good reminder that traffic analysis is still the most effective threat against network encryption.
- the8472 7y ago
  I wonder whether this attack also works when the VPN device and VPNed is put in a separate network namespace, which would have its own routing table.
  - bytK7 7y ago
    I was thinking the same thing as I read the on the description. I see no reason that a separate VPN namespace would be vulnerable to this attack. The compromised device would be able to spoof packets with whatever IPs it wanted, but they would never be received in a context where the tunnel interface would be directly accessible and therefore the device would never see a response from that address, even if correctly guessed and probed.
  - tolleydbg 7y ago
    This seems like the most popular solution, but there are concerns that it may disrupt routing, and also does not work on -some- systems.
- amluto 7y ago
  An actual high-quality DNSSEC implementation would protect against this. Using an untrusted stub resolver is a mistake; end-user OSes should verify DNSSEC responses directly.
  - colmmacc 7y ago
    Applications are the real "end" in end to end, and TLS is already e2e. Good TLS does thwart this attack; an attacker still can't generate a valid certificate. It's taken a lot of focus and attention to make TLS reliable enough to make it a default in browsers, and DNSSEC is not particularly close. DNSSEC supports out-of-date cryptography and has no negotiation mechanisms to avoid them, which makes it very hard to use as a default e2e security protocol. It also doesn't encrypt anything.
    - tolleydbg 7y ago
      Yes, it is correct that you cannot easily hijack connections that use TLS, but you can make inferences about active connections that use TLS. This is still fairly devastating for a large number of vulnerable users that rely on VPNs in nations with authoritarian information controls.
    - tssva 7y ago
      e2e TLS guarantees that you received a DNS answer from the DNS server you requested it from, no one was able to see the answer in transport and no one was able to change the answer in transport. It doesn't validate that the answer provided was correct according to the owner of the domain your query was regarding. DNSSEC tries to provide this validation. Roughly it is similar to signing a message with PGP. You can argue that DNSSEC is not up to today's crypto standards and no longer trustworthy for its intended purpose, but without some protocol which provides the guarantees DNSSEC tries too e2e encryption is a 1/2 solution to DNS security.
      - tptacek 7y ago
        The parent commenter definitely understands what DNSSEC is. His point is that DNSSEC doesn't work the way you appear to think it does. Conceptually, it's meant to prove that a DNS record in a response is actually a record created by the owner of the zone. But in practice, the cryptographic signature isn't valuable to stub resolvers on the end system; instead, end systems trust their "DNS servers" (their resolving cache server) to perform DNSSEC validation for them. The success of that validation is conveyed in a single "AD" bit in the header of the DNS response. Since this attack happens from a vantage point in between the end system and the resolving cache, DNSSEC isn't in play; the attacker will simply set the AD bit in their responses.
- tolleydbg 7y ago
  Hi Colm, were still working on a response to your email, but we appreciate the insight you provided and look forward to our conversation. This disclosure only deals with the specific threat against active TCP connections, but there are more coming.
- jlgaddis 7y ago
  Off-topic but "service.example.com" woukd have been appropriate to use in the example in your mail. It's RFC-complianr. :-)
- tialaramex 7y ago
  > Encrypted DNS queries and replies can be profiled by traffic analysis, and the reply "paused", making it easier to ensure that a DNS spoofing attempt will succeed "Encrypted" only in the sense of encrypted by the VPN which you've sidestepped. An increasing fraction of DNS traffic from real clients will have been encrypted under DPRIVE, and so sidestepping the VPN doesn't help you spoof that. In this case the connection to a "real" resolver has TLS guarantees about integrity / authenticity and so you can piggyback DNSSEC assurances on top of that if that's how you choose to do things. By the way the error side channel is very of this moment - something that for example very much concerned developers of protocols like QUIC and TLS 1.3, the reaction reminds me of an anonymous criticism of Unix using a car analogy from the UNIX-HATERS handbook: > ... If the driver makes a mistake, a giant “?” lights up in the center of the dashboard. “The experienced driver,” says Thompson, “will usually know what’s wrong.” During TLS 1.3 development more than once contributors expressed a desire for a feature - maybe an optional feature - to get more verbose or detailed errors than those provided already by the protocol in hopes that it would ease debugging. Old hands correctly urged caution, the giant ? is one less weapon for bad guys. If you send garbage to a WireGuard endpoint as I understand it nothing at all happens. QUIC is a little less tight-lipped, but still endeavours to ensure that a third party can't distinguish anything useful by injecting garbage and looking at what happens next.
  - skissane 7y ago
    The "?" story is about the error handling of the ed line editor. In the event of erroneous input, it just prints "?". The original reason for this, is that Ken Thompson had better things to do than add code for nice error messages – initially the only people using it was himself and immediate colleagues who could just ask for his help in person if they got stuck. Also, the minimalist error handling was valuable in an era of extremely constrained resources (very limited memory, 300 baud modem connections, etc) – a smaller editor could be used to edit larger files, and briefer error messages made editing over slow connections faster. As soon as there was a need for a more user-friendly editor, people built new ones and left ed as it was, and those new editors soon had much better error handling. To associate its spartan error handling with security is a bit of a retcon. Interesting analogy, but the minimalistic error handling of ed was not motivated by security concerns and had basically zero security benefits (unless you count poor usability as a form of security through obscurity.)
- jedisct1 7y ago
  Using DNSCrypt while using a VPN has become a common practice. It should prevent that specific issue.
joseluisq 7y ago
Someone know if there is some kind of pronouncement by distros implicated?
exikyut 7y ago
Huh, I wonder how the OpenBSD team will regard this one. I'm unfamiliar with exactly how to tell what's in the base system and what's in ports, but I can see openvpn* entries over at http://ftp.openbsd.org/pub/OpenBSD/6.6/packages/amd64/ http://ftp.openbsd.org/pub/OpenBSD/6.6/packages/amd64/ - does that mean there's arguably a hole in the base distro? In any case, nice work. Question. Besides randomizing packet lengths (preferably with minimum and maximum tunables (per each packet type) that each site can change, to further add entropy), what else can be done to mitigate against this family of attack? Asking as someone interested in developing "ubiquitous secure container" type protocols that are application- and use-case-specific but (theoretically) high-stakes.
- colmmacc 7y ago
  Best practice for preventing traffic analysis is to send encrypted messages of a constant size, at a constant rate. It's better to pad to a max size, than to randomize the packet length. Sending at a constant rate may not always be feasible or cost-effective, but it can still be useful to send data at a constant rate for a minimum block of time.
  - zzzcpan 7y ago
    It's just easy to implement constant rate, hence why it's best practice. But adaptive rate might be ok, like, for example, if you only pick from 4 different rates and adapt exactly every second, you only leak 2 bits of information per second. The question then becomes which leaking rate is safe enough in practice?
    - colmmacc 7y ago
      constant max rate tends not to be practical in remote roaming VPNs for cost and contention reasons. It would blast through typical mobile usage rates, for example, or get you throttled on a shared wifi network.
  - GhettoMaestro 7y ago
    Ya - the NSA calls this "LINK MASKING". Makes a communication channel look busy 100% of the time at at predefined bit rate. Pretty much worthless to traffic analysis, at the cost of burning extra bits on the wire when not actually sending traffic. Mostly only used in Military applications, and specifically where you think an advanced adversary is going to do things like traffic analysis, etc.
mondoshawan 7y ago
"This attack did not work against any Linux distribution we tested until the release of Ubuntu 19.10, and we noticed that the rp_filter settings were set to “loose” mode. We see that the default settings in sysctl.d/50-default.conf in the systemd repository were changed from “strict” to “loose” mode on November 28, 2018, so distributions using a version of systemd without modified configurations after this date are now vulnerable. Most Linux distributions we tested which use other init systems leave the value as 0, the default for the Linux kernel." Anybody happen to know why systemd decided this was something they should be messing with?
- arianvanp 7y ago
  `git blame` is your friend sysctl.d: switch net.ipv4.conf.all.rp_filter from 1 to 2 This switches the RFC3704 Reverse Path filtering from Strict mode to Loose mode. The Strict mode breaks some pretty common and reasonable use cases, such as keeping connections via one default route alive after another one appears (e.g. plugging an Ethernet cable when connected via Wi-Fi). The strict filter also makes it impossible for NetworkManager to do connectivity check on a newly arriving default route (it starts with a higher metric and is bumped lower if there's connectivity). Kernel's default is 0 (no filter), but a Loose filter is good enough. The few use cases where a Strict mode could make sense can easily override this. The distributions that don't care about the client use cases and prefer a strict filter could just ship a custom configuration in /usr/lib/sysctl.d/ to override this. https://github.com/systemd/systemd/commit/230450d4e4f1f5fc9fa4295ed9185eea5b6ea16e https://github.com/systemd/systemd/commit/230450d4e4f1f5fc9f... There was NEWS entry for this here: https://github.com/systemd/systemd/blob/230450d4e4f1f5fc9fa4295ed9185eea5b6ea16e/NEWS#L73-L80 https://github.com/systemd/systemd/blob/230450d4e4f1f5fc9fa4... as to _why_ they would make the change? Systemd trieds to be project for distros to have a sane base system. Introducing this change was probably something they deemed as being useful in a base system. Especially in the context of laptops and mobile devices this default seems sane.
  - 0xbadcafebee 7y ago
    > Systemd trieds to be project for distros to have a sane base system Systemd tries to force a monolithic operating system to behave like a microkernel, and uses propaganda and manipulation to enforce whatever preference its designers have for how all systems should run onto all the Linux distributions it can. You can call that 'sane', I'll call it totalitarian empire-building.
hamandcheese 7y ago
> This vulnerability works against OpenVPN, WireGuard, and IKEv2/IPSec > It should be noted, however, that the VPN technology used does not seem to matter I was wondering which VPNs, but after reading it seems that it doesn’t matter the VPN but is instead a kernel bug? I wonder if this also affects a host that has two network interfaces, with no VPN involved whatsoever? Also, it’s good to see Jason Donenfeld involved, which many will likely recognize as the author of Wireguard.
- tolleydbg 7y ago
  Donenfeld has been great and makes a wonderful product. We've been using WireGuard as our primary VPN solution for a long time now and don't anticipate a change anytime soon. I think this class of attack can best be mitigated at the kernel level, but unfortunately, side-channels exist anywhere that maintains state, and this might be a more general routing problem.
  - zx2c4 7y ago
    We're working around this CVE in WireGuard's wg-quick(8) with a rule something like: iptables -t raw -I PREROUTING ! -i wg0 -d 10.182.12.8 -m addrtype ! --src-type LOCAL -j DROP where wg0 is the WireGuard interface and 10.182.12.8 is the local IP of the interface. This says to drop all packets that are sent to that IP address that aren't coming from the WireGuard interface. And it's done very early in netfilter, in the "raw" table. I don't like having to use iptables in wg-quick(8), and so Willy Tarreau and I have been discussing a deeper kernel-level fix, which should be posted to netdev@ sometime soon.
    - Hello71 7y ago
      I was wondering why wireguard suddenly gained a mandatory iptables dependency...
      - zx2c4 7y ago
        Yea... not nice at all. And now we'll probably have to simultaneously support nftables. This isn't a route I really want to go down. Suggestions on alternatives are welcome.
mirages 7y ago
Sorry if I seem stupid but what are the security implications ? As far as I understood the attacker can : - Detect an active VPN connection (and maybe close it/monitor it) - Attempt to inject packets : for this part I am skeptical of the usefulness. The connection between the server and the client are normally encrypted, meaning that the injected packets will be dropped or can be used to forcefully close the connection making it a DoS attack.
- ratiolat 7y ago
  Perhaps this means that the attacker can insert traffic and kernel handles it like it came from the tunnel interface? ELI5 would be in order indeed.
- loeg 7y ago
  They can inject TCP data which looks — to the application — like it came over the VPN, but it didn't actually. The vulnerability here is unencrypted TCP streams running (purportedly) over a VPN. Not TLS streams (HTTPS and HTTP/2.0+), unless you've also got a TLS 0-day. (And maybe also unencrypted UDP sessions and unencrypted services, but that's less clear to me.)
  - bscphil 7y ago
    Oh my god. I had to read this twice, but what you said here finally made me realize the severity of this. Thanks. (I also now understand why the kernel developers might be looking at it as a vulnernablity. Neighbors in your subnet should not be able to inject packets into your computer's internal routes.)
- tetha 7y ago
  > The connection between the server and the client are normally encrypted Keyword: Normally. What about DNS? DNS without DNSSEC via an untrusted AP cannot be trusted, that's for certain. Hence, you recommend your user to activate VPN first to avoid such attacks because now it's encrypted. Suddenly, that might change and your trusted-page.internal resolves to a different hostname. This /should/ be preventable via HSTS and certs for example with HTTP, but those are assumptions again. Or what about RDP? It's not encrypted, so you hide it in a VPN - mostly due to the cleartext password. But suddenly there's a vector that might be able to inject data into an RDP stream inside a VPN connection. > - Detect an active VPN connection (and maybe close it/monitor it) Not just that. They can detect TCP connections inside the VPN connection. Hence, you can start tracking if users of an access point accept anti-gov.com even if they go through a VPN. It might be detectable on the client side and it should be possible to mitigate this via configuration, but that's still plenty scary.
jiveturkey 7y ago
This doesn't seem nearly as bad as 'hijacking' implies. > The attacker can now inject arbitrary payloads into the ongoing encrypted connection using the inferred ACK and next sequence number. First, it's hard to get to this point. Then, you're injecting garbage because you don't know the payload encryption keys. So it's just disruptive, even though yes technically it is 'hijacking'. Unless I missed something, of course! I found the writeup to be vague on the payload encryption point. It should have explicitly stated the impact one way or the other.
- aaronmdjones 7y ago
  For your first point, this isn't hard if you're in the appropriate position (adjacent or upstream), as the post details. There are already tools available that can do all of this, example invocations are also there. For your other point, they don't need to know the keys (unless the traffic travelling /over/ the VPN is /also/ encrypted). That's the whole point. This is about being able to trick the machine into accepting traffic for a connection, from an interface that the connection's traffic isn't travelling over. If it's plaintext within the VPN, this sidesteps the VPN interface and you can indeed inject your own malicious plaintext traffic into that connection.
  - jiveturkey 7y ago
    1. Yes, you have to already be in position. That's limited to specific actors. Once that is achieved, you have to probe for specific IP addresses the victim might be connected to. IOW a list you are targetting. Then the port probing and seqno guessing. These facts reduce the impact because it's not just "be a guy on the internet", eg like if there were an open database of PII sitting there for the taking, only needing discovery to find it. In no way am I claiming the attack isn't feasible. It's definitely a real risk, beyond the theoretical. 2. Thanks, got it. That makes more sense. I think I actually like this vuln. It reinforces the need for defense in depth. It reduces a takeover to an annoyance (could be critical for some apps, yes) assuming you use TLS at the app layer.
    - aaronmdjones 7y ago
GhettoMaestro 7y ago
I'm having trouble understanding why the randomly sprayed bits from the attacker to the client are even accepted at all at the crypto boundary. Shouldn't it hard-fail (not decrypt) due to an invalid key?
- loeg 7y ago
  The bits aren't VPN-framed packets; the attack isn't VPN specific. They're just sending ordinary TCP setup packets and ordinary TCP stream packets with fudged parameters until they find the right ones by brute force. It's a really noisy attack that uses weaknesses in packet routing to find existing TCP sessions and inject into them. It's also, like, an escalation from a relatively high privilege level. You need both a passive network observer (compromised router or ISP) and a noisy, active LAN device (to inject IP addresses that a router would filter as bogons). That's not to say this is crazy hard; these are definitely within the reach of a motivated attacker. Routers were the original the-S-in-IoT-is-for-security, and if you've got an IoT type device on the LAN it's probably vulnerable once you've popped the router.
knolax 7y ago
It seems for the attacker to find out about active connections to any given website they have to already know the IP of the website and then brute force the virtual ports. Being able to determine what website the target has an active connection to without prior knowledge would probably take even more brute forcing. A small solice.
- loeg 7y ago
  This is a very noisy attack, for sure.
arielb1 7y ago
> nping --tcp --flags SA --source-ip 192.168.12.1 --dest-ip 10.8.0.8 -- rate 3 -c 3 -e ap0 --dest-mac 08:00:27:9c:53:12 Why is Linux accepting packets coming from one interface into an IP address belonging to a different interface? It feels like it is "forwarding" the packets internally, but `ip_forward` is turned off. Is there any case where this behavior is legitimately useful?
- ajross 7y ago
  IP addresses don't "belong" to interfaces in the general case. It's just a hard problem. In fact there are lots of multi-homed use cases where you want to internally route packets across interfaces without an affirmative mapping of what address is supposed to be used where. For the specific case of point to point VPNs, there's a rule that makes sense. But that's not part of the network stack per se and there's no way to enforce it generically.
  - loeg 7y ago
    Do network stacks drop 127.0/8 packets from external interfaces today? Superficially (I'm not an experienced TCP/IP or routing stack developer, although I do work in the kernel) it seems like the same treatment could be used for VPN-registered interface addresses. You just need an API to specify "I'm a VPN interface" when the device is created or the IP assigned, no?
  - arielb1 7y ago
    Is there a place where I can read about these cases?
- loeg 7y ago
  From the referenced systemd commit, > such as keeping connections via one default route alive after another one appears (e.g. plugging an Ethernet cable when connected via Wi-Fi).
  - arielb1 7y ago
    How is this supposed to work? How will the packet destined to the WiFi IP address get to the Ethernet interface?
ufov2 7y ago
Is this mitigated by iptables filter firewall? For example, FORWARD policy DROP and no other rules?
- jlgaddis 7y ago
  Yes, if you are doing firewalling properly. The problm is that most people aren't and are just using their distribution defaults.
nif2ee 7y ago
I just can't understand how this attack can be implemented in real world, it not only needs having an adjacent network access to the VPN client but most importantly knowing the destination of a currently open TCP connection encrypted by the VPN and then guessing the right sequence numbers. Since most TCP connections are also carrying TLS these days, this attack is pretty useless in the vast majority of real world situations I can think of.
tinix 7y ago
Can we merge these? I posted this 10 hours earlier here: https://news.ycombinator.com/item?id=21709693 https://news.ycombinator.com/item?id=21709693
- jlgaddis 7y ago
  There's really nothing useful there that isn't also here.
Gabrielfair 7y ago
Disseminating vulnerabilities like these to less savvy people should be a critical imparitive for the next century. I imagine a team of animators at a non-profit or foundation that illustrate how these attacks work at different levels of technical backgrounds.
derpherpsson 7y ago
Correct me if I am wrong, but... If I just make sure that incomming packets that are destined for the VPN LAN are dropped, this attack does not work? Of course there are such rules in our firewalls?? Is everyone walking around without any firewall filtering nowadays? How is this a bug? Maybe I am just stupid. Did I miss something?
- rasz 7y ago
  TCP/IP stack was dropping this by default .. until systemd decided to switch the default https://news.ycombinator.com/item?id=21713479 https://news.ycombinator.com/item?id=21713479
  - yrro 7y ago
    The default behaviour of the kernel is no rp filtering at all. Older versions of systems enabled strict to filtering, no doubt causing the same sort of complaints from people who like to complain about that sort of thing. Newer versions relaxed this to loose rp filtering for the reasons explained in the commit message.
e12e 7y ago
> The access point can then determine the virtual IP of the victim by sending SYN-ACK packets to the victim device across the entire virtual IP space (the default for OpenVPN is 10.8.0.0/24). So, would a bit of address space randomization mitigate step one for ipv6? fd00::/8 is pretty big, right? Even just picking a random IP in a random /64 from that /8 should help? Or am I missing something? Also interesting comments in the reply on the list regarding "policy based" vpns. I wonder if some subset of that infrastructure could be used by Wireguard without completing it all the way out of its current nice and secure-by-simplicity design? https://seclists.org/oss-sec/2019/q4/123 https://seclists.org/oss-sec/2019/q4/123 > Only route based VPNs are impacted. In comparison, policy based VPNs are not impacted (On Linux only implementable using XFRM, which is IPsec on Linux specific) unless the XFRM policy's level is set to "use" instead of "required" (default)) because any traffic received that matches a policy (IPsec security policy) and that is not protected is dropped.
3xblah 7y ago
"We discovered a vulnerability in Linux, FreeBSD, OpenBSD, MacOS, iOS, and Android..." What about NetBSD. This also appears to be wireless only, i.e., the need for attacker to have control over the AP. Am I reading this wrong.
- 3xblah 7y ago
  I was reading it wrong. It is not wireless only. Technically, it only requires that the victim is sharing a local network with the attacker.
mirimir 7y ago
> I am reporting a vulnerability that exists on most Linux distros, and other *nix operating systems which allows a network adjacent attacker to determine if another user is connected to a VPN, the virtual IP address they have been assigned by the VPN server, and whether or not there is an active connection to a given website. I'm not sure that I understand exactly what "network adjacent attacker" means. But I'm guessing that it means an attacker on the same subnet. And that it doesn't involve actually hacking VPN encryption. But isn't it well understood that sharing LANs with untrusted neighbors is hugely risky? At least, I always segregate critical machines in protected subnets. Or am I missing something? Edit: OK, I was missing that this focuses on using VPNs via WiFi APs. And depends on the AP being malicious. So yeah, this is a huge issue, for that use case.
GMLOOKO 7y ago
@

320 points by jedisct1 7y ago

Disclaimer: I work at AWS, on Amazon Linux and our VPN products; those aren't impacted by this issue. The attack that the researchers describe is very impressive, and using traffic analysis and error messages to find the details of an open TCP connection is extremely clever. Unfortunately a similar approach can be used even more practically to target DNS on the VPN: https://www.openwall.com/lists/oss-security/2019/12/05/3 https://www.openwall.com/lists/oss-security/2019/12/05/3 Encrypted DNS queries and replies can be profiled by traffic analysis, and the reply "paused", making it easier to ensure that a DNS spoofing attempt will succeed. This is a good reminder that cryptographic protections are best done end to end; DNSSEC does not help with this attack, because it does not protect traffic between the stub resolver and the resolver. It's also a good reminder that traffic analysis is still the most effective threat against network encryption.

the8472 7y ago
I wonder whether this attack also works when the VPN device and VPNed is put in a separate network namespace, which would have its own routing table.
- bytK7 7y ago
  I was thinking the same thing as I read the on the description. I see no reason that a separate VPN namespace would be vulnerable to this attack. The compromised device would be able to spoof packets with whatever IPs it wanted, but they would never be received in a context where the tunnel interface would be directly accessible and therefore the device would never see a response from that address, even if correctly guessed and probed.
- tolleydbg 7y ago
  This seems like the most popular solution, but there are concerns that it may disrupt routing, and also does not work on -some- systems.
amluto 7y ago
An actual high-quality DNSSEC implementation would protect against this. Using an untrusted stub resolver is a mistake; end-user OSes should verify DNSSEC responses directly.
- colmmacc 7y ago
  Applications are the real "end" in end to end, and TLS is already e2e. Good TLS does thwart this attack; an attacker still can't generate a valid certificate. It's taken a lot of focus and attention to make TLS reliable enough to make it a default in browsers, and DNSSEC is not particularly close. DNSSEC supports out-of-date cryptography and has no negotiation mechanisms to avoid them, which makes it very hard to use as a default e2e security protocol. It also doesn't encrypt anything.
  - tolleydbg 7y ago
    Yes, it is correct that you cannot easily hijack connections that use TLS, but you can make inferences about active connections that use TLS. This is still fairly devastating for a large number of vulnerable users that rely on VPNs in nations with authoritarian information controls.
  - tssva 7y ago
    e2e TLS guarantees that you received a DNS answer from the DNS server you requested it from, no one was able to see the answer in transport and no one was able to change the answer in transport. It doesn't validate that the answer provided was correct according to the owner of the domain your query was regarding. DNSSEC tries to provide this validation. Roughly it is similar to signing a message with PGP. You can argue that DNSSEC is not up to today's crypto standards and no longer trustworthy for its intended purpose, but without some protocol which provides the guarantees DNSSEC tries too e2e encryption is a 1/2 solution to DNS security.
    - tptacek 7y ago
      The parent commenter definitely understands what DNSSEC is. His point is that DNSSEC doesn't work the way you appear to think it does. Conceptually, it's meant to prove that a DNS record in a response is actually a record created by the owner of the zone. But in practice, the cryptographic signature isn't valuable to stub resolvers on the end system; instead, end systems trust their "DNS servers" (their resolving cache server) to perform DNSSEC validation for them. The success of that validation is conveyed in a single "AD" bit in the header of the DNS response. Since this attack happens from a vantage point in between the end system and the resolving cache, DNSSEC isn't in play; the attacker will simply set the AD bit in their responses.
tolleydbg 7y ago
Hi Colm, were still working on a response to your email, but we appreciate the insight you provided and look forward to our conversation. This disclosure only deals with the specific threat against active TCP connections, but there are more coming.
jlgaddis 7y ago
Off-topic but "service.example.com" woukd have been appropriate to use in the example in your mail. It's RFC-complianr. :-)
tialaramex 7y ago
> Encrypted DNS queries and replies can be profiled by traffic analysis, and the reply "paused", making it easier to ensure that a DNS spoofing attempt will succeed "Encrypted" only in the sense of encrypted by the VPN which you've sidestepped. An increasing fraction of DNS traffic from real clients will have been encrypted under DPRIVE, and so sidestepping the VPN doesn't help you spoof that. In this case the connection to a "real" resolver has TLS guarantees about integrity / authenticity and so you can piggyback DNSSEC assurances on top of that if that's how you choose to do things. By the way the error side channel is very of this moment - something that for example very much concerned developers of protocols like QUIC and TLS 1.3, the reaction reminds me of an anonymous criticism of Unix using a car analogy from the UNIX-HATERS handbook: > ... If the driver makes a mistake, a giant “?” lights up in the center of the dashboard. “The experienced driver,” says Thompson, “will usually know what’s wrong.” During TLS 1.3 development more than once contributors expressed a desire for a feature - maybe an optional feature - to get more verbose or detailed errors than those provided already by the protocol in hopes that it would ease debugging. Old hands correctly urged caution, the giant ? is one less weapon for bad guys. If you send garbage to a WireGuard endpoint as I understand it nothing at all happens. QUIC is a little less tight-lipped, but still endeavours to ensure that a third party can't distinguish anything useful by injecting garbage and looking at what happens next.
- skissane 7y ago
  The "?" story is about the error handling of the ed line editor. In the event of erroneous input, it just prints "?". The original reason for this, is that Ken Thompson had better things to do than add code for nice error messages – initially the only people using it was himself and immediate colleagues who could just ask for his help in person if they got stuck. Also, the minimalist error handling was valuable in an era of extremely constrained resources (very limited memory, 300 baud modem connections, etc) – a smaller editor could be used to edit larger files, and briefer error messages made editing over slow connections faster. As soon as there was a need for a more user-friendly editor, people built new ones and left ed as it was, and those new editors soon had much better error handling. To associate its spartan error handling with security is a bit of a retcon. Interesting analogy, but the minimalistic error handling of ed was not motivated by security concerns and had basically zero security benefits (unless you count poor usability as a form of security through obscurity.)
jedisct1 7y ago
Using DNSCrypt while using a VPN has become a common practice. It should prevent that specific issue.

joseluisq 7y ago

Someone know if there is some kind of pronouncement by distros implicated?

exikyut 7y ago

Huh, I wonder how the OpenBSD team will regard this one. I'm unfamiliar with exactly how to tell what's in the base system and what's in ports, but I can see openvpn* entries over at http://ftp.openbsd.org/pub/OpenBSD/6.6/packages/amd64/ http://ftp.openbsd.org/pub/OpenBSD/6.6/packages/amd64/ - does that mean there's arguably a hole in the base distro? In any case, nice work. Question. Besides randomizing packet lengths (preferably with minimum and maximum tunables (per each packet type) that each site can change, to further add entropy), what else can be done to mitigate against this family of attack? Asking as someone interested in developing "ubiquitous secure container" type protocols that are application- and use-case-specific but (theoretically) high-stakes.

colmmacc 7y ago
Best practice for preventing traffic analysis is to send encrypted messages of a constant size, at a constant rate. It's better to pad to a max size, than to randomize the packet length. Sending at a constant rate may not always be feasible or cost-effective, but it can still be useful to send data at a constant rate for a minimum block of time.
- zzzcpan 7y ago
  It's just easy to implement constant rate, hence why it's best practice. But adaptive rate might be ok, like, for example, if you only pick from 4 different rates and adapt exactly every second, you only leak 2 bits of information per second. The question then becomes which leaking rate is safe enough in practice?
  - colmmacc 7y ago
    constant max rate tends not to be practical in remote roaming VPNs for cost and contention reasons. It would blast through typical mobile usage rates, for example, or get you throttled on a shared wifi network.
- GhettoMaestro 7y ago
  Ya - the NSA calls this "LINK MASKING". Makes a communication channel look busy 100% of the time at at predefined bit rate. Pretty much worthless to traffic analysis, at the cost of burning extra bits on the wire when not actually sending traffic. Mostly only used in Military applications, and specifically where you think an advanced adversary is going to do things like traffic analysis, etc.

mondoshawan 7y ago

"This attack did not work against any Linux distribution we tested until the release of Ubuntu 19.10, and we noticed that the rp_filter settings were set to “loose” mode. We see that the default settings in sysctl.d/50-default.conf in the systemd repository were changed from “strict” to “loose” mode on November 28, 2018, so distributions using a version of systemd without modified configurations after this date are now vulnerable. Most Linux distributions we tested which use other init systems leave the value as 0, the default for the Linux kernel." Anybody happen to know why systemd decided this was something they should be messing with?

arianvanp 7y ago
`git blame` is your friend sysctl.d: switch net.ipv4.conf.all.rp_filter from 1 to 2 This switches the RFC3704 Reverse Path filtering from Strict mode to Loose mode. The Strict mode breaks some pretty common and reasonable use cases, such as keeping connections via one default route alive after another one appears (e.g. plugging an Ethernet cable when connected via Wi-Fi). The strict filter also makes it impossible for NetworkManager to do connectivity check on a newly arriving default route (it starts with a higher metric and is bumped lower if there's connectivity). Kernel's default is 0 (no filter), but a Loose filter is good enough. The few use cases where a Strict mode could make sense can easily override this. The distributions that don't care about the client use cases and prefer a strict filter could just ship a custom configuration in /usr/lib/sysctl.d/ to override this. https://github.com/systemd/systemd/commit/230450d4e4f1f5fc9fa4295ed9185eea5b6ea16e https://github.com/systemd/systemd/commit/230450d4e4f1f5fc9f... There was NEWS entry for this here: https://github.com/systemd/systemd/blob/230450d4e4f1f5fc9fa4295ed9185eea5b6ea16e/NEWS#L73-L80 https://github.com/systemd/systemd/blob/230450d4e4f1f5fc9fa4... as to _why_ they would make the change? Systemd trieds to be project for distros to have a sane base system. Introducing this change was probably something they deemed as being useful in a base system. Especially in the context of laptops and mobile devices this default seems sane.
- 0xbadcafebee 7y ago
  > Systemd trieds to be project for distros to have a sane base system Systemd tries to force a monolithic operating system to behave like a microkernel, and uses propaganda and manipulation to enforce whatever preference its designers have for how all systems should run onto all the Linux distributions it can. You can call that 'sane', I'll call it totalitarian empire-building.

hamandcheese 7y ago

> This vulnerability works against OpenVPN, WireGuard, and IKEv2/IPSec > It should be noted, however, that the VPN technology used does not seem to matter I was wondering which VPNs, but after reading it seems that it doesn’t matter the VPN but is instead a kernel bug? I wonder if this also affects a host that has two network interfaces, with no VPN involved whatsoever? Also, it’s good to see Jason Donenfeld involved, which many will likely recognize as the author of Wireguard.

tolleydbg 7y ago
Donenfeld has been great and makes a wonderful product. We've been using WireGuard as our primary VPN solution for a long time now and don't anticipate a change anytime soon. I think this class of attack can best be mitigated at the kernel level, but unfortunately, side-channels exist anywhere that maintains state, and this might be a more general routing problem.
- zx2c4 7y ago
  We're working around this CVE in WireGuard's wg-quick(8) with a rule something like: iptables -t raw -I PREROUTING ! -i wg0 -d 10.182.12.8 -m addrtype ! --src-type LOCAL -j DROP where wg0 is the WireGuard interface and 10.182.12.8 is the local IP of the interface. This says to drop all packets that are sent to that IP address that aren't coming from the WireGuard interface. And it's done very early in netfilter, in the "raw" table. I don't like having to use iptables in wg-quick(8), and so Willy Tarreau and I have been discussing a deeper kernel-level fix, which should be posted to netdev@ sometime soon.
  - Hello71 7y ago
    I was wondering why wireguard suddenly gained a mandatory iptables dependency...
    - zx2c4 7y ago
      Yea... not nice at all. And now we'll probably have to simultaneously support nftables. This isn't a route I really want to go down. Suggestions on alternatives are welcome.

mirages 7y ago

Sorry if I seem stupid but what are the security implications ? As far as I understood the attacker can : - Detect an active VPN connection (and maybe close it/monitor it) - Attempt to inject packets : for this part I am skeptical of the usefulness. The connection between the server and the client are normally encrypted, meaning that the injected packets will be dropped or can be used to forcefully close the connection making it a DoS attack.

ratiolat 7y ago
Perhaps this means that the attacker can insert traffic and kernel handles it like it came from the tunnel interface? ELI5 would be in order indeed.
loeg 7y ago
They can inject TCP data which looks — to the application — like it came over the VPN, but it didn't actually. The vulnerability here is unencrypted TCP streams running (purportedly) over a VPN. Not TLS streams (HTTPS and HTTP/2.0+), unless you've also got a TLS 0-day. (And maybe also unencrypted UDP sessions and unencrypted services, but that's less clear to me.)
- bscphil 7y ago
  Oh my god. I had to read this twice, but what you said here finally made me realize the severity of this. Thanks. (I also now understand why the kernel developers might be looking at it as a vulnernablity. Neighbors in your subnet should not be able to inject packets into your computer's internal routes.)
tetha 7y ago
> The connection between the server and the client are normally encrypted Keyword: Normally. What about DNS? DNS without DNSSEC via an untrusted AP cannot be trusted, that's for certain. Hence, you recommend your user to activate VPN first to avoid such attacks because now it's encrypted. Suddenly, that might change and your trusted-page.internal resolves to a different hostname. This /should/ be preventable via HSTS and certs for example with HTTP, but those are assumptions again. Or what about RDP? It's not encrypted, so you hide it in a VPN - mostly due to the cleartext password. But suddenly there's a vector that might be able to inject data into an RDP stream inside a VPN connection. > - Detect an active VPN connection (and maybe close it/monitor it) Not just that. They can detect TCP connections inside the VPN connection. Hence, you can start tracking if users of an access point accept anti-gov.com even if they go through a VPN. It might be detectable on the client side and it should be possible to mitigate this via configuration, but that's still plenty scary.

jiveturkey 7y ago

This doesn't seem nearly as bad as 'hijacking' implies. > The attacker can now inject arbitrary payloads into the ongoing encrypted connection using the inferred ACK and next sequence number. First, it's hard to get to this point. Then, you're injecting garbage because you don't know the payload encryption keys. So it's just disruptive, even though yes technically it is 'hijacking'. Unless I missed something, of course! I found the writeup to be vague on the payload encryption point. It should have explicitly stated the impact one way or the other.

aaronmdjones 7y ago
For your first point, this isn't hard if you're in the appropriate position (adjacent or upstream), as the post details. There are already tools available that can do all of this, example invocations are also there. For your other point, they don't need to know the keys (unless the traffic travelling /over/ the VPN is /also/ encrypted). That's the whole point. This is about being able to trick the machine into accepting traffic for a connection, from an interface that the connection's traffic isn't travelling over. If it's plaintext within the VPN, this sidesteps the VPN interface and you can indeed inject your own malicious plaintext traffic into that connection.
- jiveturkey 7y ago
  1. Yes, you have to already be in position. That's limited to specific actors. Once that is achieved, you have to probe for specific IP addresses the victim might be connected to. IOW a list you are targetting. Then the port probing and seqno guessing. These facts reduce the impact because it's not just "be a guy on the internet", eg like if there were an open database of PII sitting there for the taking, only needing discovery to find it. In no way am I claiming the attack isn't feasible. It's definitely a real risk, beyond the theoretical. 2. Thanks, got it. That makes more sense. I think I actually like this vuln. It reinforces the need for defense in depth. It reduces a takeover to an annoyance (could be critical for some apps, yes) assuming you use TLS at the app layer.
  - aaronmdjones 7y ago

GhettoMaestro 7y ago

I'm having trouble understanding why the randomly sprayed bits from the attacker to the client are even accepted at all at the crypto boundary. Shouldn't it hard-fail (not decrypt) due to an invalid key?

loeg 7y ago
The bits aren't VPN-framed packets; the attack isn't VPN specific. They're just sending ordinary TCP setup packets and ordinary TCP stream packets with fudged parameters until they find the right ones by brute force. It's a really noisy attack that uses weaknesses in packet routing to find existing TCP sessions and inject into them. It's also, like, an escalation from a relatively high privilege level. You need both a passive network observer (compromised router or ISP) and a noisy, active LAN device (to inject IP addresses that a router would filter as bogons). That's not to say this is crazy hard; these are definitely within the reach of a motivated attacker. Routers were the original the-S-in-IoT-is-for-security, and if you've got an IoT type device on the LAN it's probably vulnerable once you've popped the router.

knolax 7y ago

It seems for the attacker to find out about active connections to any given website they have to already know the IP of the website and then brute force the virtual ports. Being able to determine what website the target has an active connection to without prior knowledge would probably take even more brute forcing. A small solice.

loeg 7y ago
This is a very noisy attack, for sure.

arielb1 7y ago

> nping --tcp --flags SA --source-ip 192.168.12.1 --dest-ip 10.8.0.8 -- rate 3 -c 3 -e ap0 --dest-mac 08:00:27:9c:53:12 Why is Linux accepting packets coming from one interface into an IP address belonging to a different interface? It feels like it is "forwarding" the packets internally, but `ip_forward` is turned off. Is there any case where this behavior is legitimately useful?

ajross 7y ago
IP addresses don't "belong" to interfaces in the general case. It's just a hard problem. In fact there are lots of multi-homed use cases where you want to internally route packets across interfaces without an affirmative mapping of what address is supposed to be used where. For the specific case of point to point VPNs, there's a rule that makes sense. But that's not part of the network stack per se and there's no way to enforce it generically.
- loeg 7y ago
  Do network stacks drop 127.0/8 packets from external interfaces today? Superficially (I'm not an experienced TCP/IP or routing stack developer, although I do work in the kernel) it seems like the same treatment could be used for VPN-registered interface addresses. You just need an API to specify "I'm a VPN interface" when the device is created or the IP assigned, no?
- arielb1 7y ago
  Is there a place where I can read about these cases?
loeg 7y ago
From the referenced systemd commit, > such as keeping connections via one default route alive after another one appears (e.g. plugging an Ethernet cable when connected via Wi-Fi).
- arielb1 7y ago
  How is this supposed to work? How will the packet destined to the WiFi IP address get to the Ethernet interface?

ufov2 7y ago

Is this mitigated by iptables filter firewall? For example, FORWARD policy DROP and no other rules?

jlgaddis 7y ago
Yes, if you are doing firewalling properly. The problm is that most people aren't and are just using their distribution defaults.

nif2ee 7y ago

I just can't understand how this attack can be implemented in real world, it not only needs having an adjacent network access to the VPN client but most importantly knowing the destination of a currently open TCP connection encrypted by the VPN and then guessing the right sequence numbers. Since most TCP connections are also carrying TLS these days, this attack is pretty useless in the vast majority of real world situations I can think of.

tinix 7y ago

Can we merge these? I posted this 10 hours earlier here: https://news.ycombinator.com/item?id=21709693 https://news.ycombinator.com/item?id=21709693

jlgaddis 7y ago
There's really nothing useful there that isn't also here.

Gabrielfair 7y ago

Disseminating vulnerabilities like these to less savvy people should be a critical imparitive for the next century. I imagine a team of animators at a non-profit or foundation that illustrate how these attacks work at different levels of technical backgrounds.

derpherpsson 7y ago

Correct me if I am wrong, but... If I just make sure that incomming packets that are destined for the VPN LAN are dropped, this attack does not work? Of course there are such rules in our firewalls?? Is everyone walking around without any firewall filtering nowadays? How is this a bug? Maybe I am just stupid. Did I miss something?

rasz 7y ago
TCP/IP stack was dropping this by default .. until systemd decided to switch the default https://news.ycombinator.com/item?id=21713479 https://news.ycombinator.com/item?id=21713479
- yrro 7y ago
  The default behaviour of the kernel is no rp filtering at all. Older versions of systems enabled strict to filtering, no doubt causing the same sort of complaints from people who like to complain about that sort of thing. Newer versions relaxed this to loose rp filtering for the reasons explained in the commit message.

e12e 7y ago

> The access point can then determine the virtual IP of the victim by sending SYN-ACK packets to the victim device across the entire virtual IP space (the default for OpenVPN is 10.8.0.0/24). So, would a bit of address space randomization mitigate step one for ipv6? fd00::/8 is pretty big, right? Even just picking a random IP in a random /64 from that /8 should help? Or am I missing something? Also interesting comments in the reply on the list regarding "policy based" vpns. I wonder if some subset of that infrastructure could be used by Wireguard without completing it all the way out of its current nice and secure-by-simplicity design? https://seclists.org/oss-sec/2019/q4/123 https://seclists.org/oss-sec/2019/q4/123 > Only route based VPNs are impacted. In comparison, policy based VPNs are not impacted (On Linux only implementable using XFRM, which is IPsec on Linux specific) unless the XFRM policy's level is set to "use" instead of "required" (default)) because any traffic received that matches a policy (IPsec security policy) and that is not protected is dropped.

3xblah 7y ago

"We discovered a vulnerability in Linux, FreeBSD, OpenBSD, MacOS, iOS, and Android..." What about NetBSD. This also appears to be wireless only, i.e., the need for attacker to have control over the AP. Am I reading this wrong.

3xblah 7y ago
I was reading it wrong. It is not wireless only. Technically, it only requires that the victim is sharing a local network with the attacker.

mirimir 7y ago

> I am reporting a vulnerability that exists on most Linux distros, and other *nix operating systems which allows a network adjacent attacker to determine if another user is connected to a VPN, the virtual IP address they have been assigned by the VPN server, and whether or not there is an active connection to a given website. I'm not sure that I understand exactly what "network adjacent attacker" means. But I'm guessing that it means an attacker on the same subnet. And that it doesn't involve actually hacking VPN encryption. But isn't it well understood that sharing LANs with untrusted neighbors is hugely risky? At least, I always segregate critical machines in protected subnets. Or am I missing something? Edit: OK, I was missing that this focuses on using VPNs via WiFi APs. And depends on the AP being malicious. So yeah, this is a huge issue, for that use case.

GMLOOKO 7y ago