10 ms·

Lies vs. Statistics: VPN Virtual Server Locations

94 points by mirimir 7y ago

mirimir 7y ago
This is an article that I wrote recently for Restore Privacy. The focus is identifying the undisclosed use of virtual locations by some popular VPN services. But that's not why I've submitted it to HN. I'd appreciate comment and criticism about the criterion that I've used for virtual locations. That is, where apparent signal velocity is greater than the speed of light, I conclude that the disclosed server location is implausible. And so it must be a virtual location. I calculate apparent signal velocity by dividing twice the calculated great circle distance between server and ping probe, by the minimum round-trip time (minrtt). For server locations, I rely primarily on whatever the VPN service has disclosed. I'd also appreciate comment and criticism about my explanation of how VPN services achieve virtual locations ("How Virtual Server Locations Are Possible"). Basically that they exploit discrepancies between location information that's associated with IP addresses, and BGP advertisements based on address announcements. Finally, for 195 servers that are apparently located near Nuland, I found an interesting pattern in the ping data.[0] For ping probes in Europe, minrtt is 1-30 msec. Then there’s a gap in observed minrtt across the Atlantic, with apparently much faster transmission. I’m guessing that's because trans-Atlantic cables don’t have many routers. Next there are groups for eastern and western North America, with faster transmission than within Europe, but slower than across the Atlantic. That probably also reflects router density. Last, there's a jump to Asia and Australia. Does that make sense? 0) https://cdn-resprivacy.pressidium.com/wp-content/uploads/2019/12/IPv4-Clusters-Colocation-Near-Nuland-by-City-and-Region.png https://cdn-resprivacy.pressidium.com/wp-content/uploads/201...
- SlowRobotAhead 7y ago
  Makes sense mostly if your only using ping from big location differences. If you were trying to use this for exclusively inside USA though, routing will instantly muddy your data. One of our offices has dual ISP connections and they are drastically different ping to another facility. One ISP pays for better routing I guess.
  - mirimir 7y ago
    Yes, I have seen that. There are some examples in a guide that I wrote for IVPN.[0] For example, the lowest-rtt probe for an IVPN server in Reykjavik, IS is in Amsterdam, NL.[1] The minrtt for a probe in Iceland is greater. However, I'm just looking at signal velocity for the probe with the smallest minrtt. So probes where routing to the server is less direct will just have greater minrtt for a given geographic separation. That is, lower signal velocity. So that doesn't affect my analysis. 0) https://www.ivpn.net/privacy-guides/how-to-verify-physical-locations-of-internet-servers https://www.ivpn.net/privacy-guides/how-to-verify-physical-l... 1) https://www.ivpn.net/blog/wp-content/img/IVPN-is1.gw_.ivpn_.net-All-Probes-under-4000-km.png https://www.ivpn.net/blog/wp-content/img/IVPN-is1.gw_.ivpn_....
- myself248 7y ago
  It all makes perfect sense, the article was just a bit tl;dr. I got about halfway through it and my eyes glazed over. I'd love to learn more about the mechanics of gathering the data. You say you used Ping.pe, CASM, and MapLatency. It sounds like you had to scrape the Ping.pe results because they don't provide an API? Did you consider using RIPE ATLAS probes? Did you do any cross-verification between services where they both claimed to have probes in the same location? It's all fascinating stuff and very, very thorough! I hope the story gets picked up by an outlet with a journalist who can distill the salient points into an article short enough for casual reading.
  - mirimir 7y ago
    Yeah. It's a lot of stuff to cover. I should maybe have put all the details in GitHub, and just presented the key results. I did scrape ping.pe using a shell script with numerous lines like this: > google-chrome --headless --run-all-compositor-stages-before-draw --virtual-time-budget=25000 --print-to-pdf='[IPv4-date].pdf' http://ping.pe/[IPv4] http://ping.pe/[IPv4] && sleep 4 Also, if I hit it too long, it started providing bullshit results. So I ended up doing it from Whonix, so each request came from a different IPv4. And I did try to contact them to arrange payment. But got no reply. I did read about RIPE ATLAS. But it seemed necessary to host a probe. And then get credits to use other probes. But maybe I misunderstood. So is there an API? I didn't do any systematic cross-verification between ping services. For each VPN server IPv4, I just got the minimum minrtt value, for all measurements from all ping services. However, I did verify that ping probes generally yielded plausible minrtts for VPN server IPv4 that didn't show implausible minrtt with any ping probe.[0 vs 1] 0) https://cdn-resprivacy.pressidium.com/wp-content/uploads/2019/12/server-probe_distance_vs_minrtt_all_ipv4_all_selected_probes.png https://cdn-resprivacy.pressidium.com/wp-content/uploads/201... 1) https://cdn-resprivacy.pressidium.com/wp-content/uploads/2019/12/server-probe_distance_vs_minrtt_excluding_supra_lightspeed_ipv4_all_selected_probes.png https://cdn-resprivacy.pressidium.com/wp-content/uploads/201...
    - bathory 7y ago
      Hosting a RIPE Atlas probe means you will generate credits, which you can then spend on measurements. All Atlas measurements require those credits as a form of payment. In short: there is no free API. You can also receive credits as a donation, e.g. by someone who hosts a probe. This is an interesting article and I’m sure someone with a constant supply of credits would surely donate a few of them, for you to continue this research.
- maxfan8 7y ago
  Nice article! I'm curious if you would be willing to open source the code and data behind this article – I'm interested in verifying other VPN providers and services as well as playing around with some of the data.
  - mirimir 7y ago
    Thanks :) The code is pretty simple, really. Just scripts for collecting data. I did the analysis in Calc, mostly. And some in Excel. But I can explain in more detail on GitHub. I'll look into open sourcing all of the data.
    - mirimir 7y ago
      I got approval to put the data on GitHub. I need to collect it from a bunch of Calc workbooks, and dump it in a text file. I also need to write up the methods in more detail, and that involves collecting stuff from multiple VMs on multiple hosts. So it'll take me a while.
- redis_mlc 7y ago
  > I’m guessing that's because trans-Atlantic cables don’t have many routers. That's correct. Also you get the fastest long-haul transmission with fiber optic, c/1.45, vs. roughly c/2 for wire. https://en.wikipedia.org/wiki/Optical_fiber#Index_of_refraction https://en.wikipedia.org/wiki/Optical_fiber#Index_of_refract...
  - mirimir 7y ago
    Cool. The cutoff does seem to be about 0.7c overall.[0] Although I allowed some slack above that, to be conservative. 0) https://cdn-resprivacy.pressidium.com/wp-content/uploads/2019/12/Location-Plausibility-vs-Ping-Velocity-Cutoff-for-Locations-Not-Disclosed-as-Virtual.png https://cdn-resprivacy.pressidium.com/wp-content/uploads/201...
- donavanm 7y ago
  I spent quite a few years working in the isp/cdn/dns space. Is “virtual location” an idiomatic term for VPN users? The more general industry would probably say youre looking at Points of Presence (“PoPs”). The core of the article is one of (mis)representation. The VPN providers are trying to present broad network coverage with a minimum of expenditure, and the clients are concerned with actual geophysical location. Virtual Locations seems to be a term designed to comingle the concepts, and obfuscate actual location and connectivity. The clustering is because there simply arent that many physical locations with connectivity to various networks (Internet Exchange Points “IXPs”). Those IXPs are themselves often clustered and associated with a few nearby physical facilities (DCs or Colo(cation) for hosting general compute servers. You mention companies “leasing” address blocks. Thats probably more correct to say theyre “delegated” from NSPs to clients like the vpn companies. In general Id find whois/rir data to be of very little use. Theres little to no association or restriction for either of those and where the IP addresses are actually announced or used. It may not matter for your use/methodology but latency for ICMP is unreliable. Many networks will use a lower traffic priority for it, introducing variable loss/jitter/delay. If at all possible actual TCP or UDP is better for evaluating RTT. One thing you might look at is the vpn provider network adjacencies in different locations. Many/most NSPs handily include some sort of location identifier (like iata code) in their device names. From those adjacent devices you can probably infer city at least. PeeringDB and/or looking glass portals are another way to deduce network topology and connectivity, possibly narrowing it down to specific facilities. Lastly your ICMP replies are inherently limited in what you see. Its very likely that there are underlying networks (ie MPLS, or GRE tunnels) that you cant see. Itd be very easy to have an interconnect in AMSIX and tunnel it over to belgium or germany. You wouldnt see that in DNS or ICMP replies, but could infer it through latency.
  - hug 7y ago
    > the clients are concerned with actual geophysical location. Are they, though? This is, of course, completely anecdotal, but the only real reason I will change the 'location' of my VPN endpoint is exactly that. My suspicion is that most of the VPN users concerns about geography match mine and actually boil down to whether or not it will trick a geofence. There's going to be some ways in which you can conflate those concepts, since obviously 'virtual locations' will be detectable by some means, as evidenced by the fact that mirmir managed it, but I suspect for a lot of consumers good enough is good enough.
- EternalAugust 7y ago
  Very interesting article. One thought I had. At the beginning where you talk about about reasons for VPN virtual locations I was reminded of an article I read like two years ago that talked a little bit about VPNs using IPs apparently registered in North Korean IP-space[0]. Since then I have scratched my head wondering why exactly a VPN provider would offer something like that. I never looked through the data to confirm that these IPs were abused, but I assume these IPs are (or would be) attractive to black hats who want to obfuscate attack patterns and confuse/alarm unknowing threat analysts. Would VPN providers like HMA actually offer such a thing to boost revenue? Or maybe they were just gimmicks. I don't know... [0] https://blog.trendmicro.com/trendlabs-security-intelligence/a-closer-look-at-north-koreas-internet/ https://blog.trendmicro.com/trendlabs-security-intelligence/... (I am pretty sure none of the IPs mentioned in the article are geolocated to NK anymore; also idk if HMA still offers IPs virtually located in NK)
  - mirimir 7y ago
    As I recall, HMA still offers a server "in" NK. I really have no clue why HMA etc offer so many odd locations. I guess it's the "at least one in every country" brag. Maybe, as you say, to confuse people. But then, HMA doesn't have a good track record for obfuscating abuse ;)
- tbenst 7y ago
  Fun read, I learned a lot! Thanks for writing. I’m curious if you used three or more probe servers in distinct locations if you might be able to roughly triangulate server location? To increase accuracy, seems the underlying topology should ideally be taken into account. Probably more work than it’s worth, would it be cool to have some kind of skeleton for the wires, to calculate distance metric with higher accuracy than an arc.
  - mirimir 7y ago
    Thanks :) It was fun to do, as well. Although often tedious. I played some with triangulation, with no joy. From the SE thread "Triangulate with Ping [closed]"[0] I get that it's impossible. Latency depends more on device delays than on distance. Within densely populated continents, router/switch count does depend more or less on distance. But there's too much variability. Or at least, none of the models that I tried converged. It might be doable, with enough information about Internet structure. So you could deconvolute device count and distance. But that would be a lot of data, and I have no real clue where to get it. The NSA? 0) https://electronics.stackexchange.com/questions/68619/triangulate-with-ping https://electronics.stackexchange.com/questions/68619/triang...
    - myself248 7y ago
      Also, a lot of circuits take paths that don't show up on traceroute, because the IP packets are just a payload bitstream riding another transport that doesn't interact with them, doesn't decrement TTL, et cetera. It's quite normal to have a router in Toledo talking to a router in Detroit but the packets ride a transparent SONET circuit to Chicago and back, so the two hops look 50 miles apart but are about 500 fiber-miles apart with consequent speed-of-light delays. And the folks who play at the IP router level are often entirely ignorant of the layers below, to much confusion. (I worked on these systems for years. A conversation with another network admin about this ignorance became the genesis of the Anything But Ethernet contest, which we ran for several years at a midwest tech con. Hijinks ensued.) Learning anything about the layers below, that don't interact with IP packets, is gonna be hard, I concur. You're flying blind except for speed-of-light, and have a lot of other delays to solve out. "Might be doable" is about my conclusion, too. You'd need a whole crapton of probes, with accurate locations, and a whole crapton of math, which is way over my head. I feel like the equations are gonna start to resemble the way GPS works, but that's the thing — GPS does work! My gut feeling is that it's worth a try.
- brian_herman__ 7y ago
  Thanks for this, great article hope to see more of your work in the future.
  - mirimir 7y ago
    Thanks :) I'll do a Show HN when the GitHub repo, with methods and data, is up. It may be a while, though.
hnews_account_1 7y ago
Tangential to this article, can someone clarify on what's good / bad about NordVPN? I've been trusting their services for 2 years now, and I signed up on some ridiculously good offer of 3 years for <$100 a year ago, so I'm locked in for 4 full years. I did this because I trusted the service from reading reviews, but places like reddit are always shitting on NordVPN, both due to their ads (obv not the issue for me) and because of some nebulous "concerns" that no one has been able to articulate properly for me. I know about the recent hack, but it seems like all of their precautions actually worked in protecting the consumers, and no logs were ever shown to have existed or been stolen.
- the_pwner224 7y ago
  They showed with the recent hack that, to them, maintaining their marketing & reputation is a higher priority than informing their users of possible security incidents, and that they would rather try to cover stuff up than be upfront about matters. Or at least that's what I gathered reading the recent HN threads.
  - hellcow 7y ago
    They knew about a vulnerability for MONTHS. They did not fix it and kept it secret. It only came out because of a whistleblower. This is not the behavior of an organization deserving ANY level of trust.
- mirimir 7y ago
  I don't have a strong opinion about the hack. And their US residential proxies do work very reliably for Disney+. There is the possibility that they're using services that obtain those proxies deceptively. But I tracked down one of them, and the guy confirmed that he's knowingly selling his bandwidth. And yes, it's just anecdotal.
- catalogia 7y ago
  > "but places like reddit are always shitting on NordVPN, both due to their ads (obv not the issue for me)" It's a problem for me. Their ads use scare tactics and make deceptive if not outright false claims. Unethical ads come from unethical corporations. I can't see any reason to give such a corporation the benefit of the doubt by assuming their unethical behavior is limited to how they advertise.
sdan 7y ago
This is why I just use AlgoVPN. I know where my traffic is going and can install it on any cloud instance I want. At the moment I'm running it on my laptop, phone, and my RPI cluster.
- EternalAugust 7y ago
  I think the problem with personal VPNs is correlating incoming with outgoing traffic is trivial for a global adversary (NSA, ISPs). When you use a service provider where a bunch of people are connecting to a single server correlating input with output becomes much more difficult, I seriously doubt it would be impossible though.
  - bradknowles 7y ago
    If your threat model seriously includes the NSA, then you shouldn’t be using IP at all. Or any kind of electronic communication, for that matter. If your threat model includes your ISP, but does not extend to nation-state level adversaries, then a good private VPN should be a decent enough solution, although a public VPN might be easier and still adequate. Source: I personally pissed off the Director of the NSA in November of 1992 (see http://www.shub-internet.org/brad/cacm92nov.html http://www.shub-internet.org/brad/cacm92nov.html ). At the time, my clearance was Top Secret/SCI, and I had been read onto multiple compartments — including the ones for ECHELON, KEYHOLE, etc.... So far as I know, I am still on their shit list, albeit not as high as Snowden or Binney.
    - lbatx 7y ago
      Clicked on your link just to get on the list (or move up, I suppose). ;-)
    - mirimir 7y ago
      Yes, this is true. So the best bet is staying off the shit lists of the NSA and other global adversaries.
    - ShorsHammer 7y ago
xt00 7y ago
It would seem that ultimately the only real way to get privacy is to either create a verifiable end to end encryption including the content server not having logs and not compromised and the other way would be pooling of user data in a way that spreads the blame across many people. Otherwise basically everything you do is trackable even if you use a vpn. Is there any other way?
- dannyw 7y ago
  If you want real privacy at the consequence of slow speeds and unfortunately a worse browsing UX, you should use Tor. Every service that is widely available / permissionless AND offers real privacy, will end up getting degraded, CAPTCHA'd or blocked. That shows it works1
  - mirimir 7y ago
    Yes, agreed. But I don't trust Tor either, so I'd first use a nested VPN chain. Just in case the entry guard is malicious. And to complicate the traffic analysis a little. I could go on about this. But I'll be writing more about all this.
- mirimir 7y ago
  Yes, everything is trackable by global adversaries like the NSA. However, even the NSA doesn't have enough resources to track all 4.5 billion Internet users. So the key is not attracting their attention.
dannyw 7y ago
What's the best VPN provider now that PIA got acquired by an Israeli spyware company?
- xvector 7y ago
  Source [1]. Truly insane. I'd never have expected this from a company that cares so much about privacy. Just cancelled my ProtonVPN subscription. I was going to recommend ProtonVPN but they have a pretty sketch background/history too. [2] Cloudflare's WireGuard-based Warp VPN [3] might be worth keeping an eye on, but Cloudflare makes it clear that users should not be expecting the privacy guarantees of a "traditional VPN" [4]: > WARP is not designed to allow you to access geo-restricted content when you’re traveling. It will not hide your IP address from the websites you visit. If you’re looking for that kind of high-security protection then a traditional VPN or a service like Tor are likely better choices for you. Perhaps we simply shouldn't use VPN services. [5] [1]: https://news.ycombinator.com/item?id=21679682 https://news.ycombinator.com/item?id=21679682 [2]: https://news.ycombinator.com/item?id=19266078 https://news.ycombinator.com/item?id=19266078 [3]: https://news.ycombinator.com/item?id=19542835 https://news.ycombinator.com/item?id=19542835 [4]: https://blog.cloudflare.com/announcing-warp-plus/ https://blog.cloudflare.com/announcing-warp-plus/ [5]: https://news.ycombinator.com/item?id=16371030 https://news.ycombinator.com/item?id=16371030
  - mirimir 7y ago
    Not to put too fine a point on it, they lost money for too long, and ended up needing a buyer. I always did wonder how they could offer so much for so little money.
    - dannyw 7y ago
      How can VPN providers lose money, when a DO Droplet for $5 a month can probably host 5 casual subscribers using things on and off?
  - deleted 7y ago
    [deleted]
  - 7y ago

94 points by mirimir 7y ago

This is an article that I wrote recently for Restore Privacy. The focus is identifying the undisclosed use of virtual locations by some popular VPN services. But that's not why I've submitted it to HN. I'd appreciate comment and criticism about the criterion that I've used for virtual locations. That is, where apparent signal velocity is greater than the speed of light, I conclude that the disclosed server location is implausible. And so it must be a virtual location. I calculate apparent signal velocity by dividing twice the calculated great circle distance between server and ping probe, by the minimum round-trip time (minrtt). For server locations, I rely primarily on whatever the VPN service has disclosed. I'd also appreciate comment and criticism about my explanation of how VPN services achieve virtual locations ("How Virtual Server Locations Are Possible"). Basically that they exploit discrepancies between location information that's associated with IP addresses, and BGP advertisements based on address announcements. Finally, for 195 servers that are apparently located near Nuland, I found an interesting pattern in the ping data.[0] For ping probes in Europe, minrtt is 1-30 msec. Then there’s a gap in observed minrtt across the Atlantic, with apparently much faster transmission. I’m guessing that's because trans-Atlantic cables don’t have many routers. Next there are groups for eastern and western North America, with faster transmission than within Europe, but slower than across the Atlantic. That probably also reflects router density. Last, there's a jump to Asia and Australia. Does that make sense? 0) https://cdn-resprivacy.pressidium.com/wp-content/uploads/2019/12/IPv4-Clusters-Colocation-Near-Nuland-by-City-and-Region.png https://cdn-resprivacy.pressidium.com/wp-content/uploads/201...

SlowRobotAhead 7y ago
Makes sense mostly if your only using ping from big location differences. If you were trying to use this for exclusively inside USA though, routing will instantly muddy your data. One of our offices has dual ISP connections and they are drastically different ping to another facility. One ISP pays for better routing I guess.
- mirimir 7y ago
  Yes, I have seen that. There are some examples in a guide that I wrote for IVPN.[0] For example, the lowest-rtt probe for an IVPN server in Reykjavik, IS is in Amsterdam, NL.[1] The minrtt for a probe in Iceland is greater. However, I'm just looking at signal velocity for the probe with the smallest minrtt. So probes where routing to the server is less direct will just have greater minrtt for a given geographic separation. That is, lower signal velocity. So that doesn't affect my analysis. 0) https://www.ivpn.net/privacy-guides/how-to-verify-physical-locations-of-internet-servers https://www.ivpn.net/privacy-guides/how-to-verify-physical-l... 1) https://www.ivpn.net/blog/wp-content/img/IVPN-is1.gw_.ivpn_.net-All-Probes-under-4000-km.png https://www.ivpn.net/blog/wp-content/img/IVPN-is1.gw_.ivpn_....
myself248 7y ago
It all makes perfect sense, the article was just a bit tl;dr. I got about halfway through it and my eyes glazed over. I'd love to learn more about the mechanics of gathering the data. You say you used Ping.pe, CASM, and MapLatency. It sounds like you had to scrape the Ping.pe results because they don't provide an API? Did you consider using RIPE ATLAS probes? Did you do any cross-verification between services where they both claimed to have probes in the same location? It's all fascinating stuff and very, very thorough! I hope the story gets picked up by an outlet with a journalist who can distill the salient points into an article short enough for casual reading.
- mirimir 7y ago
  Yeah. It's a lot of stuff to cover. I should maybe have put all the details in GitHub, and just presented the key results. I did scrape ping.pe using a shell script with numerous lines like this: > google-chrome --headless --run-all-compositor-stages-before-draw --virtual-time-budget=25000 --print-to-pdf='[IPv4-date].pdf' http://ping.pe/[IPv4] http://ping.pe/[IPv4] && sleep 4 Also, if I hit it too long, it started providing bullshit results. So I ended up doing it from Whonix, so each request came from a different IPv4. And I did try to contact them to arrange payment. But got no reply. I did read about RIPE ATLAS. But it seemed necessary to host a probe. And then get credits to use other probes. But maybe I misunderstood. So is there an API? I didn't do any systematic cross-verification between ping services. For each VPN server IPv4, I just got the minimum minrtt value, for all measurements from all ping services. However, I did verify that ping probes generally yielded plausible minrtts for VPN server IPv4 that didn't show implausible minrtt with any ping probe.[0 vs 1] 0) https://cdn-resprivacy.pressidium.com/wp-content/uploads/2019/12/server-probe_distance_vs_minrtt_all_ipv4_all_selected_probes.png https://cdn-resprivacy.pressidium.com/wp-content/uploads/201... 1) https://cdn-resprivacy.pressidium.com/wp-content/uploads/2019/12/server-probe_distance_vs_minrtt_excluding_supra_lightspeed_ipv4_all_selected_probes.png https://cdn-resprivacy.pressidium.com/wp-content/uploads/201...
  - bathory 7y ago
    Hosting a RIPE Atlas probe means you will generate credits, which you can then spend on measurements. All Atlas measurements require those credits as a form of payment. In short: there is no free API. You can also receive credits as a donation, e.g. by someone who hosts a probe. This is an interesting article and I’m sure someone with a constant supply of credits would surely donate a few of them, for you to continue this research.
maxfan8 7y ago
Nice article! I'm curious if you would be willing to open source the code and data behind this article – I'm interested in verifying other VPN providers and services as well as playing around with some of the data.
- mirimir 7y ago
  Thanks :) The code is pretty simple, really. Just scripts for collecting data. I did the analysis in Calc, mostly. And some in Excel. But I can explain in more detail on GitHub. I'll look into open sourcing all of the data.
  - mirimir 7y ago
    I got approval to put the data on GitHub. I need to collect it from a bunch of Calc workbooks, and dump it in a text file. I also need to write up the methods in more detail, and that involves collecting stuff from multiple VMs on multiple hosts. So it'll take me a while.
redis_mlc 7y ago
> I’m guessing that's because trans-Atlantic cables don’t have many routers. That's correct. Also you get the fastest long-haul transmission with fiber optic, c/1.45, vs. roughly c/2 for wire. https://en.wikipedia.org/wiki/Optical_fiber#Index_of_refraction https://en.wikipedia.org/wiki/Optical_fiber#Index_of_refract...
- mirimir 7y ago
  Cool. The cutoff does seem to be about 0.7c overall.[0] Although I allowed some slack above that, to be conservative. 0) https://cdn-resprivacy.pressidium.com/wp-content/uploads/2019/12/Location-Plausibility-vs-Ping-Velocity-Cutoff-for-Locations-Not-Disclosed-as-Virtual.png https://cdn-resprivacy.pressidium.com/wp-content/uploads/201...
donavanm 7y ago
I spent quite a few years working in the isp/cdn/dns space. Is “virtual location” an idiomatic term for VPN users? The more general industry would probably say youre looking at Points of Presence (“PoPs”). The core of the article is one of (mis)representation. The VPN providers are trying to present broad network coverage with a minimum of expenditure, and the clients are concerned with actual geophysical location. Virtual Locations seems to be a term designed to comingle the concepts, and obfuscate actual location and connectivity. The clustering is because there simply arent that many physical locations with connectivity to various networks (Internet Exchange Points “IXPs”). Those IXPs are themselves often clustered and associated with a few nearby physical facilities (DCs or Colo(cation) for hosting general compute servers. You mention companies “leasing” address blocks. Thats probably more correct to say theyre “delegated” from NSPs to clients like the vpn companies. In general Id find whois/rir data to be of very little use. Theres little to no association or restriction for either of those and where the IP addresses are actually announced or used. It may not matter for your use/methodology but latency for ICMP is unreliable. Many networks will use a lower traffic priority for it, introducing variable loss/jitter/delay. If at all possible actual TCP or UDP is better for evaluating RTT. One thing you might look at is the vpn provider network adjacencies in different locations. Many/most NSPs handily include some sort of location identifier (like iata code) in their device names. From those adjacent devices you can probably infer city at least. PeeringDB and/or looking glass portals are another way to deduce network topology and connectivity, possibly narrowing it down to specific facilities. Lastly your ICMP replies are inherently limited in what you see. Its very likely that there are underlying networks (ie MPLS, or GRE tunnels) that you cant see. Itd be very easy to have an interconnect in AMSIX and tunnel it over to belgium or germany. You wouldnt see that in DNS or ICMP replies, but could infer it through latency.
- hug 7y ago
  > the clients are concerned with actual geophysical location. Are they, though? This is, of course, completely anecdotal, but the only real reason I will change the 'location' of my VPN endpoint is exactly that. My suspicion is that most of the VPN users concerns about geography match mine and actually boil down to whether or not it will trick a geofence. There's going to be some ways in which you can conflate those concepts, since obviously 'virtual locations' will be detectable by some means, as evidenced by the fact that mirmir managed it, but I suspect for a lot of consumers good enough is good enough.
EternalAugust 7y ago
Very interesting article. One thought I had. At the beginning where you talk about about reasons for VPN virtual locations I was reminded of an article I read like two years ago that talked a little bit about VPNs using IPs apparently registered in North Korean IP-space[0]. Since then I have scratched my head wondering why exactly a VPN provider would offer something like that. I never looked through the data to confirm that these IPs were abused, but I assume these IPs are (or would be) attractive to black hats who want to obfuscate attack patterns and confuse/alarm unknowing threat analysts. Would VPN providers like HMA actually offer such a thing to boost revenue? Or maybe they were just gimmicks. I don't know... [0] https://blog.trendmicro.com/trendlabs-security-intelligence/a-closer-look-at-north-koreas-internet/ https://blog.trendmicro.com/trendlabs-security-intelligence/... (I am pretty sure none of the IPs mentioned in the article are geolocated to NK anymore; also idk if HMA still offers IPs virtually located in NK)
- mirimir 7y ago
  As I recall, HMA still offers a server "in" NK. I really have no clue why HMA etc offer so many odd locations. I guess it's the "at least one in every country" brag. Maybe, as you say, to confuse people. But then, HMA doesn't have a good track record for obfuscating abuse ;)
tbenst 7y ago
Fun read, I learned a lot! Thanks for writing. I’m curious if you used three or more probe servers in distinct locations if you might be able to roughly triangulate server location? To increase accuracy, seems the underlying topology should ideally be taken into account. Probably more work than it’s worth, would it be cool to have some kind of skeleton for the wires, to calculate distance metric with higher accuracy than an arc.
- mirimir 7y ago
  Thanks :) It was fun to do, as well. Although often tedious. I played some with triangulation, with no joy. From the SE thread "Triangulate with Ping [closed]"[0] I get that it's impossible. Latency depends more on device delays than on distance. Within densely populated continents, router/switch count does depend more or less on distance. But there's too much variability. Or at least, none of the models that I tried converged. It might be doable, with enough information about Internet structure. So you could deconvolute device count and distance. But that would be a lot of data, and I have no real clue where to get it. The NSA? 0) https://electronics.stackexchange.com/questions/68619/triangulate-with-ping https://electronics.stackexchange.com/questions/68619/triang...
  - myself248 7y ago
    Also, a lot of circuits take paths that don't show up on traceroute, because the IP packets are just a payload bitstream riding another transport that doesn't interact with them, doesn't decrement TTL, et cetera. It's quite normal to have a router in Toledo talking to a router in Detroit but the packets ride a transparent SONET circuit to Chicago and back, so the two hops look 50 miles apart but are about 500 fiber-miles apart with consequent speed-of-light delays. And the folks who play at the IP router level are often entirely ignorant of the layers below, to much confusion. (I worked on these systems for years. A conversation with another network admin about this ignorance became the genesis of the Anything But Ethernet contest, which we ran for several years at a midwest tech con. Hijinks ensued.) Learning anything about the layers below, that don't interact with IP packets, is gonna be hard, I concur. You're flying blind except for speed-of-light, and have a lot of other delays to solve out. "Might be doable" is about my conclusion, too. You'd need a whole crapton of probes, with accurate locations, and a whole crapton of math, which is way over my head. I feel like the equations are gonna start to resemble the way GPS works, but that's the thing — GPS does work! My gut feeling is that it's worth a try.
brian_herman__ 7y ago
Thanks for this, great article hope to see more of your work in the future.
- mirimir 7y ago
  Thanks :) I'll do a Show HN when the GitHub repo, with methods and data, is up. It may be a while, though.

hnews_account_1 7y ago

Tangential to this article, can someone clarify on what's good / bad about NordVPN? I've been trusting their services for 2 years now, and I signed up on some ridiculously good offer of 3 years for <$100 a year ago, so I'm locked in for 4 full years. I did this because I trusted the service from reading reviews, but places like reddit are always shitting on NordVPN, both due to their ads (obv not the issue for me) and because of some nebulous "concerns" that no one has been able to articulate properly for me. I know about the recent hack, but it seems like all of their precautions actually worked in protecting the consumers, and no logs were ever shown to have existed or been stolen.

the_pwner224 7y ago
They showed with the recent hack that, to them, maintaining their marketing & reputation is a higher priority than informing their users of possible security incidents, and that they would rather try to cover stuff up than be upfront about matters. Or at least that's what I gathered reading the recent HN threads.
- hellcow 7y ago
  They knew about a vulnerability for MONTHS. They did not fix it and kept it secret. It only came out because of a whistleblower. This is not the behavior of an organization deserving ANY level of trust.
mirimir 7y ago
I don't have a strong opinion about the hack. And their US residential proxies do work very reliably for Disney+. There is the possibility that they're using services that obtain those proxies deceptively. But I tracked down one of them, and the guy confirmed that he's knowingly selling his bandwidth. And yes, it's just anecdotal.
catalogia 7y ago
> "but places like reddit are always shitting on NordVPN, both due to their ads (obv not the issue for me)" It's a problem for me. Their ads use scare tactics and make deceptive if not outright false claims. Unethical ads come from unethical corporations. I can't see any reason to give such a corporation the benefit of the doubt by assuming their unethical behavior is limited to how they advertise.

sdan 7y ago

This is why I just use AlgoVPN. I know where my traffic is going and can install it on any cloud instance I want. At the moment I'm running it on my laptop, phone, and my RPI cluster.

EternalAugust 7y ago
I think the problem with personal VPNs is correlating incoming with outgoing traffic is trivial for a global adversary (NSA, ISPs). When you use a service provider where a bunch of people are connecting to a single server correlating input with output becomes much more difficult, I seriously doubt it would be impossible though.
- bradknowles 7y ago
  If your threat model seriously includes the NSA, then you shouldn’t be using IP at all. Or any kind of electronic communication, for that matter. If your threat model includes your ISP, but does not extend to nation-state level adversaries, then a good private VPN should be a decent enough solution, although a public VPN might be easier and still adequate. Source: I personally pissed off the Director of the NSA in November of 1992 (see http://www.shub-internet.org/brad/cacm92nov.html http://www.shub-internet.org/brad/cacm92nov.html ). At the time, my clearance was Top Secret/SCI, and I had been read onto multiple compartments — including the ones for ECHELON, KEYHOLE, etc.... So far as I know, I am still on their shit list, albeit not as high as Snowden or Binney.
  - lbatx 7y ago
    Clicked on your link just to get on the list (or move up, I suppose). ;-)
  - mirimir 7y ago
    Yes, this is true. So the best bet is staying off the shit lists of the NSA and other global adversaries.
  - ShorsHammer 7y ago

xt00 7y ago

It would seem that ultimately the only real way to get privacy is to either create a verifiable end to end encryption including the content server not having logs and not compromised and the other way would be pooling of user data in a way that spreads the blame across many people. Otherwise basically everything you do is trackable even if you use a vpn. Is there any other way?

dannyw 7y ago
If you want real privacy at the consequence of slow speeds and unfortunately a worse browsing UX, you should use Tor. Every service that is widely available / permissionless AND offers real privacy, will end up getting degraded, CAPTCHA'd or blocked. That shows it works1
- mirimir 7y ago
  Yes, agreed. But I don't trust Tor either, so I'd first use a nested VPN chain. Just in case the entry guard is malicious. And to complicate the traffic analysis a little. I could go on about this. But I'll be writing more about all this.
mirimir 7y ago
Yes, everything is trackable by global adversaries like the NSA. However, even the NSA doesn't have enough resources to track all 4.5 billion Internet users. So the key is not attracting their attention.

dannyw 7y ago

What's the best VPN provider now that PIA got acquired by an Israeli spyware company?

xvector 7y ago
Source [1]. Truly insane. I'd never have expected this from a company that cares so much about privacy. Just cancelled my ProtonVPN subscription. I was going to recommend ProtonVPN but they have a pretty sketch background/history too. [2] Cloudflare's WireGuard-based Warp VPN [3] might be worth keeping an eye on, but Cloudflare makes it clear that users should not be expecting the privacy guarantees of a "traditional VPN" [4]: > WARP is not designed to allow you to access geo-restricted content when you’re traveling. It will not hide your IP address from the websites you visit. If you’re looking for that kind of high-security protection then a traditional VPN or a service like Tor are likely better choices for you. Perhaps we simply shouldn't use VPN services. [5] [1]: https://news.ycombinator.com/item?id=21679682 https://news.ycombinator.com/item?id=21679682 [2]: https://news.ycombinator.com/item?id=19266078 https://news.ycombinator.com/item?id=19266078 [3]: https://news.ycombinator.com/item?id=19542835 https://news.ycombinator.com/item?id=19542835 [4]: https://blog.cloudflare.com/announcing-warp-plus/ https://blog.cloudflare.com/announcing-warp-plus/ [5]: https://news.ycombinator.com/item?id=16371030 https://news.ycombinator.com/item?id=16371030
- mirimir 7y ago
  Not to put too fine a point on it, they lost money for too long, and ended up needing a buyer. I always did wonder how they could offer so much for so little money.
  - dannyw 7y ago
    How can VPN providers lose money, when a DO Droplet for $5 a month can probably host 5 casual subscribers using things on and off?
- deleted 7y ago
  [deleted]
- 7y ago