8 ms·

Geolocating requests with Google Load Balancer for free

199 points by vadimska 8y ago

trunet 8y ago
finally!
deleted 8y ago
[deleted]
misterbowfinger 8y ago
Using this: https://cloud.google.com/compute/docs/load-balancing/http/backend-service#user-defined-request-headers https://cloud.google.com/compute/docs/load-balancing/http/ba... The load balancer expands variables to empty strings when it cannot determine their values, for example for geographic location variables when the IP address’s location is unknown, or for TLS parameters when TLS is not in use. Geographic values (regions, subdivisions, and cities) are estimates based on the client’s IP address. From time to time, we update the data that provides these values in order to improve accuracy and to reflect geographic and political changes.
- manigandham 8y ago
  If using the CLI, don't put the single quotes or the whole thing will become the header without a value.
  - vadimska 8y ago
    I have used the CLI and the quotes worked fine for me. Here is the complete command I have used: gcloud beta compute backend-services update app --custom-request-header 'X-Client-Geo-Location:{client_region},{client_region_subdivision},{client_city}' --custom-request-header 'X-Client-Geo-Region:{client_region_subdivision}' --custom-request-header 'X-Client-Geo-LatLong:{client_city_lat_long}' --custom-request-header 'X-Client-TLS-Version:{tls_version}' --custom-request-header 'X-Client-TLS-Chiper:{tls_cipher_suite}' --custom-request-header 'X-Client-Hostname:{tls_sni_hostname}' --custom-request-header 'X-Client-RTT:{client_rtt_msec}'
lightwin 8y ago
Just realized that Cloudflare also provides IP Geolocation, even for free plan.[1] However it only provides ISO country code of the visitors. [1] https://support.cloudflare.com/hc/en-us/articles/200168236-What-does-Cloudflare-IP-Geolocation-do- https://support.cloudflare.com/hc/en-us/articles/200168236-W...
- therealmarv 8y ago
  That's great. Do you know if this also works on all cases when I use Cloudflare only as DNS? Edit: After some thinking this cannot work when Cloudflare is used as DNS only.
ivanakimov 8y ago
Anything like this for AWS?
- lightwin 8y ago
  AWS added geolocation to cloudfront in 2014. https://aws.amazon.com/about-aws/whats-new/2014/06/26/amazon-cloudfront-device-detection-geo-targeting-host-header-cors/ https://aws.amazon.com/about-aws/whats-new/2014/06/26/amazon...
  - sheeshkebab 8y ago
    AWS cloudfront feature is incomplete - it returns only country code. Googles and maxmind features include city/lat/lng.
  - vadimska 8y ago
    Not everything goes thru CloudFront. I hope ELB/ALB will have this feature soon too!
    - philliphaydon 8y ago
      I do geo routing in Route53. Because of how often China blocks cloudfront.
iampims 8y ago
AppEngine has had this feature for years and it’s extremely convenient!
- vadimska 8y ago
  Exactly! I was using GAE for 9 years now and it's so easy to have these headers automagically being attached to your requests. Finally, Google Compute Engine has the same convenient way of geolocating requests.
SteveCoast 8y ago
I've been working on and off on a system to build this data for free - https://www.open-geo-ip.com/ https://www.open-geo-ip.com/ It uses your browser location to build the data, bootstrapped with Mechanical Turk workers around the world. It has one feedback loop, where if you use the JS library it will give you the users location if they deny browser location (if we have the data...) and if they do share location, then we update the database (but throw away the last part of their IP address). There are two or three other loops where we could build more data (which you can get here https://www.open-geo-ip.com/data/download https://www.open-geo-ip.com/data/download ). I'm probably going to do a small kickstarter to pay for some cleanup and expansion work or just kill it if that fails to get funded. Any ideas appreciated on things to do with it.
- reaperducer 8y ago
  > it will give you the users location if they deny browser location Seriously? Did you help build BonziBuddy, too?
  - SteveCoast 8y ago
    Ha. Well right now a) we most likely have no idea where an ip address is and b) even if we do, it's nearest-city accuracy.... which is exactly what everyone else does. Denying browser location doesn't stop the app or website from using your ip address against some third party service to figure out where you are.
    - mindslight 8y ago
      > Denying browser location doesn't stop the app or website from using your ip address against some third party service to figure out where you are. Just how like being uninvited from a party doesn't stop you from just showing up anyway (and trashing the bathroom for good measure).
    - reaperducer 8y ago
      >Denying browser location doesn't stop the app or website from using your ip address against some third party service to figure out where you are. I didn't state that it does. However, just because you can do something doesn't mean you should. Did it ever occur to you that if someone explicitly tells their browser that they don't want to be located that you should respect that? I hope you are never critical of Facebook's or Google's privacy standards, because there's a full-length mirror with your name on it.
avivl 8y ago
Been waiting for this for a while!
zvirog 8y ago
Finally, i am waiting for something like that for some time.
abomusa 8y ago
Awesome, save some efforts of configure ngnix to work with geoip databases.
- vadimska 8y ago
  yep. also, spares some compute cycles for your app
zvirog 8y ago
Finally, i am waiting for something like that for sometime.
shayale 8y ago
At last
brightball 8y ago
One thing I will say for Maxmind is that their API was exceptionally good at picking up anonymous proxies like Tor. I dealt with a ton of fraud a couple of jobs ago and being able to eliminate credit card transactions for people on Tor was huge in cutting down on charge backs from fake card use. The minFraud API was really beneficial as well. Don't sell the offering short.
- vadimska 8y ago
  Well, I am sure MaxMind provides a lot of value for many use-cases. However, I just need something very basic (country, city, lat and long) and getting this "out of the box" from Google's LB is blessing for me.
- dylz 8y ago
  Tor is a special case here: you can use any Tor DNSBL and it will pick up 100% of the nodes, effectively. All Tor IPs specifically and publicly advertise the fact that they are an open proxy.
- michaelmior 8y ago
  I don't doubt that there are a lot of baddies using Tor (and I wouldn't be surprised if that was the majority of Tor users in some cases). It's a shame though that cutting off Tor access is the solution. A very reasonable one from a business standpoint, but unfortunate.
  - dmoy 8y ago
    Doesn't have to be a majority of Tor users, if the tiny minority of actors are flooding tons of stolen cc transactions. A normal person may make an online purchase a day or so, while someone trying to get as much value as possible out of a bunch of stolen cc may just spam a lot of transactions. For the server seeing the incoming cc purchase requests, it's still majority fraud...
- ehsankia 8y ago
  The author does specify that for his own needs, which was basically adding 3 headers to requests, the offering was not worth it. He also does explicitly say that MaxMind is very good. So I don't think he was unfair at all in the post.
dani-doit 8y ago
No need for external services from now on.
alexg84 8y ago
Great article! It's about time that Google will come with a geo solution
tpetry 8y ago
Does anybody know how good the data is? I mean everyone of the geolocation lookup providers has faults and choosing the majority from different sources will have the best results. But maybe Google has much more accurate data?
- vadimska 8y ago
  Check this out: https://twitter.com/jasoncartwright/status/988823153714651138 https://twitter.com/jasoncartwright/status/98882315371465113...
Theodores 8y ago
If you are hosting outside of the Google eco-system and want to be able to do geo-IP for a store locator etc., or just to set a cookie on a page with the lat/lon, how does one build a minimal app in the Google cloud to do just this bit of the task? If this is easy doable then that would be an easy migration route.
- vadimska 8y ago
  It would be pretty easy to build. Probably something you can do end-to-end in one or two days.
- therealmarv 8y ago
  see this other comment for full source code: https://news.ycombinator.com/item?id=16915184 https://news.ycombinator.com/item?id=16915184
forcer 8y ago
has anyone tried to compare geolocation accuracy of this solution vs Maxmind? I would assume Country-level will be very comparable (its easy to do), but doing city level accuracy is not trivial
- vadimska 8y ago
  https://twitter.com/jasoncartwright/status/988823153714651138 https://twitter.com/jasoncartwright/status/98882315371465113...
yoram_benyaacov 8y ago
Finally we can have it as a not only on app engine :-)
Dragoneous 8y ago
Excellent! Another great capability added! This will save us some over head!
sudhirj 8y ago
For those not running on GCE, I run a free service at https://blip.runway7.net/ https://blip.runway7.net/ that piggybacks on the Google App Engine headers. Suitable for calling from the client browser / device, it allows you to ask for location specific resources straight from the client without having to do anything on the server at all. Code at https://github.com/runway7/blip https://github.com/runway7/blip
- sudhirj 8y ago
  This is especially useful in single page apps, or static sites, if that's not obvious.
yoavtco 8y ago
Wow, that's so cool! I wish my workload was running on Google Cloud
manigandham 8y ago
Note that if you're using this load balancer behind a CDN then the results can be incorrect, although probably generally within the same region. Also MaxMind does have completely free databases down to the city level, just with lower accuracy compared to their paid products: https://dev.maxmind.com/geoip/geoip2/geolite2/ https://dev.maxmind.com/geoip/geoip2/geolite2/
- vadimska 8y ago
  This probably will depend on CDN. Google's CDN is part of the Load Balancer itself and therefore it won't affect geolocation. With "3rd party" CDNs it's different, of course. Finally, the free version of MaxMind is great. Thanks for mentioning it.
htk_ 8y ago
Wow, such nice developers! offering more user private data to google huh
- tzury 8y ago
  nobody's offering Google anything, rather, Google is offering developers meta data for HTTP requests.
psanchez 8y ago
Nice! Any clue if there is a way this can be enabled automatically on balancers created using kubernetes (on GKE)? That would be really awesome.
- vadimska 8y ago
  My thoughts exactly. I am now working on this problem myself and I will share my results soon.
  - psanchez 8y ago
    Looking forward!
alam2000 8y ago
Aren't there CloudFlare, AWS and Azure have it?
ne01 8y ago
I have always said that the GCP HTTP(S) load balancer is the hidden gem in cloud. I'm glad it's becoming more and more popular (also feature rich). One feature that must be mentioned is that it allows you to cache content (could be anything: HTML, JPEG and more) for as little as 1 second! CloudFlare requires an enterprise plan and even then you cannot set a TTL lower than 30 seconds.
Shani89 8y ago
Finally!
Shani89 8y ago
Finally!! Thanks for sharing
neter145 8y ago
Sharing is caring :) Thank you
databill 8y ago
We've done some testing and found many inaccuracy in Google LB Geo-IP results, anyone know who is in charge and which channel we can feedback to Google?
- databill 8y ago
  or anyone would like to try our service, please drop me a mail data at ipip.net
geofocus 8y ago
Cool!! Location-based traffic insights have significant value in many use-cases. Show the results on a heat-map to see where you should be targeting customers.

199 points by vadimska 8y ago

trunet 8y ago

finally!

deleted 8y ago

[deleted]

misterbowfinger 8y ago

Using this: https://cloud.google.com/compute/docs/load-balancing/http/backend-service#user-defined-request-headers https://cloud.google.com/compute/docs/load-balancing/http/ba... The load balancer expands variables to empty strings when it cannot determine their values, for example for geographic location variables when the IP address’s location is unknown, or for TLS parameters when TLS is not in use. Geographic values (regions, subdivisions, and cities) are estimates based on the client’s IP address. From time to time, we update the data that provides these values in order to improve accuracy and to reflect geographic and political changes.

manigandham 8y ago
If using the CLI, don't put the single quotes or the whole thing will become the header without a value.
- vadimska 8y ago
  I have used the CLI and the quotes worked fine for me. Here is the complete command I have used: gcloud beta compute backend-services update app --custom-request-header 'X-Client-Geo-Location:{client_region},{client_region_subdivision},{client_city}' --custom-request-header 'X-Client-Geo-Region:{client_region_subdivision}' --custom-request-header 'X-Client-Geo-LatLong:{client_city_lat_long}' --custom-request-header 'X-Client-TLS-Version:{tls_version}' --custom-request-header 'X-Client-TLS-Chiper:{tls_cipher_suite}' --custom-request-header 'X-Client-Hostname:{tls_sni_hostname}' --custom-request-header 'X-Client-RTT:{client_rtt_msec}'

lightwin 8y ago

Just realized that Cloudflare also provides IP Geolocation, even for free plan.[1] However it only provides ISO country code of the visitors. [1] https://support.cloudflare.com/hc/en-us/articles/200168236-What-does-Cloudflare-IP-Geolocation-do- https://support.cloudflare.com/hc/en-us/articles/200168236-W...

therealmarv 8y ago
That's great. Do you know if this also works on all cases when I use Cloudflare only as DNS? Edit: After some thinking this cannot work when Cloudflare is used as DNS only.

ivanakimov 8y ago

Anything like this for AWS?

lightwin 8y ago
AWS added geolocation to cloudfront in 2014. https://aws.amazon.com/about-aws/whats-new/2014/06/26/amazon-cloudfront-device-detection-geo-targeting-host-header-cors/ https://aws.amazon.com/about-aws/whats-new/2014/06/26/amazon...
- sheeshkebab 8y ago
  AWS cloudfront feature is incomplete - it returns only country code. Googles and maxmind features include city/lat/lng.
- vadimska 8y ago
  Not everything goes thru CloudFront. I hope ELB/ALB will have this feature soon too!
  - philliphaydon 8y ago
    I do geo routing in Route53. Because of how often China blocks cloudfront.

iampims 8y ago

AppEngine has had this feature for years and it’s extremely convenient!

vadimska 8y ago
Exactly! I was using GAE for 9 years now and it's so easy to have these headers automagically being attached to your requests. Finally, Google Compute Engine has the same convenient way of geolocating requests.

SteveCoast 8y ago

I've been working on and off on a system to build this data for free - https://www.open-geo-ip.com/ https://www.open-geo-ip.com/ It uses your browser location to build the data, bootstrapped with Mechanical Turk workers around the world. It has one feedback loop, where if you use the JS library it will give you the users location if they deny browser location (if we have the data...) and if they do share location, then we update the database (but throw away the last part of their IP address). There are two or three other loops where we could build more data (which you can get here https://www.open-geo-ip.com/data/download https://www.open-geo-ip.com/data/download ). I'm probably going to do a small kickstarter to pay for some cleanup and expansion work or just kill it if that fails to get funded. Any ideas appreciated on things to do with it.

reaperducer 8y ago
> it will give you the users location if they deny browser location Seriously? Did you help build BonziBuddy, too?
- SteveCoast 8y ago
  Ha. Well right now a) we most likely have no idea where an ip address is and b) even if we do, it's nearest-city accuracy.... which is exactly what everyone else does. Denying browser location doesn't stop the app or website from using your ip address against some third party service to figure out where you are.
  - mindslight 8y ago
    > Denying browser location doesn't stop the app or website from using your ip address against some third party service to figure out where you are. Just how like being uninvited from a party doesn't stop you from just showing up anyway (and trashing the bathroom for good measure).
  - reaperducer 8y ago
    >Denying browser location doesn't stop the app or website from using your ip address against some third party service to figure out where you are. I didn't state that it does. However, just because you can do something doesn't mean you should. Did it ever occur to you that if someone explicitly tells their browser that they don't want to be located that you should respect that? I hope you are never critical of Facebook's or Google's privacy standards, because there's a full-length mirror with your name on it.

avivl 8y ago

Been waiting for this for a while!

zvirog 8y ago

Finally, i am waiting for something like that for some time.

abomusa 8y ago

Awesome, save some efforts of configure ngnix to work with geoip databases.

vadimska 8y ago
yep. also, spares some compute cycles for your app

zvirog 8y ago

Finally, i am waiting for something like that for sometime.

shayale 8y ago

At last

brightball 8y ago

One thing I will say for Maxmind is that their API was exceptionally good at picking up anonymous proxies like Tor. I dealt with a ton of fraud a couple of jobs ago and being able to eliminate credit card transactions for people on Tor was huge in cutting down on charge backs from fake card use. The minFraud API was really beneficial as well. Don't sell the offering short.

vadimska 8y ago
Well, I am sure MaxMind provides a lot of value for many use-cases. However, I just need something very basic (country, city, lat and long) and getting this "out of the box" from Google's LB is blessing for me.
dylz 8y ago
Tor is a special case here: you can use any Tor DNSBL and it will pick up 100% of the nodes, effectively. All Tor IPs specifically and publicly advertise the fact that they are an open proxy.
michaelmior 8y ago
I don't doubt that there are a lot of baddies using Tor (and I wouldn't be surprised if that was the majority of Tor users in some cases). It's a shame though that cutting off Tor access is the solution. A very reasonable one from a business standpoint, but unfortunate.
- dmoy 8y ago
  Doesn't have to be a majority of Tor users, if the tiny minority of actors are flooding tons of stolen cc transactions. A normal person may make an online purchase a day or so, while someone trying to get as much value as possible out of a bunch of stolen cc may just spam a lot of transactions. For the server seeing the incoming cc purchase requests, it's still majority fraud...
ehsankia 8y ago
The author does specify that for his own needs, which was basically adding 3 headers to requests, the offering was not worth it. He also does explicitly say that MaxMind is very good. So I don't think he was unfair at all in the post.

dani-doit 8y ago

No need for external services from now on.

alexg84 8y ago

Great article! It's about time that Google will come with a geo solution

tpetry 8y ago

Does anybody know how good the data is? I mean everyone of the geolocation lookup providers has faults and choosing the majority from different sources will have the best results. But maybe Google has much more accurate data?

vadimska 8y ago
Check this out: https://twitter.com/jasoncartwright/status/988823153714651138 https://twitter.com/jasoncartwright/status/98882315371465113...

Theodores 8y ago

If you are hosting outside of the Google eco-system and want to be able to do geo-IP for a store locator etc., or just to set a cookie on a page with the lat/lon, how does one build a minimal app in the Google cloud to do just this bit of the task? If this is easy doable then that would be an easy migration route.

vadimska 8y ago
It would be pretty easy to build. Probably something you can do end-to-end in one or two days.
therealmarv 8y ago
see this other comment for full source code: https://news.ycombinator.com/item?id=16915184 https://news.ycombinator.com/item?id=16915184

forcer 8y ago

has anyone tried to compare geolocation accuracy of this solution vs Maxmind? I would assume Country-level will be very comparable (its easy to do), but doing city level accuracy is not trivial

vadimska 8y ago
https://twitter.com/jasoncartwright/status/988823153714651138 https://twitter.com/jasoncartwright/status/98882315371465113...

yoram_benyaacov 8y ago

Finally we can have it as a not only on app engine :-)

Dragoneous 8y ago

Excellent! Another great capability added! This will save us some over head!

sudhirj 8y ago

For those not running on GCE, I run a free service at https://blip.runway7.net/ https://blip.runway7.net/ that piggybacks on the Google App Engine headers. Suitable for calling from the client browser / device, it allows you to ask for location specific resources straight from the client without having to do anything on the server at all. Code at https://github.com/runway7/blip https://github.com/runway7/blip

sudhirj 8y ago
This is especially useful in single page apps, or static sites, if that's not obvious.

yoavtco 8y ago

Wow, that's so cool! I wish my workload was running on Google Cloud

manigandham 8y ago

Note that if you're using this load balancer behind a CDN then the results can be incorrect, although probably generally within the same region. Also MaxMind does have completely free databases down to the city level, just with lower accuracy compared to their paid products: https://dev.maxmind.com/geoip/geoip2/geolite2/ https://dev.maxmind.com/geoip/geoip2/geolite2/

vadimska 8y ago
This probably will depend on CDN. Google's CDN is part of the Load Balancer itself and therefore it won't affect geolocation. With "3rd party" CDNs it's different, of course. Finally, the free version of MaxMind is great. Thanks for mentioning it.

htk_ 8y ago

Wow, such nice developers! offering more user private data to google huh

tzury 8y ago
nobody's offering Google anything, rather, Google is offering developers meta data for HTTP requests.

psanchez 8y ago

Nice! Any clue if there is a way this can be enabled automatically on balancers created using kubernetes (on GKE)? That would be really awesome.

vadimska 8y ago
My thoughts exactly. I am now working on this problem myself and I will share my results soon.
- psanchez 8y ago
  Looking forward!

alam2000 8y ago

Aren't there CloudFlare, AWS and Azure have it?

ne01 8y ago

I have always said that the GCP HTTP(S) load balancer is the hidden gem in cloud. I'm glad it's becoming more and more popular (also feature rich). One feature that must be mentioned is that it allows you to cache content (could be anything: HTML, JPEG and more) for as little as 1 second! CloudFlare requires an enterprise plan and even then you cannot set a TTL lower than 30 seconds.

Shani89 8y ago

Finally!

Shani89 8y ago

Finally!! Thanks for sharing

neter145 8y ago

Sharing is caring :) Thank you

databill 8y ago

We've done some testing and found many inaccuracy in Google LB Geo-IP results, anyone know who is in charge and which channel we can feedback to Google?

databill 8y ago
or anyone would like to try our service, please drop me a mail data at ipip.net

geofocus 8y ago

Cool!! Location-based traffic insights have significant value in many use-cases. Show the results on a heat-map to see where you should be targeting customers.