12 ms·
How to build a IP geolocation database from scratch?
- overcast 3y agoStep 1: Download Geolocation Database
- nonethewiser 3y agoStep 1: Download Geolocation Data Unless you think CSV is a database?
- debesyla 3y agoMaybe a dumb question (I have no knowledge), but why wouldn't we think of .CSV files as databases? It can have columns and rows filled with information and isn't that what makes a thing a database?
- nobleach 3y agoBest I can guess here, the reply is considering relational databases as "real databases" and flat files.... not real.
- nobleach 3y agoAre we really going to do the mincing of words here? Did you need the word "dump" or "export" before you understood? Although I wasn't wild about the original poster's "step 1" terseness, it's silly to think a normal person wouldn't be able to parse the sentence well enough to understand "download the database contents - perhaps stored in CSV format".
- tmpX7dMeXU 3y agoIf in your mind database implies a type of technology and not something conceptual, you’re really just outing yourself as someone that needs someone between you and the boardroom. Certainly not something to show off on Hacker News.
- Aachen 3y agoScroll down, the article is confusingly below that
- TZubiri 3y ago"how to scrape an ip geolocation database" You know you can just run a whois query per ip you want to analyze, no point in scraping the whole ipvN space.
- incolumitas 3y agoI have to scrape the whole IP address space since I offer location information as part of my API. Also I only need to scrape as many WHOIS records as there are different networks out there. So for example for the IPv4 address space, there are much less networks as there are IPv4 addresses (2^32). Also, most RIR's provide their WHOIS databases for download. Therefore, "scraping" is not really the correct word, it's an hybrid approach, but mostly based on publicly available data from the five RIR's.
- notlukesky 3y agoWhat was the easiest and the most frustrating part?
- incolumitas 3y agoLACNIC is a fucking pain in the ass since they block after like 27 records ARIN is awesome
- djbusby 3y agoThe whois data for IP is not accurate.
- gsich 3y agowhois has no sane format.
- ggm 3y agoRDAP is run by all the RIR and is Json and has all the whois data except IRR. And it does 302 redirect to best source.
- cstuder 3y agoQuestion: What’s the motivation to put coordinates in one’s own WHOIS record? (geoloc/geofeed)
- incolumitas 3y agoMany service providers actually want their clients to be able to locate them.
- noizejoy 3y agoI’m duplicating my comment elsewhere in this thread, so each serves as a direct reply to the different geolocation providers in this thread, in the hope that it will be recognized as a problem with data that implies that it’s more precise than it really is: > On one hand, I love that there’s some good alternatives in the geolocation space, but misleading geolocation precision can lead to very undesirable side effects[0]. [0] https://www.theguardian.com/technology/2016/aug/09/maxmind-mapping-lawsuit-kansas-farm-ip-address https://www.theguardian.com/technology/2016/aug/09/maxmind-m...
- dontdoxxme 3y agogeofeed is used by big CDNs, it can actually help save money for the provider by meaning a CDN uses a more optimal network location.
- mootothemax 3y agoAny suggestions for geolocating datacenter IPs, even very roughly? I'm analysing traceroute data, and while I have known start and end locations, it's the bit in the middle I'm interested in. I can infer certain details from airport codes in node hostnames, for example. It would also be possible - I guess - to infer locations based on average RTT times, presuming a given node's not having a bad day. Anyone have any other ideas? Edit: A couple of troublesome example IPs are 193.142.125.129, 129.250.6.113, and 129.250.3.250. They come up in a UK traceroute - and I believe they're in London - but geolocate all over the world.
- dontdoxxme 3y agohttps://ensa.fi/papers/geolocation_imc17.pdf https://ensa.fi/papers/geolocation_imc17.pdf has some ideas. Using RIPE atlas probes to get RTT to the IPs from known locations is close to your idea and probably the best anyway.
- tyingq 3y agoThis looked promising: "TULIP's purpose is to geolocate a specified target host (identified by IP name or address) using ping RTT delay measurements to the target from reference landmark hosts whose positions are well known (see map or table)." https://tulip.slac.stanford.edu/ https://tulip.slac.stanford.edu/ But the endpoint it posts to seems dead.
- vinay_ys 3y ago> A couple of troublesome example IPs are 193.142.125.129, 129.250.6.113, and 129.250.3.250. They come up in a UK traceroute - and I believe they're in London - but geolocate all over the world. If I'm running a popular app/web service, I would have my own AS number and I will have purchased a few blocks of IP addresses under this AS and then I would advertize these addresses from multiple owned/rented datacenters around the world. These BGP advertisements would be to my different upstream Internet service providers (ISPs) in different locations. For a given advertisement from a particular location, if you see a regional ISP as upstream, you can make an educated guess that this particular datacenter is in that region. If these are Tier 1 ISPs who provide direct connectivity around the world, then even that guess is not possible. You can see the BGP relationships in a looking glass tool like bgp.tools – https://bgp.tools/prefix/193.142.125.0/24#connectivity https://bgp.tools/prefix/193.142.125.0/24#connectivity If you have ability to do traceroute from multiple probes sprinkled across the globe with known locations, then you could triangulate by looking at the fixed IPs of the intermediate router interfaces. Even this is is defeated if I were to use a CDN like Cloudflare to advertise my IP blocks to their 200+ PoPs and ride their private networks across the globe to my datacenters.
- louison11 3y agoIf you don't want to do this yourself, you can actually just get Cloudflare to do it for you for free using a simple Worker since all Cloudflare requests contain approximate IP location information. You can also just send a request to my URL (Cloudflare Worker operated - so it should have global low latency): https://www.edenmaps.net/iplocation https://www.edenmaps.net/iplocation Use it for small applications, I don't mind. Just don't start sending me 10M requests per day ;-)
- deleted 3y ago[deleted]
- Aachen 3y agoOr you download an IP database rather than sharing with a third party which IP address is likely connecting to your service with a third party
- oh_come_on 3y ago[dead]
- tiffanyh 3y agoThis is excellent! Would you mind open sourcing the code for that?
- louison11 3y agoThis is the code running this endpoint: export function onRequest(context) { return new Response(JSON.stringify([parseFloat(context.request.cf.longitude), parseFloat(context.request.cf.latitude)]), {headers: {"Content-Type": "application/json;charset=UTF-8"}}) } This is a function on Cloudflare Pages (which is just a different name for Cloudflare Workers). Minor adjustment needed for Workers (get rid of "context", I believe)
- emadda 3y agoDoes anyone know how accurate Cloudflare geolocation is (for workers requests)?
- nonethewiser 3y agoComments seem fairly dismissive but I actually found this really interesting. It reminds me of a task I had in my first position to add PostGIS to our database and a location based search. That was based off addresses and zipcodes.
- mannyv 3y agoThat's relatively simple to do, even in mysql. One trick is to use a square instead of a circle, which avoids a lot of math.
- dboreham 3y agoInteresting but this isn't actually how geolocation is done, right? The ARIN/RIPE data isn't sufficiently accurate to be useful beyond country. Commercial geolocation involves correlating client IP vs known physical location e.g. from WiFi AP or mailing a package to the user. At least that's what I have been told over the decades.
- shortrounddev2 3y agoI work in adtech and this is how we do geolocation. There's also device geolocation but if the user doesn't consent to sharing their GPS data with us, we just use IP address for targeting. Common provider for this is Maxmind; they ship a database that you host locally and query
- dawnerd 3y agoEven the free maxmind db is accurate enough for most applications.
- tiffanyh 3y agoDoes Cloudflare have the same data as Maxmind? Because Cloudflare and Maxmind geolocate me to the exact same longitude/latitude.
- klaussilveira 3y agoCloudFlare uses Maxmind: https://developers.cloudflare.com/support/network/configuring-ip-geolocation/#troubleshooting https://developers.cloudflare.com/support/network/configurin...
- klaussilveira 3y agoSince you are in adtech: do you buy MaxMind, or roll your own? Are there any providers for US-only data, and therefore, cheaper?
- shortrounddev2 3y ago
- kiririn 3y agoA modern version of the ping-based geoip mentioned https://github.com/Ne00n/yammdb https://github.com/Ne00n/yammdb
- JoshGlazebrook 3y agoThis just links to a mmdb file that is already compiled, there isn't anything relevant to show this is a "modern" implementation of anything if the implementation isn't available.
- nanmu42 3y agoThanks for sharing. I have heard there is much effort to use BGP data to build GeoIP database.
- fasteo 3y ago>>> Consider Open Source Geolocation Projects Not the definition of "from scratch" in my book
- ChopSticksPlz 3y agoThis is a very useful .csv, what is the license? Is it free for personal and commercial use?
- T3RMINATED 3y ago[dead]
- reincoder 3y agoFirst, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the IP address through a process called multilateration. Pinging an IP address from one server gives us one dimension of location information meaning that based on certain parameters the IP address could be in any place within a certain radius on the globe. Then as we ping that IP from our other servers, the location information becomes more precise. After enough pings, we have a very precise IP location information that almost reaches zip code level precision with a high degree of accuracy. Currently, we have more than 600 probe servers across the world and it is expanding. The publicly available information that you are referring to is sometimes not very reliable in providing IP location data as: - They are often stale and not frequently updated. - They are not precise enough to be generally useful. - They provide location context at an large IP range level or even at organization level scale. And last but not least, there is no verification process with these public datasets. With IPv4 trade and VPN services being more and more popular we have seen evidence that in some instances inaccurate information is being injected in these datasets. We are happy and grateful to anyone who submits IP location corrections to us but we do verify these correction submissions for that reason. From my experience with our probe network, I can definitely say that it is far easier and cheaper to buy a server in New York than in any country in the middle of Africa. Location of an IP address greatly influences the value it can provide. We have a free IP to Country ASN database that you can use in your project if you like. https://ipinfo.io/developers/ip-to-country-asn-database https://ipinfo.io/developers/ip-to-country-asn-database
- TheClassic 3y agoYour comment is extremely interesting and what I was hoping to learn from the article (without an existing source of information, how do we determine the location of an IP address). Thank you!
- reincoder 3y ago
- jl6 3y agoIs anybody maintaining a historical archive of “IP address metadata” (which would include geolocation)? If I have logs from 10 years ago, can I look up information about that IP as it was at the time?
- alam2000 3y ago[dead]
- quectophoton 3y agoMaybe not exactly what you're looking for, but the ipfire project has a git repository[1] mapping address ranges to countries. It apparently going back to 2017 only. [1]: https://git.ipfire.org/?p=location/location-database.git;a=summary https://git.ipfire.org/?p=location/location-database.git;a=s...
- bullen 3y agoHere is a solution for those that care about speed: https://www.miyuru.lk/geoiplegacy https://www.miyuru.lk/geoiplegacy
- sneak 3y agoI feel like a more useful and accurate way would be to buy client ip and GPS location data in bulk from one of the mobile data brokers who have their spyware embedded in zillions of popular apps/games and then group it by /24 or something.
- jwie 3y agoThe easiest way to get a geolocation is to ask the user. Maybe they’ll just tell you, and if that’s good enough for your application there’s no need for such solutions.
- hddqsb 3y agoSomewhat relevant: Google Maps can learn the location of your IP based on which locations you browse in the map. If you browse a specific location enough times, it will use that as the default location when you open Google Maps, even if you clear all cookies. (I discovered this just from using Google Maps, and I'm a little concerned by the privacy implications, considering that multiple people may share an IP address.)
- gniv 3y agoI suspect it's the other way around. Google just has a very good IP geolocation db, so it uses that when you browse, absent any other info.
- netsharc 3y agoWell it has reporting beacons all over the world with GPS receivers, in the form of Android phones, and perhaps Google Maps users on iPhone too..
- hddqsb 3y agoGoogle certainly uses its geolocation DB, but it also learns based on map browsing patterns. To clarify, the scenario I described is as follows: 1. Initially, when I open Google Maps in a clean browser it defaults to my real location. 2. I repeatedly browse some other location. 3. When I open Google Maps in a clean browser, it defaults to that other location. The only reason for Google Maps to pick that other location is my map browsing.
- johnklos 3y agoI think it's interesting that the one IP range I decided to check has correct information on the ipapi.is web site, but unambiguously incorrect information in the downloadable geolocationDatabaseIPv4.csv. Somehow Bedford, New Hampshire (which came straight from WHOIS) became Bedford, Texas. How'd that happen?
- spacedcowboy 3y agoSo, at the risk of outing myself, I wrote http://www.hostip.info http://www.hostip.info a long time ago* which used a community approach to get ip address location ("is this guess wrong ? Fix it please"). The last time I checked (maybe a decade ago [grin]) it worked pretty much perfectly for a country, imperfectly for a region, and better-than-a-coin-toss for city resolution. All the data is free. I don't think they have it on the site any more, but I used to have a rotating 3D-cube thing (x,y,z were the first 3 octets of the address) for things like known-addresses, recent lookups, etc. I used different colours for different groups (country, continent,...) It was so old it was written as a Java applet. Yeah. I guess if I were to do it again, it'd be WebGL. -- *: I sold it a long time ago, with the proviso that the data must always remain free. I actually didn't believe the offer at first (it came as an email, and looked like a scam) but it went through escrow.com just fine, and I think we both walked away happy. That was almost 2 decades ago now though.
- SirMaster 3y agoThese IP geolocation lookups never seen to work for me. They are always multiple states off, and checking multiple different services pretty much never even seem to agree.
- jedberg 3y agoIt all depends on what you want to use it for and how accurate it needs to be. The best way to build a geolocation service is to have a billion devices that report their location to you at the same time they report their IP to you. That's basically Apple and Google. They have by far the best geolocation databases in the world, because they get constant updates of IP and location. The trick is basically to make an app where people willingly give you their location, and then get a lot of people to use it. That's the best way to build an accurate geo-location database, and why every app in the world now asks for your location. 4-square had the right idea, they were just ahead of their time.
- flounder3 3y agoEven 10 years ago, Apple internal privacy policies prevented itself from collecting precise lat/long. We had to use HTTP session telemetry to determine which endpoints were best for a given IP (or subnet, but not ASN), which informed our own pseudo-geoIP database so we knew which endpoint to connect to based on real world conditions. Even still, it had to be as ephemeral as possible for the sake of privacy. We weren’t allowed to use or record results from Apple Maps’ reverse geo service outside of the context of a live user request (finding nearby restaurants, etc).
- jedberg 3y agoYou don't need precise lat/lon to make a good database. Even a 1km circle would be more than enough. > but not ASN Why wasn't ASN allowed? That's what Netflix used to make endpoint routing decisions and worked really well.
- deleted 3y ago[deleted]
- flounder3 3y agoYou’re not wrong, but privacy concerns were paramount. ASNs were allowed but too vague. We needed more granularity. Corporate proxies, subdelegations, many providers aggregating announcements below /24, etc.
- bjornsing 3y agoI expected traceroute to play a bigger part in this. If you know the route to an IP address and the location of routers, perhaps even from a few different servers, then you should be able to locate it fairly well.
- bagels 3y agoSurely someone is using online shopping shipping addresses for this?
- n2dasun 3y agoStep 1. Download Visual Basic
- mannyv 3y ago[dead]
- alberth 3y agoWhat are common use cases for needing IP geolocation?
- junto 3y agoAs someone that lives in a country where the national language is not my first language, I hate websites that use IP location to make assumptions about my choice of language and it being forced on me based on a lazy assumption, when my browser is sending language headers quite clearly, and they are ignored.
- hedora 3y agoI live in the US, and IP geolocation points to the incorrect regions (plural) on all my devices. Few technologies manage to make my day-to-day internet experience than these sorts of databases. I wish they would just go away. Websites could just ask me my zipcode on first load instead of guessing it wrong every single time and then burying the flow to fix it behind multiple links and page loads. Also: There is no way to fix the database to produce the “correct” or “better” answer. I rarely want a website to use my current location. Instead, I check inventory for stores in places where I will be. This whole space is trying to solve an ill-posed problem.
- fer 3y agoI share your frustration so much that I wrote an article about it: https://www.fer.xyz/2021/04/i18n https://www.fer.xyz/2021/04/i18n
- reincoder 3y agoI work for an IP geolocations service, even I hate this thing. They call it "web experience personalization" in the industry, and it is annoying. I have never recommended anyone to do that. The best way to do website personalization through IP geolocation: - Taxes and stuff (if applicable) - Delivery costs (if applicable) - Putting the user's country first in those country selection drop-down menu And that's about it from the top of my head. In my experience, these translations never work and only create distractions. Regardless of the positive intention the website has, using Google Translate to create a native language version of the website is just not a good idea.
- junto 3y agoA fair number of popular internationalization frameworks also drive the idea that region and language are fixed pairs. The example I often use to illustrate this problem is that there are roughly 4 million Norwegian speakers in the world, but 14 million speakers of Catalan. Visit an international website in Spain and you rarely get given the option to have it in Catalan. Good example is Amazon.es https://www.amazon.es/customer-preferences/edit?from=mobile&ie=UTF8&preferencesReturnUrl=%2F&ref_=footer_lang https://www.amazon.es/customer-preferences/edit?from=mobile&...
- ggm 3y agorfc8805 https://datatracker.ietf.org/doc/rfc8805/ https://datatracker.ietf.org/doc/rfc8805/ rfc9092 https://datatracker.ietf.org/doc/rfc9092/ https://datatracker.ietf.org/doc/rfc9092/
- ipapi-co 3y ago[dead]
- fabioyy 3y agoa long time ago i build a project like that but instead of relying on whois. i did a traceroute to every ipv4 address avaliable. several router hops, have a reverse dns that uses some names that include city codes, (like airport codes ). most providers have a single hop for a city. so its easy to correlate the latest router hop to a city.
- Fileformat 3y agoShameless self-promotion: I built a page to compare IP geolocation providers: https://resolve.rs/ip/geolocation.html https://resolve.rs/ip/geolocation.html I'll work on adding ipapi.is shortly!
- XCSme 3y agoI am using the free version of https://db-ip.com https://db-ip.com, seems pretty good.
- reincoder 3y agoFantastic stuff. I work for IPinfo.io and I actually came across your site about a month ago. I was planning to contact to you about mentioning the free IP databases [0]. However, when I saw that a few API didn't return any response, I thought maybe the site was not maintained. [0] https://ipinfo.io/products/free-ip-database https://ipinfo.io/products/free-ip-database ---- Tangent I find the geographic coordinate values returning up to 15 decimal places is absurd for an IP geolocation response. IP geolocation is never that precise and, this level of "precision" is not warranted and frankly distracting. Like at best it should be 4 decimal places. relevant xkcd: https://xkcd.com/2170/ https://xkcd.com/2170/