10 ms·
Ask HN: Who here would be interested in a free daily updated GeoIP database?
I recently was playing around with Maxmind's free GeoIP database but was disappointed at how inaccurate the free version was.
I started hacking away yesterday at making an alternative database that seems to be more accurate, but it's still in the works. Who here would be interested in a free alternative, including and excluding cities, updated on a daily basis, and if so, which format's would you like to see?
- dutchbrit 11y agoOn a related note, what's the best way to host big databases for free? As a release on Github?
- onethumb 11y agoVery interested, but worried that if it's free, it won't last or be maintained, etc. But in specific answer to this question, if it's really public and free, why not see about adding it to Amazon's public datasets [1]? S3 is fast, durable, available, and I've enjoyed these other data sets. [1] http://aws.amazon.com/datasets/ http://aws.amazon.com/datasets/
- fale 11y agoI would be interested, personally and I believe many other developers would also be interested :-)
- dangrossman 11y agoHow/where do you get more accurate data than MaxMind does? And under what license do you have access to this data, and under what license would you be able to offer it to others? If all the legal stuff looks right, and the data is indeed more accurate, then of course there's going to be interest in it. I think it would take a while for word to spread and links to propagate, but eventually you'd see quite a bit of sustained traffic on that project. IP-to-location data is a very common need.
- 13 11y agoProbably a lot of sources, depends how well connected you are. The biggest one is just routing information and rDNS hosts which give away a lot of the information required anyway. If you were really thrifty you would make agreements with services (or use services you run yourself) that have users supply their location information and use that to construct a more granular database. I suspect this already happens to a degree, some of the fake information I give out has ended up in at least one IP address database. I strongly considered doing this at one point but lacked the connections to pull it off properly (not enough sources of data I could get without approaching random people, basically). A secondary concept would just be to crowd source it and have the information freely available, but I was never sure how to deal with people poisoning the database or just honestly supplying conflicting information.
- clinta 11y agoMy only experience with MaxMind was the trouble we had when we got a new IPv4 assignment from ARIN. I filled out the form on their website the first week, our address was in the WhoIS records for our IPs, and our BGP announcements made our location obvious, yet they had our IP's (obviously from ARIN) showing up as originating from South Africa for months. Seems like anyone who can look at BGP tables and whois information can get more current and accurate data than MaxMind.
- rebelde 11y agoI have long thought that we could do better than maxmind. I have a website with tens of millions of users all over the world that could help you build this database is your want to lead the effort. I imagine showing users what country we think they are in and they correct it if it is wrong.
- mike-cardwell 11y agoI imagine browser language settings would be a useful indicator too. Not perfect obviously, but highly useful given enough data.
- rebelde 11y agoThat, too. We could use a number of different indicators to help figure it out.
- Gys 11y agoSo it would be a 'simple' table dump ? Updated daily by adding only the changes ? I would be interested. Although a service like ip-api.com is not expensive ($8 per month or so if commercial use). The structure of the data seems very simple: ip address range plus a location, I guess. So maybe just dump as a simple csv text file ? Always easy to process. You have to make sure the legal side is covered: how do you get your changes ?
- mgkimsal 11y agoI'm not sure I want a free alternative. I want an inexpensive option that I'm pretty sure won't get acquired and shutdown, but also has reasonable enough revenue to remain in business, occasionally blessing me with new features. A free service like locationkit (seen here a few days ago IIRC) looks OK, and I'm reasonably sure it's supported by revenues for other services from the parent company. However, even that feels like it might get shut down if it becomes too big a burden. Something with a good understandable license that I can pay a reasonable amount ($5-$20/year, for most low-mid volume use cases) would be great, as long as I can really use it for commercial purposes. I had some high hopes for Yahoo services a few years back, but they seemed determined to plague everything with confusing or commercial-hostile licensing.
- pvg 11y agoThat sounds a lot like some of MaxMind's products.
- mgkimsal 11y agoI think it is. I've not tested their quality, but saw their pricing - buy credits then they get used up per API call, similar to Twilio's per minute deductions from a credit. If the OP has a way to best MaxMind's quality,and can charge, that'd be good. But the OP was talking about a 'free' model, which I'm not really interested in.
- lucb1e 11y agoI agree, paying a small fee for it sounds good. However it might gain a lot of popularity and support if there is also a free version, e.g. for non-commercial purposes only and updated only once a year or so. I would personally not want to pay up front without knowing anything about this database from a company (or private person) that only asks 5 bucks a year for it.
- dennisgorelik 11y ago> $5-$20/year That seems to be way too low. $20/month+ could make sense, but less than that should be simply free (limited edition). Dealing with payments is just too much overhead.
- wineisfine 11y agoTons of devs are interested
- kenetik 11y agoWould LOVE this, possibly in json? Id even consider hosting it for you.
- nicolewhite 11y agoI would be interested. We pay for Maxmind and still get a lot of inaccurate data.
- TheSwordsman 11y agoI think a lot of people would be interested. I'm also curious of the implementation details. Open source? :p To echo others, aim for low-priced and not free. That'll give you some operating revenue to avoid it going the way of the dodo.
- deleted 11y ago[deleted]
- mike-cardwell 11y agoI would be very interested in such a thing. Where are you getting your data from? Does it work for IPv6 too?
- JosephRedfern 11y agoHmm. Can you determine such things using BGP? I guess that wouldn't be very accurate, though...
- JosephRedfern 11y agoHmm. Can you determine such things using BGP? I guess that wouldn't be very accurate, though...
- PythonicAlpha 11y agoIn the EU, the sales tax now (I think, since this year) must be paid to the country, the buyer comes from (before, the sellers country was decisive). Also the height of the tax depends on this. So, at least in Europe there are plenty of small companies that could need such a thing, at least as long they don't have a payment-provider that handles the sales-tax for them. I myself, could also be interested, but I think, I will leave that to my payment-provider ... (ugly, EU!) Edit: Ok, I should not, but why the downvote, please? When I wrote something wrong, than please explain!
- _delirium 11y agoYou almost certainly don't want to be using IP geolocation to compute sales tax, rather some kind of billing address explicitly provided by the customer.
- PythonicAlpha 11y agoIn that case, the customer could cheat. I am not totally sure about the current discussion on that, but the problem is, that you definitively can not rely on the address given of the customer, because in this case the customer will always give a country with the lowest sales tax. Also you must provide proofs for the location. The IP address is IMHO (and not only my opinion) better and will be more likely be recognized as proof instead of relying solely on some customer input. But, you (because you are so sure about the best method) by sure know, that the laws are explicitly for non-physical goods, where no delivery-address must be provided, right? Here is a link, how one company is doing it (forget about government information, they seldom go so much in details!): https://rachelandrew.co.uk/archives/2014/12/08/consensus-and-vatmoss/ https://rachelandrew.co.uk/archives/2014/12/08/consensus-and... They use the IP address to validate the user given address and for later proof. BTW: Downvoting all the time does not help the discussion. I try to discuss this topic seriously and I was bringing some very relevant topic here, and am constantly downvoted. (It is not my fault, that the EU came up with such laws, and provide no useful guide how to implement it!)
- _delirium 11y ago
- karmajunkie 11y agoI'll echo the "low cost, not free" thought. Stuff takes money to maintain. I don't want to build a service that depends on an unsustainable business model.
- notatoad 11y agoMaxmind is low cost, not free, and has a pretty good track record. What would you want from an alternative?
- rebelde 11y agoI would want: - Easier updates. Now it is a minor hassle to update my MySQL database table from CVS. I need to do it again soon. - Availability in different formats. For my personal needs, I would love to download a MySQL MyISAM table. - More accurate data, including "ambiguous country" for employees of multinational firms. - Related open source tools through github.
- oalders 11y ago(Disclaimer: I work for MaxMind) As far as formats go, if anyone is thinking of putting together a custom database with information keyed on IP address or IP ranges, I'd encourage you to look into our MaxMind DB format: http://maxmind.github.io/MaxMind-DB/ http://maxmind.github.io/MaxMind-DB/ There are open source file readers available in many languages: https://github.com/maxmind/?utf8=%E2%9C%93&query=MaxMind-DB https://github.com/maxmind/?utf8=%E2%9C%93&query=MaxMind-DB There's also an open source writer https://metacpan.org/pod/MaxMind::DB::Writer https://metacpan.org/pod/MaxMind::DB::Writer Our officially supported libraries are actively maintained and pull requests are always welcome. I'm actually about to begin writing a blog post on how you can create your own custom databases. I'll share it on HN once it's ready.
- dutchbrit 11y agoAwesome, thanks!
- Firegarden 11y agoCan you please confirm that the free versions are not 'good enough'? http://dev.maxmind.com/geoip/geoip2/geolite2/ http://dev.maxmind.com/geoip/geoip2/geolite2/
- oalders 11y agoIf you want to compare the accuracy of different products, you can do so on this page: https://www.maxmind.com/en/geoip2-city-database-accuracy https://www.maxmind.com/en/geoip2-city-database-accuracy To get the actual product comparisons, you'll need to enter a country in the input box. For instance: https://www.maxmind.com/en/geoip2-city-database-accuracy?country=United+States&resolution=50 https://www.maxmind.com/en/geoip2-city-database-accuracy?cou... You can try refining the distances to see how the different products compare. If you fiddle with it a bit, you'll see that the results do vary from country to country. So, "good enough" will depend on some combination of your tolerance for error + which countries you care about.
- pczzy 11y ago
- tokenizerrr 11y agoThat would be amazing. I'd love a datadump that could be accessed on my own servers instead of having to hit an API. The format of the datadump doesn't really matter to me, as long as there's documentation on how to read it.
- dsl 11y agoMost geo data providers (like MaxMind) provide datafile downloads so you don't have to use an API.
- tokenizerrr 11y agoRight, but those aren't usually free ;) On the other hand there are plenty of free (limited) APIs.
- wise_young_man 11y agoThey do have a free version which is just a bit less accurate than the commercial version. http://dev.maxmind.com/geoip/geoip2/downloadable/ http://dev.maxmind.com/geoip/geoip2/downloadable/
- tokenizerrr 11y agoOh, I didn't know that. How much less accurate? Will it accurately determine country of origin?
- egze 11y agoYes, it will
- deleted 11y ago[deleted]
- RandyRanderson 11y agoNote that you could build a simple API with Google App Engine: https://cloud.google.com/appengine/docs/python/#Request_Headers https://cloud.google.com/appengine/docs/python/#Request_Head... and it would likely be free if the number of requests was below a certain threshold per month.
- sajal83 11y agoId be interested, especially if it used same file format as Maxmind's binary dat file ... so it can be a drop in replacement.
- anjanb 11y agoYes, one initial download and then subsequent deltas on a regular basis. Will be willing to pay. I know of plenty of Fortune 500 companies will be willing to pay for it.
- bisdata 11y agoOn a related note, would anyone in the community be interested in an inexpensive service that differentiates dynamic from static IPs? Dynamic IPs would be accompanied by an estimate of lease duration.
- lucb1e 11y agoI wouldn't want this to exist because it discriminates my dynamic-looking IP address from server IP addresses (even though mine is statically assigned to my Internet subscription) and I have enough trouble hosting things at home already. But I understand that's not really an argument: you might say you don't want nukes to exist because they hurt people. I just wanted to mention it. That said, maybe. Usually an rdns lookup does the trick already though (a4-8e-ef-78.dynamic.ziggo.nl or cpe-12-34-56-78.rr.com or 80-100-131-150.adsl.xs4all.nl is clear enough).
- Gys 11y agoYes, clear enough if you are Dutch ;-) But what about other countries ? Singapore, India, Brasil.......
- bisdata 11y agoYeah rDNS is helpful, but as you pointed out with your home IP, many dynamic IPs (those that are rotated by an ISP) are actually granted very long leases. The service I'm proposing would make an additional effort to estimate the duration of dynamic leases.
- adminprof 11y agoI'd be interested if it included cities. If there was a javascript plugin to automatically help contribute (is that possible), I'd add it to some of my web pages. CSV is fine.
- clouserw 11y agoMozilla runs an open/free geolocation service: https://location.services.mozilla.com/ https://location.services.mozilla.com/
- vardars 11y agohttps://mozilla-ichnaea.readthedocs.org/en/latest/config.html#geoip https://mozilla-ichnaea.readthedocs.org/en/latest/config.htm... "The geoip section contains settings related to the maxmind GeoIP database." If I don't get it wrong, this service relies on MaxMind's db for GeoIP.
- cpeterso 11y agoThe Mozilla Location Service does use MaxMind for GeoIP, but the service's primary use is to map Wi-Fi SSIDs to longitude/latitude position. Mozilla has its own Wi-Fi database continually fed new data from the Mozilla Stumbler app (on the Google Play Store and F-Droid markets) and Firefox for Android. The Mozilla database also has a partnership with commercial Wi-Fi company Combain and the OpenCellID project for fallback results. https://blog.mozilla.org/services/2015/03/06/combain-deal-helps-expand-mozilla-location-service/ https://blog.mozilla.org/services/2015/03/06/combain-deal-he... (I work at Mozilla and I used to work on the Location project. :)
- cpeterso 11y agoI pitched the idea of Mozilla creating a free GeoIP database by collecting (opt-in) traceroutes from Firefox users to some Mozilla servers. The traceroutes could then be overlaid to create a physical map of the Internet. I know other projects have mapped the internet, but this approach would work from the endpoints in, not from some centralized crawler.
- openservices 11y agoThere seems to be heavy interest in a low-cost service, even free if it was sustainable. Just curious if anyone has seen innovative pricing around services, similar to how open-source software made expensive software much cheaper or free, where the vendor made the main "service/data" much less expensive but monetized in a different way.
- openservices 11y agoSo what I'm envisioning are services designed for zero profit. Prices are indexed to AWS/Azure/Google prices; as they fall, the service's prices falls. Customers who need predictable pricing can buy long term contracts, as they do on AWS. Customers who use the service more, pay more, because the service offers utility-based pricing. The goal is to make the service as cheap as possible and maintained by the community. The source code and all data, except private data (e.g., API keys), is open and editable by the community. Think it would be an interesting way to deliver services and data, just not sure how it turns into a business (since the API/service portion is designed to be zero profit).
- lucb1e 11y agoI use GeoIP fairly regularly. How I would set this up: - Free version: only updated once [every 6 months / a year] and non-commercial use only - Paid version: $9.99 for a year of free updates - Updates come both in diff and in full version (for the commercial version) - Use a common format, or use multiple formats. People love csv, but more serious applications probably use an existing format (I see maxmind a lot in this thread). - Perhaps an API with heavy ratelimiting for people who need only a very occasional lookup (for very occasional use, I use ip2location.com/demo currently; for more serious stuff I download a database). Having a free version with a nice license might also get you included in a lot of open source software, which increases name visibility and makes people write parsers for your format of choice. A minor note on payment: accept Paypal and/or Bitcoin, or I (and many other Europeans) can never pay you. These credit card-only services are impossible to order from. Idea for the diff: if you have a binary format, perhaps use this: https://www.chromium.org/developers/design-documents/software-updates-courgette https://www.chromium.org/developers/design-documents/softwar...
- JoshGlazebrook 11y agoI actually use ip2location's demo page as well. I put together my own api that grabs ip to location data from the 4-5 sources that provide the information for free, and then look for similarities. So if a (city, region, country) is present 3/5 times, it defaults to that. If they're not any similarities, it will default to maxmind geoip2's demo (if it hasn't already used up the daily free lookups). And then cache it all of course. Works well for the limited amount of lookups I need.
- dutchbrit 11y agoI haven't thought about making this a for-profit project yet. Maybe a paid API in the future, but I really want to offer an accurate & free option that's up to date. Re: CC's, I'm also based in Europe (The Netherlands) and also hate services that only allow credit cards. Thanks for the link regarding diffs, I'll definitely give it a read.
- raquo 11y agoWhy is paying with credit cards hard for Europeans?
- cryptodev 11y agoCan you provide some contact info? Would like to help adding changes, I personally find Maxmind inaccurate as well.
- dutchbrit 11y agoShoot me an email at sam.granger@gmail.com and I'll provide you with updates
- cryptodev 11y agoCan you provide some contact information? I would like to contribute as well, I find Maxmind extremely unreliable when it comes to Home ISPs.
- dutchbrit 11y agoShoot me an email at sam.granger@gmail.com and I'll provide you with updates
- SteveCoast 11y agoI've been building exactly this: http://browserlocation.co/ http://browserlocation.co/ I have a JS library coming also for use when the w3c geolocation fails. The data is open and downloadable with cities etc (CSV right now). There's a mailing list for updates, and anyone can contribute just by opening the website in a browser!
- heyalexej 11y agoIt shouldn't be free, it should be reasonably priced so you don't run out of money and can keep this thing running for years to come. There's huge demand for it - just search for geoip on Github. Some more pointers are here [0],[1]. So many projects, but most of them work with Maxmind data, which is the problem. Inconsistencies in the data they provide makes it painful to make performant fallback solutions. Performance across geolocations and lack thereof in "exotic" geos is another one. It seems none of these services is deployed across the globe. There's a need for a blazing fast, reliable service with granular, comprehensive data, that is performing well for millions of requests across the globe, be it for mobile or web traffic. I found a great base[2] and I think Go is a perfect language for this kind of service. Here are the free checkers I tested the other day, the IP is a good example for incomplete data: checkers = [ 'http://freegeoip.net/json/103.1.29.57', 'http://geoip.nekudo.com/api/103.1.29.57', 'http://ip-api.com/json/103.1.29.57', 'http://ipinfo.io/103.1.29.57/json', 'http://www.telize.com/geoip/103.1.29.57' ] [0] http://ip-api.com/docs/statistics http://ip-api.com/docs/statistics [1] http://geuis.com/2012/09/21/jsonip-com-close-to-serving-10-million-requests-a-month/ http://geuis.com/2012/09/21/jsonip-com-close-to-serving-10-m... [2] https://github.com/fiorix/freegeoip https://github.com/fiorix/freegeoip
- vld 11y agoip-api guy here. > It seems none of these services is deployed across the globe Our free service has servers in USA, EU and Asia > There's a need for a blazing fast, reliable service with granular, comprehensive data, that is performing well for millions of requests across the globe, be it for mobile or web traffic. We have more than 2 billion requests per month!
- hannob 11y agohonest opinion: I hate geoip. I think judging your visitors from their geolocation is more and more becoming a problem. It goes against the fundamental value of the Internet being a global network. I don't want Netflix to sell a sub-standard product in some countries because some licensing foo. I want everyone on this planet to have access to the same Internet. If I could I would redesign internet routing in a way that the webpage doesn't know where a request comes from.
- escaped_hn 11y agoor pay 5 bucks for a vpn.
- martin-adams 11y agoSadly, with the whole European VAT requirements the guidelines are as follows: https://www.gov.uk/government/publications/vat-supplying-digital-services-to-private-consumers/vat-businesses-supplying-digital-services-to-private-consumers https://www.gov.uk/government/publications/vat-supplying-dig... "the business making the supply must obtain and keep 2 pieces of non-contradictory information to support and evidence the member state where the customer is normally located." One of those being: "the Internet Protocol (IP) address of the device used by the customer" So sign up for something while abroad and you could get caught out.
- atmosx 11y ago> I want everyone on this planet to have access to the same Internet. Really? Do you want everyone on this planet to get taxed equally too? Because if he answer is 'yes', then you're strategy isn't working.
- geofft 11y ago> I don't want Netflix to sell a sub-standard product in some countries because some licensing foo. "some licensing foo" is a great way of eliding that in some countries, the service would be illegal. The alternative is that the product could not exist at all, if there is no way to negotiate a contract for it. You can certainly argue that products should not exist if that would involve contradicting some abstract ideal for society, or that it's good for things on the internet to blatantly violate mundane real-world laws until real-world governments become irrelevant. Respectable people hold both of those positions. But they're not very widely-accepted positions.
- Firegarden 11y agoI have a domain name you could use localipy.com. Just tell me your nameservers.
- Axsuul 11y agoI like the generosity here but no one in their right mind should consider this kind of offer unless the domain was transferred to them. There's too much risk involved and things that can go wrong unless a domain is 100% owned by you.
- Firegarden 11y agoRight. I have so many names and I never put them to good use :S
- social_quotient 11y agoI have a domain I've not done much with that might be perfect for this geoip.io Let me know if interested or if we could maybe partner on this
- bytesandbots 11y agoYou seem to be solving a very sought after problem which also makes it so much more difficult. Where do you plan to source this much info? Can somebody make me understand why ipv4 assignments along with their requests are not made public?
- devicenull 11y ago> Can somebody make me understand why ipv4 assignments along with their requests are not made public? What do you mean? With any of the RIR's you can get it down to a specific company. With ARIN, you're supposed to be able to get it down to a particular customer. What more information do you want?
- rajacombinator 11y agoI'd be interested. I'm not sure daily updates add much value, how often do these things change? If you have a way to provide more accuracy though, that is very interesting.
- ars 11y agoI'd definitely be interested, but I'd also be worried that you could really sustain interest in doing this for a decade.
- sintaxi 11y agohell yes!
- Hengjie 11y agoA slight tangent but good feedback to OP regardless: I use MaxMind at work but because we're on the RoR stack, we have to download and commit the massive 30MB databases into our repository in order for it to be used on Heroku. We use https://github.com/yhirose/maxminddb https://github.com/yhirose/maxminddb which seems to be the only good option available that reads the new "mmdb" format. It's a complete pain to have to commit such massive DBs for it to work and would prefer for Rails to fetch the database upon initialization, cache it for at most 2 days and redownload if it hasn't already been restarted by Heroku. If you can provide a solution to that, it would give us enough value for me to pay a few dollars per month for it.
- ericclemmons 11y agoI'm in a similar boat, but just have a license key on the machine (committed) and use the maxmind updater to fetch our licensed db. It happens once per provision, and a cron job to keep it up to date.
- jack9 11y agoWe do millions of lookups per day. We do not use Maxmind.
- openservices 11y agoWhat do you use instead of Maxmind?
- heyalexej 11y agoMind to elaborate a little bit on this?
- vivekv 11y agoI love a good free service where the data quality is improved. I have played around with maxmind's service a bit but the data for India is far from accurate. I have been hunting for a local database (no service) since I run a IP lookup service http://ipof.in http://ipof.in where I want the service subsumed in. This is a free service for anybody who uses it. It supports XML/JSON and plain text.
- dg81 11y agoLocation to Timezone would be of value to me. No IP involved, want to map a Google Places / Location City value ~ to a timezone. Would pay monthly fee.
- thecodemonkey 11y agoWe offer this at http://geocod.io http://geocod.io :)
- dg81 11y agoI took a look thanks. Pricing seems good, however we have a very international audience. We need to get timezones for geocodes (preferably Google Place IDs) from all over Asia and the Middle East. How good is your global timezone data? Is it possible to map Google Places (from their places autocomplete form) to a timezone?
- deleted 11y ago[deleted]
- blablabla123 11y agoMe! Once or twice a year I need that stuff on different occasions and each time I find it really annoying.
- dg81 11y agoTwo considerations that help avoid the IP issue, but require user permission and so have a big obvious disadvantage: 1) We can prompt mobile users to enable location/gps and get their location that way. 2) We can prompt modern browsers to allow location services and get their location that way. This is if a user wants to have their location 'auto-set' instead of via pulldowns. 3) A hard problem to solve is - mapping a unified set of city names to timezones, and updating the timezones as per Daylight Savings Time and general timezone updates. Anyone have references for 3) ?
- matiasb 11y agoWould it be possible to run some kind of GeoIP DB over a P2P/DHT network?
- david_ar 11y agoAlso see http://hostip.info http://hostip.info
- linshibo 11y agoI prefer csv. I care more about the content it provided, such as location , city, longitude , latitude， isp. In my company, we need a accurate database very much.And now we use IP2LOCATION-LITE-DB11.CSV and ipip.net .
- dutchbrit 11y agoThank you all for your feedback, I will continue working on this and make a new post once this project goes live.
- j45 11y agoThis would be very welcome. Also having SQL load files ready to go would be something that could be very easy to become dependant on and start paying for more frequent updates.
- wiml 11y agoHave you considered simply contributing to the service at http://www.hostip.info/ http://www.hostip.info/ ? They seem to be doing what you describe, and have been around long enough to have reasonably good information (in my experience)— I'm sure they would make use of any additional data, or hacking on their backend, which you might provide.