4 ms·

Just cache the responses so that only the first time a new IP connects it needs to be looked up (until cache fills) Integrating maxmind is a bit of a pain and

Just cache the responses so that only the first time a new IP connects it needs to be looked up (until cache fills) Integrating maxmind is a bit of a pain and it’s a massive database. Not well suited to use cases like single purpose or ephemeral containers. You’ll probably end up storing the maxmind db in a separate node on your local network, anyway. So you still have a latency issue (although lower, obviously). More importantly you now need to maintain an extra box per replicated cluster. It could make sense to outsource that management if the latency between your data center and ipdata is acceptable. Such low latency is easily achievable if ipdata has BGP anycast on servers in the same cloud region as your app.

mywittyname 9y ago
We already do something like that, marking up the user information with geoip information when we first see them, but we just get a lot of unique users. (Yes, we could do batch lookups, but that would be a complete rewrite of our workflow.) I found Maxmind integration to be pretty straight-forward. They have APIs for most popular languages and most of the time, usage amounts to instantiating a reader object with the path to the database, then calling methods on that object with the IP in question. So, db.city(ip) returns all of the city information, etc. Massive is a relative term I guess. But the databases are segmented, so you only take what you need and all the ones I've worked with have been tens of megabytes in size.
ddorian43 9y ago
It's a 100mb db. You load it in-memory (mmap) on every webserver.
- chatmasta 9y ago
  I consider 100mb to be quite large. That could easily double memory consumption of an application. Most http services are light but widely replicated. Adding 100mb of memory usage to each replica is certainly a tradeoff.
  - ddorian43 9y ago
    Assuming mmap 100mb will happen only once in each web-server.
    - chatmasta 9y ago
      What exactly do you mean by web server? Generally you will have at least one server process wherever you do the maxmind lookup, which could be at the LB/proxy (Nginx) or app server (node, python, go, etc). Nowadays these services are often heavily decoupled and therefore built to be as lightweight as possible. Think about small single purpose containers. Adding 100mb of memory requirements per instance could be quite expensive, depending which existing process is calling maxmind.
      - boundlessdreamz 9y ago
        are you passing the original ip addresses to all these containers? Why don't you just decode the ip to location at the edge of your network (typically load balancer) and add a header with location?
        chatmasta 9y ago
        That’s the best way to do it of course. But then you’re adding 100mb to each instance in your load balancing layer.
      - ddorian43 9y ago
        Say you have python/ruby/js processes. You usually build 1 box/container with multiple processes inside. If you have 100 services and you distribute 1 process of each in each box then THAT is your problem. "Nowadays" people are wrong.

mywittyname 9y ago

We already do something like that, marking up the user information with geoip information when we first see them, but we just get a lot of unique users. (Yes, we could do batch lookups, but that would be a complete rewrite of our workflow.) I found Maxmind integration to be pretty straight-forward. They have APIs for most popular languages and most of the time, usage amounts to instantiating a reader object with the path to the database, then calling methods on that object with the IP in question. So, db.city(ip) returns all of the city information, etc. Massive is a relative term I guess. But the databases are segmented, so you only take what you need and all the ones I've worked with have been tens of megabytes in size.

ddorian43 9y ago

It's a 100mb db. You load it in-memory (mmap) on every webserver.

chatmasta 9y ago
I consider 100mb to be quite large. That could easily double memory consumption of an application. Most http services are light but widely replicated. Adding 100mb of memory usage to each replica is certainly a tradeoff.
- ddorian43 9y ago
  Assuming mmap 100mb will happen only once in each web-server.
  - chatmasta 9y ago
    What exactly do you mean by web server? Generally you will have at least one server process wherever you do the maxmind lookup, which could be at the LB/proxy (Nginx) or app server (node, python, go, etc). Nowadays these services are often heavily decoupled and therefore built to be as lightweight as possible. Think about small single purpose containers. Adding 100mb of memory requirements per instance could be quite expensive, depending which existing process is calling maxmind.
    - boundlessdreamz 9y ago
      are you passing the original ip addresses to all these containers? Why don't you just decode the ip to location at the edge of your network (typically load balancer) and add a header with location?
      - chatmasta 9y ago
        That’s the best way to do it of course. But then you’re adding 100mb to each instance in your load balancing layer.
    - ddorian43 9y ago
      Say you have python/ruby/js processes. You usually build 1 box/container with multiple processes inside. If you have 100 services and you distribute 1 process of each in each box then THAT is your problem. "Nowadays" people are wrong.