5 ms·

How Ipdata Serves 25M API Calls from 10 Global Endpoints for $150 a Month

sandGorgon 9y ago
super cool article! I do wonder if you didnt try Apistar (https://github.com/encode/apistar https://github.com/encode/apistar) . Its built by the author of django rest framework .and also is much more frequently updated
- jonathan-kosgei 9y ago
  Thanks! :) I had no idea apistar existed, it certainly didn't come up when I was doing my research. But it looks pretty full featured, thank you for this!
- chimen 9y ago
  What is apistar supposed to be solving in this context?
  - jonathan-kosgei 9y ago
    I haven't worked with it yet, but it looks to me to be a Python framework just like Japronto or Flask but one that is focused heavily on building APIs.
    - chimen 9y ago
      It was a top/root comment that's why I asked. I thought it does something special. Also worth a look: http://python-eve.org/ http://python-eve.org/
      - jonathan-kosgei 9y ago
        No prob, Eve looks pretty cool. I'd love to benchmark it alongside japronto and apistar. Thanks!
seanp2k2 9y ago
That does seem good, but when you consider that it’s basically just doing a DB query for an IP (and there might be a clever way to store CIDRs where the DB finds it within a range) + authentication, authorization, and accounting ( AAA), it becomes a bit less impressive. I’m guessing again that the AAA stuff is actually more work than the actual lookups. I’ve made some services which use geoip before with in-memory lookups. Since it doesn’t change that frequently, aside from AAA, you don’t really need an external DB at all — just have the actual executable contain the lookup info, now you’re down to just lambda. Actually, don’t even build a service — just distribute a binary with the tables inside, so then customers don’t even need to make a call, wait, and pay for each call. Consider how well that would scale (and indeed this is already the business model for some of the MaxMind products (e.g. https://www.maxmind.com/en/geoip2-city https://www.maxmind.com/en/geoip2-city ), but it doesn’t make for as interesting of an article.
- ddorian43 9y ago
  How I scaled my single-function + single-db-query api!
  - jonathan-kosgei 9y ago
    If anything it's closer to - How I built a multi-regional apigateway API with multiple creative workarounds despite there being no official way to do it and it only becoming possible to do so 5 months ago :)
    - deleted 9y ago
      [deleted]
    - kreetx 9y ago
      Hear hear! Clearly a significant amount of effort has went into figuring out this design, and it's interesting to read about it. EDIT: And you're the author -- thanks!
- jonathan-kosgei 9y ago
  The article is about building a fast and highly available API distribution architecture. You could just as well apply it to anything you want to build. I will agree that it's not for everyone. And would probably more likely appeal to indie makers like myself.
orf 9y ago
Interesting read, thanks for posting, but I think you can do better. I've been playing with Fastly and VCL recently and you could write your whole app in VCL and have it served right from the CDN. 25 million requests would cost about $20 for that. If you include the API keys in the log output and batch-process that each day/hour/whatever to keep usage stats, that would remove the need for Dynamo? You'd then keep a table of banned API keys and update that through the Fastly API dynamically. Just a thought. Still, $150 p/m is not bad!
- jonathan-kosgei 9y ago
  Awesome! thanks for sharing! I'm always looking for optimizations. Any idea whether fastly would support running out of specific regions? Or how many endpoints they have?
  - orf 9y ago
    Not sure if you can run them on specific regions. They have a lot of endpoints though, there is a list on their status page: https://status.fastly.com/ https://status.fastly.com/. I count 55.
    - jonathan-kosgei 9y ago
      Thanks! I'm excited to try this and benchmark it against my current setup!
- zzzcpan 9y ago
  Considering it's like 20 simple requests per second, you can do it on a couple of tiny virtual servers (for geo and redundancy), totaling less than $10 per month.
  - orf 9y ago
    the hard bit is the API key and usage monitoring. you could write the whole thing as Cloudflare workers for $5 per month else. $20 for 55 'servers' spread worldwide isn't bad, with no maintenance overhead.
samwillis 9y ago
Out of interest where did you get the regional data for IP addresses from? Did you collect it yourself, if so how? There seem to be very few sources that allow you to resell it.
- jonathan-kosgei 9y ago
  We aggregate the data from a couple of sources, see https://ipdata.co/about.html https://ipdata.co/about.html
xstartup 9y ago
Why not push to Kinesis directly in a background worker? I understand printing log is simple but you still have to push that log to cloudwatch... it's still being done on the same system. So, you end up paying that cost either way. You can use MongoDB atlas with multi-region replication for low latency queries instead of DynamoDB.

super cool article! I do wonder if you didnt try Apistar (https://github.com/encode/apistar https://github.com/encode/apistar) . Its built by the author of django rest framework .and also is much more frequently updated

jonathan-kosgei 9y ago
Thanks! :) I had no idea apistar existed, it certainly didn't come up when I was doing my research. But it looks pretty full featured, thank you for this!
chimen 9y ago
What is apistar supposed to be solving in this context?
- jonathan-kosgei 9y ago
  I haven't worked with it yet, but it looks to me to be a Python framework just like Japronto or Flask but one that is focused heavily on building APIs.
  - chimen 9y ago
    It was a top/root comment that's why I asked. I thought it does something special. Also worth a look: http://python-eve.org/ http://python-eve.org/
    - jonathan-kosgei 9y ago
      No prob, Eve looks pretty cool. I'd love to benchmark it alongside japronto and apistar. Thanks!

seanp2k2 9y ago

That does seem good, but when you consider that it’s basically just doing a DB query for an IP (and there might be a clever way to store CIDRs where the DB finds it within a range) + authentication, authorization, and accounting ( AAA), it becomes a bit less impressive. I’m guessing again that the AAA stuff is actually more work than the actual lookups. I’ve made some services which use geoip before with in-memory lookups. Since it doesn’t change that frequently, aside from AAA, you don’t really need an external DB at all — just have the actual executable contain the lookup info, now you’re down to just lambda. Actually, don’t even build a service — just distribute a binary with the tables inside, so then customers don’t even need to make a call, wait, and pay for each call. Consider how well that would scale (and indeed this is already the business model for some of the MaxMind products (e.g. https://www.maxmind.com/en/geoip2-city https://www.maxmind.com/en/geoip2-city ), but it doesn’t make for as interesting of an article.

ddorian43 9y ago
How I scaled my single-function + single-db-query api!
- jonathan-kosgei 9y ago
  If anything it's closer to - How I built a multi-regional apigateway API with multiple creative workarounds despite there being no official way to do it and it only becoming possible to do so 5 months ago :)
  - deleted 9y ago
    [deleted]
  - kreetx 9y ago
    Hear hear! Clearly a significant amount of effort has went into figuring out this design, and it's interesting to read about it. EDIT: And you're the author -- thanks!
jonathan-kosgei 9y ago
The article is about building a fast and highly available API distribution architecture. You could just as well apply it to anything you want to build. I will agree that it's not for everyone. And would probably more likely appeal to indie makers like myself.

orf 9y ago

Interesting read, thanks for posting, but I think you can do better. I've been playing with Fastly and VCL recently and you could write your whole app in VCL and have it served right from the CDN. 25 million requests would cost about $20 for that. If you include the API keys in the log output and batch-process that each day/hour/whatever to keep usage stats, that would remove the need for Dynamo? You'd then keep a table of banned API keys and update that through the Fastly API dynamically. Just a thought. Still, $150 p/m is not bad!

jonathan-kosgei 9y ago
Awesome! thanks for sharing! I'm always looking for optimizations. Any idea whether fastly would support running out of specific regions? Or how many endpoints they have?
- orf 9y ago
  Not sure if you can run them on specific regions. They have a lot of endpoints though, there is a list on their status page: https://status.fastly.com/ https://status.fastly.com/. I count 55.
  - jonathan-kosgei 9y ago
    Thanks! I'm excited to try this and benchmark it against my current setup!
zzzcpan 9y ago
Considering it's like 20 simple requests per second, you can do it on a couple of tiny virtual servers (for geo and redundancy), totaling less than $10 per month.
- orf 9y ago
  the hard bit is the API key and usage monitoring. you could write the whole thing as Cloudflare workers for $5 per month else. $20 for 55 'servers' spread worldwide isn't bad, with no maintenance overhead.

samwillis 9y ago

Out of interest where did you get the regional data for IP addresses from? Did you collect it yourself, if so how? There seem to be very few sources that allow you to resell it.

jonathan-kosgei 9y ago
We aggregate the data from a couple of sources, see https://ipdata.co/about.html https://ipdata.co/about.html

xstartup 9y ago

Why not push to Kinesis directly in a background worker? I understand printing log is simple but you still have to push that log to cloudwatch... it's still being done on the same system. So, you end up paying that cost either way. You can use MongoDB atlas with multi-region replication for low latency queries instead of DynamoDB.