5 ms·

An Alternative to Required API Keys

30 points by EzGraphs 13y ago

martin-adams 13y ago
To me it seems like hanging would be more of a deterrent than failing requests as developers could simply retry a request on a simulated failure. Some developers could even try to fire multiple requests and use the fastest one, effectively reducing the failure rate to the app. It an interesting problem to solve. The one that I would like to see a solution on is allowing end users interact directly with a third party service without exposing the API key or requiring the to do any authentication.
- pearkes 13y ago
  When Google phased out their undocumented weather API that a lot of people used[1], leading up to it for a few weeks they just randomly returned 403 to random requests. Good way to confuse developers and motivate them to use a real, supported API. [1] http://thenextweb.com/google/2012/08/28/did-google-just-quietly-kill-private-weather-api/ http://thenextweb.com/google/2012/08/28/did-google-just-quie...
- jsharpe 13y ago
  Another approach to discourage use in production would be to randomly return incorrect or garbled data. It's still easy to use for learning, but would lead to embarrassing errors or omissions if used for something real.
  - martin-adams 13y ago
    Maybe, but that would just result in less trust in quality of the service. Which errors and hangs would do also. I'd rather see simulated data if it's purely for development purposes. You would have far more control over the conditions you need to test.
ProblemFactory 13y ago
One good bit of advice I have seen for people who are just starting out with publishing an API: require a "developer email" parameter in the request instead of API key. Yes, it could be spoofed or faked. But: * You can detect honest, accidental excessive usage and follow up by email, * You can block new email addresses but keep the API up for existing users if hit by a randomised bot, * You can follow up with developers, and interview them to see what they are using the API for, * There is an obvious way to get in touch with developers to tell them about API changes, * And you can start out with this by just logging requests to a database/file/cache, without building a full "developer portal".
- junto 13y ago
  Good idea, but it does have one failing. When an API key has been compromised (e.g. a developer leaves the company on bad terms), it is much easier to change a randomly generated API key without impacting anything else. Changing an email address is a hassle for the developer, should you need to revoke the 'key' due to misuse by a third party. Also, this should be obvious, but many people forget to use a secure transport for their APIs, which then leaks private information, such as API keys and other HTTP headers parameters whilst being sent over the wire. SSL is a must for APIs.
  - dayglogee 13y ago
    I suppose you could have your mail system forward all mails with the following format to your real mailbox: api-<guid>@yourdomain.com
  - ProblemFactory 13y ago
    Once the API gets to the stage where you need authentication, access controls, blocking access, proper rate-limiting or billing, you have outgrown the "email address in request" method. After all, the abuser might not be a former developer, but anyone in the world who knows your email address. But for a first experimental API, it's much better than having no info on who your users are besides an IP address pointing to AWS. And logging API requests to a text file and doing analytics manually with grep is much faster than setting up developer accounts.
TillE 13y ago
> Rate limiting is one approach but it is easily gamed. Is it? Simple IP-based rate limiting seems quite effective to all but the most determined abuser.
- lazyjones 13y ago
  > Simple IP-based rate limiting seems quite effective to all but the most determined abuser. Most (noticeable) abusers are quite determined. They will try Tor, then various anonymizer proxies (forcing you to block all of them or give up), then sometimes Amazon EC2 instances or other VPS, or large public proxies / ISP proxies that you cannot block. In Germany, they can also use cheap DSL where they can switch to new IP addresses almost instantaneously (T-Online is a godsend for abusers/scrapers).
  - dubcanada 13y ago
    What's to stop them from creating a bit that creates API keys every 5 minutes?
    - iaskwhy 13y ago
      A delay on the API key generation or approval?
  - dspillett 13y ago
    > In Germany, they can also use cheap DSL where they can switch to new IP addresses almost instantaneously That one is easy to reduce the effect of: as well as a per-IP limit have a per-/24 limit that is much lower then 256 times the single address limit. To avoid seeming to be too unfair to other people on the same ISP who happen to be at an address next to a heavy-but-not-abusive user make this limit's bucket refill more often (i.e. if you limit to 100 requests/day per IP, try 100 requests/hour per 256 address block). You might consider different granularities instead of or as well as /32 and /24. It is not uncommon for a small organisation to have a /29 or /28 allocated (heck, I have a /29 on my line at home as I host a couple of services from it that need their own address). Perhaps set limits for granularities of /32, /28, /24 and /20? Of course this requires development effort, implementing an efficient method of enforcing these limits, but not lots. > They will try Tor, then various anonymizer proxies (forcing you to block all of them or give up) Are there really enough reliable cost-free options out there to make that a massive problem? A few determined people will use more resource than you wont by these methods, but the majority will either put up with the rate limit, pay what-ever you are asking for a less limited option, or do somewhere else (the third option being something you'd prefer didn't happen, of course, but you'll always lose some users by refusing to give them the moon on a stick). > Amazon EC2 instances or other VPS, or large public proxies / ISP proxies that you cannot block While you can't block them, you can apply rate limits at multiple granularities as mentioned above to limit their effect. To reduce the effect of ISP proxies on HTTP APIs force the use of HTTPS (less HTTPS traffic goes via a proxy than does HTTP traffic - ISPs that use transparent proxies for HTTP caching don't do the same for HTTPS as there would be no point). THE THING NO ONE HAS MENTIONED YET Of course this all changes a little once you consider IPv6. A per address limit is meaningless when everyone has a /64 or wider to play with so you are going to have to implement ranged limits (perhaps with gradiated limits like above for /64, /56, /48, ...) if you are limiting by address, and you are going to limit by ISP when trying to limit by individual more often than you will filtering IPv4 addresses this way (some VPS providers hand out small numbers of v6 addresses to each VM so they presumably use one /64 for everyone, where-as I have a /64 allocated to my home line.
onion2k 13y ago
If you're trying to put people off abusing it, wouldn't randomly failing make the situation worse? People will just code apps to try the request again until it doesn't. It'd be like inviting people to DDoS your API. So long as the display bit is asynchronous then the user will just see a spinner until it's worked and the results are sent across.
chiph 13y ago
Required keys make it much harder for developers to learn new things. Really? You fill out a form, click submit, get your key via email soon after. Or at worst, the following Monday. IMO, that's not a lot of friction.
- pavel_lishin 13y ago
  > Or at worst, the following Monday. For someone just starting out, having to wait the entire weekend that you planned to use for studying is pretty much a dead end.
  - chiph 13y ago
    I would say that their attention span is too short, then. And that they're more of a dilettante than a student.
  - btgeekboy 13y ago
    Tell that to the Google Cast team. Saturday morning: "Hmm, how does this stuff work? Oh, I've got to get a key..." Monday night, 11PM: "Here's your key!"
moron4hire 13y ago
how hard is it to run a second, smaller server for "learning" that doesn't get rate limited but also doesn't have full access? Maybe it even only has access to a fake data set, no real data.

30 points by EzGraphs 13y ago

To me it seems like hanging would be more of a deterrent than failing requests as developers could simply retry a request on a simulated failure. Some developers could even try to fire multiple requests and use the fastest one, effectively reducing the failure rate to the app. It an interesting problem to solve. The one that I would like to see a solution on is allowing end users interact directly with a third party service without exposing the API key or requiring the to do any authentication.

pearkes 13y ago
When Google phased out their undocumented weather API that a lot of people used[1], leading up to it for a few weeks they just randomly returned 403 to random requests. Good way to confuse developers and motivate them to use a real, supported API. [1] http://thenextweb.com/google/2012/08/28/did-google-just-quietly-kill-private-weather-api/ http://thenextweb.com/google/2012/08/28/did-google-just-quie...
jsharpe 13y ago
Another approach to discourage use in production would be to randomly return incorrect or garbled data. It's still easy to use for learning, but would lead to embarrassing errors or omissions if used for something real.
- martin-adams 13y ago
  Maybe, but that would just result in less trust in quality of the service. Which errors and hangs would do also. I'd rather see simulated data if it's purely for development purposes. You would have far more control over the conditions you need to test.

ProblemFactory 13y ago

One good bit of advice I have seen for people who are just starting out with publishing an API: require a "developer email" parameter in the request instead of API key. Yes, it could be spoofed or faked. But: * You can detect honest, accidental excessive usage and follow up by email, * You can block new email addresses but keep the API up for existing users if hit by a randomised bot, * You can follow up with developers, and interview them to see what they are using the API for, * There is an obvious way to get in touch with developers to tell them about API changes, * And you can start out with this by just logging requests to a database/file/cache, without building a full "developer portal".

junto 13y ago
Good idea, but it does have one failing. When an API key has been compromised (e.g. a developer leaves the company on bad terms), it is much easier to change a randomly generated API key without impacting anything else. Changing an email address is a hassle for the developer, should you need to revoke the 'key' due to misuse by a third party. Also, this should be obvious, but many people forget to use a secure transport for their APIs, which then leaks private information, such as API keys and other HTTP headers parameters whilst being sent over the wire. SSL is a must for APIs.
- dayglogee 13y ago
  I suppose you could have your mail system forward all mails with the following format to your real mailbox: api-<guid>@yourdomain.com
- ProblemFactory 13y ago
  Once the API gets to the stage where you need authentication, access controls, blocking access, proper rate-limiting or billing, you have outgrown the "email address in request" method. After all, the abuser might not be a former developer, but anyone in the world who knows your email address. But for a first experimental API, it's much better than having no info on who your users are besides an IP address pointing to AWS. And logging API requests to a text file and doing analytics manually with grep is much faster than setting up developer accounts.

TillE 13y ago

> Rate limiting is one approach but it is easily gamed. Is it? Simple IP-based rate limiting seems quite effective to all but the most determined abuser.

lazyjones 13y ago
> Simple IP-based rate limiting seems quite effective to all but the most determined abuser. Most (noticeable) abusers are quite determined. They will try Tor, then various anonymizer proxies (forcing you to block all of them or give up), then sometimes Amazon EC2 instances or other VPS, or large public proxies / ISP proxies that you cannot block. In Germany, they can also use cheap DSL where they can switch to new IP addresses almost instantaneously (T-Online is a godsend for abusers/scrapers).
- dubcanada 13y ago
  What's to stop them from creating a bit that creates API keys every 5 minutes?
  - iaskwhy 13y ago
    A delay on the API key generation or approval?
- dspillett 13y ago
  > In Germany, they can also use cheap DSL where they can switch to new IP addresses almost instantaneously That one is easy to reduce the effect of: as well as a per-IP limit have a per-/24 limit that is much lower then 256 times the single address limit. To avoid seeming to be too unfair to other people on the same ISP who happen to be at an address next to a heavy-but-not-abusive user make this limit's bucket refill more often (i.e. if you limit to 100 requests/day per IP, try 100 requests/hour per 256 address block). You might consider different granularities instead of or as well as /32 and /24. It is not uncommon for a small organisation to have a /29 or /28 allocated (heck, I have a /29 on my line at home as I host a couple of services from it that need their own address). Perhaps set limits for granularities of /32, /28, /24 and /20? Of course this requires development effort, implementing an efficient method of enforcing these limits, but not lots. > They will try Tor, then various anonymizer proxies (forcing you to block all of them or give up) Are there really enough reliable cost-free options out there to make that a massive problem? A few determined people will use more resource than you wont by these methods, but the majority will either put up with the rate limit, pay what-ever you are asking for a less limited option, or do somewhere else (the third option being something you'd prefer didn't happen, of course, but you'll always lose some users by refusing to give them the moon on a stick). > Amazon EC2 instances or other VPS, or large public proxies / ISP proxies that you cannot block While you can't block them, you can apply rate limits at multiple granularities as mentioned above to limit their effect. To reduce the effect of ISP proxies on HTTP APIs force the use of HTTPS (less HTTPS traffic goes via a proxy than does HTTP traffic - ISPs that use transparent proxies for HTTP caching don't do the same for HTTPS as there would be no point). THE THING NO ONE HAS MENTIONED YET Of course this all changes a little once you consider IPv6. A per address limit is meaningless when everyone has a /64 or wider to play with so you are going to have to implement ranged limits (perhaps with gradiated limits like above for /64, /56, /48, ...) if you are limiting by address, and you are going to limit by ISP when trying to limit by individual more often than you will filtering IPv4 addresses this way (some VPS providers hand out small numbers of v6 addresses to each VM so they presumably use one /64 for everyone, where-as I have a /64 allocated to my home line.

onion2k 13y ago

If you're trying to put people off abusing it, wouldn't randomly failing make the situation worse? People will just code apps to try the request again until it doesn't. It'd be like inviting people to DDoS your API. So long as the display bit is asynchronous then the user will just see a spinner until it's worked and the results are sent across.

chiph 13y ago

Required keys make it much harder for developers to learn new things. Really? You fill out a form, click submit, get your key via email soon after. Or at worst, the following Monday. IMO, that's not a lot of friction.

pavel_lishin 13y ago
> Or at worst, the following Monday. For someone just starting out, having to wait the entire weekend that you planned to use for studying is pretty much a dead end.
- chiph 13y ago
  I would say that their attention span is too short, then. And that they're more of a dilettante than a student.
- btgeekboy 13y ago
  Tell that to the Google Cast team. Saturday morning: "Hmm, how does this stuff work? Oh, I've got to get a key..." Monday night, 11PM: "Here's your key!"

moron4hire 13y ago

how hard is it to run a second, smaller server for "learning" that doesn't get rate limited but also doesn't have full access? Maybe it even only has access to a fake data set, no real data.