3 ms·

From the site: The NPS Data API is open and accessible to all developers who wish to use NPS data in their projects. From their "API Guides": Limits ar

From the site: The NPS Data API is open and accessible to all developers who wish to use NPS data in their projects. From their "API Guides": Limits are placed on the number of API requests you may make using your API key. Rate limits may vary by service, but the defaults are: Hourly Limit: 1,000 requests per hour For each API key, these limits are applied across all developer.nps.gov API requests. Exceeding these limits will lead to your API key being temporarily blocked from making further requests. The block will automatically be lifted by waiting an hour. That, along with their ToS[0], hardly seems to qualify as a "cargo-culting scourge." 0 - https://www.nps.gov/aboutus/disclaimer.htm https://www.nps.gov/aboutus/disclaimer.htm

AnthonyMouse 3y ago
API keys were invented as a tracking device. You sign up and then they associate all your use with one person and can do things like revoke your keys if you e.g. try to compete with the company's own products. Neither of these should be relevant to public data on a government service. Rate limits are straight forward to implement per-IP address without having any other information about anyone. The sort of person willing to bypass them by using a thousand IP addresses is the same sort of person who would sign up for a thousand API keys using fake names. How are you supposed to rate limit by API key if "anyone" can get an API key? You'd need to use some means to rate limit how many API keys someone could request, which was the original problem.
- mlhpdx 3y ago
  Per IPv6 address? It’s very difficult (impossible?) to even make IPv4 based rate limiting work.
  - AnthonyMouse 3y ago
    With IPv6 you use address blocks instead of individual addresses. IP-based rate limiting is extremely effective because it bifurcates the internet into IP addresses controlled by the attacker and ones that aren't. The attacker can only issue requests at a rate of rate limit per IP address times number of IP addresses (or IPv6 blocks) they control. Then the IP addresses under their control get denied while the IP addresses not under their control, i.e. all of the other users, are unaffected. This only becomes a problem if they control on the order of millions of IP addresses, but then you're dealing with a sophisticated criminal organization and are probably screwed anyway.
- AdieuToLogic 3y ago
  > API keys were invented as a tracking device. Yes, by definition. Apparently, you did not review the "Disclaimer" link I provided. In it is the following: Not all information or content on this website has been created or is owned by the NPS. Some content is protected by third party rights, such as copyright, trademark, rights of publicity, privacy, and contractual restrictions. The NPS endeavors to provide information that it possesses about the copyright status of the content and to identify any other terms and conditions that may apply to use of the content (such as, trademark, rights of privacy or publicity, donor restrictions, etc.); however, the NPS can offer no guarantee or assurance that all pertinent information is provided or that the information is correct in each circumstance. It is your responsibility to determine what permission(s) you need in order to use the content and, if necessary, to obtain such permission. Notice the first sentence; "Not all information or content on this website has been created or is owned by the NPS." Perhaps there is a need for use to be "tracked" in order to ensure legal agreement to the Terms of Use?
  - AnthonyMouse 3y ago
    That isn't a terms of use, it's a disclaimer. It's informing you that some of the information on the website might not be in the public domain, which is a simple factual statement that doesn't require you to agree to anything in order for it to be true or applicable.
    - AdieuToLogic 3y ago
      My bad, I thought of it as a ToS. Thanks for the clarification.
- vegardx 3y ago
  > API keys were invented as a tracking device And that's exactly how they're used as well. They need a method to track the usage of these services because there is often a cost involved with providing them. You also need a way to block or rate limit usage that is not IP bound. As an example, when Yr[0] opened up their APIs for free world-wide weather forecast it quickly spiralled out of control. I don't recall the specifics of it, but in short a major phone manufacturer started using their APIs on their phones and it took down the service because of the increased load. They could have solved it by just adding more hardware, things like this is highly cacheable, but when you're dealing with tax payers money you generally don't want to subsidise for-profit companies. So you implement a token and tell them to implement their own caching layer on top of it, and everyone is happy. I don't see how you'd solve something like that with anything other than a token. The methods you've mentioned in other posts simply don't work when a couple of hundred million phones ping your API every time they unlock their phone and it refreshes the weather widget. It also create no incentive for the developers to do things right, like not checking for updates every time the user does something, even though the initial request also came with a TTL and cache-control header that clearly states when this would be updated again. [0] https://developer.yr.no https://developer.yr.no
  - AnthonyMouse 3y ago
    > They could have solved it by just adding more hardware, things like this is highly cacheable, but when you're dealing with tax payers money you generally don't want to subsidise for-profit companies. So you implement a token and tell them to implement their own caching layer on top of it, and everyone is happy. The for-profit company is happy, anyway. They get free data and you've priced the competition out of the market. What things like this are really useful for is to create the app equivalent of weather.gov. Most for-profit "repackage government data" websites and apps are ad-laden spyware that will spin your CPU at 100% and shovel every byte of data they can hoover up into a data warehouse that sells to anyone with a buck while doing little more than displaying the government data. If you want to create an open source one which is free and promises not to track the user, you can, but then you need the data. If you end up with millions of users, who has more resources to set up caching servers, some individual idealist with zero revenue or the United States Government? This shouldn't even be a question. The government has to operate infrastructure that can handle millions of users for many other reasons. This should be something they're experienced in, and something like this should just fit into a slot in existing infrastructure. This is what it's for. If all you want is to provide the data for various scummy middlemen to wrap in ads and spyware then why is it an API at all instead of a static data dump / live feed with the latest changes?
mlhpdx 3y ago
Yep. Very likely they are using API Gateway with Usage Plans, which is a very simple and effective way to do rate limiting and quotas.

AnthonyMouse 3y ago

API keys were invented as a tracking device. You sign up and then they associate all your use with one person and can do things like revoke your keys if you e.g. try to compete with the company's own products. Neither of these should be relevant to public data on a government service. Rate limits are straight forward to implement per-IP address without having any other information about anyone. The sort of person willing to bypass them by using a thousand IP addresses is the same sort of person who would sign up for a thousand API keys using fake names. How are you supposed to rate limit by API key if "anyone" can get an API key? You'd need to use some means to rate limit how many API keys someone could request, which was the original problem.

mlhpdx 3y ago
Per IPv6 address? It’s very difficult (impossible?) to even make IPv4 based rate limiting work.
- AnthonyMouse 3y ago
  With IPv6 you use address blocks instead of individual addresses. IP-based rate limiting is extremely effective because it bifurcates the internet into IP addresses controlled by the attacker and ones that aren't. The attacker can only issue requests at a rate of rate limit per IP address times number of IP addresses (or IPv6 blocks) they control. Then the IP addresses under their control get denied while the IP addresses not under their control, i.e. all of the other users, are unaffected. This only becomes a problem if they control on the order of millions of IP addresses, but then you're dealing with a sophisticated criminal organization and are probably screwed anyway.
AdieuToLogic 3y ago
> API keys were invented as a tracking device. Yes, by definition. Apparently, you did not review the "Disclaimer" link I provided. In it is the following: Not all information or content on this website has been created or is owned by the NPS. Some content is protected by third party rights, such as copyright, trademark, rights of publicity, privacy, and contractual restrictions. The NPS endeavors to provide information that it possesses about the copyright status of the content and to identify any other terms and conditions that may apply to use of the content (such as, trademark, rights of privacy or publicity, donor restrictions, etc.); however, the NPS can offer no guarantee or assurance that all pertinent information is provided or that the information is correct in each circumstance. It is your responsibility to determine what permission(s) you need in order to use the content and, if necessary, to obtain such permission. Notice the first sentence; "Not all information or content on this website has been created or is owned by the NPS." Perhaps there is a need for use to be "tracked" in order to ensure legal agreement to the Terms of Use?
- AnthonyMouse 3y ago
  That isn't a terms of use, it's a disclaimer. It's informing you that some of the information on the website might not be in the public domain, which is a simple factual statement that doesn't require you to agree to anything in order for it to be true or applicable.
  - AdieuToLogic 3y ago
    My bad, I thought of it as a ToS. Thanks for the clarification.
vegardx 3y ago
> API keys were invented as a tracking device And that's exactly how they're used as well. They need a method to track the usage of these services because there is often a cost involved with providing them. You also need a way to block or rate limit usage that is not IP bound. As an example, when Yr[0] opened up their APIs for free world-wide weather forecast it quickly spiralled out of control. I don't recall the specifics of it, but in short a major phone manufacturer started using their APIs on their phones and it took down the service because of the increased load. They could have solved it by just adding more hardware, things like this is highly cacheable, but when you're dealing with tax payers money you generally don't want to subsidise for-profit companies. So you implement a token and tell them to implement their own caching layer on top of it, and everyone is happy. I don't see how you'd solve something like that with anything other than a token. The methods you've mentioned in other posts simply don't work when a couple of hundred million phones ping your API every time they unlock their phone and it refreshes the weather widget. It also create no incentive for the developers to do things right, like not checking for updates every time the user does something, even though the initial request also came with a TTL and cache-control header that clearly states when this would be updated again. [0] https://developer.yr.no https://developer.yr.no
- AnthonyMouse 3y ago
  > They could have solved it by just adding more hardware, things like this is highly cacheable, but when you're dealing with tax payers money you generally don't want to subsidise for-profit companies. So you implement a token and tell them to implement their own caching layer on top of it, and everyone is happy. The for-profit company is happy, anyway. They get free data and you've priced the competition out of the market. What things like this are really useful for is to create the app equivalent of weather.gov. Most for-profit "repackage government data" websites and apps are ad-laden spyware that will spin your CPU at 100% and shovel every byte of data they can hoover up into a data warehouse that sells to anyone with a buck while doing little more than displaying the government data. If you want to create an open source one which is free and promises not to track the user, you can, but then you need the data. If you end up with millions of users, who has more resources to set up caching servers, some individual idealist with zero revenue or the United States Government? This shouldn't even be a question. The government has to operate infrastructure that can handle millions of users for many other reasons. This should be something they're experienced in, and something like this should just fit into a slot in existing infrastructure. This is what it's for. If all you want is to provide the data for various scummy middlemen to wrap in ads and spyware then why is it an API at all instead of a static data dump / live feed with the latest changes?

mlhpdx 3y ago

Yep. Very likely they are using API Gateway with Usage Plans, which is a very simple and effective way to do rate limiting and quotas.