3 ms·

I think it can be done responsibly. If you're not imposing any more traffic on them than you would be visiting their site manually, then using their APIs to get

I think it can be done responsibly. If you're not imposing any more traffic on them than you would be visiting their site manually, then using their APIs to get at structured data is actually win-win compared to the alternative (load the whole thing and scrape). Where I'll agree with you is cases where people do this and impose way more traffic than is typical and often more than is necessary (i.e. no caching). But that's not really specific to reverse engineering apis, that's just about being a good internet citizen in general. I'm of the opinion that a user agent is a user agent, and website owners shouldn't pick and choose what user agents they support. Target the behaviors that affect your infrastructure, not the means of access and algorithms used to process what you send.

Eikon 2y ago
> If you're not imposing any more traffic on them than you would be visiting their site manually, then using their APIs to get at structured data is actually win-win compared to the alternative (load the whole thing and scrape). I agree but that's not usually how it goes. From what I've seen, it's mostly very poorly written scripts with not rate limiting and no backoff strategy that will be hitting your api servers.
- lolinder 2y ago
  Would you rather have those poorly-written scripts hitting your APIs or have a poorly-written puppeteer script loading every asset you have—hitting the APIs along with everything else? Casting shade on API reverse engineering when what you actually have is a failure to rate limit is throwing the baby out with the bathwater. Abusive users will abuse until you build in a technological method to stop them, and user-agent sniffing provably doesn't work to stop bad actors. The concept of a flexible, customizable User Agent that operates on my behalf is a key idea that's foundational to the web, and I'm not willing to cede that cultural ground in the vague hope that we can make the bad guys feel bad and start just using Chrome like civilized people.

Eikon 2y ago

> If you're not imposing any more traffic on them than you would be visiting their site manually, then using their APIs to get at structured data is actually win-win compared to the alternative (load the whole thing and scrape). I agree but that's not usually how it goes. From what I've seen, it's mostly very poorly written scripts with not rate limiting and no backoff strategy that will be hitting your api servers.

lolinder 2y ago
Would you rather have those poorly-written scripts hitting your APIs or have a poorly-written puppeteer script loading every asset you have—hitting the APIs along with everything else? Casting shade on API reverse engineering when what you actually have is a failure to rate limit is throwing the baby out with the bathwater. Abusive users will abuse until you build in a technological method to stop them, and user-agent sniffing provably doesn't work to stop bad actors. The concept of a flexible, customizable User Agent that operates on my behalf is a key idea that's foundational to the web, and I'm not willing to cede that cultural ground in the vague hope that we can make the bad guys feel bad and start just using Chrome like civilized people.