4 ms·

This approach is generally seen as unwanted by website owners (it's worth noting that automated API clients are distinct from regular user agents). As a “revers

by Eikon 2y ago

This approach is generally seen as unwanted by website owners (it's worth noting that automated API clients are distinct from regular user agents). As a “reverse engineer”, you have no idea how expensive or not an endpoint is to process a request. Instead, I'd recommend reaching out to the website owners directly to discuss your API needs - they're often interested in hearing about potential integrations and use cases. If you don't receive a response, proceeding with unauthorized API usage is mostly abusive and poor internet citizenship.

gsky 2y ago
Unspoken rule is that maintaining 3 seconds delay between requests
lolinder 2y ago
I think it can be done responsibly. If you're not imposing any more traffic on them than you would be visiting their site manually, then using their APIs to get at structured data is actually win-win compared to the alternative (load the whole thing and scrape). Where I'll agree with you is cases where people do this and impose way more traffic than is typical and often more than is necessary (i.e. no caching). But that's not really specific to reverse engineering apis, that's just about being a good internet citizen in general. I'm of the opinion that a user agent is a user agent, and website owners shouldn't pick and choose what user agents they support. Target the behaviors that affect your infrastructure, not the means of access and algorithms used to process what you send.
- Eikon 2y ago
  > If you're not imposing any more traffic on them than you would be visiting their site manually, then using their APIs to get at structured data is actually win-win compared to the alternative (load the whole thing and scrape). I agree but that's not usually how it goes. From what I've seen, it's mostly very poorly written scripts with not rate limiting and no backoff strategy that will be hitting your api servers.
  - lolinder 2y ago
    Would you rather have those poorly-written scripts hitting your APIs or have a poorly-written puppeteer script loading every asset you have—hitting the APIs along with everything else? Casting shade on API reverse engineering when what you actually have is a failure to rate limit is throwing the baby out with the bathwater. Abusive users will abuse until you build in a technological method to stop them, and user-agent sniffing provably doesn't work to stop bad actors. The concept of a flexible, customizable User Agent that operates on my behalf is a key idea that's foundational to the web, and I'm not willing to cede that cultural ground in the vague hope that we can make the bad guys feel bad and start just using Chrome like civilized people.
MathMonkeyMan 2y ago
Nah. Bots looking for exploits is rude, spamming an endpoint with more traffic than normal is rude.. but a human trying to figure out the API that you exposed to the internet? That's just fair play. Also, better to ask for forgiveness than to ask for permission. The author is adding value to the world while hurting nobody, and the answer would likely be an automatic "no" anyway.
boolemancer 2y ago
In my personal view, this seems a little overbearing. If you expose an API, and you want to tell a user that they are "unauthorized" to use it, it should return a 401 status code so that the caller knows they're unauthorized. If you can't do that because their traffic looks like normal usage of the API by your web app, then I question why their usage is problematic for you. At the end of the day, you don't get to control what 'browser' the user uses to interact with your service. Sure, it might be Chrome, but it just as easily might be Firefox, or Lynx, or something the user built from scratch, or someone manually typing out HTTP requests in netcat, or, in this case, someone building a custom client for your specific service. If you host a web server, it's on you to remember that and design accordingly, not on the user to limit how they use your service.
- AlienRobot 2y ago
  That's like saying if someone accepts cash that means you should be allowed to pay a $100 bill with a thousand dimes. Just because you're right doesn't mean you aren't wrong.
  - lolinder 2y ago
    The $100 tab paid in dimes causes severe inconvenience to the person trying to count them and to the person who has to take them to the bank to cash them in and wait for them to be counted again. Their very reasonable question was: if you can't distinguish the reverse engineered traffic from the traffic through your own app in order to block it, then what harm is the traffic doing? Presumably it's flying under your rate limits, and the traffic has a valid session token from a real customer. If you're unable to single it out and return a 4xx, why does it matter where it's coming from? I can think of a few reasons it might, but I'm not particularly sympathetic to them. They generally boil down to "I won't be able to use my app to manipulate the user into taking actions they'd otherwise not take." I'd be interested to hear if there are better reasons.
    - AlienRobot 2y ago
      "if you can't distinguish the reverse engineered traffic from the traffic through your own app in order to block it, then what harm is the traffic doing?" If you really believe this you'll use a custom user agent instead of spoofing Chrome. :-) Some websites use HTTP referer to block traffic. Ask yourself if any reverse engineer would be stopped by what is obviously the website telling you not to access an endpoint. I'll add that end users don't have complete information about the website. They can't know how many resources a website has to deal to reverse engineering (webmasters can't just play cat and mouse with you just because you're wasting their money) nor do they know the cost of an endpoint. I mean, most tech inclined use ad blockers when it's obvious 90% of the websites pay the cost of their endpoints by showing ads, so I doubt they would respect anything more subtle than that.
gaeb69 2y ago
I agree. When scraping my school's portal which uses Canvas, my school actually allows it. Sometimes you can get the green light just by reading API docs/School's privacy policy as they're usually obliged to have one (ofc this primarily applies to school APIs like in OP's article)
mindslight 2y ago
A company can always document their API if they want power users to be informed about the company's preferred ways of doing things. They generally don't because they want to create a bunch of market friction so you're forced into using their proprietary client apps to interact with their services. I for one applaud efforts like the original post, and think it's an instance of outstanding Internet citizenship, where citizenship means working to preserve Internet societal norms like decentralization.
rmbyrro 2y ago
This would be akin to asking a shop owner if I'm allowed to pick a panflet from an endless stack of panflets placed on the sidewalk. If they don't want the public to pick panflets, don't put them where the public can reach'em!
- mtnGoat 2y ago
  Same could be said about things in your front yard? I’m not sure taking liberties with things you don’t own, is always the best policy, nor is putting the entire responsibility on the owner. I don’t think this is something you can boil down to a simple black and white.
  - barnabee 2y ago
    Accessing a publicly available web service is not “taking liberties with things you don’t own”. If you put a server on the public internet and I send it a message (assuming I’m not using ill-gotten credentials, etc.), anything it responds with is your problem, not mine.
  - Chris2048 2y ago
    > Same could be said about things in your front yard? No. It could be said, but wouldn't be true - objects in a domestic front yard are nothing like pamphlets placed on the sidewalk.
    - mtnGoat 2y ago
      Things on sidewalks are not free to take because you can touch them. Just like things in your front yard are not. Just because you can get to it, doesn’t mean it’s yours to take and act upon at will.
      - Chris2048 2y ago
        Pamphlets are by convention. And even if they weren't, they could be removed on the basis of being on public property anyway.
tomjen3 2y ago
This is a weird attitude. The Internet we used to know meant that you could do things with it. Certainly you should not reverse engineer so as to get access to others accounts in the first place, but that should be impossible anyway. I am aware that lots of companies have ideas(TM) about how you should be able to use their products(TM) and may even add these to their Terms of Service, a document that has somehow become the last refuge for the bureaucratic organisation desperate to maintain control when forced to connect things to great unbureaucratic internet. To that, I say: too bad. I never signed up for the new version of the internet and I do not consider TOS to be anything but noise. I used Pidgin back in the day and would again if it worked. This absurd idea that website owners should have any say about what runs on your computer/device is nonsense.
- jimz 2y ago
  Well, during the first days of the Pandemic I discovered that theathletic was using the same hardcoded API key for EVERY SINGLE ACCOUNT on their app. Granted, sports news when there were no sports and the near absence of interactive elements made it pretty meaningless, except you could impersonate any user to leave comments. So, bizarrely, sometimes you can reverse engineer right into other people's accounts, I just am not quite sure how the devs (I think, looking at the comments in the code, that they were Czech) managed to get the gig, considering how much the site was able to gather talent and create great content in spite of the paywall, and was sold to the NY Times for quite a bit of cash ($550 million). A $550 million app should not be using a hardcoded key in production. The Times is really not a great tech company in any sense. If I were a bit less lazy/busy I'd get more into their audio app, but frankly their reporting has gone downhill. I guess they're running the referral mill strategy now with all the ads they put into the app where there were none. Maybe they can hire some better programmers, or better reporters, for that matter.
- leptons 2y ago
  >This absurd idea that website owners should have any say about what runs on your computer/device is nonsense. No, they don't get a say about what software you run on your computer. But if your computer is accessing private APIs that I pay for, then I get a say in how you get to use it. It's also up to me to secure the APIs and prevent abuse. If I don't do that then you're essentially free to do what you like with the API until such time that I do lock it down. I'm also free to block your IP address and delete your account if you break the rules of use of the API that I am paying for. Don't like it? Too bad. You can pay for infrastructure to run your own damn APIs. For public APIs, the same rules about public usage of any physical space should apply. If you can see it "from public" aka logged-out, then you can take photos or record it (aka access the API). If it's a restricted area, then the public isn't allowed there and it's up to the entity trying to protect it to secure it. I make my living for the last 7 years reverse-engineering non-public APIs from a service my company pays for. The service gets to set a rate-limit, and they enforce it. They know what we're doing and we are in contact often with their managers and engineers. They let us know if we're straining their systems and we respond by limiting use of some of their more expensive APIs. We've almost DDOS their system before, and this is a system millions of people subscribe to, that serves billions of pages per day. It's in everyone's best interest to get along and not abuse the APIs, and not cut us off from using them in a different way than they intended. I would love it if this service took developers seriously and actually had a real developer program, but they do not, and they likely never will. It's more geared to consumers. But we depend on them in a very big way, so my job is reverse-engineering and scale up something that was never meant to be scaled. It's interesting work, but it also requires having an adult attitude and playing nicely with others. A little mutual respect can go a long way.
hipadev23 2y ago
This is such a cute take.
barnabee 2y ago
What’s wanted (or not) by website operators is irrelevant. Adversarial interoperability is a cornerstone of the value of the internet and something we should fight hard to keep.
jerjerjer 2y ago
Serving ads is mostly abusive and poor internet citizenship but here we are.
sethaurus 2y ago
It's always fascinating to discover a comment on Hacker News arguing against using network services in any way without the written permission of their owners and operators. We'd have a very different internet today if people had been following this advice for the last 30 years. A much worse internet, to be clear.

by Eikon 2y ago

gsky 2y ago

Unspoken rule is that maintaining 3 seconds delay between requests

lolinder 2y ago

I think it can be done responsibly. If you're not imposing any more traffic on them than you would be visiting their site manually, then using their APIs to get at structured data is actually win-win compared to the alternative (load the whole thing and scrape). Where I'll agree with you is cases where people do this and impose way more traffic than is typical and often more than is necessary (i.e. no caching). But that's not really specific to reverse engineering apis, that's just about being a good internet citizen in general. I'm of the opinion that a user agent is a user agent, and website owners shouldn't pick and choose what user agents they support. Target the behaviors that affect your infrastructure, not the means of access and algorithms used to process what you send.

Eikon 2y ago
> If you're not imposing any more traffic on them than you would be visiting their site manually, then using their APIs to get at structured data is actually win-win compared to the alternative (load the whole thing and scrape). I agree but that's not usually how it goes. From what I've seen, it's mostly very poorly written scripts with not rate limiting and no backoff strategy that will be hitting your api servers.
- lolinder 2y ago
  Would you rather have those poorly-written scripts hitting your APIs or have a poorly-written puppeteer script loading every asset you have—hitting the APIs along with everything else? Casting shade on API reverse engineering when what you actually have is a failure to rate limit is throwing the baby out with the bathwater. Abusive users will abuse until you build in a technological method to stop them, and user-agent sniffing provably doesn't work to stop bad actors. The concept of a flexible, customizable User Agent that operates on my behalf is a key idea that's foundational to the web, and I'm not willing to cede that cultural ground in the vague hope that we can make the bad guys feel bad and start just using Chrome like civilized people.

MathMonkeyMan 2y ago

Nah. Bots looking for exploits is rude, spamming an endpoint with more traffic than normal is rude.. but a human trying to figure out the API that you exposed to the internet? That's just fair play. Also, better to ask for forgiveness than to ask for permission. The author is adding value to the world while hurting nobody, and the answer would likely be an automatic "no" anyway.

boolemancer 2y ago

In my personal view, this seems a little overbearing. If you expose an API, and you want to tell a user that they are "unauthorized" to use it, it should return a 401 status code so that the caller knows they're unauthorized. If you can't do that because their traffic looks like normal usage of the API by your web app, then I question why their usage is problematic for you. At the end of the day, you don't get to control what 'browser' the user uses to interact with your service. Sure, it might be Chrome, but it just as easily might be Firefox, or Lynx, or something the user built from scratch, or someone manually typing out HTTP requests in netcat, or, in this case, someone building a custom client for your specific service. If you host a web server, it's on you to remember that and design accordingly, not on the user to limit how they use your service.

AlienRobot 2y ago
That's like saying if someone accepts cash that means you should be allowed to pay a $100 bill with a thousand dimes. Just because you're right doesn't mean you aren't wrong.
- lolinder 2y ago
  The $100 tab paid in dimes causes severe inconvenience to the person trying to count them and to the person who has to take them to the bank to cash them in and wait for them to be counted again. Their very reasonable question was: if you can't distinguish the reverse engineered traffic from the traffic through your own app in order to block it, then what harm is the traffic doing? Presumably it's flying under your rate limits, and the traffic has a valid session token from a real customer. If you're unable to single it out and return a 4xx, why does it matter where it's coming from? I can think of a few reasons it might, but I'm not particularly sympathetic to them. They generally boil down to "I won't be able to use my app to manipulate the user into taking actions they'd otherwise not take." I'd be interested to hear if there are better reasons.
  - AlienRobot 2y ago
    "if you can't distinguish the reverse engineered traffic from the traffic through your own app in order to block it, then what harm is the traffic doing?" If you really believe this you'll use a custom user agent instead of spoofing Chrome. :-) Some websites use HTTP referer to block traffic. Ask yourself if any reverse engineer would be stopped by what is obviously the website telling you not to access an endpoint. I'll add that end users don't have complete information about the website. They can't know how many resources a website has to deal to reverse engineering (webmasters can't just play cat and mouse with you just because you're wasting their money) nor do they know the cost of an endpoint. I mean, most tech inclined use ad blockers when it's obvious 90% of the websites pay the cost of their endpoints by showing ads, so I doubt they would respect anything more subtle than that.

gaeb69 2y ago

I agree. When scraping my school's portal which uses Canvas, my school actually allows it. Sometimes you can get the green light just by reading API docs/School's privacy policy as they're usually obliged to have one (ofc this primarily applies to school APIs like in OP's article)

mindslight 2y ago

A company can always document their API if they want power users to be informed about the company's preferred ways of doing things. They generally don't because they want to create a bunch of market friction so you're forced into using their proprietary client apps to interact with their services. I for one applaud efforts like the original post, and think it's an instance of outstanding Internet citizenship, where citizenship means working to preserve Internet societal norms like decentralization.

rmbyrro 2y ago

This would be akin to asking a shop owner if I'm allowed to pick a panflet from an endless stack of panflets placed on the sidewalk. If they don't want the public to pick panflets, don't put them where the public can reach'em!

mtnGoat 2y ago
Same could be said about things in your front yard? I’m not sure taking liberties with things you don’t own, is always the best policy, nor is putting the entire responsibility on the owner. I don’t think this is something you can boil down to a simple black and white.
- barnabee 2y ago
  Accessing a publicly available web service is not “taking liberties with things you don’t own”. If you put a server on the public internet and I send it a message (assuming I’m not using ill-gotten credentials, etc.), anything it responds with is your problem, not mine.
- Chris2048 2y ago
  > Same could be said about things in your front yard? No. It could be said, but wouldn't be true - objects in a domestic front yard are nothing like pamphlets placed on the sidewalk.
  - mtnGoat 2y ago
    Things on sidewalks are not free to take because you can touch them. Just like things in your front yard are not. Just because you can get to it, doesn’t mean it’s yours to take and act upon at will.
    - Chris2048 2y ago
      Pamphlets are by convention. And even if they weren't, they could be removed on the basis of being on public property anyway.

tomjen3 2y ago

This is a weird attitude. The Internet we used to know meant that you could do things with it. Certainly you should not reverse engineer so as to get access to others accounts in the first place, but that should be impossible anyway. I am aware that lots of companies have ideas(TM) about how you should be able to use their products(TM) and may even add these to their Terms of Service, a document that has somehow become the last refuge for the bureaucratic organisation desperate to maintain control when forced to connect things to great unbureaucratic internet. To that, I say: too bad. I never signed up for the new version of the internet and I do not consider TOS to be anything but noise. I used Pidgin back in the day and would again if it worked. This absurd idea that website owners should have any say about what runs on your computer/device is nonsense.

jimz 2y ago
Well, during the first days of the Pandemic I discovered that theathletic was using the same hardcoded API key for EVERY SINGLE ACCOUNT on their app. Granted, sports news when there were no sports and the near absence of interactive elements made it pretty meaningless, except you could impersonate any user to leave comments. So, bizarrely, sometimes you can reverse engineer right into other people's accounts, I just am not quite sure how the devs (I think, looking at the comments in the code, that they were Czech) managed to get the gig, considering how much the site was able to gather talent and create great content in spite of the paywall, and was sold to the NY Times for quite a bit of cash ($550 million). A $550 million app should not be using a hardcoded key in production. The Times is really not a great tech company in any sense. If I were a bit less lazy/busy I'd get more into their audio app, but frankly their reporting has gone downhill. I guess they're running the referral mill strategy now with all the ads they put into the app where there were none. Maybe they can hire some better programmers, or better reporters, for that matter.
leptons 2y ago
>This absurd idea that website owners should have any say about what runs on your computer/device is nonsense. No, they don't get a say about what software you run on your computer. But if your computer is accessing private APIs that I pay for, then I get a say in how you get to use it. It's also up to me to secure the APIs and prevent abuse. If I don't do that then you're essentially free to do what you like with the API until such time that I do lock it down. I'm also free to block your IP address and delete your account if you break the rules of use of the API that I am paying for. Don't like it? Too bad. You can pay for infrastructure to run your own damn APIs. For public APIs, the same rules about public usage of any physical space should apply. If you can see it "from public" aka logged-out, then you can take photos or record it (aka access the API). If it's a restricted area, then the public isn't allowed there and it's up to the entity trying to protect it to secure it. I make my living for the last 7 years reverse-engineering non-public APIs from a service my company pays for. The service gets to set a rate-limit, and they enforce it. They know what we're doing and we are in contact often with their managers and engineers. They let us know if we're straining their systems and we respond by limiting use of some of their more expensive APIs. We've almost DDOS their system before, and this is a system millions of people subscribe to, that serves billions of pages per day. It's in everyone's best interest to get along and not abuse the APIs, and not cut us off from using them in a different way than they intended. I would love it if this service took developers seriously and actually had a real developer program, but they do not, and they likely never will. It's more geared to consumers. But we depend on them in a very big way, so my job is reverse-engineering and scale up something that was never meant to be scaled. It's interesting work, but it also requires having an adult attitude and playing nicely with others. A little mutual respect can go a long way.

hipadev23 2y ago

This is such a cute take.

barnabee 2y ago

What’s wanted (or not) by website operators is irrelevant. Adversarial interoperability is a cornerstone of the value of the internet and something we should fight hard to keep.

jerjerjer 2y ago

Serving ads is mostly abusive and poor internet citizenship but here we are.

sethaurus 2y ago

It's always fascinating to discover a comment on Hacker News arguing against using network services in any way without the written permission of their owners and operators. We'd have a very different internet today if people had been following this advice for the last 30 years. A much worse internet, to be clear.