10 ms·

All the data can be yours: reverse engineering APIs

595 points by noleary 2y ago

Eikon 2y ago
This approach is generally seen as unwanted by website owners (it's worth noting that automated API clients are distinct from regular user agents). As a “reverse engineer”, you have no idea how expensive or not an endpoint is to process a request. Instead, I'd recommend reaching out to the website owners directly to discuss your API needs - they're often interested in hearing about potential integrations and use cases. If you don't receive a response, proceeding with unauthorized API usage is mostly abusive and poor internet citizenship.
- gsky 2y ago
  Unspoken rule is that maintaining 3 seconds delay between requests
- lolinder 2y ago
  I think it can be done responsibly. If you're not imposing any more traffic on them than you would be visiting their site manually, then using their APIs to get at structured data is actually win-win compared to the alternative (load the whole thing and scrape). Where I'll agree with you is cases where people do this and impose way more traffic than is typical and often more than is necessary (i.e. no caching). But that's not really specific to reverse engineering apis, that's just about being a good internet citizen in general. I'm of the opinion that a user agent is a user agent, and website owners shouldn't pick and choose what user agents they support. Target the behaviors that affect your infrastructure, not the means of access and algorithms used to process what you send.
  - Eikon 2y ago
    > If you're not imposing any more traffic on them than you would be visiting their site manually, then using their APIs to get at structured data is actually win-win compared to the alternative (load the whole thing and scrape). I agree but that's not usually how it goes. From what I've seen, it's mostly very poorly written scripts with not rate limiting and no backoff strategy that will be hitting your api servers.
    - lolinder 2y ago
      Would you rather have those poorly-written scripts hitting your APIs or have a poorly-written puppeteer script loading every asset you have—hitting the APIs along with everything else? Casting shade on API reverse engineering when what you actually have is a failure to rate limit is throwing the baby out with the bathwater. Abusive users will abuse until you build in a technological method to stop them, and user-agent sniffing provably doesn't work to stop bad actors. The concept of a flexible, customizable User Agent that operates on my behalf is a key idea that's foundational to the web, and I'm not willing to cede that cultural ground in the vague hope that we can make the bad guys feel bad and start just using Chrome like civilized people.
- MathMonkeyMan 2y ago
  Nah. Bots looking for exploits is rude, spamming an endpoint with more traffic than normal is rude.. but a human trying to figure out the API that you exposed to the internet? That's just fair play. Also, better to ask for forgiveness than to ask for permission. The author is adding value to the world while hurting nobody, and the answer would likely be an automatic "no" anyway.
- boolemancer 2y ago
  In my personal view, this seems a little overbearing. If you expose an API, and you want to tell a user that they are "unauthorized" to use it, it should return a 401 status code so that the caller knows they're unauthorized. If you can't do that because their traffic looks like normal usage of the API by your web app, then I question why their usage is problematic for you. At the end of the day, you don't get to control what 'browser' the user uses to interact with your service. Sure, it might be Chrome, but it just as easily might be Firefox, or Lynx, or something the user built from scratch, or someone manually typing out HTTP requests in netcat, or, in this case, someone building a custom client for your specific service. If you host a web server, it's on you to remember that and design accordingly, not on the user to limit how they use your service.
  - AlienRobot 2y ago
    That's like saying if someone accepts cash that means you should be allowed to pay a $100 bill with a thousand dimes. Just because you're right doesn't mean you aren't wrong.
    - lolinder 2y ago
      The $100 tab paid in dimes causes severe inconvenience to the person trying to count them and to the person who has to take them to the bank to cash them in and wait for them to be counted again. Their very reasonable question was: if you can't distinguish the reverse engineered traffic from the traffic through your own app in order to block it, then what harm is the traffic doing? Presumably it's flying under your rate limits, and the traffic has a valid session token from a real customer. If you're unable to single it out and return a 4xx, why does it matter where it's coming from? I can think of a few reasons it might, but I'm not particularly sympathetic to them. They generally boil down to "I won't be able to use my app to manipulate the user into taking actions they'd otherwise not take." I'd be interested to hear if there are better reasons.
      - AlienRobot 2y ago
        "if you can't distinguish the reverse engineered traffic from the traffic through your own app in order to block it, then what harm is the traffic doing?" If you really believe this you'll use a custom user agent instead of spoofing Chrome. :-) Some websites use HTTP referer to block traffic. Ask yourself if any reverse engineer would be stopped by what is obviously the website telling you not to access an endpoint. I'll add that end users don't have complete information about the website. They can't know how many resources a website has to deal to reverse engineering (webmasters can't just play cat and mouse with you just because you're wasting their money) nor do they know the cost of an endpoint. I mean, most tech inclined use ad blockers when it's obvious 90% of the websites pay the cost of their endpoints by showing ads, so I doubt they would respect anything more subtle than that.
- gaeb69 2y ago
  I agree. When scraping my school's portal which uses Canvas, my school actually allows it. Sometimes you can get the green light just by reading API docs/School's privacy policy as they're usually obliged to have one (ofc this primarily applies to school APIs like in OP's article)
- mindslight 2y ago
  A company can always document their API if they want power users to be informed about the company's preferred ways of doing things. They generally don't because they want to create a bunch of market friction so you're forced into using their proprietary client apps to interact with their services. I for one applaud efforts like the original post, and think it's an instance of outstanding Internet citizenship, where citizenship means working to preserve Internet societal norms like decentralization.
- rmbyrro 2y ago
  This would be akin to asking a shop owner if I'm allowed to pick a panflet from an endless stack of panflets placed on the sidewalk. If they don't want the public to pick panflets, don't put them where the public can reach'em!
  - mtnGoat 2y ago
    Same could be said about things in your front yard? I’m not sure taking liberties with things you don’t own, is always the best policy, nor is putting the entire responsibility on the owner. I don’t think this is something you can boil down to a simple black and white.
    - barnabee 2y ago
      Accessing a publicly available web service is not “taking liberties with things you don’t own”. If you put a server on the public internet and I send it a message (assuming I’m not using ill-gotten credentials, etc.), anything it responds with is your problem, not mine.
    - Chris2048 2y ago
      > Same could be said about things in your front yard? No. It could be said, but wouldn't be true - objects in a domestic front yard are nothing like pamphlets placed on the sidewalk.
      - mtnGoat 2y ago
        Things on sidewalks are not free to take because you can touch them. Just like things in your front yard are not. Just because you can get to it, doesn’t mean it’s yours to take and act upon at will.
        Chris2048 2y ago
        Pamphlets are by convention. And even if they weren't, they could be removed on the basis of being on public property anyway.
- tomjen3 2y ago
  This is a weird attitude. The Internet we used to know meant that you could do things with it. Certainly you should not reverse engineer so as to get access to others accounts in the first place, but that should be impossible anyway. I am aware that lots of companies have ideas(TM) about how you should be able to use their products(TM) and may even add these to their Terms of Service, a document that has somehow become the last refuge for the bureaucratic organisation desperate to maintain control when forced to connect things to great unbureaucratic internet. To that, I say: too bad. I never signed up for the new version of the internet and I do not consider TOS to be anything but noise. I used Pidgin back in the day and would again if it worked. This absurd idea that website owners should have any say about what runs on your computer/device is nonsense.
  - jimz 2y ago
    Well, during the first days of the Pandemic I discovered that theathletic was using the same hardcoded API key for EVERY SINGLE ACCOUNT on their app. Granted, sports news when there were no sports and the near absence of interactive elements made it pretty meaningless, except you could impersonate any user to leave comments. So, bizarrely, sometimes you can reverse engineer right into other people's accounts, I just am not quite sure how the devs (I think, looking at the comments in the code, that they were Czech) managed to get the gig, considering how much the site was able to gather talent and create great content in spite of the paywall, and was sold to the NY Times for quite a bit of cash ($550 million). A $550 million app should not be using a hardcoded key in production. The Times is really not a great tech company in any sense. If I were a bit less lazy/busy I'd get more into their audio app, but frankly their reporting has gone downhill. I guess they're running the referral mill strategy now with all the ads they put into the app where there were none. Maybe they can hire some better programmers, or better reporters, for that matter.
  - leptons 2y ago
    >This absurd idea that website owners should have any say about what runs on your computer/device is nonsense. No, they don't get a say about what software you run on your computer. But if your computer is accessing private APIs that I pay for, then I get a say in how you get to use it. It's also up to me to secure the APIs and prevent abuse. If I don't do that then you're essentially free to do what you like with the API until such time that I do lock it down. I'm also free to block your IP address and delete your account if you break the rules of use of the API that I am paying for. Don't like it? Too bad. You can pay for infrastructure to run your own damn APIs. For public APIs, the same rules about public usage of any physical space should apply. If you can see it "from public" aka logged-out, then you can take photos or record it (aka access the API). If it's a restricted area, then the public isn't allowed there and it's up to the entity trying to protect it to secure it. I make my living for the last 7 years reverse-engineering non-public APIs from a service my company pays for. The service gets to set a rate-limit, and they enforce it. They know what we're doing and we are in contact often with their managers and engineers. They let us know if we're straining their systems and we respond by limiting use of some of their more expensive APIs. We've almost DDOS their system before, and this is a system millions of people subscribe to, that serves billions of pages per day. It's in everyone's best interest to get along and not abuse the APIs, and not cut us off from using them in a different way than they intended. I would love it if this service took developers seriously and actually had a real developer program, but they do not, and they likely never will. It's more geared to consumers. But we depend on them in a very big way, so my job is reverse-engineering and scale up something that was never meant to be scaled. It's interesting work, but it also requires having an adult attitude and playing nicely with others. A little mutual respect can go a long way.
- hipadev23 2y ago
  This is such a cute take.
- barnabee 2y ago
  What’s wanted (or not) by website operators is irrelevant. Adversarial interoperability is a cornerstone of the value of the internet and something we should fight hard to keep.
- jerjerjer 2y ago
  Serving ads is mostly abusive and poor internet citizenship but here we are.
- sethaurus 2y ago
  It's always fascinating to discover a comment on Hacker News arguing against using network services in any way without the written permission of their owners and operators. We'd have a very different internet today if people had been following this advice for the last 30 years. A much worse internet, to be clear.
sccxy 2y ago
> Mobile apps have no choice but to use HTTP APIs. You can easily download a lot of iOS apps through the Mac App Store, then run strings on their bundles to look for endpoints. Are there any good tutorials for that? 'strings' is not the greatest name for searching good information.
- mooreds 2y ago
  I googled a bit and found this, which points to some other tools. https://www.corellium.com/blog/ios-mobile-reverse-engineering https://www.corellium.com/blog/ios-mobile-reverse-engineerin... (No affiliation.)
- out-of-ideas 2y ago
  gnu strings: (first google result from "gnu strings" search) https://sourceware.org/binutils/docs/binutils/strings.html https://sourceware.org/binutils/docs/binutils/strings.html
- syntaxing 2y ago
  I was about to post the exact thing, I dont see any thing API related to iOS apps on Google. Edit: After some Google-ing and trying it on my macbook, there is a native CLI tool called "strings". Supposedly it does the following: strings is primarily used to find and display printable character sequences in binary files, object files, and executables. Which means the author is probably looking at the app to see the hardcorded characters in the app binary(?) and searching for the API end points.
  - emilamlom 2y ago
    Just for context, strings is super commonly used when reverse-engineering anything. It's a great first-step because it's easy, fast, and get's some decent clues to help you get your bearings in an unknown binary file.
    - anthk 2y ago
      Among 'nm' and 'binwalk e $FILE'.
treyd 2y ago
I wonder how difficult it would be to combine many of these techniques into some automated script that dumps a manifest of the different types of undocumented APIs there are. LLMs have also been shown to be pretty good at answering semantic questions about large blobs of minified code, perhaps there could be some success there?
- cobertos 2y ago
  There's too many fiddly bits of which endpoints return what data in which shape that requires a custom solution each time. An LLM would probably have a very easy shot at this problem though. It's much more straightforward if you can find a GraphQL, Swagger, or OpenAPI spec to automate conversion I'd imagine.
- justo-rivera 2y ago
  Launch HN: Integuru – Reverse-engineer internal APIs using LLMs https://news.ycombinator.com/item?id=41983409 https://news.ycombinator.com/item?id=41983409
matthewfcarlson 2y ago
I went to the 75grand app listed in the article and saw a listing for Cafe Mac and did a double take. Apple's employee cafe is caffe Macs, so I was quite confused for a second
gaeb69 2y ago
Sick app btw. Funny this comes up because I'm working on the exact same thing for my school. Note that if your school uses Canvas; Canvas' API is well documented and has GraphQL endpoints.
- 0x3444ac53 2y ago
  Canvas, and several other LMS's all use scorm cloud on the backend. Discovered this while researching for something work related. Getting into the internals of an LMS is tricky, but surprisingly very interesting and very fun.
deleted 2y ago
[deleted]
Lucasoato 2y ago
> 75grand’s success was even met with jealousy from the college That's a common story; in my university (in Padova, UniPD) happened something even worse. They tried hard to shut down an unofficial app (Uniweb) that was installed by most of the students in favor of the "official" one, that was completely unusable (and probably was born out of a rigged contract). At the end the best one won and became official, but that was after a lot of struggle.
- mtnGoat 2y ago
  I’m just not sure jealousy is the correct word for this though. Most systems don’t like these kinds of things for a number us reasons.
  - TeMPOraL 2y ago
    In my (perhaps limited) experience, most of those reasons are just different ways of spelling "our official solution is shit, because we don't care and/or make money in some underhanded way".
  - wingerlang 2y ago
    I think the word usage is tongue in cheek
  - klabetron 2y ago
    Agreed that I’m not sure jealousy is the right interpretation of imitation being the sincerest form of flattery. Having said that, speaking as a Macalester alumnus, I wouldn’t put it past them to be a bit petty behind the scenes :)
- mock-possum 2y ago
  > They tried hard to shut down an unofficial app (Uniweb) that was installed by most of the students in favor of the "official" one, that was completely unusable Sounds like what happened with the Apollo app and Reddit
- matheusmoreira 2y ago
  My high school used a system so bad I stopped caring about my grades, I just didn't want to log into that crap more than once a year. I couldn't believe my eyes when I got into university and saw they were using the exact same shitty system. Reverse engineered just enough of that .net monstrosity to get the data I cared about. Replacing the school's portal with my own script made me so happy. They kept breaking my scraper constantly though, got to the point it was too much effort to maintain it. Worst part is I don't even think they were doing it on purpose. They were just bad in general and couldn't keep their site together.
z3c0 2y ago
> The error messages helpfully suggested fields I hadn’t known about by “correcting” my typos. Glad to see this being called out. Sure, I get why it's convenient. Misspelling a field by one character is a daily occurrence ("activty" and "heirarchy" are my regulars). The catch is that spellchecking queries and returning valid fields in the error effectively reduces entropy by both character space and message length, varying by the type of distance used in the spellcheck.
colesantiago 2y ago
Most of these techniques are extremely old and very outdated. Teams that I've seen working on apps now implement much stronger checks on APIs especially Android apps such as SafetyCheck and DeviceCheck and other methods, which makes using strings rather basic to see them. And most apps are now encrypted so you just see junk in the logs.
- rozap 2y ago
  And on the web side, fingerprinting is rampant and there are JS challenges in cloudflare, imperva, etc which make it trickier. Frustrating to run a whole browser with a virtual screen, load the whole page which is ofc like 15mb of JS and other trash, just to do a very simple thing. Granted, smaller fish like the ones OP is referring to generally don't have aggressive anti automation measures in place, so it can be easy...but generally these techniques don't work if the operator has put the proper measures in place.
  - skilbjo 2y ago
    take a look at https://xhr.dev/ https://xhr.dev/, a product I built to avoid bot detection from things like cloudflare, imperva, aws waf, and others
    - Zopieux 2y ago
      >self host (Docker): $60k/yr lmao ok
    - enahs-sf 2y ago
      What does the $500 a month get me? Infinite resources to scrape all of LinkedIn?
  - imp0cat 2y ago
    Frustrating? Yeah! but it works SO great! I especially like Playwright in this context, it can do pretty much anything and is a joy to use.
- filleokus 2y ago
danielvaughn 2y ago
At a former job, we reverse engineered the trading APIs of most American retail stock brokerages (Fidelity, E-Trade, Robinhood, TD Ameritrade, etc). We did it by rooting an iPhone and using Charles Proxy to grab the unencrypted traffic. I learned a lot from that experience, and it's also just plain fun to do. We did get some strongly worded letters from Robinhood though, lol. They tried blocking our servers but we just set up this automated system in Digital Ocean that would spin up a new droplet each time we detected a blockage, and they were never able to stop us after that. Fun times.
- dangsux 2y ago
  [dead]
- packetlost 2y ago
  I did almost this exact same thing back in 2015~ ish when I was in high school over Christmas break. I reverse engineered the anime streaming site Crunchyroll's API via their Android and PS3 app using some HTTP proxy application and trial + error. I ended having a proper HLS-based streaming player and Android TV app back when their Android app was still Flash based. It was lots of fun!
  - danielvaughn 2y ago
    2015/2016 was exactly when I was doing the above job. We could've hired you as an intern!
    - packetlost 2y ago
      Maybe! I considered pursuing a job there at the time, but I opted to get a degree instead. Being located in the Midwest would've made it rather challenging anyways.
    - packetlost 2y ago
      Also this would have been winter of 2014 and into early 2015, so a bit before :) Sadly the CR forums are gone, so my rather popular thread that had feedback and support is long gone.
stoplight 2y ago
This is how I made a better version of the nhl.com site [1] that has a better UI (you can see scores/schedules much more easily), is mobile first, has no ads, and responsiveness built in. I did the same for the AHL [2], and the PWHL [3]. [1] https://nhl-remix.vercel.app/ https://nhl-remix.vercel.app/ [2] https://ahl-remix.vercel.app/ https://ahl-remix.vercel.app/ [3] https://pwhl-remix.vercel.app/ https://pwhl-remix.vercel.app/
- kmoser 2y ago
  For a minute I thought PWHL was short for "pwn-NHL".
- 12345hn6789 2y ago
  Poked around a bit. It's responsive and looks great on mobile. Kudos
- bluetidepro 2y ago
  Make one for the NFL, please! I can’t stand how hard it is to get scores on their website. Haha
  - TechDebtDevin 2y ago
    There's like a gazzillion alternative scoring sites. Googling the team name also works while they're playing, Google usually has a good feed.
  - Hackbraten 2y ago
    Funny you would mention that. A few weeks ago, I wrote a Python-based client library [0] for the website kicker.de [1]. It supports the NFL, too: import asyncio from kickerde_api_client import Api from kickerde_api_client.model import LeagueId api = Api() query = {'league': LeagueId.NFL, 'season': '2024/25'} season = asyncio.run(api.league_season(**query)) print(season['longName']) # 'National Football League' print(season['country']['longName']) # 'USA' print([ team['shortName'] for team in season['teams'].values() if team['shortName'].startswith('B') ]) # ['Buffalo', 'Baltimore'] day = season['gamedays'][18] print(str(day['dateFrom'].date())) # '2025-01-05' [0]: https://kickerde-api-client.readthedocs.io/en/stable/autoapi/kickerde_api_client/index.html https://kickerde-api-client.readthedocs.io/en/stable/autoapi... [1]: https://www.kicker.de/ https://www.kicker.de/
rcpt 2y ago
Anyone have this for Twitter? I want to remove most of my tweets but the official API costs $200
- s09dfhks 2y ago
  Their delete post endpoints probably require auth. What’s to stop you from deleting someone else’s posts
  - samtheprogram 2y ago
    …you can see what auth/headers are passed to the endpoint from said developer tools. I agree with sibling commenters that automating the browser with DOM APIs is an easier route to go though.
- markerz 2y ago
  Maybe reverse it from the web app. I deleted a tweet and saw this request: HTTP POST https://x.com/i/api/graphql/VstuveVgh5q5jk7lmnVopqr/DeleteTweet { "variables": { "tweet_id":"12344567899123", "dark_request":false }, "queryId":"VstuveVgh5q5jk7lmnVopqr" } You can execute these from javascript in the browser if the auth part is too complicated. ### Update, this is the pure javascript console way, if you don't want to write your own client doing HTTP posts I played with the console more and got these parts: // Find all tweets on screen (this gives you the tweet IDs too) document.querySelectorAll('a > time') // Click the "more" button on the first tweet document.querySelectorAll('a > time')[0].parentElement.parentElement.parentElement.parentElement.parentElement.parentElement.parentElement.parentElement.parentElement.parentElement.parentElement.querySelector('button').click() // Click delete on the tweet document.querySelectorAll('[data-testid="Dropdown"]')[0].children[0].click() // Confirm delete document.querySelectorAll('[data-testid="confirmationSheetConfirm"]')[0].click()
  - 0x3444ac53 2y ago
    Shortly before they destroyed the API I made a little script to delete all my tweets. Was really really handy
aswprojectry1 2y ago
[dead]
rubslopes 2y ago
I do exactly this, but for the company that I work for. I'm on the dashboards and integrations team, and I don't have direct access to the codebase of the main product. As the internal APIs have no documentation at all, I'm always "hacking" our own system using the browser inspector to find out how our endpoints work.
- strunz 2y ago
  There are so many of us doing this, haha. If you work at a big enough company or one with poorly documented APIs, it's just faster to reverse engineer the existing apps/UIs.
  - TeMPOraL 2y ago
    Just make sure you don't end up accidentally making one of those APIs semi-official. I've seen a case where one team developed a temporary hack to automate some process in their product, then allowed another team making a sibling product to use it to test a possible feature; soon after, the possible feature became an actual one, everyone seemingly forgot the API was a throwaway test. Over the years, as both products evolved, that feature got pretty flaky, and in one of many cross-team attempts at fixing it, someone from the original time finally pointed out that the whole thing is still relying on a temporary hack in the original product that was never intended to be productized...
- leptons 2y ago
  Same here, but I do have access to the codebase. But there's no documentation about all the different APIs. Reading through the code also doesn't always explain what the API does, or what it's for. I work on a separate system that benefits from using these APIs from the main product, there is almost no overlap or communication between engineering departments. It's far from ideal.
arnau_cruz 2y ago
[flagged]
- arnau_cruz 2y ago
  [flagged]
dchuk 2y ago
I’ve reverse engineered a few industry conference apps to more easily get the list of attendees (some conferences will literally only give you a pdf of scanned paper lists of contact info for attendees, which is insane, especially if you are paying to have a booth there). I’ve either decompiled the Android app, or ran mitmproxy on the device, or both, to figure it out. Does anyone have a recommendation for having a pre-rooted Android simulator with the tools you need installed and ready to go to make this a quicker process? I’d love to just drag and drop an apk into a simulator and start inspecting vs having to use a real device and all that jazz.
- SeriousStorm 2y ago
  I found this recently, but I haven't tried it yet. https://github.com/budtmo/docker-android https://github.com/budtmo/docker-android
  - uber1geek 2y ago
    that is a cool find :)
ideashower 2y ago
Someone did this to the Ring mobile app to get a dataset of all (public) Ring cameras across the US https://gizmodo.com/ring-s-hidden-data-let-us-map-amazons-sprawling-home-su-1840312279 https://gizmodo.com/ring-s-hidden-data-let-us-map-amazons-sp...
speakspokespok 2y ago
Can someone recommend a long form guide that covers best practices for handling authentication and the like?
- gessha 2y ago
  Is this what you’re looking for: https://thecopenhagenbook.com/ https://thecopenhagenbook.com/ It was shared last or week or so on here.
tomblomfield 2y ago
W
Gamemaster1379 2y ago
I actually enjoy reversing APIs. A game I really liked to play announced EOS last year. I ended up capturing a bunch of mitm traffic and built my own API instead and now head up a private server for it. I don't dare monetize that project, but I wish I could use my skills to make some extra money at reversing APIs. Wouldn't know where to begin though.
- mdaniel 2y ago
  I have the same wish, and wish that for you, but tangentially I wanted to offer that if you were to publish the game server you built, to the very best of my knowledge alternative implementations are protected by US law (assuming that applies to you, of course) and then everyone in a similar boat could sponsor your project, straight up donate to you, or at very least you'll have made the world a better place for the players in a similar situation as you were
  - Gamemaster1379 2y ago
    The game is Gundam Evolution, so Bandai Namco is the owner. They are known to be very litigious and protective of Gundam IP. The game was a multiplayer FPS and involved cosmetic micro transactions. Due to mismanagement they shut down in one year. It launched in late 2022 and EOSed late 2024. I'm in the US, yes. The company is Japanese though and they may not care as much about US law when choosing to pursue. Even if they ultimately don't have a case, there's still enough gray area to torment-by-lawyer. For that reason I chose not to take any cash for the project. I also don't distribute any game files. Trying to minimize risk as much as possible. But yeah! It was a good community building project. We're about to hit 10,000 members I hope it leads to some connections that are profit rearing.
    - matheusmoreira 2y ago
      You're a hero. I couldn't care less what laws say, companies that go after people like you deserve to be boycotted mercilessly. I haven't bought a single EA game since they C&D'd the people reverse engineering Battlefield 2 and 2142 network protocols. They just wanted to keep some beloved games alive. I will never forgive them for it. Whatever you're doing, I hope you get away with it.
nicbou 2y ago
bund.dev is a group of volunteers doing exactly that for APIs run by the German government. I met Lilith at a meetup recently. Her talk was super interesting. https://bund.dev/ https://bund.dev/ https://links.lilithwittmann.de/ https://links.lilithwittmann.de/
purple-leafy 2y ago
I’ve built the leading satay transparency tool in my country and one other, reverse engineering an API. Took me a few hours, now 6000 people use the tool. I think my tool is responsible for their changes to rate limiting lol
geniium 2y ago
Very interesting read! Gives some idea on how to integrate data that are not easy to access. Thanks
faizshah 2y ago
Anyone come up with good techniques for reverse engineering websockets? Its especially annoying since many use binary message formats and there isnt a great way to document an arbitrary binary message protocol. A couple techniques im trying out: - websocat and wsrepl for reverse engineering interactively: https://github.com/doyensec/wsrepl https://github.com/doyensec/wsrepl - kaitai struct for documenting the binary message formats: https://kaitai.io/ https://kaitai.io/
- pimterry 2y ago
  I'm very interested in this, I do a lot of protocol debugging. Kaitai looks very neat - is that the most popular format for this kind of thing, or are there other popular options I should look at too?
  - pdimitar 2y ago
    Just lately somebody posted about the Imhex tool's DSL: https://xy2i.blogspot.com/2024/11/using-imhexs-pattern-language-to-parse.html https://xy2i.blogspot.com/2024/11/using-imhexs-pattern-langu...
    - S0y 2y ago
      I recently wanted to reverse engineer some Websocket packets for a game I was playing. I used BurpSuite as a proxy to bypass the SSL encryption. It also has a pretty handy tool that will monitor all websocket traffic. After that I used ImHex, pretty much exactly like in that blog to reverse engineer the websocket packets. The DSL is a little finicky but once you wrap your head around it, it`s very nice and powerful.
      - pdimitar 2y ago
        Thanks for sharing your experience, it is valuable as I am considering using Imhex and its DSL lately.
- rtrgrd 2y ago
lyime 2y ago
If you want to do this and get paid for it come talk to us at Terminal49
- pdimitar 2y ago
  Yes, I would like to do that, RE and network APIs were always a passion. What's the best channel to reach you? I'll email you if you are not seeing this comment.
  - Jeremy1026 2y ago
    Reverse engineer their website CMS and find the admin email.
    - pdimitar 2y ago
      I got downvoted for not taking the hint, I suppose.
miki123211 2y ago
I think good APIs are one of te most important and least obvious advantages of SPAs.
saagarjha 2y ago
I did the same thing long (well, not that long) ago for my school’s system too: https://github.com/saagarjha/break https://github.com/saagarjha/break. In retrospect this was an excellent use of time because when I made this I barely knew what JSON was and ended up reimplementing some protocols from scratch (HTTP basic auth, WebDAV) that I learned much later were actually standardized and I had actually known them all along ;) Alas, the service got bought out by some vulture/bodyshop PE firm and the app ended up outliving the servers it was hitting. But until then I got a lot of great email from high school kids telling me how much I improved their experience or asking if they could build their own thing on the APIs they found in my app after trawling GitHub.
klabetron 2y ago
Macalester ‘07 here. Back in my day their LDAP directory was public (at least on the campus network) which I used to scrape student & professor lists.
6510 2y ago
I one time talk with a guy running a scraping company. A client of his wanted all products from all of his suppliers in a single interface. One really slow website was constantly changing things specifically to make scraping impossible. He had to fix it every few hours then they changed everything again. I gave him the perfect solution: Why don't you just call them and ask how much money they want for access to their product catalog? Give them tiny bits of information if they don't approve immediately. Tell them it is for one of their customers and that their site is to slow. They are the only supplier not in the interface which is bad for them and bad for you. If they still refuse offer them access to their competitors data at a modest fee. He couldn't stop laughing, he never considered it. When the sun set the next morning he put in the call. They gave him access immediately, they were so happy to finally get rid of his crawler messing up their analytics data. Apparently the people on the other end of the tube also didn't sleep. They had a good laugh about it.
xfeeefeee 2y ago
It's really fascinating watching the constant push and pull in projects like yt-dlp and places like YouTube and TikTok. So many interesting techniques to make things more difficult for reverse engineering or making unofficial requests. There is even a tiny JavaScript engine iirc that calculates a special value that YouTube uses to verify requests.
- anigbrowl 2y ago
  Companies should take the existence of things like this as a signal that something is wrong with their product. There's a lot of things wrong with YouTube these days. Their ad engine is just out of control, often sticking ads at both the beginning and the end of short videos, sticking ads in the middle of songs on music performance videos and so on. Amazingly, they've managed to make it worse than TV.
  - jfil 2y ago
    This logic only applies for companies in a competitive business environment. YouTube is a video monopolist - they know that it is an awful service for viewers, but there's no need to improve things because where else would viewers go? Unfortunately, they are also part of a conglomerate that has reached full dominance in several online markets that are not growing. And they need to show quarter-on-quarter growth to shareholders. So you can't capture more market share, and there aren't any more people discovering the Web, so you stick additional ads everywhere. These people know exactly what they are doing.
brutecat 2y ago
The problem w/ this is a lot of the time is as they keep ramping up their anti-abuse measures, your old code breaks and needs to be rewritten (especially if it's an API you intend to use for a long term project). It's just an endless cat & mouse game.
- pavel_lishin 2y ago
  Sure, but all software rots.
sodality2 2y ago
Did this with Redlib to allow access to Reddit, and have been building tooling to allow academic researchers to access data at a large scale at no cost :) https://github.com/redlib-org/redlib https://github.com/redlib-org/redlib
anothername12 2y ago
There’s probably a group of poor bastards on call at 5am, poring over alerts, and logs in Datadog etc wondering WTF is wrong with their applications. Give ‘em a shout out by saying “hi” in a request parameter or something while you’re reverse engineering.
thesurlydev 2y ago
As I read through these comments I'm reminded of recon techniques commonly used in bug bounties. Reverse proxies, decompiling APKs, parsing JS files to find endpoints, etc. I often went down rabbit holes in the recon phase and never found bugs but I had fun in the process so I considered it a win anyway. I also implemented a bot mitigation system for a large international company so I got to see techniques used from the other side. Mobile phone farms and traffic from China was the most difficult to mitigate.
- pdimitar 2y ago
  Why was China in particular difficult to mitigate?
hudsonpeter0147 2y ago
[dead]
raudette 2y ago
A technique not covered here is setting up a wifi hotspot to impersonate a device. I wanted to reverse engineer the protocol used by an inexpensive action camera. I ran the Android app through a decompiler (JADX), and the app supported many different types of cameras, so it wasn't clear which calls were related to the one I owned. So I setup a Raspberry Pi with the SSID as the camera, logged the first API call. Then I ran that API against the camera, and then wrote a small python script on the Pi to return the same API result. I did that call by call until I'd worked everything out. I wrote it up here: https://www.hotelexistence.ca/reverse-engineer-akaso-ek7000/ https://www.hotelexistence.ca/reverse-engineer-akaso-ek7000/
zzo38computer 2y ago
I had also sometimes done, but usually because the existing UI isn't very good (and sometimes it doesn't work at all).
nurettin 2y ago
Browser debugger is my #1 resource. There is almost always a way to get stuff by replaying network requests. No matter how much they obfuscate (I'm looking at you, quora) it is still possible to replicate the requests and use the site like an API. Some real estate businesses and shadier data providers use browser detection to get around automation, but it is usually a minor inconvenience since you can run browsers headless or on xvfb if they have to render.

595 points by noleary 2y ago

This approach is generally seen as unwanted by website owners (it's worth noting that automated API clients are distinct from regular user agents). As a “reverse engineer”, you have no idea how expensive or not an endpoint is to process a request. Instead, I'd recommend reaching out to the website owners directly to discuss your API needs - they're often interested in hearing about potential integrations and use cases. If you don't receive a response, proceeding with unauthorized API usage is mostly abusive and poor internet citizenship.

gsky 2y ago
Unspoken rule is that maintaining 3 seconds delay between requests
lolinder 2y ago
I think it can be done responsibly. If you're not imposing any more traffic on them than you would be visiting their site manually, then using their APIs to get at structured data is actually win-win compared to the alternative (load the whole thing and scrape). Where I'll agree with you is cases where people do this and impose way more traffic than is typical and often more than is necessary (i.e. no caching). But that's not really specific to reverse engineering apis, that's just about being a good internet citizen in general. I'm of the opinion that a user agent is a user agent, and website owners shouldn't pick and choose what user agents they support. Target the behaviors that affect your infrastructure, not the means of access and algorithms used to process what you send.
- Eikon 2y ago
  > If you're not imposing any more traffic on them than you would be visiting their site manually, then using their APIs to get at structured data is actually win-win compared to the alternative (load the whole thing and scrape). I agree but that's not usually how it goes. From what I've seen, it's mostly very poorly written scripts with not rate limiting and no backoff strategy that will be hitting your api servers.
  - lolinder 2y ago
    Would you rather have those poorly-written scripts hitting your APIs or have a poorly-written puppeteer script loading every asset you have—hitting the APIs along with everything else? Casting shade on API reverse engineering when what you actually have is a failure to rate limit is throwing the baby out with the bathwater. Abusive users will abuse until you build in a technological method to stop them, and user-agent sniffing provably doesn't work to stop bad actors. The concept of a flexible, customizable User Agent that operates on my behalf is a key idea that's foundational to the web, and I'm not willing to cede that cultural ground in the vague hope that we can make the bad guys feel bad and start just using Chrome like civilized people.
MathMonkeyMan 2y ago
Nah. Bots looking for exploits is rude, spamming an endpoint with more traffic than normal is rude.. but a human trying to figure out the API that you exposed to the internet? That's just fair play. Also, better to ask for forgiveness than to ask for permission. The author is adding value to the world while hurting nobody, and the answer would likely be an automatic "no" anyway.
boolemancer 2y ago
In my personal view, this seems a little overbearing. If you expose an API, and you want to tell a user that they are "unauthorized" to use it, it should return a 401 status code so that the caller knows they're unauthorized. If you can't do that because their traffic looks like normal usage of the API by your web app, then I question why their usage is problematic for you. At the end of the day, you don't get to control what 'browser' the user uses to interact with your service. Sure, it might be Chrome, but it just as easily might be Firefox, or Lynx, or something the user built from scratch, or someone manually typing out HTTP requests in netcat, or, in this case, someone building a custom client for your specific service. If you host a web server, it's on you to remember that and design accordingly, not on the user to limit how they use your service.
- AlienRobot 2y ago
  That's like saying if someone accepts cash that means you should be allowed to pay a $100 bill with a thousand dimes. Just because you're right doesn't mean you aren't wrong.
  - lolinder 2y ago
    The $100 tab paid in dimes causes severe inconvenience to the person trying to count them and to the person who has to take them to the bank to cash them in and wait for them to be counted again. Their very reasonable question was: if you can't distinguish the reverse engineered traffic from the traffic through your own app in order to block it, then what harm is the traffic doing? Presumably it's flying under your rate limits, and the traffic has a valid session token from a real customer. If you're unable to single it out and return a 4xx, why does it matter where it's coming from? I can think of a few reasons it might, but I'm not particularly sympathetic to them. They generally boil down to "I won't be able to use my app to manipulate the user into taking actions they'd otherwise not take." I'd be interested to hear if there are better reasons.
    - AlienRobot 2y ago
      "if you can't distinguish the reverse engineered traffic from the traffic through your own app in order to block it, then what harm is the traffic doing?" If you really believe this you'll use a custom user agent instead of spoofing Chrome. :-) Some websites use HTTP referer to block traffic. Ask yourself if any reverse engineer would be stopped by what is obviously the website telling you not to access an endpoint. I'll add that end users don't have complete information about the website. They can't know how many resources a website has to deal to reverse engineering (webmasters can't just play cat and mouse with you just because you're wasting their money) nor do they know the cost of an endpoint. I mean, most tech inclined use ad blockers when it's obvious 90% of the websites pay the cost of their endpoints by showing ads, so I doubt they would respect anything more subtle than that.
gaeb69 2y ago
I agree. When scraping my school's portal which uses Canvas, my school actually allows it. Sometimes you can get the green light just by reading API docs/School's privacy policy as they're usually obliged to have one (ofc this primarily applies to school APIs like in OP's article)
mindslight 2y ago
A company can always document their API if they want power users to be informed about the company's preferred ways of doing things. They generally don't because they want to create a bunch of market friction so you're forced into using their proprietary client apps to interact with their services. I for one applaud efforts like the original post, and think it's an instance of outstanding Internet citizenship, where citizenship means working to preserve Internet societal norms like decentralization.
rmbyrro 2y ago
This would be akin to asking a shop owner if I'm allowed to pick a panflet from an endless stack of panflets placed on the sidewalk. If they don't want the public to pick panflets, don't put them where the public can reach'em!
- mtnGoat 2y ago
  Same could be said about things in your front yard? I’m not sure taking liberties with things you don’t own, is always the best policy, nor is putting the entire responsibility on the owner. I don’t think this is something you can boil down to a simple black and white.
  - barnabee 2y ago
    Accessing a publicly available web service is not “taking liberties with things you don’t own”. If you put a server on the public internet and I send it a message (assuming I’m not using ill-gotten credentials, etc.), anything it responds with is your problem, not mine.
  - Chris2048 2y ago
    > Same could be said about things in your front yard? No. It could be said, but wouldn't be true - objects in a domestic front yard are nothing like pamphlets placed on the sidewalk.
    - mtnGoat 2y ago
      Things on sidewalks are not free to take because you can touch them. Just like things in your front yard are not. Just because you can get to it, doesn’t mean it’s yours to take and act upon at will.
      - Chris2048 2y ago
        Pamphlets are by convention. And even if they weren't, they could be removed on the basis of being on public property anyway.
tomjen3 2y ago
This is a weird attitude. The Internet we used to know meant that you could do things with it. Certainly you should not reverse engineer so as to get access to others accounts in the first place, but that should be impossible anyway. I am aware that lots of companies have ideas(TM) about how you should be able to use their products(TM) and may even add these to their Terms of Service, a document that has somehow become the last refuge for the bureaucratic organisation desperate to maintain control when forced to connect things to great unbureaucratic internet. To that, I say: too bad. I never signed up for the new version of the internet and I do not consider TOS to be anything but noise. I used Pidgin back in the day and would again if it worked. This absurd idea that website owners should have any say about what runs on your computer/device is nonsense.
- jimz 2y ago
  Well, during the first days of the Pandemic I discovered that theathletic was using the same hardcoded API key for EVERY SINGLE ACCOUNT on their app. Granted, sports news when there were no sports and the near absence of interactive elements made it pretty meaningless, except you could impersonate any user to leave comments. So, bizarrely, sometimes you can reverse engineer right into other people's accounts, I just am not quite sure how the devs (I think, looking at the comments in the code, that they were Czech) managed to get the gig, considering how much the site was able to gather talent and create great content in spite of the paywall, and was sold to the NY Times for quite a bit of cash ($550 million). A $550 million app should not be using a hardcoded key in production. The Times is really not a great tech company in any sense. If I were a bit less lazy/busy I'd get more into their audio app, but frankly their reporting has gone downhill. I guess they're running the referral mill strategy now with all the ads they put into the app where there were none. Maybe they can hire some better programmers, or better reporters, for that matter.
- leptons 2y ago
  >This absurd idea that website owners should have any say about what runs on your computer/device is nonsense. No, they don't get a say about what software you run on your computer. But if your computer is accessing private APIs that I pay for, then I get a say in how you get to use it. It's also up to me to secure the APIs and prevent abuse. If I don't do that then you're essentially free to do what you like with the API until such time that I do lock it down. I'm also free to block your IP address and delete your account if you break the rules of use of the API that I am paying for. Don't like it? Too bad. You can pay for infrastructure to run your own damn APIs. For public APIs, the same rules about public usage of any physical space should apply. If you can see it "from public" aka logged-out, then you can take photos or record it (aka access the API). If it's a restricted area, then the public isn't allowed there and it's up to the entity trying to protect it to secure it. I make my living for the last 7 years reverse-engineering non-public APIs from a service my company pays for. The service gets to set a rate-limit, and they enforce it. They know what we're doing and we are in contact often with their managers and engineers. They let us know if we're straining their systems and we respond by limiting use of some of their more expensive APIs. We've almost DDOS their system before, and this is a system millions of people subscribe to, that serves billions of pages per day. It's in everyone's best interest to get along and not abuse the APIs, and not cut us off from using them in a different way than they intended. I would love it if this service took developers seriously and actually had a real developer program, but they do not, and they likely never will. It's more geared to consumers. But we depend on them in a very big way, so my job is reverse-engineering and scale up something that was never meant to be scaled. It's interesting work, but it also requires having an adult attitude and playing nicely with others. A little mutual respect can go a long way.
hipadev23 2y ago
This is such a cute take.
barnabee 2y ago
What’s wanted (or not) by website operators is irrelevant. Adversarial interoperability is a cornerstone of the value of the internet and something we should fight hard to keep.
jerjerjer 2y ago
Serving ads is mostly abusive and poor internet citizenship but here we are.
sethaurus 2y ago
It's always fascinating to discover a comment on Hacker News arguing against using network services in any way without the written permission of their owners and operators. We'd have a very different internet today if people had been following this advice for the last 30 years. A much worse internet, to be clear.

sccxy 2y ago

> Mobile apps have no choice but to use HTTP APIs. You can easily download a lot of iOS apps through the Mac App Store, then run strings on their bundles to look for endpoints. Are there any good tutorials for that? 'strings' is not the greatest name for searching good information.

mooreds 2y ago
I googled a bit and found this, which points to some other tools. https://www.corellium.com/blog/ios-mobile-reverse-engineering https://www.corellium.com/blog/ios-mobile-reverse-engineerin... (No affiliation.)
out-of-ideas 2y ago
gnu strings: (first google result from "gnu strings" search) https://sourceware.org/binutils/docs/binutils/strings.html https://sourceware.org/binutils/docs/binutils/strings.html
syntaxing 2y ago
I was about to post the exact thing, I dont see any thing API related to iOS apps on Google. Edit: After some Google-ing and trying it on my macbook, there is a native CLI tool called "strings". Supposedly it does the following: strings is primarily used to find and display printable character sequences in binary files, object files, and executables. Which means the author is probably looking at the app to see the hardcorded characters in the app binary(?) and searching for the API end points.
- emilamlom 2y ago
  Just for context, strings is super commonly used when reverse-engineering anything. It's a great first-step because it's easy, fast, and get's some decent clues to help you get your bearings in an unknown binary file.
  - anthk 2y ago
    Among 'nm' and 'binwalk e $FILE'.

treyd 2y ago

I wonder how difficult it would be to combine many of these techniques into some automated script that dumps a manifest of the different types of undocumented APIs there are. LLMs have also been shown to be pretty good at answering semantic questions about large blobs of minified code, perhaps there could be some success there?

cobertos 2y ago
There's too many fiddly bits of which endpoints return what data in which shape that requires a custom solution each time. An LLM would probably have a very easy shot at this problem though. It's much more straightforward if you can find a GraphQL, Swagger, or OpenAPI spec to automate conversion I'd imagine.
justo-rivera 2y ago
Launch HN: Integuru – Reverse-engineer internal APIs using LLMs https://news.ycombinator.com/item?id=41983409 https://news.ycombinator.com/item?id=41983409

matthewfcarlson 2y ago

I went to the 75grand app listed in the article and saw a listing for Cafe Mac and did a double take. Apple's employee cafe is caffe Macs, so I was quite confused for a second

gaeb69 2y ago

Sick app btw. Funny this comes up because I'm working on the exact same thing for my school. Note that if your school uses Canvas; Canvas' API is well documented and has GraphQL endpoints.

0x3444ac53 2y ago
Canvas, and several other LMS's all use scorm cloud on the backend. Discovered this while researching for something work related. Getting into the internals of an LMS is tricky, but surprisingly very interesting and very fun.

deleted 2y ago

[deleted]

Lucasoato 2y ago

> 75grand’s success was even met with jealousy from the college That's a common story; in my university (in Padova, UniPD) happened something even worse. They tried hard to shut down an unofficial app (Uniweb) that was installed by most of the students in favor of the "official" one, that was completely unusable (and probably was born out of a rigged contract). At the end the best one won and became official, but that was after a lot of struggle.

mtnGoat 2y ago
I’m just not sure jealousy is the correct word for this though. Most systems don’t like these kinds of things for a number us reasons.
- TeMPOraL 2y ago
  In my (perhaps limited) experience, most of those reasons are just different ways of spelling "our official solution is shit, because we don't care and/or make money in some underhanded way".
- wingerlang 2y ago
  I think the word usage is tongue in cheek
- klabetron 2y ago
  Agreed that I’m not sure jealousy is the right interpretation of imitation being the sincerest form of flattery. Having said that, speaking as a Macalester alumnus, I wouldn’t put it past them to be a bit petty behind the scenes :)
mock-possum 2y ago
> They tried hard to shut down an unofficial app (Uniweb) that was installed by most of the students in favor of the "official" one, that was completely unusable Sounds like what happened with the Apollo app and Reddit
matheusmoreira 2y ago
My high school used a system so bad I stopped caring about my grades, I just didn't want to log into that crap more than once a year. I couldn't believe my eyes when I got into university and saw they were using the exact same shitty system. Reverse engineered just enough of that .net monstrosity to get the data I cared about. Replacing the school's portal with my own script made me so happy. They kept breaking my scraper constantly though, got to the point it was too much effort to maintain it. Worst part is I don't even think they were doing it on purpose. They were just bad in general and couldn't keep their site together.

z3c0 2y ago

> The error messages helpfully suggested fields I hadn’t known about by “correcting” my typos. Glad to see this being called out. Sure, I get why it's convenient. Misspelling a field by one character is a daily occurrence ("activty" and "heirarchy" are my regulars). The catch is that spellchecking queries and returning valid fields in the error effectively reduces entropy by both character space and message length, varying by the type of distance used in the spellcheck.

colesantiago 2y ago

Most of these techniques are extremely old and very outdated. Teams that I've seen working on apps now implement much stronger checks on APIs especially Android apps such as SafetyCheck and DeviceCheck and other methods, which makes using strings rather basic to see them. And most apps are now encrypted so you just see junk in the logs.

rozap 2y ago
And on the web side, fingerprinting is rampant and there are JS challenges in cloudflare, imperva, etc which make it trickier. Frustrating to run a whole browser with a virtual screen, load the whole page which is ofc like 15mb of JS and other trash, just to do a very simple thing. Granted, smaller fish like the ones OP is referring to generally don't have aggressive anti automation measures in place, so it can be easy...but generally these techniques don't work if the operator has put the proper measures in place.
- skilbjo 2y ago
  take a look at https://xhr.dev/ https://xhr.dev/, a product I built to avoid bot detection from things like cloudflare, imperva, aws waf, and others
  - Zopieux 2y ago
    >self host (Docker): $60k/yr lmao ok
  - enahs-sf 2y ago
    What does the $500 a month get me? Infinite resources to scrape all of LinkedIn?
- imp0cat 2y ago
  Frustrating? Yeah! but it works SO great! I especially like Playwright in this context, it can do pretty much anything and is a joy to use.
filleokus 2y ago

danielvaughn 2y ago

At a former job, we reverse engineered the trading APIs of most American retail stock brokerages (Fidelity, E-Trade, Robinhood, TD Ameritrade, etc). We did it by rooting an iPhone and using Charles Proxy to grab the unencrypted traffic. I learned a lot from that experience, and it's also just plain fun to do. We did get some strongly worded letters from Robinhood though, lol. They tried blocking our servers but we just set up this automated system in Digital Ocean that would spin up a new droplet each time we detected a blockage, and they were never able to stop us after that. Fun times.

dangsux 2y ago
[dead]
packetlost 2y ago
I did almost this exact same thing back in 2015~ ish when I was in high school over Christmas break. I reverse engineered the anime streaming site Crunchyroll's API via their Android and PS3 app using some HTTP proxy application and trial + error. I ended having a proper HLS-based streaming player and Android TV app back when their Android app was still Flash based. It was lots of fun!
- danielvaughn 2y ago
  2015/2016 was exactly when I was doing the above job. We could've hired you as an intern!
  - packetlost 2y ago
    Maybe! I considered pursuing a job there at the time, but I opted to get a degree instead. Being located in the Midwest would've made it rather challenging anyways.
  - packetlost 2y ago
    Also this would have been winter of 2014 and into early 2015, so a bit before :) Sadly the CR forums are gone, so my rather popular thread that had feedback and support is long gone.

stoplight 2y ago

This is how I made a better version of the nhl.com site [1] that has a better UI (you can see scores/schedules much more easily), is mobile first, has no ads, and responsiveness built in. I did the same for the AHL [2], and the PWHL [3]. [1] https://nhl-remix.vercel.app/ https://nhl-remix.vercel.app/ [2] https://ahl-remix.vercel.app/ https://ahl-remix.vercel.app/ [3] https://pwhl-remix.vercel.app/ https://pwhl-remix.vercel.app/

kmoser 2y ago
For a minute I thought PWHL was short for "pwn-NHL".
12345hn6789 2y ago
Poked around a bit. It's responsive and looks great on mobile. Kudos
bluetidepro 2y ago
Make one for the NFL, please! I can’t stand how hard it is to get scores on their website. Haha
- TechDebtDevin 2y ago
  There's like a gazzillion alternative scoring sites. Googling the team name also works while they're playing, Google usually has a good feed.
- Hackbraten 2y ago
  Funny you would mention that. A few weeks ago, I wrote a Python-based client library [0] for the website kicker.de [1]. It supports the NFL, too: import asyncio from kickerde_api_client import Api from kickerde_api_client.model import LeagueId api = Api() query = {'league': LeagueId.NFL, 'season': '2024/25'} season = asyncio.run(api.league_season(**query)) print(season['longName']) # 'National Football League' print(season['country']['longName']) # 'USA' print([ team['shortName'] for team in season['teams'].values() if team['shortName'].startswith('B') ]) # ['Buffalo', 'Baltimore'] day = season['gamedays'][18] print(str(day['dateFrom'].date())) # '2025-01-05' [0]: https://kickerde-api-client.readthedocs.io/en/stable/autoapi/kickerde_api_client/index.html https://kickerde-api-client.readthedocs.io/en/stable/autoapi... [1]: https://www.kicker.de/ https://www.kicker.de/

rcpt 2y ago

Anyone have this for Twitter? I want to remove most of my tweets but the official API costs $200

s09dfhks 2y ago
Their delete post endpoints probably require auth. What’s to stop you from deleting someone else’s posts
- samtheprogram 2y ago
  …you can see what auth/headers are passed to the endpoint from said developer tools. I agree with sibling commenters that automating the browser with DOM APIs is an easier route to go though.
markerz 2y ago
Maybe reverse it from the web app. I deleted a tweet and saw this request: HTTP POST https://x.com/i/api/graphql/VstuveVgh5q5jk7lmnVopqr/DeleteTweet { "variables": { "tweet_id":"12344567899123", "dark_request":false }, "queryId":"VstuveVgh5q5jk7lmnVopqr" } You can execute these from javascript in the browser if the auth part is too complicated. ### Update, this is the pure javascript console way, if you don't want to write your own client doing HTTP posts I played with the console more and got these parts: // Find all tweets on screen (this gives you the tweet IDs too) document.querySelectorAll('a > time') // Click the "more" button on the first tweet document.querySelectorAll('a > time')[0].parentElement.parentElement.parentElement.parentElement.parentElement.parentElement.parentElement.parentElement.parentElement.parentElement.parentElement.querySelector('button').click() // Click delete on the tweet document.querySelectorAll('[data-testid="Dropdown"]')[0].children[0].click() // Confirm delete document.querySelectorAll('[data-testid="confirmationSheetConfirm"]')[0].click()
- 0x3444ac53 2y ago
  Shortly before they destroyed the API I made a little script to delete all my tweets. Was really really handy

aswprojectry1 2y ago

[dead]

rubslopes 2y ago

I do exactly this, but for the company that I work for. I'm on the dashboards and integrations team, and I don't have direct access to the codebase of the main product. As the internal APIs have no documentation at all, I'm always "hacking" our own system using the browser inspector to find out how our endpoints work.

strunz 2y ago
There are so many of us doing this, haha. If you work at a big enough company or one with poorly documented APIs, it's just faster to reverse engineer the existing apps/UIs.
- TeMPOraL 2y ago
  Just make sure you don't end up accidentally making one of those APIs semi-official. I've seen a case where one team developed a temporary hack to automate some process in their product, then allowed another team making a sibling product to use it to test a possible feature; soon after, the possible feature became an actual one, everyone seemingly forgot the API was a throwaway test. Over the years, as both products evolved, that feature got pretty flaky, and in one of many cross-team attempts at fixing it, someone from the original time finally pointed out that the whole thing is still relying on a temporary hack in the original product that was never intended to be productized...
leptons 2y ago
Same here, but I do have access to the codebase. But there's no documentation about all the different APIs. Reading through the code also doesn't always explain what the API does, or what it's for. I work on a separate system that benefits from using these APIs from the main product, there is almost no overlap or communication between engineering departments. It's far from ideal.

arnau_cruz 2y ago

[flagged]

arnau_cruz 2y ago
[flagged]

dchuk 2y ago

I’ve reverse engineered a few industry conference apps to more easily get the list of attendees (some conferences will literally only give you a pdf of scanned paper lists of contact info for attendees, which is insane, especially if you are paying to have a booth there). I’ve either decompiled the Android app, or ran mitmproxy on the device, or both, to figure it out. Does anyone have a recommendation for having a pre-rooted Android simulator with the tools you need installed and ready to go to make this a quicker process? I’d love to just drag and drop an apk into a simulator and start inspecting vs having to use a real device and all that jazz.

SeriousStorm 2y ago
I found this recently, but I haven't tried it yet. https://github.com/budtmo/docker-android https://github.com/budtmo/docker-android
- uber1geek 2y ago
  that is a cool find :)

ideashower 2y ago

Someone did this to the Ring mobile app to get a dataset of all (public) Ring cameras across the US https://gizmodo.com/ring-s-hidden-data-let-us-map-amazons-sprawling-home-su-1840312279 https://gizmodo.com/ring-s-hidden-data-let-us-map-amazons-sp...

speakspokespok 2y ago

Can someone recommend a long form guide that covers best practices for handling authentication and the like?

gessha 2y ago
Is this what you’re looking for: https://thecopenhagenbook.com/ https://thecopenhagenbook.com/ It was shared last or week or so on here.

tomblomfield 2y ago

Gamemaster1379 2y ago

I actually enjoy reversing APIs. A game I really liked to play announced EOS last year. I ended up capturing a bunch of mitm traffic and built my own API instead and now head up a private server for it. I don't dare monetize that project, but I wish I could use my skills to make some extra money at reversing APIs. Wouldn't know where to begin though.

mdaniel 2y ago
I have the same wish, and wish that for you, but tangentially I wanted to offer that if you were to publish the game server you built, to the very best of my knowledge alternative implementations are protected by US law (assuming that applies to you, of course) and then everyone in a similar boat could sponsor your project, straight up donate to you, or at very least you'll have made the world a better place for the players in a similar situation as you were
- Gamemaster1379 2y ago
  The game is Gundam Evolution, so Bandai Namco is the owner. They are known to be very litigious and protective of Gundam IP. The game was a multiplayer FPS and involved cosmetic micro transactions. Due to mismanagement they shut down in one year. It launched in late 2022 and EOSed late 2024. I'm in the US, yes. The company is Japanese though and they may not care as much about US law when choosing to pursue. Even if they ultimately don't have a case, there's still enough gray area to torment-by-lawyer. For that reason I chose not to take any cash for the project. I also don't distribute any game files. Trying to minimize risk as much as possible. But yeah! It was a good community building project. We're about to hit 10,000 members I hope it leads to some connections that are profit rearing.
  - matheusmoreira 2y ago
    You're a hero. I couldn't care less what laws say, companies that go after people like you deserve to be boycotted mercilessly. I haven't bought a single EA game since they C&D'd the people reverse engineering Battlefield 2 and 2142 network protocols. They just wanted to keep some beloved games alive. I will never forgive them for it. Whatever you're doing, I hope you get away with it.

nicbou 2y ago

bund.dev is a group of volunteers doing exactly that for APIs run by the German government. I met Lilith at a meetup recently. Her talk was super interesting. https://bund.dev/ https://bund.dev/ https://links.lilithwittmann.de/ https://links.lilithwittmann.de/

purple-leafy 2y ago

I’ve built the leading satay transparency tool in my country and one other, reverse engineering an API. Took me a few hours, now 6000 people use the tool. I think my tool is responsible for their changes to rate limiting lol

geniium 2y ago

Very interesting read! Gives some idea on how to integrate data that are not easy to access. Thanks

faizshah 2y ago

Anyone come up with good techniques for reverse engineering websockets? Its especially annoying since many use binary message formats and there isnt a great way to document an arbitrary binary message protocol. A couple techniques im trying out: - websocat and wsrepl for reverse engineering interactively: https://github.com/doyensec/wsrepl https://github.com/doyensec/wsrepl - kaitai struct for documenting the binary message formats: https://kaitai.io/ https://kaitai.io/

pimterry 2y ago
I'm very interested in this, I do a lot of protocol debugging. Kaitai looks very neat - is that the most popular format for this kind of thing, or are there other popular options I should look at too?
- pdimitar 2y ago
  Just lately somebody posted about the Imhex tool's DSL: https://xy2i.blogspot.com/2024/11/using-imhexs-pattern-language-to-parse.html https://xy2i.blogspot.com/2024/11/using-imhexs-pattern-langu...
  - S0y 2y ago
    I recently wanted to reverse engineer some Websocket packets for a game I was playing. I used BurpSuite as a proxy to bypass the SSL encryption. It also has a pretty handy tool that will monitor all websocket traffic. After that I used ImHex, pretty much exactly like in that blog to reverse engineer the websocket packets. The DSL is a little finicky but once you wrap your head around it, it`s very nice and powerful.
    - pdimitar 2y ago
      Thanks for sharing your experience, it is valuable as I am considering using Imhex and its DSL lately.
rtrgrd 2y ago

lyime 2y ago

If you want to do this and get paid for it come talk to us at Terminal49

pdimitar 2y ago
Yes, I would like to do that, RE and network APIs were always a passion. What's the best channel to reach you? I'll email you if you are not seeing this comment.
- Jeremy1026 2y ago
  Reverse engineer their website CMS and find the admin email.
  - pdimitar 2y ago
    I got downvoted for not taking the hint, I suppose.

miki123211 2y ago

I think good APIs are one of te most important and least obvious advantages of SPAs.

saagarjha 2y ago

I did the same thing long (well, not that long) ago for my school’s system too: https://github.com/saagarjha/break https://github.com/saagarjha/break. In retrospect this was an excellent use of time because when I made this I barely knew what JSON was and ended up reimplementing some protocols from scratch (HTTP basic auth, WebDAV) that I learned much later were actually standardized and I had actually known them all along ;) Alas, the service got bought out by some vulture/bodyshop PE firm and the app ended up outliving the servers it was hitting. But until then I got a lot of great email from high school kids telling me how much I improved their experience or asking if they could build their own thing on the APIs they found in my app after trawling GitHub.

klabetron 2y ago

Macalester ‘07 here. Back in my day their LDAP directory was public (at least on the campus network) which I used to scrape student & professor lists.

6510 2y ago

I one time talk with a guy running a scraping company. A client of his wanted all products from all of his suppliers in a single interface. One really slow website was constantly changing things specifically to make scraping impossible. He had to fix it every few hours then they changed everything again. I gave him the perfect solution: Why don't you just call them and ask how much money they want for access to their product catalog? Give them tiny bits of information if they don't approve immediately. Tell them it is for one of their customers and that their site is to slow. They are the only supplier not in the interface which is bad for them and bad for you. If they still refuse offer them access to their competitors data at a modest fee. He couldn't stop laughing, he never considered it. When the sun set the next morning he put in the call. They gave him access immediately, they were so happy to finally get rid of his crawler messing up their analytics data. Apparently the people on the other end of the tube also didn't sleep. They had a good laugh about it.

xfeeefeee 2y ago

It's really fascinating watching the constant push and pull in projects like yt-dlp and places like YouTube and TikTok. So many interesting techniques to make things more difficult for reverse engineering or making unofficial requests. There is even a tiny JavaScript engine iirc that calculates a special value that YouTube uses to verify requests.

anigbrowl 2y ago
Companies should take the existence of things like this as a signal that something is wrong with their product. There's a lot of things wrong with YouTube these days. Their ad engine is just out of control, often sticking ads at both the beginning and the end of short videos, sticking ads in the middle of songs on music performance videos and so on. Amazingly, they've managed to make it worse than TV.
- jfil 2y ago
  This logic only applies for companies in a competitive business environment. YouTube is a video monopolist - they know that it is an awful service for viewers, but there's no need to improve things because where else would viewers go? Unfortunately, they are also part of a conglomerate that has reached full dominance in several online markets that are not growing. And they need to show quarter-on-quarter growth to shareholders. So you can't capture more market share, and there aren't any more people discovering the Web, so you stick additional ads everywhere. These people know exactly what they are doing.

brutecat 2y ago

The problem w/ this is a lot of the time is as they keep ramping up their anti-abuse measures, your old code breaks and needs to be rewritten (especially if it's an API you intend to use for a long term project). It's just an endless cat & mouse game.

pavel_lishin 2y ago
Sure, but all software rots.

sodality2 2y ago

Did this with Redlib to allow access to Reddit, and have been building tooling to allow academic researchers to access data at a large scale at no cost :) https://github.com/redlib-org/redlib https://github.com/redlib-org/redlib

anothername12 2y ago

There’s probably a group of poor bastards on call at 5am, poring over alerts, and logs in Datadog etc wondering WTF is wrong with their applications. Give ‘em a shout out by saying “hi” in a request parameter or something while you’re reverse engineering.

thesurlydev 2y ago

As I read through these comments I'm reminded of recon techniques commonly used in bug bounties. Reverse proxies, decompiling APKs, parsing JS files to find endpoints, etc. I often went down rabbit holes in the recon phase and never found bugs but I had fun in the process so I considered it a win anyway. I also implemented a bot mitigation system for a large international company so I got to see techniques used from the other side. Mobile phone farms and traffic from China was the most difficult to mitigate.

pdimitar 2y ago
Why was China in particular difficult to mitigate?

hudsonpeter0147 2y ago

[dead]

raudette 2y ago

A technique not covered here is setting up a wifi hotspot to impersonate a device. I wanted to reverse engineer the protocol used by an inexpensive action camera. I ran the Android app through a decompiler (JADX), and the app supported many different types of cameras, so it wasn't clear which calls were related to the one I owned. So I setup a Raspberry Pi with the SSID as the camera, logged the first API call. Then I ran that API against the camera, and then wrote a small python script on the Pi to return the same API result. I did that call by call until I'd worked everything out. I wrote it up here: https://www.hotelexistence.ca/reverse-engineer-akaso-ek7000/ https://www.hotelexistence.ca/reverse-engineer-akaso-ek7000/

zzo38computer 2y ago

I had also sometimes done, but usually because the existing UI isn't very good (and sometimes it doesn't work at all).

nurettin 2y ago

Browser debugger is my #1 resource. There is almost always a way to get stuff by replaying network requests. No matter how much they obfuscate (I'm looking at you, quora) it is still possible to replicate the requests and use the site like an API. Some real estate businesses and shadier data providers use browser detection to get around automation, but it is usually a minor inconvenience since you can run browsers headless or on xvfb if they have to render.