8 ms·
MitmProxy2Swagger: Automagically reverse-engineer REST APIs
- Gamemaster1379 2y agoThis is a nice tool. A game I liked to play announced end of service back in 2023. They gave enough notice to let me capture some logs from their cooridinator service. I captured them in mitmproxy and ran those through this to help me identify all the endpoints and their general structure. (A few things were a misnomer, like the examples suggesting certain values were able to be floats when they could only be integers) I was able to get a team together and we were able to stand up private servers as a result.
- simonjgreen 2y agoAmazing! What game was this for? I was involved in the RE efforts around UO way back in the day.
- kirici 2y agoGundam Evolution, going by comment history.
- ge96 2y agoDifferent plot/game mechanics but armored core 6 is great if you like mecha
- Gamemaster1379 2y agoGundam Evolution, as someone else noted from my comment history.
- andrewstuart 2y agoThis is something that would be easy to do an ordinary job of, missing lots of edge cases and not making something thorough and complete. A really professional and thorough job would be extremely time consuming and hard.
- matthewolfe 2y agoI do this a lot for my work. A tool like this that can help get me to a nice starting point is huge. Instead of developing a mental model of the API in my head by manually looking through API requests/responses in ProxyMan, this can start me off much more quickly. From there, the edge cases can be worked out.
- waseemmalik 2y ago[flagged]
- efilife 2y ago[flagged]
- dang 2y agoThey aren't.
- NicholasGurr 2y ago[dead]
- zython 2y agoThis is so cool. Thanks for sharing !
- tinchox5 2y agoCoool!
- zebomon 2y agoI looked through this earlier today when I saw it mentioned in that thread about the closed source tool for the same purpose. Having done a good bit of this type of reverse engineering the hard way over the years, it's a very exciting find. I had been talking with my partner about building something similar for the past six months. How exciting to learn that it's already out there and open source too!
- colesantiago 2y agoAgain, this is the very easy part of the reverse engineering API process that most tools can do, similar to API Parrot and the rest of them. This is not hard to do. The hard part is that inevitably, all these internal APIs will just add aggressive CAPTCHAs, Device Check, fingerprinting, etc to prevent common drive by re'ing. Easy to add these on the defence side, and extremely difficult to bypass on the other side. I can imagine all developer teams now upping their security with the combination of the above mentioned to prevent this.
- sebmellen 2y agoDepends on the age of the tool. We work with a lot of legacy systems that actually want us to integrate with them but don’t have the dev resources to build a proper API surface. As a result, we end up doing a lot of painful reverse engineering. These tools look promising for purposes like this.
- devjab 2y agoI curious as to why people would have a public API to begin with if they wanted to protect it from people using it. Then again, why would anyone have a public undocumented API in 2024 when a LLM can give you a cli tool to auto-generate 90% of the OpenAPI spec in a couple of hours? The last question isn't serious, I've worked in enterprise for decades and almost none of the tools organisations end up buying have good documentation for their API's. Not that those are publicly available, but still.
- lesuorac 2y agoI think you have a misunderstanding here. The API needs to be "public" because the app uses the internet to communicate back to the home server. The API is not "public" in the sense that the app developers want anybody to use it; they just want their app to use this API. So they don't write publicly accessible documentation about it because they don't want to encourage its use. A tool like MitmProxy2Swagger lets you run the app and record all of its API calls so that you can use this unadvertised API.
- 2y ago
- notcrazylol 2y agoI was wondering how it would take in graphql endpoints and convert it to swagger, since its just a single POST API with change in params. But thats more of a swagger issue than the tools. Has anyone dealt with this? Would be really helpful if you could share your ideas too :)
- asabla 2y agoWhy would you tho? If you're working against an GraphQL based API, you should be able to pull a schema file. And use that to implement your own API. All you would get from an Mitmproxy is example queries and mutations. With the additional complexity of extra tooling to stich together the schema file
- jampekka 2y agoPulling the schema file can, and often is, disabled server side. And GraphQL APIs can, and often do, decline to serve other than persisted queries, and those can't be really inferred even with known schema.
- notcrazylol 2y agoSo I am working with a new company that has a ton of graphql queries. What I wanted to do was write an integration test for them in the fastest and easiest way possible. I don't want to sit and read each query to identity where it is in the user flow. So I was thinking if I run this in the background and go through a happy flow, I can get the APIs in order and write an integration test.
- swyx 2y agodid i miss something or why are there TWO (2) "magically reverse engineer REST APIs" projects on the HN front page right now? is there some offline beef going on? (screenshot in case this goes away https://x.com/swyx/status/1874762725383188502 https://x.com/swyx/status/1874762725383188502)
- Quarrel 2y agoPresumably, because the closed source one got some traction, so people are pointing out the open source alternative.
- littlestymaar 2y agoLikely because of this comment[1] in the other thread which made people submit this link, and when multiple independent people submit the same link in a short period of time you're very likely to end up on the front page (this exact situation happened to me once) [1] https://news.ycombinator.com/item?id=42568121 https://news.ycombinator.com/item?id=42568121
- AbuAssar 2y agoYeah, that's where I got the link from.
- TechDebtDevin 2y ago[flagged]
- 2y ago
- youngNed 2y agoperhaps a n00b question, but would this work, or is there something similar for apps, specifically android apps?
- rhaps0dy 2y agoDepends on the app. If it uses some online functionality probably yes. You could also try decompilation, it’s decent on java apps like android’s.
- whilenot-dev 2y agoA MITM proxy isn't specific to any app, it's a forward proxy for your outgoing network connection. In case of an Android app you'd need to run mitmproxy on a machine in your network and setup the connection as proxy in your Android's network settings. Then you'd need follow http://mitm.it http://mitm.it to install mitmproxys root certificate on the Android device (to trust the connection with TLS) and off you go. EDIT: or rather follow the docs[0] [0]: https://docs.mitmproxy.org/stable/howto-install-system-trusted-ca-android/ https://docs.mitmproxy.org/stable/howto-install-system-trust...
- tecleandor 2y agoI've used this specific tool to help me reverse engineer the private API of an Android App. The thing is, depending on how hardened the app is, you'll have to play with Android to allow this interception, mostly because of certificate pinning. Also I remember something about apps not using the system wide trusted certificates you install (IIRC). I remember using a rooted device with LineageOS, and downloading the APK and modifying it with a tool so the self signed certificate for the mitm proxy works with it. The mitm proxy docs have some links to tools that can do that [0] and you could also use an Android emulator if you don't have an extra phone to mess with it [1] 0: https://docs.mitmproxy.org/stable/concepts-certificates/ 1: https://docs.mitmproxy.org/stable/howto-install-system-trusted-ca-android/
- jazz9k 2y agoI use burp suite combined with Frida (which can remove root check and override ssl pinning).
- _5hxt 2y agoYeah - does this get nullabilities right?
- tecleandor 2y agoI've used this tool in the past with success. Not perfect but it accelerates the work greatly if you can launch a mitm proxy quickly and are familiar with the tool. I've been fighting lately with an API, though. It's not very, let's say, RESTy. It has only one endpoint, and the different "sections" of the API are defined in parameters, so MitmProxy2Swagger doesn't detect them properly :(
- nejsjsjsbsb 2y agoNothing is RESTy
- quectophoton 2y ago> It's not very, let's say, RESTy. It has only one endpoint, To be fair, from what I understand an actual(tm) REST API would only have a single defined endpoint[1]: the entry point. With every other endpoint being discovered from the responses. And also from your message I'm guessing a URI still uniquely identifies a resource (specifically through the "query" part of the URI, instead of the more common "path"). So, technically, assuming there's nothing too weird with that API, it seems like MitmProxy2Swagger is failing to detect a REST API. [1]: Corollary: If an API is RESTful, it should be possible to rename any endpoint (except the entry point) at any moment in time without prior notice, and clients would not break as long as the response types/schemas are still supported by the clients. In-flight requests might fail with a 4xx, but after a retry they should go to the correct endpoint without any code change required.
- zdragnar 2y agoThis is HATEOAS, basically the core feature of REST that very few people actually use. Most of what the industry calls REST or RESTful is just structured and inefficient RPC.
- tecleandor 2y agoTrue, I almost never see the endpoint discovery thing, I almost forgot about it...
- srameshc 2y agoObvious question: How to protect against this ?
- smallnix 2y agoWhat specifically do you want to protect?
- soheil 2y ago[flagged]
- erk__ 2y agoOnly as long as you cannot load your own certificates, which you are able to in a lot of cases. Though on Android you can lock certificates allowed in a app, this can be circumvented though it adds another step. I am unsure if the same is a case for Apples devices, at least you might need jailbreak there.
- iBotPeaches 2y agoA good deal of APIs don't pin SSL certs so MITM works for a solid amount of them.
- deleted 2y ago[deleted]
- RandomRandy 2y agoYou can read SSL traffic if you're able to install a root certificate on your device and the website/app doesn't use certificate pinning. I recently used HttpToolkit to reverse engineer a REST endpoint that used SSL encryption
- pimterry 2y agoEven if it does use certificate pinning, you can generally disable that using tools like Frida (https://frida.re https://frida.re) with scripts like https://github.com/httptoolkit/frida-interception-and-unpinning/ https://github.com/httptoolkit/frida-interception-and-unpinn...
- soheil 2y ago[flagged]
- dtn 2y agoIsn't that the point of mitmproxy? https://github.com/mitmproxy/mitmproxy https://github.com/mitmproxy/mitmproxy
- cenamus 2y agoOption b could be more about breaking into some office that happens to contain those keys ;)
- bandrami 2y agoWait, it works fine on production APIs, what it doesn't work on is "production clients". You're deliberately man-in-the-middling yourself.
- CubsFan1060 2y agohttps://docs.mitmproxy.org/stable/overview-getting-started/#configure-your-browser-or-device https://docs.mitmproxy.org/stable/overview-getting-started/#... Seems like the proxy handles all the SSL, and likely strips any SSL Specific headers, etc.. But also, many, many, many companies do this exact thing. Just one example: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClEZCA0 https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?...
- daemonhorn 2y ago1) Having the TLS stack trust a "custom CA" provided by any number of debug tools (like mitmproxy or OWASP Zap) is relatively simple operation and can be done by anyone on any OS as long as you have admin/root. 2) There are a number of additional debug ways to decode the encryption from an application endpoint (e.g. https://wiki.wireshark.org/TLS https://wiki.wireshark.org/TLS and look at SSLKEYLOGFILE environment variable supported by most major TLS stacks and all major browser). Since MitmProxy2Swagger also supports HAR format ingest (e.g. https://github.com/alufers/mitmproxy2swagger#HAR https://github.com/alufers/mitmproxy2swagger#HAR ), this can easily be exported from any browser built-in debug tools (which also removes the encryption). Modern TLS is great, but there are limitations on what it actually provides especially around the CA trust model. These mitm tools are not designed to take random traffic from the internet you intercepted, they require privileged endpoint access to enable specific debug features or configurations.
- mkagenius 2y agoIf only someone could automate[1] the clicking and navigating part by writing in plaintext something like "Open airbnb and explore all the features as much as possible" :) 1. https://github.com/BandarLabs/clickclickclick https://github.com/BandarLabs/clickclickclick - It does that and I am one of the authors.
- deleted 2y ago[deleted]
- Zinkay 2y ago[flagged]
- Zinkay 2y ago[flagged]