8 ms·

Mitmproxy 7.0

258 points by mhils 5y ago

mhils 5y ago
Mitmproxy dev here, happy to answer any questions! :)
- chmaynard 5y ago
  I'm interested in how you generate your docs. I'd like to do something similar using Jekyll.
  - mhils 5y ago
    Our main docs are built with Hugo (https://github.com/mitmproxy/mitmproxy/tree/main/docs https://github.com/mitmproxy/mitmproxy/tree/main/docs). For our API docs we use pdoc (https://pdoc.dev https://pdoc.dev), which integrates well with most static site generators. pdoc is also maintained by us. :)
    - chmaynard 5y ago
      Thanks. When you began working on the main docs, did you start with a pre-built theme or did you start from scratch?
      - mhils 5y ago
        If I'm not mistaken they are built from scratch, but that part wasn't built by me. IMO the by far hardest part is actually writing the docs. :)
- alias_neo 5y ago
  How practical is it (or not) to intercept TLS traffic for IoT devices on my network where I don't have the option of setting a proxy? I've been on a crusade the past few years to try and get my self-signed CA onto "smart" devices in my home, but with some it's difficult or I risk damaging expensive hardware. I've tried various ARP poisoning techniques and such in the past with limited success.
  - egberts1 5y ago
    Many IoTs have local search and scan, so block the DNS server at your firewall then provide your own DNS server via your own DHCP server.
  - mhils 5y ago
    I think that's a classic "it depends". I see two main challenges for IoT monitoring: 1. Getting the IoT traffic to your proxy machine. As you say this is fairly trivial when you can set a proxy, but a bit tricky otherwise as you need to run mitmproxy in transparent mode (https://docs.mitmproxy.org/stable/concepts-modes/ https://docs.mitmproxy.org/stable/concepts-modes/). With some luck you can set a custom gateway on your device which simplifies the setup, otherwise you need to have the capability to set iptables rules on the router. I personally have a small $20 OpenWRT router dedicated for that. Alternatively you can run your own DNS server and point everything to mitmproxy. v7 now also supports SNI/host-header based proxying! The good news is that this setup is the same for all your devices and once it works, it works. 2. Getting the device to trust your (personal) mitmproxy CA cert. This one really depends on the particular IoT device and may involve some serious reverse-engineering if you are unlucky. I wish there would be a more of a "I should be able to inspect my own traffic" movement, but it's a tricky problem to solve.
    - atatatat 5y ago
      ...because of enforcement on cheap Chinese etc goods.
    - alias_neo 5y ago
      The first part is the easy part, I have routers and run DNS, and have experience with IPTables, I can usually find ways to get hold of the traffic. The second part is the challenge, and for some devices I've been unable to embed my own CA. I was hoping there was a proxy trick or some magic sauce I was missing. Perhaps where DHCP could automatically configure clients with a (insecure) proxy or something along those lines?
      - oauea 5y ago
- aritmo 5y ago
  Thanks for mitmproxy! Suppose I have set an interception filter and I am viewing a live packet. I am about to edit it, etc. But when a new packet arrives, the text UI moves to the new packet! I have lost the packet that I was looking a few seconds ago. Is this an issue in mitmproxy or am doing things the wrong way?
  - mhils 5y ago
    Could you clarify what you mean by packet? Are you talking about HTTP requests or about the new non-HTTP TCP streams? The UI/UX for the latter is very much work-in-progress.
    - aritmo 5y ago
      I am talking about the HTTP requests.
      - mhils 5y ago
        The currently focused flow should never change when a new one comes in. If that's happening to you, please file a bug over on GitHub or ping me on our dev Slack and we'll get this fixed ASAP. :)
      - KORraN 5y ago
        Do you have by any chance enabled focus to follow new requests? Keyboard shortcut Shift + F. Although it's disabled by default (IIRC), you may have enabled it by accident.
- monai 5y ago
  mitmproxy is a great tool, but it lacks upstream certificate lookup and attributes transfer to a generated host certificate. For example, Charles does that. Some applications perform attribute inspection to detect MITM proxies, namely Spotify. Are you planning to implement this feature? Or maybe you already did it in version 7?
  - mhils 5y ago
    mitmproxy does lookup the upstream certificate by default, but at the moment we only transfer common name, subject alternative names and (new) organization name. The good news is that with version 7 it's very easy to add more here, I'd be more than happy to accept contributions. :)
    - monai 5y ago
      Good news indeed! I tried to contribute 1–2 years ago and quickly realized that such a feature would require a substantial rewrite after a quick code inspection.
      - mhils 5y ago
        If you (or anyone else reading this) is interested, please do reach out on GitHub or our dev Slack. Happy to help you get started. :)
        prinzhorn 5y ago
        Is this related to https://github.com/mitmproxy/mitmproxy/issues/4575 https://github.com/mitmproxy/mitmproxy/issues/4575 ?
        mhils 5y ago
        Yes, exactly!
- punnerud 5y ago
  I am still using version 2.0 after the big changes in shortcuts. I used mitmproxy weekly, now I have switched to mostly using Charles, but would love to switch back. Any plans for a recommended shortcut setup again? Not writing 30 characters for copy body of a POST request, but fast key bindings.
  - mhils 5y ago
    I don't recall what changed with v3, but sorry for letting you down. Shortcuts are fully configurable since a few releases: https://docs.mitmproxy.org/stable/concepts-commands/#custom-key-bindings https://docs.mitmproxy.org/stable/concepts-commands/#custom-.... If you have specific feedback about where the keyboard UX sucks I'd be happy if you could open an issue on GitHub so that we can take a closer look!
- cjbprime 5y ago
  Thanks for Mitmproxy! It would be amazing to be able to see non-HTTPS (but still TLS) traffic in the Flows view, and even to intercept it live instead of having to drop to raw TCP mode and write Python. Any idea whether that's on the roadmap?
  - mhils 5y ago
    Starting with this release you do see TCP-over-TLS in the flows view, we have a small screenshot in the blog post. Interception/modification is still something where we need to iron out the UX for, with version 7 you can at least inject messages with the `inject.tcp`/`inject.websocket` commands. This is not very well documented yet, but the technical foundations are there now. :)
- rsgrn 5y ago
  If I wanted to use mitmproxy to mitm and observe traffic during automated testing, what mode/settings should I be using on Windows? The setup is: A client (dev machine or pipeline) running tests that communicate to a Windows VM that hosts the app under test. The client RPCs to call automation APIs on the host/server. However I couldn't get both communicating and using mitmproxy to observe traffic to work at the same time.
  - mhils 5y ago
    If your server is really just one service on one port, I would just run mitmproxy as a reverse proxy and point your client to that (https://docs.mitmproxy.org/stable/concepts-modes/ https://docs.mitmproxy.org/stable/concepts-modes/). If there are multiple services, having it running as a regular proxy sounds best to me.
- c6401 5y ago
  I know that's not the purpose of mitmproxy, but wondering if it's possible to create automation scripts, like for puppeteer or export requests to curl/httpie/vscode rest client. I tried through web-interface but couldn't find anything like.
  - mhils 5y ago
    You can export to cURL/httpie export in mitmproxy, it'll be exposed in mitmweb with the next release. :) For automation, take a look at https://docs.mitmproxy.org/stable/overview-features/#server-side-replay https://docs.mitmproxy.org/stable/overview-features/#server-...!
filleokus 5y ago
Mitmproxy is so awesome, I use it all the time! Can recommend it for anyone who needs to do some fiddling with HTTP(S) traffic for debugging / reverse engineering purposes. I'm slightly too much of a GUI person to use it in a "explorative" phase, then I'm more a fan of Charles/Proxyman, but whenever I wanna modify the traffic or do some scripting to i.e log certain traffic, I reach for Mitmproxy.
- MostKaleido 5y ago
  I'd highly recommend https://httptoolkit.tech/ https://httptoolkit.tech/ for that explorative GUI phase. I found it recently and the rule configuration, UI and interception setup is significantly better than Charles/Fiddler/Proxyman.
  - deleted 5y ago
    [deleted]
  - enumjorge 5y ago
    It’s such a bummer that a tool like this has a subscription model pricing. I don’t know what it’s like in other companies, but where I work getting a recurring subscription approved is a hassle compared to a one-time thing. I’ve been looking for something like this for work but a recurring cost is almost an instant dealbreaker.
    - pimterry 5y ago
      Hi! I'm the dev behind HTTP Toolkit. There is a subscription for Pro features, but most users don't need it - you can do most things common use cases need (all interception & inspection, manual traffic rewriting, almost everything that doesn't need advanced configuration) with just the free version. It's also 100% open-source, even the paid bits. That said, imo subscriptions provide lots of benefits as a model for both sides. Basically by aligning how much value users get with how much they pay - cheap for quick use, expensive for extensive heavy use, strong incentive to fix bugs & support existing users instead of chasing the next shiny launch. I hear your point about approvals anyway. If subscriptions are impossible, I can do one-off indefinite licenses - send me an email at tim@httptoolkit.tech
ericb 5y ago
Such an amazing and well-maintained tool!!! Anyone interested in sharing use-cases where it has helped you?
- Proven 5y ago
  Tried it last week to see what APIs are used to communicate with home appliance. Couldn't get it to work.
- rbdixon 5y ago
  Scripting flow manipulations in mitmproxy is easy. There are a lot of great examples to learn from which help. The filter syntax is expressive and simplifies writing code to mangle flows. I've done body conversion [0], weird types of authorization testing, manipulated system update packages on the fly, all sorts of bizarre things. Auto-reloading scripts on change is such a time saver. If you want a python-based programmable HTTPS proxy for testing I have not found anything better. [0]: https://carvesystems.com/news/proxying-wbxml-services/ https://carvesystems.com/news/proxying-wbxml-services/
- mhils 5y ago
  We have a super diverse userbase, so this is a fun question! Some pointers: Development: - Debug your own apps/devices (which do not come with Chrome Devtools). - Simple Python scripts to replace/inject website content. (https://docs.mitmproxy.org/stable/addons-overview/ https://docs.mitmproxy.org/stable/addons-overview/) - Capture a web page/API and then replay the responses offline. (https://docs.mitmproxy.org/stable/overview-features/#server-side-replay https://docs.mitmproxy.org/stable/overview-features/#server-...). - Replay requests for testing. (https://docs.mitmproxy.org/stable/overview-features/#client-side-replay https://docs.mitmproxy.org/stable/overview-features/#client-...) Privacy: - Figure out what your Covid app is sending out (https://seancoates.com/blogs/how-i-helped-fix-canadas-covid-alert-app https://seancoates.com/blogs/how-i-helped-fix-canadas-covid-...) - Look at how evil IoT devices for pets are (https://www.nytimes.com/2018/12/11/realestate/spying-on-the-dog-or-cat.html https://www.nytimes.com/2018/12/11/realestate/spying-on-the-...) - Generally find more evil stuff (https://mitmproxy.org/publications/ https://mitmproxy.org/publications/) Security: - Reverse-engineering custom binary protocols is never fun, but with mitmproxy you can at least write custom contentviews to dissect stuff. (https://github.com/mitmproxy/mitmproxy/blob/main/examples/addons/contentview.py https://github.com/mitmproxy/mitmproxy/blob/main/examples/ad...) - Fiddle with generic TLS-protected TCP streams. (https://github.com/mitmproxy/mitmproxy/blob/main/examples/addons/tcp-simple.py https://github.com/mitmproxy/mitmproxy/blob/main/examples/ad...) - Dump TLS master secrets for WireShark. (https://docs.mitmproxy.org/stable/howto-wireshark-tls/ https://docs.mitmproxy.org/stable/howto-wireshark-tls/)
beermonster 5y ago
I’ve used this tool to identify and fix security issues and have also taught software testers how to use it just to spread the love and knowledge. Kudos!
pedrovhb 5y ago
Does anyone know how the story currently is for MITM'ing Android? I had fun reverse engineering app APIs a few years back, but I looked into it again more recently and found that Android has started pinning certificates by default even on apps that didn't pin anything themselves. I also had trouble getting my custom certificate to be used at all, when it used to be pretty easy. Can Frida still bypass pinning, even this new default one, or is it done at OS level?
- pimterry 5y ago
  In short: * By default, all modern Android apps only trust CA system certificates, and on a normal device you cannot change those. * You can change system certificates on rooted devices and most (but not all) emulators. * Apps can opt-in to trusting user-installed CA certificates within their manifest config, but almost all don't. If you're on a non-rooted device, the only way to MITM traffic is by modifying the application itself, so that it opts into trusting your cert. You can either inject Frida-Gadget to do that, do it manually, or use https://github.com/shroudedcode/apk-mitm https://github.com/shroudedcode/apk-mitm which tries to do it automatically (but it's a bit hit & miss whether it works). If you're on an emulator/rooted device, it's totally possible, see https://httptoolkit.tech/blog/intercepting-android-https/#injecting-ca-certificates-into-rooted-devices https://httptoolkit.tech/blog/intercepting-android-https/#in... for how that works. That's separate to explicit certificate pinning. That's also possible on Android (with some standard config settings, or manually in code) although it's got much less popular I think since the defaults were tightened up. In that case you do usually need Frida, and it's generally easiest to just use a rooted device. I wrote a blog post about exactly how to do that last week: https://httptoolkit.tech/blog/frida-certificate-pinning/#how-can-you-remove-certificate-pinning-with-frida https://httptoolkit.tech/blog/frida-certificate-pinning/#how...
  - mirashii 5y ago
    > By default, all modern Android apps only trust CA system certificates, and on a normal device you cannot change those. Sure you can. It's in a settings pane, Settings -> Security -> Encryption and Credentials -> Install a certificate. MDM profiles can also install certificates. There's pretty much zero possibility of this being removed as a feature as it is essential to a wide variety of corporate environments.
EvanAnderson 5y ago
Mitmproxy was an interesting solution to a problem I had a few weeks ago. One of my IT support gigs was trying to use "Mevo"-branded cameras to stream some live events to Facebook using an iPad. We had to use the venue's Wi-Fi network. We found the Mevo app would not detect that we were "connected to the Internet" and wouldn't allow us to stream. The venue's network admin said they weren't filtering anything outbound to the Internet. We could access websites from the iPad just fine. Same w/ Facebook, Youtube, etc. I put up Mitmproxy, an adhoc Wi-Fi network on a second Wi-Fi NIC, a DHCP server, and iptables NAT on my laptop. I set the iPad to use my adhoc SSID and my machine as an explicit HTTP proxy. My intention was to snoop on the traffic to determine what Mevo was using for criteria to consider being "connected". For whatever reason the Mevo app just started working in this configuration. I ended up sticking a spare Wi-Fi router and a PC w/ Mitmproxy installed in the venue for the duration of the event. I never did figure out what the Mevo app didn't like about the venue's network.
- derimagia 5y ago
  If I had to guess why you had issues, the app probably had certificate pinning. Apple makes it pretty easy to pin domains, but it's also not too difficult to circumvent it. This looks to cover it pretty well: https://www.guardsquare.com/blog/leveraging-infoplist-based-certificate-pinning-ios-and-making-its-shortcomings https://www.guardsquare.com/blog/leveraging-infoplist-based-...
- Forbo 5y ago
  If I had to guess, there may have been some sort of captive portal in place. I've seen even the "invisible" kind that don't require user intervention cause interference with connectivity checks for some devices.
gumby 5y ago
Is anyone using this for page rewriting? Not just for the all-important ad blocking but removing the whitespace where the removed ad was, rewriting autoplay videos and gifs into click-to-play, etc. So many apps have web views that the doing it in the browser is a step backwards. I used to do all my rewrite via a proxy but it stopped being particularly useful when https became almost universal.
- zo1 5y ago
  It's getting harder and harder to do even with HTTPS! Look at the other commenter on here that described the absolute wasteland with apps and OS-vendors (Android,IOS) being actively hostile to device-owners. Edit. Typo.
eklavya 5y ago
That sans io core sure sounds like a free monad interpreter.

Mitmproxy 7.0

258 points by mhils 5y ago

mhils 5y ago

Mitmproxy dev here, happy to answer any questions! :)

chmaynard 5y ago
I'm interested in how you generate your docs. I'd like to do something similar using Jekyll.
- mhils 5y ago
  Our main docs are built with Hugo (https://github.com/mitmproxy/mitmproxy/tree/main/docs https://github.com/mitmproxy/mitmproxy/tree/main/docs). For our API docs we use pdoc (https://pdoc.dev https://pdoc.dev), which integrates well with most static site generators. pdoc is also maintained by us. :)
  - chmaynard 5y ago
    Thanks. When you began working on the main docs, did you start with a pre-built theme or did you start from scratch?
    - mhils 5y ago
      If I'm not mistaken they are built from scratch, but that part wasn't built by me. IMO the by far hardest part is actually writing the docs. :)
alias_neo 5y ago
How practical is it (or not) to intercept TLS traffic for IoT devices on my network where I don't have the option of setting a proxy? I've been on a crusade the past few years to try and get my self-signed CA onto "smart" devices in my home, but with some it's difficult or I risk damaging expensive hardware. I've tried various ARP poisoning techniques and such in the past with limited success.
- egberts1 5y ago
  Many IoTs have local search and scan, so block the DNS server at your firewall then provide your own DNS server via your own DHCP server.
- mhils 5y ago
  I think that's a classic "it depends". I see two main challenges for IoT monitoring: 1. Getting the IoT traffic to your proxy machine. As you say this is fairly trivial when you can set a proxy, but a bit tricky otherwise as you need to run mitmproxy in transparent mode (https://docs.mitmproxy.org/stable/concepts-modes/ https://docs.mitmproxy.org/stable/concepts-modes/). With some luck you can set a custom gateway on your device which simplifies the setup, otherwise you need to have the capability to set iptables rules on the router. I personally have a small $20 OpenWRT router dedicated for that. Alternatively you can run your own DNS server and point everything to mitmproxy. v7 now also supports SNI/host-header based proxying! The good news is that this setup is the same for all your devices and once it works, it works. 2. Getting the device to trust your (personal) mitmproxy CA cert. This one really depends on the particular IoT device and may involve some serious reverse-engineering if you are unlucky. I wish there would be a more of a "I should be able to inspect my own traffic" movement, but it's a tricky problem to solve.
  - atatatat 5y ago
    ...because of enforcement on cheap Chinese etc goods.
  - alias_neo 5y ago
    The first part is the easy part, I have routers and run DNS, and have experience with IPTables, I can usually find ways to get hold of the traffic. The second part is the challenge, and for some devices I've been unable to embed my own CA. I was hoping there was a proxy trick or some magic sauce I was missing. Perhaps where DHCP could automatically configure clients with a (insecure) proxy or something along those lines?
    - oauea 5y ago
aritmo 5y ago
Thanks for mitmproxy! Suppose I have set an interception filter and I am viewing a live packet. I am about to edit it, etc. But when a new packet arrives, the text UI moves to the new packet! I have lost the packet that I was looking a few seconds ago. Is this an issue in mitmproxy or am doing things the wrong way?
- mhils 5y ago
  Could you clarify what you mean by packet? Are you talking about HTTP requests or about the new non-HTTP TCP streams? The UI/UX for the latter is very much work-in-progress.
  - aritmo 5y ago
    I am talking about the HTTP requests.
    - mhils 5y ago
      The currently focused flow should never change when a new one comes in. If that's happening to you, please file a bug over on GitHub or ping me on our dev Slack and we'll get this fixed ASAP. :)
    - KORraN 5y ago
      Do you have by any chance enabled focus to follow new requests? Keyboard shortcut Shift + F. Although it's disabled by default (IIRC), you may have enabled it by accident.
monai 5y ago
mitmproxy is a great tool, but it lacks upstream certificate lookup and attributes transfer to a generated host certificate. For example, Charles does that. Some applications perform attribute inspection to detect MITM proxies, namely Spotify. Are you planning to implement this feature? Or maybe you already did it in version 7?
- mhils 5y ago
  mitmproxy does lookup the upstream certificate by default, but at the moment we only transfer common name, subject alternative names and (new) organization name. The good news is that with version 7 it's very easy to add more here, I'd be more than happy to accept contributions. :)
  - monai 5y ago
    Good news indeed! I tried to contribute 1–2 years ago and quickly realized that such a feature would require a substantial rewrite after a quick code inspection.
    - mhils 5y ago
      If you (or anyone else reading this) is interested, please do reach out on GitHub or our dev Slack. Happy to help you get started. :)
      - prinzhorn 5y ago
        Is this related to https://github.com/mitmproxy/mitmproxy/issues/4575 https://github.com/mitmproxy/mitmproxy/issues/4575 ?
        mhils 5y ago
        Yes, exactly!
punnerud 5y ago
I am still using version 2.0 after the big changes in shortcuts. I used mitmproxy weekly, now I have switched to mostly using Charles, but would love to switch back. Any plans for a recommended shortcut setup again? Not writing 30 characters for copy body of a POST request, but fast key bindings.
- mhils 5y ago
  I don't recall what changed with v3, but sorry for letting you down. Shortcuts are fully configurable since a few releases: https://docs.mitmproxy.org/stable/concepts-commands/#custom-key-bindings https://docs.mitmproxy.org/stable/concepts-commands/#custom-.... If you have specific feedback about where the keyboard UX sucks I'd be happy if you could open an issue on GitHub so that we can take a closer look!
cjbprime 5y ago
Thanks for Mitmproxy! It would be amazing to be able to see non-HTTPS (but still TLS) traffic in the Flows view, and even to intercept it live instead of having to drop to raw TCP mode and write Python. Any idea whether that's on the roadmap?
- mhils 5y ago
  Starting with this release you do see TCP-over-TLS in the flows view, we have a small screenshot in the blog post. Interception/modification is still something where we need to iron out the UX for, with version 7 you can at least inject messages with the `inject.tcp`/`inject.websocket` commands. This is not very well documented yet, but the technical foundations are there now. :)
rsgrn 5y ago
If I wanted to use mitmproxy to mitm and observe traffic during automated testing, what mode/settings should I be using on Windows? The setup is: A client (dev machine or pipeline) running tests that communicate to a Windows VM that hosts the app under test. The client RPCs to call automation APIs on the host/server. However I couldn't get both communicating and using mitmproxy to observe traffic to work at the same time.
- mhils 5y ago
  If your server is really just one service on one port, I would just run mitmproxy as a reverse proxy and point your client to that (https://docs.mitmproxy.org/stable/concepts-modes/ https://docs.mitmproxy.org/stable/concepts-modes/). If there are multiple services, having it running as a regular proxy sounds best to me.
c6401 5y ago
I know that's not the purpose of mitmproxy, but wondering if it's possible to create automation scripts, like for puppeteer or export requests to curl/httpie/vscode rest client. I tried through web-interface but couldn't find anything like.
- mhils 5y ago
  You can export to cURL/httpie export in mitmproxy, it'll be exposed in mitmweb with the next release. :) For automation, take a look at https://docs.mitmproxy.org/stable/overview-features/#server-side-replay https://docs.mitmproxy.org/stable/overview-features/#server-...!

filleokus 5y ago

Mitmproxy is so awesome, I use it all the time! Can recommend it for anyone who needs to do some fiddling with HTTP(S) traffic for debugging / reverse engineering purposes. I'm slightly too much of a GUI person to use it in a "explorative" phase, then I'm more a fan of Charles/Proxyman, but whenever I wanna modify the traffic or do some scripting to i.e log certain traffic, I reach for Mitmproxy.

MostKaleido 5y ago
I'd highly recommend https://httptoolkit.tech/ https://httptoolkit.tech/ for that explorative GUI phase. I found it recently and the rule configuration, UI and interception setup is significantly better than Charles/Fiddler/Proxyman.
- deleted 5y ago
  [deleted]
- enumjorge 5y ago
  It’s such a bummer that a tool like this has a subscription model pricing. I don’t know what it’s like in other companies, but where I work getting a recurring subscription approved is a hassle compared to a one-time thing. I’ve been looking for something like this for work but a recurring cost is almost an instant dealbreaker.
  - pimterry 5y ago
    Hi! I'm the dev behind HTTP Toolkit. There is a subscription for Pro features, but most users don't need it - you can do most things common use cases need (all interception & inspection, manual traffic rewriting, almost everything that doesn't need advanced configuration) with just the free version. It's also 100% open-source, even the paid bits. That said, imo subscriptions provide lots of benefits as a model for both sides. Basically by aligning how much value users get with how much they pay - cheap for quick use, expensive for extensive heavy use, strong incentive to fix bugs & support existing users instead of chasing the next shiny launch. I hear your point about approvals anyway. If subscriptions are impossible, I can do one-off indefinite licenses - send me an email at tim@httptoolkit.tech

ericb 5y ago

Such an amazing and well-maintained tool!!! Anyone interested in sharing use-cases where it has helped you?

Proven 5y ago
Tried it last week to see what APIs are used to communicate with home appliance. Couldn't get it to work.
rbdixon 5y ago
Scripting flow manipulations in mitmproxy is easy. There are a lot of great examples to learn from which help. The filter syntax is expressive and simplifies writing code to mangle flows. I've done body conversion [0], weird types of authorization testing, manipulated system update packages on the fly, all sorts of bizarre things. Auto-reloading scripts on change is such a time saver. If you want a python-based programmable HTTPS proxy for testing I have not found anything better. [0]: https://carvesystems.com/news/proxying-wbxml-services/ https://carvesystems.com/news/proxying-wbxml-services/
mhils 5y ago
We have a super diverse userbase, so this is a fun question! Some pointers: Development: - Debug your own apps/devices (which do not come with Chrome Devtools). - Simple Python scripts to replace/inject website content. (https://docs.mitmproxy.org/stable/addons-overview/ https://docs.mitmproxy.org/stable/addons-overview/) - Capture a web page/API and then replay the responses offline. (https://docs.mitmproxy.org/stable/overview-features/#server-side-replay https://docs.mitmproxy.org/stable/overview-features/#server-...). - Replay requests for testing. (https://docs.mitmproxy.org/stable/overview-features/#client-side-replay https://docs.mitmproxy.org/stable/overview-features/#client-...) Privacy: - Figure out what your Covid app is sending out (https://seancoates.com/blogs/how-i-helped-fix-canadas-covid-alert-app https://seancoates.com/blogs/how-i-helped-fix-canadas-covid-...) - Look at how evil IoT devices for pets are (https://www.nytimes.com/2018/12/11/realestate/spying-on-the-dog-or-cat.html https://www.nytimes.com/2018/12/11/realestate/spying-on-the-...) - Generally find more evil stuff (https://mitmproxy.org/publications/ https://mitmproxy.org/publications/) Security: - Reverse-engineering custom binary protocols is never fun, but with mitmproxy you can at least write custom contentviews to dissect stuff. (https://github.com/mitmproxy/mitmproxy/blob/main/examples/addons/contentview.py https://github.com/mitmproxy/mitmproxy/blob/main/examples/ad...) - Fiddle with generic TLS-protected TCP streams. (https://github.com/mitmproxy/mitmproxy/blob/main/examples/addons/tcp-simple.py https://github.com/mitmproxy/mitmproxy/blob/main/examples/ad...) - Dump TLS master secrets for WireShark. (https://docs.mitmproxy.org/stable/howto-wireshark-tls/ https://docs.mitmproxy.org/stable/howto-wireshark-tls/)

beermonster 5y ago

I’ve used this tool to identify and fix security issues and have also taught software testers how to use it just to spread the love and knowledge. Kudos!

pedrovhb 5y ago

Does anyone know how the story currently is for MITM'ing Android? I had fun reverse engineering app APIs a few years back, but I looked into it again more recently and found that Android has started pinning certificates by default even on apps that didn't pin anything themselves. I also had trouble getting my custom certificate to be used at all, when it used to be pretty easy. Can Frida still bypass pinning, even this new default one, or is it done at OS level?

pimterry 5y ago
In short: * By default, all modern Android apps only trust CA system certificates, and on a normal device you cannot change those. * You can change system certificates on rooted devices and most (but not all) emulators. * Apps can opt-in to trusting user-installed CA certificates within their manifest config, but almost all don't. If you're on a non-rooted device, the only way to MITM traffic is by modifying the application itself, so that it opts into trusting your cert. You can either inject Frida-Gadget to do that, do it manually, or use https://github.com/shroudedcode/apk-mitm https://github.com/shroudedcode/apk-mitm which tries to do it automatically (but it's a bit hit & miss whether it works). If you're on an emulator/rooted device, it's totally possible, see https://httptoolkit.tech/blog/intercepting-android-https/#injecting-ca-certificates-into-rooted-devices https://httptoolkit.tech/blog/intercepting-android-https/#in... for how that works. That's separate to explicit certificate pinning. That's also possible on Android (with some standard config settings, or manually in code) although it's got much less popular I think since the defaults were tightened up. In that case you do usually need Frida, and it's generally easiest to just use a rooted device. I wrote a blog post about exactly how to do that last week: https://httptoolkit.tech/blog/frida-certificate-pinning/#how-can-you-remove-certificate-pinning-with-frida https://httptoolkit.tech/blog/frida-certificate-pinning/#how...
- mirashii 5y ago
  > By default, all modern Android apps only trust CA system certificates, and on a normal device you cannot change those. Sure you can. It's in a settings pane, Settings -> Security -> Encryption and Credentials -> Install a certificate. MDM profiles can also install certificates. There's pretty much zero possibility of this being removed as a feature as it is essential to a wide variety of corporate environments.

EvanAnderson 5y ago

Mitmproxy was an interesting solution to a problem I had a few weeks ago. One of my IT support gigs was trying to use "Mevo"-branded cameras to stream some live events to Facebook using an iPad. We had to use the venue's Wi-Fi network. We found the Mevo app would not detect that we were "connected to the Internet" and wouldn't allow us to stream. The venue's network admin said they weren't filtering anything outbound to the Internet. We could access websites from the iPad just fine. Same w/ Facebook, Youtube, etc. I put up Mitmproxy, an adhoc Wi-Fi network on a second Wi-Fi NIC, a DHCP server, and iptables NAT on my laptop. I set the iPad to use my adhoc SSID and my machine as an explicit HTTP proxy. My intention was to snoop on the traffic to determine what Mevo was using for criteria to consider being "connected". For whatever reason the Mevo app just started working in this configuration. I ended up sticking a spare Wi-Fi router and a PC w/ Mitmproxy installed in the venue for the duration of the event. I never did figure out what the Mevo app didn't like about the venue's network.

derimagia 5y ago
If I had to guess why you had issues, the app probably had certificate pinning. Apple makes it pretty easy to pin domains, but it's also not too difficult to circumvent it. This looks to cover it pretty well: https://www.guardsquare.com/blog/leveraging-infoplist-based-certificate-pinning-ios-and-making-its-shortcomings https://www.guardsquare.com/blog/leveraging-infoplist-based-...
Forbo 5y ago
If I had to guess, there may have been some sort of captive portal in place. I've seen even the "invisible" kind that don't require user intervention cause interference with connectivity checks for some devices.

gumby 5y ago

Is anyone using this for page rewriting? Not just for the all-important ad blocking but removing the whitespace where the removed ad was, rewriting autoplay videos and gifs into click-to-play, etc. So many apps have web views that the doing it in the browser is a step backwards. I used to do all my rewrite via a proxy but it stopped being particularly useful when https became almost universal.

zo1 5y ago
It's getting harder and harder to do even with HTTPS! Look at the other commenter on here that described the absolute wasteland with apps and OS-vendors (Android,IOS) being actively hostile to device-owners. Edit. Typo.

eklavya 5y ago

That sans io core sure sounds like a free monad interpreter.