5 ms·

Mitmproxy 9: WireGuard Mode

158 points by mhils 4y ago

ericb 4y ago
Is the UDP work a precursor to HTTP/3 support? I don't see HTTP/3 mentioned in the announcement, so I assume that something else is needed to make it work still? Edit: I take that back, I see "Add HTTP/3 binary frame content view" in the commits. So does that mean it works? I would have thought that would be a headline-level announcement, though? Either way, amazing stuff, and thanks for the brilliant work!
- mhils 4y ago
  Yes, the UDP parts are preparation for QUIC/HTTP/3! We will be there soon. :-)
csdvrx 4y ago
Do you have links for the Magisk module? (and ideally a quick android tutorial)
- jeroenhd 4y ago
  The source code seems to be here: https://github.com/jorants/mitmproxy/blob/d3ceb1b0f8a5a09e6399d94ae8dd2a939821924b/mitmproxy/utils/magisk.py https://github.com/jorants/mitmproxy/blob/d3ceb1b0f8a5a09e63... Installation steps seems to be: - Navigate to /cert/magisk and download the ZIP file - Load the ZIP file in your Magisk app - Might be necessary: reboot The link to the Magisk module is also on the normal Mitmproxy cert page. Because each install generates (or should generate) its own certificate, the module needs to be generated on the fly.
- mhils 4y ago
  I haven't played around with this myself yet, but basically: 1. Configure your device to use mitmproxy. 2. Visit our magic mitm.it domain on the device. 3. Click on "Show Instructions" for Android.
mhils 4y ago
mitmproxy dev here, happy to answer questions once I'm back home later! :)
- Avicebron 4y ago
  First of all thanks! Not sure if you know any of the Linux maintainers, but if you do I'd love to know how to get in touch with some of them.
  - mhils 4y ago
    Are you looking for mitmproxy-on-linux maintainers, or Linux kernel maintainers? You have found the former I suppose, but I'm the wrong person to ask for the latter. :)
    - Avicebron 4y ago
      I was looking for a quick way to update my version from the command line (ubuntu), but --options didn't show anything off hand like an --update/--upgrade. I just briefly skimmed the docs/google for something that would avoid having to reinstall the binary. EDIT: also is there a way to configure the IDE that pops up when I go to edit path? It's vim which is fine, I'm just curious if I can either integrate a custom vim installation or route it to another IDE.
      - mhils 4y ago
        The recommended way to install mitmproxy on Linux is to download new standalone binaries. They are self-contained and can just be dropped into /usr/local/bin. If you want some way to update automatically, use our Docker images, install via pipx (https://docs.mitmproxy.org/stable/overview-installation/#installation-from-the-python-package-index-pypi https://docs.mitmproxy.org/stable/overview-installation/#ins...), or (for Windows users) use the Microsoft Store. > also is there a way to configure the IDE that pops up when I go to edit path? You can set $MITMPROXY_EDITOR or more generally $EDITOR. :)
- clumsysmurf 4y ago
cjbprime 4y ago
This looks great! My only frustration is not with mitmproxy, but macOS -- setting a proxy in System Preferences->Network isn't a guarantee that it will be used by apps, and there doesn't appear to be a clean way to force all traffic through a proxy.
- mdaniel 4y ago
  Isn't that closer to what a VPN would try to do than a proxy? I guess it depends on whether you mean all traffic on the machine or only the kind of traffic that travels over proxies (I think socks4/5 are special cases and presumably not what you meant on a post about mitmproxy)
  - cjbprime 4y ago
    I was just thinking TCP, although since mitmproxy now supports UDP perhaps it can be all traffic?
- mhils 4y ago
  If you can run mitmproxy on another device, then our new WireGuard mode does exactly fix that problem. At least as long as WireGuard is able to capture all traffic. :) Transparent same-device interception is something where we still need to do some work on. I've started some work for Windows (https://github.com/mitmproxy/mitmproxy/pull/5543 https://github.com/mitmproxy/mitmproxy/pull/5543), but the top priority is HTTP/3 first. :)
lordgrenville 4y ago
I don't know anything about this project, can somebody explain what the (nonmalicious) use case would be?
- nibbleshifter 4y ago
  Debugging first or third party applications that are acting up.
- lancebeet 4y ago
  I use mitmproxy (mainly mitmweb as a reverse proxy) on a daily basis to test and debug front-end applications. Mitmproxy allows me to reach application states that would otherwise be unreachable without modifying the application code. There are or course alternatives but mitmproxy is very capable, the mitmweb ui is really convenient and the scripting interface is a pleasure to use.
- sangnoir 4y ago
  What I've used it for on the past: 1. Debugging client/server API calls to isolate problems in complex JSON interactions. You can even set breakpoints to modify server responses on the fly (within the client timeout period) 2. Capturing live responses from API for replaying in client regression tests. 3. Faking server responses for a subset of routes that haven't been implemented on the server-side yet, while keeping all other routes intact (e.g. auth). This enables parallel client & server development. Scripting Mitmproxy in Python is easy. 4. Non-software engineering use case: replacing resources on third party websites. e.g. injecting a dark mode css file into a website rhat doesn't officially support one.
spockz 4y ago
Is there something like this that can introduce network latency, but on the TLS and/or HTTP level? E.g. delay TLS handshake (to simulate CRL checks), send headers but delay streaming, etc.
- mdaniel 4y ago
  I think that's what comcast (https://github.com/tylertreat/comcast#readme https://github.com/tylertreat/comcast#readme) is trying to do more than mitmproxy. I think there's a competing project but that one is the easiest to remember :-)
syntaxing 4y ago
Is there a good mitmproxy configuration that blocks ads network wide particularly YouTube? I tried using mitm-adblock but it was so slow it was unusable.
- mhils 4y ago
  I don't think I'm aware of any. If possible I would recommend using a browser extension like uBlock Origin instead, being in the browser context makes adblocking more efficient. (I realize not every client is a browser, I simply don't have good recommendations for this then)
saghul 4y ago
Nice work! Does this mean that intercepting WebRTC media traffic is now possible? It's DTLS, so I guess I'll have to check it out!
- mhils 4y ago
  We can definitely intercept DTLS now, there are no specific contentviews (pretty-printers) for WebRTC yet. I don't know how much of a binary protocol WebRTC is that would make pretty-printing necessary. :)
nanomonkey 4y ago
A friend of mine claims to have found mitmproxy on her phone, and is worried that her ex husband is using it to track her movements and interactions. I am unable to find anything concrete to say whether or not this is a possibility. Anyone heard of something similar being done, and know how this could have been accomplished. She says he didn't have access to her phone, but I know he is technically capable.
- atonse 4y ago
  Is it iOS or Android? I’d suspect it’s way easier to do this sort of thing on Android. But iOS also has certain network extensions you can use (like for VPN) that can redirect traffic. Not staying that’s what happened, just saying that alters the likeliness.
- mhils 4y ago
  mitmproxy would not really be helpful for what you describe, this sounds like she is confusing it with some other software/stalkerware.
dontbenebby 4y ago
Thanks for this! I haven't really played around with packet captures since back before HTTPS was widespread but I've heard a lot of people gripe about it over the years. Big shoutout to one of the poor bastards who had me as their student back when Pitt segmented the school of information science away from the rest of the network. I have no idea if that's still the case, because those moron librarians added a card reader, and I don't know how to parkour onto the roof of that building. I got hung up on the fact an extra credit question go marked wrong because I said based on the SSID, the packets were from a BSD operating system. The TA marked it wrong and said "no, it's from an OSX system"... I ended up in the actual professor's office, with a listout of the various codes for manufacturers and pointing out there's no "OSX" specific one. (In retrospect, it was a situation where I was rules lawyering because I was pissed because I kept bumping into a string of people who'd tell me "It's not my job to teach you" only... they weren't some cute woman from Iowa in town for a book signing meeting me for espresso, they were literally a professor teaching, in one case, a "statistics for psychology students" course I selected EXPLICITLY because their students also haven't had Calculus... and they were tanking my GPA. Anyways, I'm re-hash-ing very old gripes, but it felt like something fundemental shifted when https went widespread, while universities seem to focus on weird CTF bullshit. (You can just message them on a dating application, though then they'll make a show of refusing to hire you and reporting you to the FBI as a possible cyberterrorist... like uh, I get that it's spooky season or whatever but if someone made an offer I wouldn't be a cyberterrorist anymore!!) Anyways, I am looking forward to playing with this after I clear my todos. Thanks for the hard work.
armitron 4y ago
This is crying to be rewritten in a faster language. Python is too inefficient for a lot of production mitm use cases.
- brian_cloutier 4y ago
  Which use-cases are you thinking of? I'm sure they exist but they've never come up for me; I've used mitmproxy many times and it's always worked gloriously, but I never use it to capture / re-write more than a few dozen requests at a time.
- svnpenn 4y ago
  https://github.com/google/martian https://github.com/google/martian
GekkePrutser 4y ago
Is the wireguard option also supported on the Basis? Especially FreeBSD which now has wireguard kernel support.
- mhils 4y ago
  What do you mean by "on the basis"? :)
  - GekkePrutser 4y ago
    I mean on the BSDs. Sorry. I was typing from my phone and it autocorrected it, normally I notice but here I didn't.
    - mhils 4y ago
      Our WireGuard mode is entirely user space and works on all operating systems mitmproxy runs on. :)
      - GekkePrutser 4y ago
        Oh great, I will try it thanks so much for writing it! I really love terminal UI applications for their speed and low footprint.

158 points by mhils 4y ago

Is the UDP work a precursor to HTTP/3 support? I don't see HTTP/3 mentioned in the announcement, so I assume that something else is needed to make it work still? Edit: I take that back, I see "Add HTTP/3 binary frame content view" in the commits. So does that mean it works? I would have thought that would be a headline-level announcement, though? Either way, amazing stuff, and thanks for the brilliant work!

mhils 4y ago
Yes, the UDP parts are preparation for QUIC/HTTP/3! We will be there soon. :-)

csdvrx 4y ago

Do you have links for the Magisk module? (and ideally a quick android tutorial)

jeroenhd 4y ago
The source code seems to be here: https://github.com/jorants/mitmproxy/blob/d3ceb1b0f8a5a09e6399d94ae8dd2a939821924b/mitmproxy/utils/magisk.py https://github.com/jorants/mitmproxy/blob/d3ceb1b0f8a5a09e63... Installation steps seems to be: - Navigate to /cert/magisk and download the ZIP file - Load the ZIP file in your Magisk app - Might be necessary: reboot The link to the Magisk module is also on the normal Mitmproxy cert page. Because each install generates (or should generate) its own certificate, the module needs to be generated on the fly.
mhils 4y ago
I haven't played around with this myself yet, but basically: 1. Configure your device to use mitmproxy. 2. Visit our magic mitm.it domain on the device. 3. Click on "Show Instructions" for Android.

mhils 4y ago

mitmproxy dev here, happy to answer questions once I'm back home later! :)

Avicebron 4y ago
First of all thanks! Not sure if you know any of the Linux maintainers, but if you do I'd love to know how to get in touch with some of them.
- mhils 4y ago
  Are you looking for mitmproxy-on-linux maintainers, or Linux kernel maintainers? You have found the former I suppose, but I'm the wrong person to ask for the latter. :)
  - Avicebron 4y ago
    I was looking for a quick way to update my version from the command line (ubuntu), but --options didn't show anything off hand like an --update/--upgrade. I just briefly skimmed the docs/google for something that would avoid having to reinstall the binary. EDIT: also is there a way to configure the IDE that pops up when I go to edit path? It's vim which is fine, I'm just curious if I can either integrate a custom vim installation or route it to another IDE.
    - mhils 4y ago
      The recommended way to install mitmproxy on Linux is to download new standalone binaries. They are self-contained and can just be dropped into /usr/local/bin. If you want some way to update automatically, use our Docker images, install via pipx (https://docs.mitmproxy.org/stable/overview-installation/#installation-from-the-python-package-index-pypi https://docs.mitmproxy.org/stable/overview-installation/#ins...), or (for Windows users) use the Microsoft Store. > also is there a way to configure the IDE that pops up when I go to edit path? You can set $MITMPROXY_EDITOR or more generally $EDITOR. :)
clumsysmurf 4y ago

cjbprime 4y ago

This looks great! My only frustration is not with mitmproxy, but macOS -- setting a proxy in System Preferences->Network isn't a guarantee that it will be used by apps, and there doesn't appear to be a clean way to force all traffic through a proxy.

mdaniel 4y ago
Isn't that closer to what a VPN would try to do than a proxy? I guess it depends on whether you mean all traffic on the machine or only the kind of traffic that travels over proxies (I think socks4/5 are special cases and presumably not what you meant on a post about mitmproxy)
- cjbprime 4y ago
  I was just thinking TCP, although since mitmproxy now supports UDP perhaps it can be all traffic?
mhils 4y ago
If you can run mitmproxy on another device, then our new WireGuard mode does exactly fix that problem. At least as long as WireGuard is able to capture all traffic. :) Transparent same-device interception is something where we still need to do some work on. I've started some work for Windows (https://github.com/mitmproxy/mitmproxy/pull/5543 https://github.com/mitmproxy/mitmproxy/pull/5543), but the top priority is HTTP/3 first. :)

lordgrenville 4y ago

I don't know anything about this project, can somebody explain what the (nonmalicious) use case would be?

nibbleshifter 4y ago
Debugging first or third party applications that are acting up.
lancebeet 4y ago
I use mitmproxy (mainly mitmweb as a reverse proxy) on a daily basis to test and debug front-end applications. Mitmproxy allows me to reach application states that would otherwise be unreachable without modifying the application code. There are or course alternatives but mitmproxy is very capable, the mitmweb ui is really convenient and the scripting interface is a pleasure to use.
sangnoir 4y ago
What I've used it for on the past: 1. Debugging client/server API calls to isolate problems in complex JSON interactions. You can even set breakpoints to modify server responses on the fly (within the client timeout period) 2. Capturing live responses from API for replaying in client regression tests. 3. Faking server responses for a subset of routes that haven't been implemented on the server-side yet, while keeping all other routes intact (e.g. auth). This enables parallel client & server development. Scripting Mitmproxy in Python is easy. 4. Non-software engineering use case: replacing resources on third party websites. e.g. injecting a dark mode css file into a website rhat doesn't officially support one.

spockz 4y ago

Is there something like this that can introduce network latency, but on the TLS and/or HTTP level? E.g. delay TLS handshake (to simulate CRL checks), send headers but delay streaming, etc.

mdaniel 4y ago
I think that's what comcast (https://github.com/tylertreat/comcast#readme https://github.com/tylertreat/comcast#readme) is trying to do more than mitmproxy. I think there's a competing project but that one is the easiest to remember :-)

syntaxing 4y ago

Is there a good mitmproxy configuration that blocks ads network wide particularly YouTube? I tried using mitm-adblock but it was so slow it was unusable.

mhils 4y ago
I don't think I'm aware of any. If possible I would recommend using a browser extension like uBlock Origin instead, being in the browser context makes adblocking more efficient. (I realize not every client is a browser, I simply don't have good recommendations for this then)

saghul 4y ago

Nice work! Does this mean that intercepting WebRTC media traffic is now possible? It's DTLS, so I guess I'll have to check it out!

mhils 4y ago
We can definitely intercept DTLS now, there are no specific contentviews (pretty-printers) for WebRTC yet. I don't know how much of a binary protocol WebRTC is that would make pretty-printing necessary. :)

nanomonkey 4y ago

A friend of mine claims to have found mitmproxy on her phone, and is worried that her ex husband is using it to track her movements and interactions. I am unable to find anything concrete to say whether or not this is a possibility. Anyone heard of something similar being done, and know how this could have been accomplished. She says he didn't have access to her phone, but I know he is technically capable.

atonse 4y ago
Is it iOS or Android? I’d suspect it’s way easier to do this sort of thing on Android. But iOS also has certain network extensions you can use (like for VPN) that can redirect traffic. Not staying that’s what happened, just saying that alters the likeliness.
mhils 4y ago
mitmproxy would not really be helpful for what you describe, this sounds like she is confusing it with some other software/stalkerware.

dontbenebby 4y ago

Thanks for this! I haven't really played around with packet captures since back before HTTPS was widespread but I've heard a lot of people gripe about it over the years. Big shoutout to one of the poor bastards who had me as their student back when Pitt segmented the school of information science away from the rest of the network. I have no idea if that's still the case, because those moron librarians added a card reader, and I don't know how to parkour onto the roof of that building. I got hung up on the fact an extra credit question go marked wrong because I said based on the SSID, the packets were from a BSD operating system. The TA marked it wrong and said "no, it's from an OSX system"... I ended up in the actual professor's office, with a listout of the various codes for manufacturers and pointing out there's no "OSX" specific one. (In retrospect, it was a situation where I was rules lawyering because I was pissed because I kept bumping into a string of people who'd tell me "It's not my job to teach you" only... they weren't some cute woman from Iowa in town for a book signing meeting me for espresso, they were literally a professor teaching, in one case, a "statistics for psychology students" course I selected EXPLICITLY because their students also haven't had Calculus... and they were tanking my GPA. Anyways, I'm re-hash-ing very old gripes, but it felt like something fundemental shifted when https went widespread, while universities seem to focus on weird CTF bullshit. (You can just message them on a dating application, though then they'll make a show of refusing to hire you and reporting you to the FBI as a possible cyberterrorist... like uh, I get that it's spooky season or whatever but if someone made an offer I wouldn't be a cyberterrorist anymore!!) Anyways, I am looking forward to playing with this after I clear my todos. Thanks for the hard work.

armitron 4y ago

This is crying to be rewritten in a faster language. Python is too inefficient for a lot of production mitm use cases.

brian_cloutier 4y ago
Which use-cases are you thinking of? I'm sure they exist but they've never come up for me; I've used mitmproxy many times and it's always worked gloriously, but I never use it to capture / re-write more than a few dozen requests at a time.
svnpenn 4y ago
https://github.com/google/martian https://github.com/google/martian

GekkePrutser 4y ago

Is the wireguard option also supported on the Basis? Especially FreeBSD which now has wireguard kernel support.

mhils 4y ago
What do you mean by "on the basis"? :)
- GekkePrutser 4y ago
  I mean on the BSDs. Sorry. I was typing from my phone and it autocorrected it, normally I notice but here I didn't.
  - mhils 4y ago
    Our WireGuard mode is entirely user space and works on all operating systems mitmproxy runs on. :)
    - GekkePrutser 4y ago
      Oh great, I will try it thanks so much for writing it! I really love terminal UI applications for their speed and low footprint.