7 ms·

Show HN: Mitmproxy 1.0 released, now with a web interface

248 points by mhils 10y ago

mhils 10y ago
Author(s) here - happy to answer any questions and take feedback! :)
- echelon 10y ago
  Why the sudden bump to a 1.0 version number? And why the planned jump to 2.0 in February? (I'd really like to see that blog post you're alluding to. :) Congrats on the release!
  - cortesi 10y ago
    It probably has to do with psychology more than anything. My mental model of this sort of development (if you'll permit me to be a bit florid) is of an eternal pilgrimage of small incremental improvements, without any fixed destination or many real landmarks along the way. The TODO list for mitmproxy has never grown shorter (and never will), so I just never reached a point where declaring a 1.0 was obviously right. I realise this is eccentric, doesn't reflect the maturity of the project, and is probably not the way my co-maintainers think about this, hence the change. For much the same reason, I'd like to de-emphasise major version increments as flag-fall events. Incrementing the major version inline with our development cadence makes sense from that perspective.
    - echelon 10y ago
      Excellent answer! Makes a lot of sense. Thanks!
- capt8bit 10y ago
  Many thanks to you, cortesi, and the rest of the team. I hope this doesn't mean that the command-line interface is going to become a second-class interface? The CLI is what originally attracted me to mitmproxy. Having CLI keyboard shortcuts helped me immediately speed up my workflow faster than I was ever able to customize ZAP or Burp. I love mitmproxy and use it for all my web application pentests. The flexibility is amazing, and allows me to quickly adapt, or write a rewrite rule or plugin, in any situation. Regex rules for limits and intercepts are amazing too. Keep up the good work.
  - cortesi 10y ago
    The console interface will always remain a first-class interface. We plan to bring mitmweb up to feature parity with the console, after which we'll make sure that neither tool has capabilities the other doesn't. I'm super excited about the new possibilities the web interface opens up for the project (watch this space), but the command-line tool is wired into my fingers. :)
  - mhils 10y ago
    > I hope this doesn't mean that the command-line interface is going to become a second-class interface? No worries - the command line interface is definitely staying a first-class interface. We're also planning to mirror the CLI keyboard shortcuts in the web interface (to the extent possible). Some are already in, others are coming. :)
  - tptacek 10y ago
    That's interesting to me. Can you describe your Burp workflow, and how you accomplish the same things in mitmproxy? One of the things we're doing this year for our clients is selling them on the idea of doing basic security integration testing as part of their normal dev process, which might involve these companies buying copies of Burp for their team. But I could probably be convinced that mitmproxy would work just as well. (I am, for what it's worth, extremely familiar with Burp, but only casually acquainted with mitmproxy).
    - capt8bit 10y ago
      > (I am, for what it's worth, extremely familiar with Burp, but only casually acquainted with mitmproxy). You may be disappointed, since I am the opposite. I use burp in rare cases now, since I've been using mitmproxy for so long. >That's interesting to me. Can you describe your Burp workflow, and how you accomplish the same things in mitmproxy? You may also be disappointed on how my methodology might differ. I work for a shop that performs grey-box (authenticated) web application tests that focus on manual comprehensive coverage. So my work flow goes something like this: - Manually walk the entire authenticated application, or designated portion. Populating all areas and using all application functionality. - Analyze mitmproxy logs to identify all application entry points, and other OWASP areas of interest. (mitmproxy's "Limit" feature makes this fast and easy) - Manually Test every endpoint, or other item, by duplicating the request, editing, and resending. This is pretty basic, so I'm sure you see how this could be done with burp's repeater and intruder, which were the main areas I used. In mitmproxy you can easily duplicate a flow, edit it, and resend it. Then, based on the response, resend it again. That's what I do for API testing, for web application testing it is usually closer to: - Stage requests in Hackbar on Firefox and send the request, which goes through mitmproxy. - Use Mitmproxy for fine-tuning anything, watching the details, and scripting the tedious. > One of the things we're doing this year for our clients is selling them on the idea of doing basic security integration testing as part of their normal dev process, which might involve these companies buying copies of Burp for their team. But I could probably be convinced that mitmproxy would work just as well. My basic common work flow is probably not very helpful in this case, but I think mitmproxy could still be very helpful to you. You probably know that mitmproxy can replay a series of requests, and has a great scripting API. I think these two features could be used together to meet a lot of your needs. Let me tell you about some of my experiences: In some cases a customer just can't get their fix quite right, so I will need to repeatedly perform a series of tedious steps to retest an issue. So, I automate the process by using mitmproxy to save a file of the steps involved in testing the vulnerability. This may be something like: Log in -> Create widget -> edit widget -> destroy widget -> submit note Then, I can pass this to mitmdump to be replayed, with sticky-cookies enabled, and I just watch the output to make sure the output is correct. As far as scripting goes, you may be interested in cases where I wanted to quickly test every input parameter for input validation of a list of characters. The script I wrote would be run on every request I made to the site I was testing, if it was a post request it would do the following: - Duplicate the request for every POST parameter in the request - Inject the list of characters in to the post parameter - Run/Replay the new request Hopefully you can make sense of my examples. I think by replaying request, or writing simple testing scripts, you could automate a lot of the basic security integration testing. The beautiful thing about mitmproxy is that it has a simple, powerful, scriptable, core. By using the powerful building-blocks it's provided I can make it do whatever I want it to do. But, it may require a little python scripting. Let me know if want additional attempts at clarification. edited for formatting
- Eridrus 10y ago
  I tried to use mitmproxy to fiddle around with ad requests a while back and found it really hard to get good performance from it. All I wanted to do was find all responses that looked like XML (regardless of headers) and do some simple replacements. Even after fiddling with settings to make it not buffer large responses it was still slow enough that it really affected my general web browsing through it. I ended up using Fiddler instead, and found the performance much better right out of the gate. Which is unfortunate since I can't really share Fiddler scripts with people on osx/linux.
  - jaimex2 10y ago
    Its written in Python so performance will never be its strong point. Your regex might have been too greedy or you were throwing an entire network at it?
    - Eridrus 10y ago
      I was running it on my laptop, MITMing only my own browsing. I was using something like "<?xml" in body to check for xml, then do a few additional checks, so unless the string in operator sucks perf wise I don't think there was much to do. I'm not expecting miracles, just reporting what I experienced. Maybe some form of multiprocess architecture would help alleviate these issues, maybe running it via Jython would have been better, I don't really know much about Python performance.
- rahrahrah 10y ago
  Can you mitm HTTPS? I'm guessing that the client would get some sort of certificate warning?
  - 8ig8 10y ago
    Yes... http://docs.mitmproxy.org/en/latest/certinstall.html http://docs.mitmproxy.org/en/latest/certinstall.html
  - chrisweekly 10y ago
    Yes. You simply need to install a mitmproxy cert. https://mitmproxy.org/doc/certinstall.html https://mitmproxy.org/doc/certinstall.html
colemickens 10y ago
Nice but, I'd pay money for built in SAZ support. Maybe just exposing the web interface to colleagues might be sufficient, but generally if they can't open it in Fiddler, it may have never happened, even if I provide complete instructions for opening the flow. (And I've spent a nontrivial amount of time to make the SAZ post processor thing to work but after spending an hour or two trying to get compatible versions and python's environment setup, I gave up, installed a Windows VM and made a Fiddler capture...) edit: Now I feel guilty for a feature request in an announcement post. Thank you for mitmproxy! It's definitely something I use weekly and I very much appreciate the support that Maximilian has given me in various GitHub issues, Slack, etc.
- nimrody 10y ago
  There's a Fiddler version for Linux (based on mono) which, in my experience, works quite well. Easier than starting up a windows VM. The killer feature of Fiddler, for me, is the timeline display which we use to optimise network heavy apps.
  - colemickens 10y ago
    Kind of a non-starter for me, as this is a headless cloud VM with more RAM and CPU than any PC I've ever owned. If I were to actually install X, Xrdp, etc... I'm not in a better situation than simply using Windows in the first place. And I suppose it's been long enough since trying Fiddler for Linux that I should reserve my experiences from last time around. I've gotten pretty used to mitmproxy. Seems to be the defacto, goto tool for HTTP captures on Linux.
    - feld 10y ago
      Why wouldn't X window forwarding be ok?
      - colemickens 10y ago
        I've never had X forwarding work well enough to actually be usable, and last time I checked NX was still a massive (/non-free) pain to setup. Besides, I already have a workflow that works perfectly fine, is headless as I prefer, works with mitmproxy, etc. Also, I installed Fiddler from AUR again this afternoon and it just looked awful. The settings menus were unreadable, for example.
janvdberg 10y ago
Looks cool! How does this compare to Burp Suite? I am specifically looking for something more lightweight.
- tptacek 10y ago
  It's getting there, and if you hate Burp's Java UI and mostly use Burp solely to capture and manually edit requests, it might be an acceptable substitute. I don't think pathoc/pathos are really close to what the Burp "Intruder" fuzzer does (and I'm not in love with Intruder either). A lot of people also rely on Burp's spider, for which there's no mitmproxy analog. Obviously, there's Burp's scanner, but I think scanners are a bad idea and wouldn't count this against mitmproxy.
  - mkagenius 10y ago
    > are really close to what the Burp "Intruder" fuzzer does (and I'm not in love with Intruder either). Have you achieved any good results with the intruder? if yes, an example would be awesome. (Personally I have never used Burp)
    - tptacek 10y ago
      Burp is the industry standard web pentesting tool, so that's a bit like asking if a writer has ever obtained good results from Microsoft Word.
      - mkagenius 10y ago
        I was asking about intruder in specific. I am not doubting the capabilities of Burp.
- capt8bit 10y ago
  I have used mitmproxy for performing web application pentest for years. If you want something to create/intercept/edit/tamper/replay requests, this is your tool. If you want to script any of those things, this is still your tool. However, burp comes with a lot of bells and whistles that don't make a lot of sense to build in to mitmproxy, but you can script yourself. For example, there is no intruder, spider, or scanner tool. But, they have an easy to use interface to write scripts that will be run on every request you make, or individual requests. Or, you can just pass all mitmproxy traffic out to burp and get the best of both worlds.
Goopplesoft 10y ago
Forgot this was written in python. Scanned the code to see how they do http/2. This lib seems useful for general python use: https://github.com/mitmproxy/mitmproxy/tree/master/pathod https://github.com/mitmproxy/mitmproxy/tree/master/pathod https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/net https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy... The whole package very well written. I'll definitely read through it and add it to my list of repos people should scan if they work with python.
- mhils 10y ago
  Thanks! We actually use https://github.com/python-hyper/hyper-h2 https://github.com/python-hyper/hyper-h2 to implement mitmproxy's HTTP/2 support and have our own HTTP/2 mini-stack for fuzzing that (which is also exposed in pathod).
SimeVidas 10y ago
The web interface is available only after installing the Python tool, or?
- mhils 10y ago
  Not sure if this answers your question: You can download our precompiled standalone binaries from https://github.com/mitmproxy/mitmproxy/releases/ https://github.com/mitmproxy/mitmproxy/releases/, run the mitmweb executable and your browser should open.
mstade 10y ago
TIL a new word: abstruse. Mitmproxy is a great tool for debugging web applications at a low level, particularly security related issues. Even for a n00b like myself the CLI was easy to use, but I'm happy to see a web interface as well. Keep up the good work!
lox 10y ago
I'd love to figure out how to use Mitmproxy to intercept traffic from Docker containers for debugging on macOS, I've never been able to make it work.
- Steeeve 10y ago
  I've done this a few times - intercepting traffic to applications running within docker, not intercepting docker's internal communications. I usually use a container specifically for MITMProxy rather than install it alongside whatever I'm troubleshooting. It was pretty straightforward. I don't recall having to do anything tricky to make it work.
lauren21247 10y ago
https://www.youtube.com/channel/UCYdf53MwhmY0roTmHddpHyw https://www.youtube.com/channel/UCYdf53MwhmY0roTmHddpHyw

248 points by mhils 10y ago

mhils 10y ago

Author(s) here - happy to answer any questions and take feedback! :)

echelon 10y ago
Why the sudden bump to a 1.0 version number? And why the planned jump to 2.0 in February? (I'd really like to see that blog post you're alluding to. :) Congrats on the release!
- cortesi 10y ago
  It probably has to do with psychology more than anything. My mental model of this sort of development (if you'll permit me to be a bit florid) is of an eternal pilgrimage of small incremental improvements, without any fixed destination or many real landmarks along the way. The TODO list for mitmproxy has never grown shorter (and never will), so I just never reached a point where declaring a 1.0 was obviously right. I realise this is eccentric, doesn't reflect the maturity of the project, and is probably not the way my co-maintainers think about this, hence the change. For much the same reason, I'd like to de-emphasise major version increments as flag-fall events. Incrementing the major version inline with our development cadence makes sense from that perspective.
  - echelon 10y ago
    Excellent answer! Makes a lot of sense. Thanks!
capt8bit 10y ago
Many thanks to you, cortesi, and the rest of the team. I hope this doesn't mean that the command-line interface is going to become a second-class interface? The CLI is what originally attracted me to mitmproxy. Having CLI keyboard shortcuts helped me immediately speed up my workflow faster than I was ever able to customize ZAP or Burp. I love mitmproxy and use it for all my web application pentests. The flexibility is amazing, and allows me to quickly adapt, or write a rewrite rule or plugin, in any situation. Regex rules for limits and intercepts are amazing too. Keep up the good work.
- cortesi 10y ago
  The console interface will always remain a first-class interface. We plan to bring mitmweb up to feature parity with the console, after which we'll make sure that neither tool has capabilities the other doesn't. I'm super excited about the new possibilities the web interface opens up for the project (watch this space), but the command-line tool is wired into my fingers. :)
- mhils 10y ago
  > I hope this doesn't mean that the command-line interface is going to become a second-class interface? No worries - the command line interface is definitely staying a first-class interface. We're also planning to mirror the CLI keyboard shortcuts in the web interface (to the extent possible). Some are already in, others are coming. :)
- tptacek 10y ago
  That's interesting to me. Can you describe your Burp workflow, and how you accomplish the same things in mitmproxy? One of the things we're doing this year for our clients is selling them on the idea of doing basic security integration testing as part of their normal dev process, which might involve these companies buying copies of Burp for their team. But I could probably be convinced that mitmproxy would work just as well. (I am, for what it's worth, extremely familiar with Burp, but only casually acquainted with mitmproxy).
  - capt8bit 10y ago
    > (I am, for what it's worth, extremely familiar with Burp, but only casually acquainted with mitmproxy). You may be disappointed, since I am the opposite. I use burp in rare cases now, since I've been using mitmproxy for so long. >That's interesting to me. Can you describe your Burp workflow, and how you accomplish the same things in mitmproxy? You may also be disappointed on how my methodology might differ. I work for a shop that performs grey-box (authenticated) web application tests that focus on manual comprehensive coverage. So my work flow goes something like this: - Manually walk the entire authenticated application, or designated portion. Populating all areas and using all application functionality. - Analyze mitmproxy logs to identify all application entry points, and other OWASP areas of interest. (mitmproxy's "Limit" feature makes this fast and easy) - Manually Test every endpoint, or other item, by duplicating the request, editing, and resending. This is pretty basic, so I'm sure you see how this could be done with burp's repeater and intruder, which were the main areas I used. In mitmproxy you can easily duplicate a flow, edit it, and resend it. Then, based on the response, resend it again. That's what I do for API testing, for web application testing it is usually closer to: - Stage requests in Hackbar on Firefox and send the request, which goes through mitmproxy. - Use Mitmproxy for fine-tuning anything, watching the details, and scripting the tedious. > One of the things we're doing this year for our clients is selling them on the idea of doing basic security integration testing as part of their normal dev process, which might involve these companies buying copies of Burp for their team. But I could probably be convinced that mitmproxy would work just as well. My basic common work flow is probably not very helpful in this case, but I think mitmproxy could still be very helpful to you. You probably know that mitmproxy can replay a series of requests, and has a great scripting API. I think these two features could be used together to meet a lot of your needs. Let me tell you about some of my experiences: In some cases a customer just can't get their fix quite right, so I will need to repeatedly perform a series of tedious steps to retest an issue. So, I automate the process by using mitmproxy to save a file of the steps involved in testing the vulnerability. This may be something like: Log in -> Create widget -> edit widget -> destroy widget -> submit note Then, I can pass this to mitmdump to be replayed, with sticky-cookies enabled, and I just watch the output to make sure the output is correct. As far as scripting goes, you may be interested in cases where I wanted to quickly test every input parameter for input validation of a list of characters. The script I wrote would be run on every request I made to the site I was testing, if it was a post request it would do the following: - Duplicate the request for every POST parameter in the request - Inject the list of characters in to the post parameter - Run/Replay the new request Hopefully you can make sense of my examples. I think by replaying request, or writing simple testing scripts, you could automate a lot of the basic security integration testing. The beautiful thing about mitmproxy is that it has a simple, powerful, scriptable, core. By using the powerful building-blocks it's provided I can make it do whatever I want it to do. But, it may require a little python scripting. Let me know if want additional attempts at clarification. edited for formatting
Eridrus 10y ago
I tried to use mitmproxy to fiddle around with ad requests a while back and found it really hard to get good performance from it. All I wanted to do was find all responses that looked like XML (regardless of headers) and do some simple replacements. Even after fiddling with settings to make it not buffer large responses it was still slow enough that it really affected my general web browsing through it. I ended up using Fiddler instead, and found the performance much better right out of the gate. Which is unfortunate since I can't really share Fiddler scripts with people on osx/linux.
- jaimex2 10y ago
  Its written in Python so performance will never be its strong point. Your regex might have been too greedy or you were throwing an entire network at it?
  - Eridrus 10y ago
    I was running it on my laptop, MITMing only my own browsing. I was using something like "<?xml" in body to check for xml, then do a few additional checks, so unless the string in operator sucks perf wise I don't think there was much to do. I'm not expecting miracles, just reporting what I experienced. Maybe some form of multiprocess architecture would help alleviate these issues, maybe running it via Jython would have been better, I don't really know much about Python performance.
rahrahrah 10y ago
Can you mitm HTTPS? I'm guessing that the client would get some sort of certificate warning?
- 8ig8 10y ago
  Yes... http://docs.mitmproxy.org/en/latest/certinstall.html http://docs.mitmproxy.org/en/latest/certinstall.html
- chrisweekly 10y ago
  Yes. You simply need to install a mitmproxy cert. https://mitmproxy.org/doc/certinstall.html https://mitmproxy.org/doc/certinstall.html

colemickens 10y ago

Nice but, I'd pay money for built in SAZ support. Maybe just exposing the web interface to colleagues might be sufficient, but generally if they can't open it in Fiddler, it may have never happened, even if I provide complete instructions for opening the flow. (And I've spent a nontrivial amount of time to make the SAZ post processor thing to work but after spending an hour or two trying to get compatible versions and python's environment setup, I gave up, installed a Windows VM and made a Fiddler capture...) edit: Now I feel guilty for a feature request in an announcement post. Thank you for mitmproxy! It's definitely something I use weekly and I very much appreciate the support that Maximilian has given me in various GitHub issues, Slack, etc.

nimrody 10y ago
There's a Fiddler version for Linux (based on mono) which, in my experience, works quite well. Easier than starting up a windows VM. The killer feature of Fiddler, for me, is the timeline display which we use to optimise network heavy apps.
- colemickens 10y ago
  Kind of a non-starter for me, as this is a headless cloud VM with more RAM and CPU than any PC I've ever owned. If I were to actually install X, Xrdp, etc... I'm not in a better situation than simply using Windows in the first place. And I suppose it's been long enough since trying Fiddler for Linux that I should reserve my experiences from last time around. I've gotten pretty used to mitmproxy. Seems to be the defacto, goto tool for HTTP captures on Linux.
  - feld 10y ago
    Why wouldn't X window forwarding be ok?
    - colemickens 10y ago
      I've never had X forwarding work well enough to actually be usable, and last time I checked NX was still a massive (/non-free) pain to setup. Besides, I already have a workflow that works perfectly fine, is headless as I prefer, works with mitmproxy, etc. Also, I installed Fiddler from AUR again this afternoon and it just looked awful. The settings menus were unreadable, for example.

janvdberg 10y ago

Looks cool! How does this compare to Burp Suite? I am specifically looking for something more lightweight.

tptacek 10y ago
It's getting there, and if you hate Burp's Java UI and mostly use Burp solely to capture and manually edit requests, it might be an acceptable substitute. I don't think pathoc/pathos are really close to what the Burp "Intruder" fuzzer does (and I'm not in love with Intruder either). A lot of people also rely on Burp's spider, for which there's no mitmproxy analog. Obviously, there's Burp's scanner, but I think scanners are a bad idea and wouldn't count this against mitmproxy.
- mkagenius 10y ago
  > are really close to what the Burp "Intruder" fuzzer does (and I'm not in love with Intruder either). Have you achieved any good results with the intruder? if yes, an example would be awesome. (Personally I have never used Burp)
  - tptacek 10y ago
    Burp is the industry standard web pentesting tool, so that's a bit like asking if a writer has ever obtained good results from Microsoft Word.
    - mkagenius 10y ago
      I was asking about intruder in specific. I am not doubting the capabilities of Burp.
capt8bit 10y ago
I have used mitmproxy for performing web application pentest for years. If you want something to create/intercept/edit/tamper/replay requests, this is your tool. If you want to script any of those things, this is still your tool. However, burp comes with a lot of bells and whistles that don't make a lot of sense to build in to mitmproxy, but you can script yourself. For example, there is no intruder, spider, or scanner tool. But, they have an easy to use interface to write scripts that will be run on every request you make, or individual requests. Or, you can just pass all mitmproxy traffic out to burp and get the best of both worlds.

Goopplesoft 10y ago

Forgot this was written in python. Scanned the code to see how they do http/2. This lib seems useful for general python use: https://github.com/mitmproxy/mitmproxy/tree/master/pathod https://github.com/mitmproxy/mitmproxy/tree/master/pathod https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/net https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy... The whole package very well written. I'll definitely read through it and add it to my list of repos people should scan if they work with python.

mhils 10y ago
Thanks! We actually use https://github.com/python-hyper/hyper-h2 https://github.com/python-hyper/hyper-h2 to implement mitmproxy's HTTP/2 support and have our own HTTP/2 mini-stack for fuzzing that (which is also exposed in pathod).

SimeVidas 10y ago

The web interface is available only after installing the Python tool, or?

mhils 10y ago
Not sure if this answers your question: You can download our precompiled standalone binaries from https://github.com/mitmproxy/mitmproxy/releases/ https://github.com/mitmproxy/mitmproxy/releases/, run the mitmweb executable and your browser should open.

mstade 10y ago

TIL a new word: abstruse. Mitmproxy is a great tool for debugging web applications at a low level, particularly security related issues. Even for a n00b like myself the CLI was easy to use, but I'm happy to see a web interface as well. Keep up the good work!

lox 10y ago

I'd love to figure out how to use Mitmproxy to intercept traffic from Docker containers for debugging on macOS, I've never been able to make it work.

Steeeve 10y ago
I've done this a few times - intercepting traffic to applications running within docker, not intercepting docker's internal communications. I usually use a container specifically for MITMProxy rather than install it alongside whatever I'm troubleshooting. It was pretty straightforward. I don't recall having to do anything tricky to make it work.

lauren21247 10y ago

https://www.youtube.com/channel/UCYdf53MwhmY0roTmHddpHyw https://www.youtube.com/channel/UCYdf53MwhmY0roTmHddpHyw