3 ms·

a) This has less features than mitmproxy, burp suite or owasp zap. b) Export request is a very essential feature that's available for free in either of these a

a) This has less features than mitmproxy, burp suite or owasp zap. b) Export request is a very essential feature that's available for free in either of these alternatives and its pay-walled here. c) Intercept request in a terminal very badly broken. It broke (go get) and probably breaks others. I expect it adds an environment variable (which can be ignored by an application) or uses LD_PRELOAD (which doesn't work in statically linked applications). Other than that, it functions like you would expect it to. Worked out of the box for Firefox and curl.

pimterry 5y ago
I'm the author. I'm not a Go developer though, can you give me an example I can reproduce for `go get` that doesn't work for you? For the terminal, there's a few mechanisms, but environment variables are the catch-all there, yes (full list: https://github.com/httptoolkit/httptoolkit-server/blob/master/src/interceptors/terminal/terminal-env-overrides.ts https://github.com/httptoolkit/httptoolkit-server/blob/maste...). Those do work for most cases, but it is absolutely not a hard guarantee for applications that actively ignore standard proxy configuration (handling that is very hard, and definitely out of scope here). Go does generally observes `http_proxy` correctly by default in other cases I've tested, so this vert simple code from the test suite is automatically intercepted for example: https://github.com/httptoolkit/httptoolkit-server/blob/master/test/fixtures/docker/go/app.go https://github.com/httptoolkit/httptoolkit-server/blob/maste.... Very happy to look into any failing cases you can share.
- IceWreck 5y ago
  Yes, go based programs observe the proxy variable unless specifically compiled not to. But that doesn't break applications, they just ignore the env var. Go's default package manager doesn't seem to work at all with HTTP Toolkit. Here's what I did. * Intercept Tab > Fresh terminal. * In the terminal, do your usual stuff, it intercepts curl, etc. * If you try to use go's package manager, example: `go get golang.org/x/oauth2` It errors out with ``` go get: module golang.org/x/oauth2: reading http://127.0.0.1:8000/golang.org/x/oauth2/@v/list http://127.0.0.1:8000/golang.org/x/oauth2/@v/list: 500 Server error ``` Ideally it shouldn't break the application, just ignore if it can't intercept. > absolutely not a hard guarantee for applications that actively ignore standard proxy configuration (handling that is very hard, and definitely out of scope here). I encountered a usecase where this was needed and LD_PRELOAD trick (used by proxychains) etc failed because the application was statically compiled. I ended up using https://github.com/hmgle/graftcp https://github.com/hmgle/graftcp which somehow manages to force tcp traffic through a socks5 proxy.
  - pimterry 5y ago
    Thanks, that's super useful. > If you try to use go's package manager, example: `go get golang.org/x/oauth2` I just tested, and `go get golang.org/x/oauth2` seems to work fine for me, I can see all the requests being happily intercepted immediately: https://imgur.com/a/Cb1y9Q2 https://imgur.com/a/Cb1y9Q2 Can you see the 500 in HTTP Toolkit, and any more info there (in the body or as an error at the top) related to that? Or can you see a "certificate rejected" message? If nothing turns up there at all then yes, something must be overriding the proxy configuration. Maybe you have some other Go package manager configuration that conflicts with this? I'd be very interested to know about that if so, I'm sure there's others with the same thing. It's always very hard to know if my configuration is representative of normal devs for any given language/tool. Probably best to debug this outside of a HN thread though :-). You can file a proper issue about this at https://github.com/httptoolkit/httptoolkit/issues/new https://github.com/httptoolkit/httptoolkit/issues/new, I'd love to know what's going on there and get this fixed. > I ended up using https://github.com/hmgle/graftcp https://github.com/hmgle/graftcp which somehow manages to force tcp traffic through a socks5 proxy. Really interesting, thanks! I'll look into that.
    - IceWreck 5y ago
      This is the error I see in the response body in HTTP Toolkit. ``` Error: Passthrough loop detected. This probably means you're sending a request directly to a passthrough endpoint, which is forwarding it to the target URL, which is a passthrough endpoint, which is forwarding it to the target URL, which is a passthrough endpoint... You should either explicitly mock a response for this URL (http://127.0.0.1:8000/golang.org/x/oauth2/@v/list http://127.0.0.1:8000/golang.org/x/oauth2/@v/list), or use the server as a proxy, instead of making requests to it directly. ```
      - pimterry 5y ago
        That's really interesting! That's what appears if you HTTP Toolkit receives a request, proxies to the provided destination, and then finds itself on the other end of its own outgoing socket. This avoids the simplest infinite proxy loops. That'll make if you try to proxy a request to `127.0.0.1:8000` via HTTP Toolkit for example. That shouldn't be happening here though - there's no way Go should be using the proxy address as its target URL. I.e. to see this result, it must be sending a request like: GET /golang.org/x/oauth2/@v/list Host: 127.0.0.1:8000 You can probably see the full details in HTTP Toolkit's request data. Any chance you've added a rule in HTTP Toolkit that's redirecting traffic back to itself? That could do this, and would show up as two identical requests in HTTP Toolkit. If not (you can reset the rules by hitting the trash icon on the mock page) then this means something in your Go setup is either configured very strangely or its interpreting HTTP Toolkit's configuration incorrectly. Sorry I can't be more helpful, I have no idea why `go get` would send a request like that! If you can pin it down to something I can reproduce (e.g. a specific Go version or config setting or something) than I'm happy to investigate further, but it seems to work correctly everywhere I've tested so far.

pimterry 5y ago

I'm the author. I'm not a Go developer though, can you give me an example I can reproduce for `go get` that doesn't work for you? For the terminal, there's a few mechanisms, but environment variables are the catch-all there, yes (full list: https://github.com/httptoolkit/httptoolkit-server/blob/master/src/interceptors/terminal/terminal-env-overrides.ts https://github.com/httptoolkit/httptoolkit-server/blob/maste...). Those do work for most cases, but it is absolutely not a hard guarantee for applications that actively ignore standard proxy configuration (handling that is very hard, and definitely out of scope here). Go does generally observes `http_proxy` correctly by default in other cases I've tested, so this vert simple code from the test suite is automatically intercepted for example: https://github.com/httptoolkit/httptoolkit-server/blob/master/test/fixtures/docker/go/app.go https://github.com/httptoolkit/httptoolkit-server/blob/maste.... Very happy to look into any failing cases you can share.

IceWreck 5y ago
Yes, go based programs observe the proxy variable unless specifically compiled not to. But that doesn't break applications, they just ignore the env var. Go's default package manager doesn't seem to work at all with HTTP Toolkit. Here's what I did. * Intercept Tab > Fresh terminal. * In the terminal, do your usual stuff, it intercepts curl, etc. * If you try to use go's package manager, example: `go get golang.org/x/oauth2` It errors out with ``` go get: module golang.org/x/oauth2: reading http://127.0.0.1:8000/golang.org/x/oauth2/@v/list http://127.0.0.1:8000/golang.org/x/oauth2/@v/list: 500 Server error ``` Ideally it shouldn't break the application, just ignore if it can't intercept. > absolutely not a hard guarantee for applications that actively ignore standard proxy configuration (handling that is very hard, and definitely out of scope here). I encountered a usecase where this was needed and LD_PRELOAD trick (used by proxychains) etc failed because the application was statically compiled. I ended up using https://github.com/hmgle/graftcp https://github.com/hmgle/graftcp which somehow manages to force tcp traffic through a socks5 proxy.
- pimterry 5y ago
  Thanks, that's super useful. > If you try to use go's package manager, example: `go get golang.org/x/oauth2` I just tested, and `go get golang.org/x/oauth2` seems to work fine for me, I can see all the requests being happily intercepted immediately: https://imgur.com/a/Cb1y9Q2 https://imgur.com/a/Cb1y9Q2 Can you see the 500 in HTTP Toolkit, and any more info there (in the body or as an error at the top) related to that? Or can you see a "certificate rejected" message? If nothing turns up there at all then yes, something must be overriding the proxy configuration. Maybe you have some other Go package manager configuration that conflicts with this? I'd be very interested to know about that if so, I'm sure there's others with the same thing. It's always very hard to know if my configuration is representative of normal devs for any given language/tool. Probably best to debug this outside of a HN thread though :-). You can file a proper issue about this at https://github.com/httptoolkit/httptoolkit/issues/new https://github.com/httptoolkit/httptoolkit/issues/new, I'd love to know what's going on there and get this fixed. > I ended up using https://github.com/hmgle/graftcp https://github.com/hmgle/graftcp which somehow manages to force tcp traffic through a socks5 proxy. Really interesting, thanks! I'll look into that.
  - IceWreck 5y ago
    This is the error I see in the response body in HTTP Toolkit. ``` Error: Passthrough loop detected. This probably means you're sending a request directly to a passthrough endpoint, which is forwarding it to the target URL, which is a passthrough endpoint, which is forwarding it to the target URL, which is a passthrough endpoint... You should either explicitly mock a response for this URL (http://127.0.0.1:8000/golang.org/x/oauth2/@v/list http://127.0.0.1:8000/golang.org/x/oauth2/@v/list), or use the server as a proxy, instead of making requests to it directly. ```
    - pimterry 5y ago
      That's really interesting! That's what appears if you HTTP Toolkit receives a request, proxies to the provided destination, and then finds itself on the other end of its own outgoing socket. This avoids the simplest infinite proxy loops. That'll make if you try to proxy a request to `127.0.0.1:8000` via HTTP Toolkit for example. That shouldn't be happening here though - there's no way Go should be using the proxy address as its target URL. I.e. to see this result, it must be sending a request like: GET /golang.org/x/oauth2/@v/list Host: 127.0.0.1:8000 You can probably see the full details in HTTP Toolkit's request data. Any chance you've added a rule in HTTP Toolkit that's redirecting traffic back to itself? That could do this, and would show up as two identical requests in HTTP Toolkit. If not (you can reset the rules by hitting the trash icon on the mock page) then this means something in your Go setup is either configured very strangely or its interpreting HTTP Toolkit's configuration incorrectly. Sorry I can't be more helpful, I have no idea why `go get` would send a request like that! If you can pin it down to something I can reproduce (e.g. a specific Go version or config setting or something) than I'm happy to investigate further, but it seems to work correctly everywhere I've tested so far.