8 ms·
I've been MITM'ing my traffic to, among other things, strip ads and rewrite pages with things like custom CSS, ever since I discovered The Proxomitron over 2 de
by userbinator 3y ago
I've been MITM'ing my traffic to, among other things, strip ads and rewrite pages with things like custom CSS, ever since I discovered The Proxomitron over 2 decades ago. It does tend to get me profiled as a "bot" by CloudFlare and such, but there are not-so-trivial ways around that too. It also shows why things like remote attestation are hazardous for user freedom.
- greyface- 3y ago> It does tend to get me profiled as a "bot" by CloudFlare Since Cloudflare employees are known to lurk here, I'd like to know: is this considered a false positive, or working as intended?
- rs_rs_rs_rs_rs 3y agoNot a CF employee but I am pretty sure it's working as intended as they built and use this: https://github.com/cloudflare/mitmengine https://github.com/cloudflare/mitmengine
- fulafel 3y agoHere's hoping that, if they do it, they'll start blocking corporate TLS mitm regimes too.
- michaelt 3y agoBlocking? Cloudflare provides corporate TLS MITM regimes.
- sgammon 3y ago"Corporate MITM" is built into the very idea of PKI. Running your own custom PKI is a completely valid way to operate a network. This is a feature because it allows anyone to establish their own trust tree, completely outside of the public certificate trust network. "Regimes" sounds pejorative but in truth, companies have a duty and in many cases a legal obligation to protect their networks. Prima facie, I don't see any reason at all why interception of traffic in this circumstance is "bad," except maybe a potential for political misuse like any other written medium. I actually think the reverse would be substantially worse: if _only_ the public trust chain was valid in major browsers, we would be completely hosed and there would be no distinguishing factor at all between remote attestation and trust. Thus, corporate TLS interception is, at worst, a necessary byproduct of a very well chosen tradeoff.
- userbinator 3y agoAnd as a personal user, I also should have the right to intercept the traffic on my network.
- josephcsible 3y agoI have a minor quibble with "my network". You should have the right to intercept the traffic originating or terminating at your devices, but not to intercept any traffic going between other people's devices just because it's on your network.
- userbinator 3y agoThen it wouldn't be my network anymore.
- josephcsible 3y agoShould your ISP be able to MITM all of your traffic just because it's going over their network? If not, then what's the difference?
- sgammon 3y agoThat's what I'm standing up for in this comment, but you've characterized it as opposition. I don't understand, perhaps you'd like to clarify.
- userbinator 3y agoI'm not opposing, just adding more context.
- rightbyte 3y agoEver since Google started pushing HTTPS I understood that non-encryption would be a future privacy problem.
- yrro 3y agoWhy? This isn't the 90s any more. You shouldn't use your employer's systems for anything other than work.
- hedora 3y agoWhy should they only allow MItM’ed traffic when a third party is the one defeating the encryption?
- netsharc 3y agoHah, remember how Amazon invented "Cyber-Monday"? They claim the Monday after Thanksgiving when people go back to work it'll be their first encounter with Internet-connected computers after the Thanksgiving vacation, and it's the time where they'll be most busy shopping for Christmas presents...
- yrro 3y agoSo that's where thta came from...
- mynameisvlad 3y agoAmazon was barely a blip when Cyber Monday first came out, let alone the inventors of the term. > The term "Cyber Monday" was coined by Ellen Davis, and was first used within the ecommerce community during the 2005 holiday season. According to Scott Silverman, the head of Shop.org, the term was coined based on 2004 research showing "one of the biggest online shopping days of the year" was the Monday after Thanksgiving (12th-biggest day historically). Retailers also noted the most significant shopping period was December 5 through 15 of the previous year. In late November 2005, The New York Times reported: "The name Cyber Monday grew out of the observation that millions of otherwise productive working Americans, fresh off a Thanksgiving weekend of window shopping, were returning to high-speed Internet connections at work Monday and buying what they liked." At the time, a lot of people had slow Internet at home. The idea for having such a holiday was created by Tony Valado, in 2003 while working at 1800Flowers.com, and coined "White Wednesday" to be the day before Thanksgiving for online retailers. https://en.wikipedia.org/wiki/Cyber_Monday https://en.wikipedia.org/wiki/Cyber_Monday
- cbsmith 3y agoCorporate TLS mitm is so often misunderstood. There are, unfortunately, a lot of legit reasons to do so.
- thegeomaster 3y agoUse this to evade it: https://github.com/lwthiker/curl-impersonate https://github.com/lwthiker/curl-impersonate Greetings from a country whose (almost) entire IP space is blacklisted by Cloudflare!
- issafram 3y agoWhich country?
- kyrra 3y agoFrom his profile: > Software engineer from Belgrade, Serbia
- SOLAR_FIELDS 3y agoWow thanks, I was looking for this exact piece of software a few weeks ago when testing against some geo restrictions
- pmarreck 3y agonecessity: still the mother of invention greetings from the other side of the world, commandline brother. I did not know of this tool, but I do now. Thanks
- foobarbazetc 3y agoWow. Need this as a golang net/http shim.
- rollcat 3y ago> We consider an HTTPS connection to be intercepted when there is a mismatch between the expected client request signature corresponding to the browser identified by the User Agent, and the actual client request fingerprint of the request. Sounds like something that would be "trivial" to defeat, by means of "emulating" other TLS implementations more closely?
- homarp 3y agowhich is what https://github.com/lwthiker/curl-impersonate https://github.com/lwthiker/curl-impersonate does: it's a special build of curl that can impersonate Chrome & Firefox
- nwellinghoff 3y agoCan this be used as a proxy to your real browser? Can you paint the whole picture/setup for a noob?
- cbsmith 3y agoPresumably so. You'd just need to compile the curl-impersonate for the appropriate browser you want to impersonate, and then link it in to a proxy that used libcurl for its https traffic.
- voytec 3y agoI have no releation to CF but from my perspective, if a human is marked as a bot - that's a false positive.
- aardvarkr 3y ago99.9999% of humans won’t be decrypting protobuf traffic but a large % of bots are out of necessity to interact with the website. That’s what I’d call a good filter.
- adrr 3y agoSince you’re not a Cloudflare customer, you need to think like a customer who is using Cloudflare’s bot protection. Do you want to filter people who are blocking your ads through mitm? I would guess yes.
- 1B05H1N 3y agoYou should be okay as long as you solve the challenge(s).
- 1vuio0pswjnm7 3y agoI also run a forward proxy and found adding a user-agent header was all that was necessary.
- brainbag 3y agowould you mind providing more details about this? Is it any user agent header, or do you need to do some thing like curl-impersonate?
- 1vuio0pswjnm7 3y agoNo, actually it just needed to be well-formed. UA headers follow a certain format. Some folks like to try to "blend in" and use known headers seen in the wild but personally I like to make up new UA headers that reveal nothing, not even fake details. On the whole, I very rarely send a UA header. The number of sites that demand one is very small for me. I had just assumed Cloudflare was doing TLS fingerprinting or something more involved than just checking for presence of a UA header. But it worked. The StackExchange family of sites is one example where a UA header is needed. They will block otherwise. Not using Cloudflare to do the blocking though.
- 1vuio0pswjnm7 3y agos/StackExchange/StackOverflow/
- faraggi 3y ago> among other things what else do you MITM for?
- 1vuio0pswjnm7 3y agoMITM is kind of a silly term for what the computer owner is doing when using a forward proxy. If the computer owner binds the proxy to a localhost address, the unencrypted requests need only go over the loopback. There need be no unencrypted requests travelling over the LAN. The computer owner, the "MITM", is on their own computer sitting in between an application and the network interface. That is exactly where they should be, IMHO. Corporations do it. They own the computer and they own their internal networks. It's no different for home users. And WFH is blurring the distinction anyway. Once the requests leave the computer and travel onto the internet destined for another computer, then of course "MITM" makes sense as a concept. We all want to prevent that. The computer owner controls the proxy and it's the proxy, not the untrusted application, like a "modern" web browser for example, that handles authentication of the remote peer. Compiling and fully controlling a "modern" browser is a PITA. Almost no one does it, even software developers. Instead people beg for an advertising company or their partner to make changes to a browser. That does not seem to work. Sometimes when people complain it stops the company from making undesired changes. But only temporarily. Whereas compiling a proxy is easy and the user can fully control it. Having used many different applications that implement support for TLS, I actually trust the proxy's implementation more than most applications. It's arguably easier to audit one program, the proxy, than it is to check every application to make sure the developer didn't make a mistake when adding TLS support. I recall socat as one example. That mistake went undetected for a long time. Elinks was another. At the time, it was dropped from OpenBSD ports as a result.
- docmars 3y agoFor any gamers out there, Reshade's installer is another example where TLS istn't implemented properly. It fails to download effect/shader packages for basic setup on up-to-date Windows machines because it requires that you disable TLS 1.3 globally on Windows so it can use 1.1 or 1.2. [1] https://reshade.me/forum/troubleshooting/8746-reshade-v-5-8-0-cant-download-effect-packages-any-ideas https://reshade.me/forum/troubleshooting/8746-reshade-v-5-8-...
- ttctciyf 3y ago> The Proxomitron Googling this, development ended in 2004? An informed summary of the current state of play would be interesting, since a lot of different "continuations" of it seem to be around. Also, is it Windows-only? (I've been casually looking for a simple proxy that would enable injection of local links into remote content.)
- d2323 3y agoPrivacy might be what you're after. Open source and cross platform.
- mrAssHat 3y agoThat's impossible to google.
- scoopertrooper 3y agoDidn't help that they spelled the name incorrectly, but I found it :D https://github.com/barre/privaxy https://github.com/barre/privaxy
- userbinator 3y agoIn this case "incorrectly" is what makes it possible to find it, search-engine-autocorrect-stupidity aside. The name is obviously a combination of privacy and proxy. There's another one called Privoxy which works on similar principles.
- d2323 3y agoApologies, I meant Privoxy. Privaxy looks pretty decent too, but I've never used it.
- Phemist 3y agoAuto-correct issue?
- deleted 3y ago[deleted]
- sneak 3y agoPresumably this involves installing new CA certs on your devices, no?
- userbinator 3y agoYes, and this also answers the other comment about what else I MITM for: blocking devices that I can't inspect the traffic of is another reason, since the proxy is the only way out of the network.
- pdimitar 3y agoPardon the lay question, but is it really the only way out of the network? Can't programs initiate connections while skipping the system-wide configured proxy?
- haraldooo 3y agoSure they can.. but they can’t get around your outgoing firewall rule that reroutes alle traffic for certain ports to the proxy.
- pdimitar 3y agoHm, I have to see if Mikrotik has rule syntax for this. I can already force every app who thinks they will use their own DNS server to use mine but not sure how I could do the same with a proxy. Maybe just force ports 80 and 443? But what's stopping these apps to communicate on non-standard ports?
- emidln 3y agoThere's no reason to allow arbitrary traffic in either direction other than convenience. If you want a more secure network, you block everything by default and narrowly open as needed.
- 3y ago
- slacka 3y agoNetwork level ad-blocking like Privoxy and pi-hole have so many downsides like they can't handle inline adverts. My pi4 pi-hole is unplugged right now because it just didn't work well. After hours wasted trying to make it work with all the services, I've thrown in the towel. The time involved is just not worth it for a home network. What works are browser based ad-blockers and app patchers like ReVanced. As my savings have gone up, I've relied more and more on just paying for ad-free services like YouTube Premium, Hulu, Netfix, and Max for the cases those 2 can't handle.
- KerrAvon 3y agoWhat services are you having trouble with? My RPi PiHole has been working great for whole-house blocking for a half-decade now.
- ryaneager 3y agoLikewise. My biggest tip is set your secondary DNS to something outside your networks so if your pi-hole goes down your entire network doesn’t go down you just get ads.
- smolyeet 3y agoThere’s also nextdns which I was planning on switching to anyways
- berniedurfee 3y agoI’ve been using NextDNS for a few years now. I don’t have time to mess with an ‘on prem’ solution. It works really well and is simple to manage. It runs on my mobile devices easily. I turn it off occasionally to shop, but immediately turn it back on as the internet is a whole different place without some sort of robust ad blocking. It’s nice to see the logs fill up with blocked telemetry and other crap all day.
- sizzle 3y agoIs it okay to use cloudflare dns?
- venusenvy47 3y agoI haven't thought about Proxomitron in about two decades - are you still using it? I never used it for anything you described, but it was great to use as a proxy with our company firewall. Many programs back in the day couldn't connect with the Internet because our firewall back then required login information to make outbound connections.
- BlindWanderer 3y agoI loved Proxomitron! I hadn't thought of using it to MITM YouTube!