8 ms·

Show HN: Privaxy – Adblocking / tracker blocking by MITMing HTTPS traffic

86 points by pierrebarre 4y ago

pkulak 4y ago
What does it mean when: "The service may not tolerate TLS interception." I figured the proxy would be making the request entirely independently. How would an external entity even know the data was later being passed on?
- mhio 4y ago
  TLS connections are tunnelled through proxies directly to the endpoint (HTTP CONNECT method) rather than the "client request to proxy" followed by "proxy request to endpoint" method of proxying. This remote interception then involves turning a CONNECT back into the classic proxy connection. First a TLS session from your client to the proxy, then a TLS session from the proxy to the real endpoint. The proxy needs to present itself to the client as valid for the real endpoint of the TLS connection. This is usually done by adding your own CA into the clients trust so you can sign any certificates required for the client -> proxy half. As you note, the connection from proxy -> endpoint is normally the easy part of that as it works like a normal client. Two examples of not "tolerating" that interception are certificate pinning and client certificates. Certificate pinning - The client validates extra information about the presented certificate beyond CA trust. Usually the x509 SHA-256 digest presented to the client. In this case the external entity doesn't enforce anything, you could modify the client to work. Client certificates - Client cert authentication includes verification of the server certificate, so the forged proxy certificate will not be valid for the client cert. They are a pair. This would require a forged client cert for client -> proxy. Then the real client cert for proxy -> endpoint half. So it's more convincing the client to tolerate the interception rather than the external endpoint.
  - pkulak 4y ago
    Interesting. Thanks!
hereme888 4y ago
Some people seem to be saying that apps and devices bypass your DNS settings. If I set NextDNS with DoT in my Android under the "private DNS" setting, and turn on the NextDNS setting with DNS rebinding protection, would the phone and some apps still find a way around it? I also use NetGuard, but it's more cumbersome and doesn't allow DoT.
- mhio 4y ago
  It's possible. Applications don't have to rely on the OS provided mechanisms to lookup names, or even rely on DNS to get an IP for something. Chromium contains its own DNS resolver so connects directly to a DNS server rather than use the OS, but it would normally default to your OS settings (and only use DoH when they find a matching entry in their list of DoH providers). Desktop Firefox is an example of an app that defaults to DoH from 1.1.1.1 (in some places).
bilekas 4y ago
> Privaxy is also way more capable than DNS-based blockers as it is able to operate directly on URLs and to inject resources into web pages. I'm not sure I understand why it would be more capable than a DNS blocker ? If it's just because you can inject into the traffic that's comparing apples and oranges ? Or am I missing something ?
- ThePhysicist 4y ago
  Because you can modify HTML and other resources on the fly, i.e. you can remove tracker scripts before they would even be able to send stuff to a third party.
- captn3m0 4y ago
  Let’s say a text based ad shows up in a div with the id “advert”. A DNS based blocker will not be able to block it, but an extension or a proxy based blocker that looks at the HTML content will be able to block it. So yeah, inject as well as as modify the HTML directly. It could do things like shimming advertising libraries as well defanging them potentially.
  - bilekas 4y ago
    Okay, that makes a bit more sense now actually!
  - sumtechguy 4y ago
    To add to that. DNS block is basically 'built in' for this type of filtering as you can just make your filter strings your list of DNS sites. It does have the downside that not everything is http. That is where a real DNS filter comes into play with known malicious endpoints. So a combination is very nice to have.
- Septem9er 4y ago
  Simply because it isn't always enough to look at the domain to decide if it should be filtered (for serving ads or whatever). That's one reasons why DNS blockers can filter less effectively than e.g. browser addons. So yes, the reason is exactly as stated in the quote. It is more capable because it can operate on URLs and on the resources of the website directly.
teddyh 4y ago
I fear that MITMing ads is a dead end: 1. IIUC, when SNI is encrypted (in TLS 1.3?) almost everything is out the window. 2. Local devices can do DNS over HTTPS (DoH) and DNS over QUIC (DoQ) to look up their stuff, so DNS-based blocking will soon be obsolete. 3. The browser itself is controlled by the biggest ad-vendor around (Google), so you’ll probably get no help there. The only solutions are: A. Use browsers not controlled by Google (i.e. not any Chrome fork either). B. Use only apps and devices locally which do not display ads. (This is, in a way, a generalization of A.) C. Legislate away the business models of ads and the media and “smart” devices which use ads. (A very similar argument can be made for user tracking and telemetry.)
- baxuz 4y ago
  Does AdGuard also use this approach? https://kb.adguard.com/en/general/how-malware-protection-works https://kb.adguard.com/en/general/how-malware-protection-wor...
- fomine3 4y ago
  4. Deliver ads from same host as content (like Twitter, YouTube)
- deleted 4y ago
  [deleted]
- ThePhysicist 4y ago
  In my understanding ECH/ESNI shouldn't be an issue in this setup as long as the browser issues a domain-specific CONNECT request (i.e. "CONNECT google.com" instead of "CONNECT 24.154.13.11"). I think even with ECH enabled you should be able to impersonate the web server if you have a valid root CA certificate in the browsers' trust store. Remember, you're not performing "hostile" MITM-ing, but explicitly configure a proxy and root certificate in your browser. DNS shouldn't be an issue either as the browser leaves domain resolution to the proxy.
  - teddyh 4y ago
    This is, of course, assuming that you can trust the browser to obey its proxy settings. (And proxy setting do not apply at all to local “smart” devices.)
ThePhysicist 4y ago
I really like this, built something similar in Golang a while ago (not open-source for various reasons). In general it's a good approach I think, you can also inject JS that can do additional stuff in the browser to suppress tracking/ads.
2Gkashmiri 4y ago
Why build something fresh and not join forces with pihole? Reinventing the wheel for a niche function doesn't get traction much. I don't know the reason why the devs of this project think they need to start afresh, there are already tools like Firefox+unlock origin+ pihole which should solve most if not all of the problems. Why not incorporate the defining feature into pihole so that people don't have to add more complexity? Do I switch off my pihole and set this up?
- autoexec 4y ago
  > Why build something fresh and not join forces with pihole? Reinventing the wheel for a niche function doesn't get traction much. What harm does it do? Sure, some combination of 3 or more other things might give you most of the same functionally but why shouldn't people have the option to chose whichever works best for them? Even if the capabilities were 100% identical it's still worth it because it gives you an option if the thing you're using goes evil or stops updating or turns out to contain a vulnerability that takes months to fix etc. Even better, it could lead to innovation. Maybe Privaxy does something better than pihole does, or has some nice feature they don't and pihole sees it, loves the idea, and makes that improvement or adds that feature too and suddenly everybody is better off. Maybe just having competition helps improve things. I'm really struggling to understand how anyone loses here, or why it's preferable to have our options limited.
- dredmorbius 4y ago
  Does PiHole do anything other than DNS-based blocking?
  - 2Gkashmiri 4y ago
    i dont know. my point is the "fragmentation" thing
    - dredmorbius 4y ago
      As Privaxy includes blocklists, I'd argue that it is a superset of PiHole functionality. DNS blocklisting is actually pretty straightforward, and there are many tools which do it. PiHole is only one. That said, which would be better suited to incorporate the other is an interesting question.
Saint_Genet 4y ago
Used to run privoxy back in the day, but stopped when adblock extensions came along. It was simply more convenient to manage adblocking from the browser rather than figuring out regexps to put in its config. Also, it didn't do https.
adamzochowski 4y ago
There was a proxy, proxomitron in early 2000s, that allowed you to change the html/js as it went through the proxy. people used it for adblocking and removing page annoyances, like removing sounds / animated gifs / etc. Here is a list of random old filters people had built at one time: https://proxomitron.info/45/help/Default-Web-Filters.html https://proxomitron.info/45/help/Default-Web-Filters.html
- dredmorbius 4y ago
  There were numerous of these. Privoxy, dansguardian, Squid (AFAIR), and others. The notion that SSL/TLS means that ONLY the webserver origin and web browser client are permitted to see or mitigate content ... is itself harmful. Trusted proxies under your control do have a place, though yes, that introduces new points of contention as well.
  - captn3m0 4y ago
    I used to swear by Privoxy till the internet realised HTTPS was actually important and it stopped working everywhere.
    - dredmorbius 4y ago
      Largely the same. Privaxy actually looks pretty sweet in that regard.
sidpatil 4y ago
Not to be confused with Privoxy: https://www.privoxy.org/ https://www.privoxy.org/
- dredmorbius 4y ago
  My understanding is that Privoxy either cannot deal with SSL/TLS traffic, or deals very poorly with it. The FAQ doesn't seem to discuss the issue at all, which is not a good sign: https://www.privoxy.org/faq/index.html https://www.privoxy.org/faq/index.html
mhils 4y ago
This approach is a natural escalation step as DNS-based blocking is getting increasingly difficult. But it's not without its drawbacks. For example, browsers tend to have by far the best TLS implementations. By MITMing yourself, you essentially trust the proxy's TLS implementation instead, which will receive much less scrutiny. There's a lot of precedent for TLS vulnerabilities introduced by middleboxes. If browser extensions are possible they should be preferred. But the author does have a point that this can't be taken for granted anymore!
- randomhodler84 4y ago
  Why is DNS based blocking getting difficult? You run a bind server and tell it what it can and cannot resolve. It can even listen on DoH so you get transport security between peer and local dns server.
  - rsync 4y ago
    Your browser (or your tv) can just skip your entire dns infra and make its own lookups over https- which you won’t see. That’s the evil genius of doh- you can’t block 443 and their “dns server” could be the same hostname as the site you visit … and now we’re discussing mitm’ing ourselves… Sigh.
    - randomhodler84 4y ago
      Could, but do? I have never seen DNS or DOH pinning. Seems fragile. Would likely fall back to host resolver anyway.
  - mhils 4y ago
    AdTech increasingly uses CNAME cloaking-style tricks to evade DNS blocking. Some of those tricks are detectable, but DNS blocking will inevitably fail once ads are served from the first party domain. It's still rare, but simple CNAME cloaks specifically have seen an uptick in the last few years.
- trasz 4y ago
  A TLS proxy is something that’s trivially easy to sandbox; a browser is the exact opposite.
cal85 4y ago
What are the potential benefits of a ‘MITM’ approach, compared to other approaches like acting as DNS (like pihole)? Edit: I should have read the About section more carefully: > Privaxy is also way more capable than DNS-based blockers as it is able to operate directly on URLs and to inject resources into web pages. Makes sense. So it potentially has the fine-grained control of a browser-based blocker but also has good performance like a pihole. Sounds compelling. Now I’m interested to know why it’s not been done this way before? Is it just a hard problem to solve, and no one has attempted it yet?
- randomhodler84 4y ago
  It’s been done for years and years but it’s considered a very bad idea these days. MITM https sessions is a trivial problem today. It’s just a bad idea as it breaks the entire trust model of the internet. Most commercial firewalls for the last decade plus have such features.
geoffeg 4y ago
I've really wanted a server-side uBlock Origin like this for a while now for devices that can't run uBlock (mobile, etc) or where uBlock is limited in functionality (Chrome). This looks like a great start.
- itintheory 4y ago
  In case you weren't aware, firefox on android can run uBlock Origin without root or any other modifications. This proxy would be nice to have system level ad blocking though!
randomhodler84 4y ago
I said it before and I will say it again, MITM for ad blocking is not a way forward. Cert pinning defeats this on 99% of consumer devices and introduces a security hole in the browser by subverting the trust model. Unless the proxy is doing 100% of the same thing the browser is doing, and it isn’t, you are weakening browser security too. Instrument the endpoint (browser plug-in) or control name resolution (filtering DNS server that uses DoH to prevent upstream filtering).
- gumby 4y ago
  The problem with browser plug ins is that they only work in browsers. I read most html or other "web pages" in programs other than browsers (mail client, RSS readers, Electron apps, etc)
- pkulak 4y ago
  It's not about this being some end-all solution, it's about it being an option. Personally, I love it. I used to use Privoxy, back when nothing was encrypted, and it was wonderful. A central place to store all my ad-blocking config that could be connected to at will by most devices on my network. I mostly have that now with DNS blocking, but once ad networks stop putting ads on separate domains, that's done. Keep in mind that ad-blocking browser plugins aren't exactly secure either. They have access, not only to every network request, but every keystroke, mouse wiggle, etc. And all it takes to all fall down is for whoever is maintaining it to cash out and sell to a bad actor: you'll helpfully be automatically updated to the new, state-owned version.
idrock 4y ago
I used to deploy privoxy everywhere and loved the ability to intercept and script just about everything... will def check it out.

86 points by pierrebarre 4y ago

What does it mean when: "The service may not tolerate TLS interception." I figured the proxy would be making the request entirely independently. How would an external entity even know the data was later being passed on?

mhio 4y ago
TLS connections are tunnelled through proxies directly to the endpoint (HTTP CONNECT method) rather than the "client request to proxy" followed by "proxy request to endpoint" method of proxying. This remote interception then involves turning a CONNECT back into the classic proxy connection. First a TLS session from your client to the proxy, then a TLS session from the proxy to the real endpoint. The proxy needs to present itself to the client as valid for the real endpoint of the TLS connection. This is usually done by adding your own CA into the clients trust so you can sign any certificates required for the client -> proxy half. As you note, the connection from proxy -> endpoint is normally the easy part of that as it works like a normal client. Two examples of not "tolerating" that interception are certificate pinning and client certificates. Certificate pinning - The client validates extra information about the presented certificate beyond CA trust. Usually the x509 SHA-256 digest presented to the client. In this case the external entity doesn't enforce anything, you could modify the client to work. Client certificates - Client cert authentication includes verification of the server certificate, so the forged proxy certificate will not be valid for the client cert. They are a pair. This would require a forged client cert for client -> proxy. Then the real client cert for proxy -> endpoint half. So it's more convincing the client to tolerate the interception rather than the external endpoint.
- pkulak 4y ago
  Interesting. Thanks!

hereme888 4y ago

Some people seem to be saying that apps and devices bypass your DNS settings. If I set NextDNS with DoT in my Android under the "private DNS" setting, and turn on the NextDNS setting with DNS rebinding protection, would the phone and some apps still find a way around it? I also use NetGuard, but it's more cumbersome and doesn't allow DoT.

mhio 4y ago
It's possible. Applications don't have to rely on the OS provided mechanisms to lookup names, or even rely on DNS to get an IP for something. Chromium contains its own DNS resolver so connects directly to a DNS server rather than use the OS, but it would normally default to your OS settings (and only use DoH when they find a matching entry in their list of DoH providers). Desktop Firefox is an example of an app that defaults to DoH from 1.1.1.1 (in some places).

bilekas 4y ago

> Privaxy is also way more capable than DNS-based blockers as it is able to operate directly on URLs and to inject resources into web pages. I'm not sure I understand why it would be more capable than a DNS blocker ? If it's just because you can inject into the traffic that's comparing apples and oranges ? Or am I missing something ?

ThePhysicist 4y ago
Because you can modify HTML and other resources on the fly, i.e. you can remove tracker scripts before they would even be able to send stuff to a third party.
captn3m0 4y ago
Let’s say a text based ad shows up in a div with the id “advert”. A DNS based blocker will not be able to block it, but an extension or a proxy based blocker that looks at the HTML content will be able to block it. So yeah, inject as well as as modify the HTML directly. It could do things like shimming advertising libraries as well defanging them potentially.
- bilekas 4y ago
  Okay, that makes a bit more sense now actually!
- sumtechguy 4y ago
  To add to that. DNS block is basically 'built in' for this type of filtering as you can just make your filter strings your list of DNS sites. It does have the downside that not everything is http. That is where a real DNS filter comes into play with known malicious endpoints. So a combination is very nice to have.
Septem9er 4y ago
Simply because it isn't always enough to look at the domain to decide if it should be filtered (for serving ads or whatever). That's one reasons why DNS blockers can filter less effectively than e.g. browser addons. So yes, the reason is exactly as stated in the quote. It is more capable because it can operate on URLs and on the resources of the website directly.

teddyh 4y ago

I fear that MITMing ads is a dead end: 1. IIUC, when SNI is encrypted (in TLS 1.3?) almost everything is out the window. 2. Local devices can do DNS over HTTPS (DoH) and DNS over QUIC (DoQ) to look up their stuff, so DNS-based blocking will soon be obsolete. 3. The browser itself is controlled by the biggest ad-vendor around (Google), so you’ll probably get no help there. The only solutions are: A. Use browsers not controlled by Google (i.e. not any Chrome fork either). B. Use only apps and devices locally which do not display ads. (This is, in a way, a generalization of A.) C. Legislate away the business models of ads and the media and “smart” devices which use ads. (A very similar argument can be made for user tracking and telemetry.)

baxuz 4y ago
Does AdGuard also use this approach? https://kb.adguard.com/en/general/how-malware-protection-works https://kb.adguard.com/en/general/how-malware-protection-wor...
fomine3 4y ago
4. Deliver ads from same host as content (like Twitter, YouTube)
deleted 4y ago
[deleted]
ThePhysicist 4y ago
In my understanding ECH/ESNI shouldn't be an issue in this setup as long as the browser issues a domain-specific CONNECT request (i.e. "CONNECT google.com" instead of "CONNECT 24.154.13.11"). I think even with ECH enabled you should be able to impersonate the web server if you have a valid root CA certificate in the browsers' trust store. Remember, you're not performing "hostile" MITM-ing, but explicitly configure a proxy and root certificate in your browser. DNS shouldn't be an issue either as the browser leaves domain resolution to the proxy.
- teddyh 4y ago
  This is, of course, assuming that you can trust the browser to obey its proxy settings. (And proxy setting do not apply at all to local “smart” devices.)

ThePhysicist 4y ago

I really like this, built something similar in Golang a while ago (not open-source for various reasons). In general it's a good approach I think, you can also inject JS that can do additional stuff in the browser to suppress tracking/ads.

2Gkashmiri 4y ago

Why build something fresh and not join forces with pihole? Reinventing the wheel for a niche function doesn't get traction much. I don't know the reason why the devs of this project think they need to start afresh, there are already tools like Firefox+unlock origin+ pihole which should solve most if not all of the problems. Why not incorporate the defining feature into pihole so that people don't have to add more complexity? Do I switch off my pihole and set this up?

autoexec 4y ago
> Why build something fresh and not join forces with pihole? Reinventing the wheel for a niche function doesn't get traction much. What harm does it do? Sure, some combination of 3 or more other things might give you most of the same functionally but why shouldn't people have the option to chose whichever works best for them? Even if the capabilities were 100% identical it's still worth it because it gives you an option if the thing you're using goes evil or stops updating or turns out to contain a vulnerability that takes months to fix etc. Even better, it could lead to innovation. Maybe Privaxy does something better than pihole does, or has some nice feature they don't and pihole sees it, loves the idea, and makes that improvement or adds that feature too and suddenly everybody is better off. Maybe just having competition helps improve things. I'm really struggling to understand how anyone loses here, or why it's preferable to have our options limited.
dredmorbius 4y ago
Does PiHole do anything other than DNS-based blocking?
- 2Gkashmiri 4y ago
  i dont know. my point is the "fragmentation" thing
  - dredmorbius 4y ago
    As Privaxy includes blocklists, I'd argue that it is a superset of PiHole functionality. DNS blocklisting is actually pretty straightforward, and there are many tools which do it. PiHole is only one. That said, which would be better suited to incorporate the other is an interesting question.

Saint_Genet 4y ago

Used to run privoxy back in the day, but stopped when adblock extensions came along. It was simply more convenient to manage adblocking from the browser rather than figuring out regexps to put in its config. Also, it didn't do https.

adamzochowski 4y ago

There was a proxy, proxomitron in early 2000s, that allowed you to change the html/js as it went through the proxy. people used it for adblocking and removing page annoyances, like removing sounds / animated gifs / etc. Here is a list of random old filters people had built at one time: https://proxomitron.info/45/help/Default-Web-Filters.html https://proxomitron.info/45/help/Default-Web-Filters.html

dredmorbius 4y ago
There were numerous of these. Privoxy, dansguardian, Squid (AFAIR), and others. The notion that SSL/TLS means that ONLY the webserver origin and web browser client are permitted to see or mitigate content ... is itself harmful. Trusted proxies under your control do have a place, though yes, that introduces new points of contention as well.
- captn3m0 4y ago
  I used to swear by Privoxy till the internet realised HTTPS was actually important and it stopped working everywhere.
  - dredmorbius 4y ago
    Largely the same. Privaxy actually looks pretty sweet in that regard.

sidpatil 4y ago

Not to be confused with Privoxy: https://www.privoxy.org/ https://www.privoxy.org/

dredmorbius 4y ago
My understanding is that Privoxy either cannot deal with SSL/TLS traffic, or deals very poorly with it. The FAQ doesn't seem to discuss the issue at all, which is not a good sign: https://www.privoxy.org/faq/index.html https://www.privoxy.org/faq/index.html

mhils 4y ago

This approach is a natural escalation step as DNS-based blocking is getting increasingly difficult. But it's not without its drawbacks. For example, browsers tend to have by far the best TLS implementations. By MITMing yourself, you essentially trust the proxy's TLS implementation instead, which will receive much less scrutiny. There's a lot of precedent for TLS vulnerabilities introduced by middleboxes. If browser extensions are possible they should be preferred. But the author does have a point that this can't be taken for granted anymore!

randomhodler84 4y ago
Why is DNS based blocking getting difficult? You run a bind server and tell it what it can and cannot resolve. It can even listen on DoH so you get transport security between peer and local dns server.
- rsync 4y ago
  Your browser (or your tv) can just skip your entire dns infra and make its own lookups over https- which you won’t see. That’s the evil genius of doh- you can’t block 443 and their “dns server” could be the same hostname as the site you visit … and now we’re discussing mitm’ing ourselves… Sigh.
  - randomhodler84 4y ago
    Could, but do? I have never seen DNS or DOH pinning. Seems fragile. Would likely fall back to host resolver anyway.
- mhils 4y ago
  AdTech increasingly uses CNAME cloaking-style tricks to evade DNS blocking. Some of those tricks are detectable, but DNS blocking will inevitably fail once ads are served from the first party domain. It's still rare, but simple CNAME cloaks specifically have seen an uptick in the last few years.
trasz 4y ago
A TLS proxy is something that’s trivially easy to sandbox; a browser is the exact opposite.

cal85 4y ago

What are the potential benefits of a ‘MITM’ approach, compared to other approaches like acting as DNS (like pihole)? Edit: I should have read the About section more carefully: > Privaxy is also way more capable than DNS-based blockers as it is able to operate directly on URLs and to inject resources into web pages. Makes sense. So it potentially has the fine-grained control of a browser-based blocker but also has good performance like a pihole. Sounds compelling. Now I’m interested to know why it’s not been done this way before? Is it just a hard problem to solve, and no one has attempted it yet?

randomhodler84 4y ago
It’s been done for years and years but it’s considered a very bad idea these days. MITM https sessions is a trivial problem today. It’s just a bad idea as it breaks the entire trust model of the internet. Most commercial firewalls for the last decade plus have such features.

geoffeg 4y ago

I've really wanted a server-side uBlock Origin like this for a while now for devices that can't run uBlock (mobile, etc) or where uBlock is limited in functionality (Chrome). This looks like a great start.

itintheory 4y ago
In case you weren't aware, firefox on android can run uBlock Origin without root or any other modifications. This proxy would be nice to have system level ad blocking though!

randomhodler84 4y ago

I said it before and I will say it again, MITM for ad blocking is not a way forward. Cert pinning defeats this on 99% of consumer devices and introduces a security hole in the browser by subverting the trust model. Unless the proxy is doing 100% of the same thing the browser is doing, and it isn’t, you are weakening browser security too. Instrument the endpoint (browser plug-in) or control name resolution (filtering DNS server that uses DoH to prevent upstream filtering).

gumby 4y ago
The problem with browser plug ins is that they only work in browsers. I read most html or other "web pages" in programs other than browsers (mail client, RSS readers, Electron apps, etc)
pkulak 4y ago
It's not about this being some end-all solution, it's about it being an option. Personally, I love it. I used to use Privoxy, back when nothing was encrypted, and it was wonderful. A central place to store all my ad-blocking config that could be connected to at will by most devices on my network. I mostly have that now with DNS blocking, but once ad networks stop putting ads on separate domains, that's done. Keep in mind that ad-blocking browser plugins aren't exactly secure either. They have access, not only to every network request, but every keystroke, mouse wiggle, etc. And all it takes to all fall down is for whoever is maintaining it to cash out and sell to a bad actor: you'll helpfully be automatically updated to the new, state-owned version.

idrock 4y ago

I used to deploy privoxy everywhere and loved the ability to intercept and script just about everything... will def check it out.