15 ms·

The Quantum Ad-List contains domains used by ads, trackers, malware

158 points by URfejk 6y ago

gray_-_wolf 6y ago
Seems to be at least partially compilation of other lists, but without giving them fair credit. [0] does make some pretty convincing claims. 0: https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/issues/13 https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/i...
OneFunFellow 6y ago
Feedback: Perhaps add info about Pi-Hole usage.
- deleted 6y ago
  [deleted]
- pdimitar 6y ago
  Shouldn't just adding the URL[0] work? [0] https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/raw/master/For%20hosts%20file/The_Quantum_Ad-List.txt https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/r...
  - walshemj 6y ago
    Yes just login to the web interface go to group management and add that url - just done that on my pi hole running on a Model B Rev 2
    - pdimitar 6y ago
      Yep, just did the same. Thanks for confirming that it also works for you.
  - dangerboysteve 6y ago
    I was asking the same and then looked at my pihole and came to the same conclusion.
  - Merman_Mike 6y ago
    I had 1,217,258 blocked domains on my Pi-Hole to begin with, and this added 515,284 new domains.
    - pdimitar 6y ago
      Yes, I have seen a rough add of ~500K domains indeed. Had to whitelist api.instabug.com since with it blocked my 9GAG app goes crazy and tries to connect to it non-stop, lighting my phone on fire.
      - Merman_Mike 6y ago
        I'm surprised to see that there is any overlap between 9GAG and HN :)
        pdimitar 6y ago
        Next you'll tell me you are surprised that a 9gag and an HN regular gets laid! ;) In all seriousness, I casually browse it for 30-40 minutes a day. Helps put my mind at ease here and there (although it too became heavily politicized and not as much fun as it was years ago).
        justinlloyd 6y ago
        You leave him alone, he is a renowned Field Marshal, though as OP, also known to suck a lot of #!@% P.S. 9gag in-joke, you wouldn't get it.
    - yumraj 6y ago
      I went from 2.039 million to 2.043 million, so about 5,000 new domains.
gbh444g 6y ago
TBH, at this point we may as well start using whitelists: 1st party domains and known 3rd party CDNs for static content and maybe media, 1st party scripts for frequently visited sites and a special button hidden in a safe place to enable 3rd party scripts for those who want to live dangerously.
- ignoramous 6y ago
  That's uMatrix (at least in-browser)... Sadly it appears to have been discontinued by gorhill. IP whitelisting isn't fool-proof though. For example, I run an anti-censorship DoH resolver "IP fronted" by Cloudflare specifically to be a hard target to block at IP layer. I mean, you can block Cloudflare's IPs but that'd mean you'd end up blocking other websites too.
  - hosteur 6y ago
    UMatrix is vital. Did anyone pick up the project after gorhill?
    - ignoramous 6y ago
      Don't think u/gorhill would be inclined at the idea of handing over the reins after the uBlock.org fiasco. Forks may survive, but the extension looks to be all but dead.
- eli 6y ago
  Known 3rd party CDNs are used for all kinds of content from CSS needed to make a site work to ads and tracking scripts.
- CivBase 6y ago
  That's basically what I do with uMatrix.
  - nashashmi 6y ago
    Yup. I was going to post this too. Why keep 34000 site Peter Lowe’s ad blocker when a 50 site white list will do the same. But umatrix is complex. Ublock is good enough. I just wished I could whitelist all cdns
Merman_Mike 6y ago
I wish there were more details about how this list of domains was compiled. This is the explanation on the Gitlab: > We were testing an AI that could show some basic emotions about internet content, and turns out it was very precise at getting “annoyed” by ads and “unsolicited” third party connections… > From that, I forked our own project and tweaked it in a specific way to basically only focus on ads, trackers, etc. and act like a web crawler, turns out to be very effective!
- lightswitch05 6y ago
  I agree. I think its very likely 'import a ton of other block lists and then apply some sort of logic to whitelist potential problem domains', but I would love to be wrong.
- ComodoHacker 6y ago
  Explanation is what our current AI tech struggles with most.
caterama 6y ago
This really makes me wish there was a way to include files in /etc/hosts files...
- sebow 6y ago
  HostsMan is a good utility for windows, which from what i recall lets you use lists and even add update URLs; Apparently a less-fledged foss alternative is SwitchHosts! for linux, which also seem to have this crucial feature
- mech422 6y ago
  Just generate /etc/hosts from a cron job... Pretty easy to concat, sort, and filter however you like in a short script.
  - coldpie 6y ago
    You can also set up a systemd service to monitor files in a directory and refresh the /etc/hosts when they are changed.
- justinlloyd 6y ago
  If you are okay with a GUI on Windows, Mac or Linux, I can recommend Hozz. https://blog.zhangruipeng.me/Hozz/ https://blog.zhangruipeng.me/Hozz/ No longer under development but works just fine.
- TheQuantumAlpha 6y ago
  I made an installer...
deleted 6y ago
[deleted]
smashah 6y ago
Awesome work! For anyone interested in using The Quantum ad-list in their adguard setups, I've added this to my self-updating ad gaurd block list generator (smashblock - which is also based on hblock). https://github.com/smashah/smashblock https://github.com/smashah/smashblock
deleted 6y ago
[deleted]
sciencesama 6y ago
Is there a way we can block part of the subdomain, like a proxy firewall. Used to see this thing called mcafee web gateway. That used to block all ads,banner ads and even instagram ads !! Is there one such open source project out there ?? Where we can download and add the fanboy type block list to block the ad crap !!
- _wldu 6y ago
  DNS and IP blocks are like hitting a tiny finishing nail with a 10 pound hammer. Sure, you smash it, but you smash everything around it too. Try Squid. It's a dynamic higher layer HTTP/web proxy that can block specific URL content, etc. https://en.wikipedia.org/wiki/Squid_(software) https://en.wikipedia.org/wiki/Squid_(software)
secondcoming 6y ago
Interesting. My company is on there, and one of our domains listed is not used in any for ad serving. It's the URL of our customer portal.
TheQuantumAlpha 6y ago
I see... Thanks!!
TheQuantumAlpha 6y ago
Thus, some other ad-list holders are panicking that they may become irrelevant, therefore trying to silence us... They threaten to copystrike us as we have similar domains they have... Ads are ads, right?
TheQuantumAlpha 6y ago
https://bitbucket.org/The_Quantum_Alpha/the-quantum-ad-list/src/master/ https://bitbucket.org/The_Quantum_Alpha/the-quantum-ad-list/...
Havoc 6y ago
Having used it for a while, on thing it seems to pick up that the standard pihole lists don't is >clients.l.google.com >clients4.google.com Thousands of requests though so pushed up the overall block rate by a good 5%

158 points by URfejk 6y ago

Seems to be at least partially compilation of other lists, but without giving them fair credit. [0] does make some pretty convincing claims. 0: https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/issues/13 https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/i...

OneFunFellow 6y ago

Feedback: Perhaps add info about Pi-Hole usage.

deleted 6y ago
[deleted]
pdimitar 6y ago
Shouldn't just adding the URL[0] work? [0] https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/raw/master/For%20hosts%20file/The_Quantum_Ad-List.txt https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/r...
- walshemj 6y ago
  Yes just login to the web interface go to group management and add that url - just done that on my pi hole running on a Model B Rev 2
  - pdimitar 6y ago
    Yep, just did the same. Thanks for confirming that it also works for you.
- dangerboysteve 6y ago
  I was asking the same and then looked at my pihole and came to the same conclusion.
- Merman_Mike 6y ago
  I had 1,217,258 blocked domains on my Pi-Hole to begin with, and this added 515,284 new domains.
  - pdimitar 6y ago
    Yes, I have seen a rough add of ~500K domains indeed. Had to whitelist api.instabug.com since with it blocked my 9GAG app goes crazy and tries to connect to it non-stop, lighting my phone on fire.
    - Merman_Mike 6y ago
      I'm surprised to see that there is any overlap between 9GAG and HN :)
      - pdimitar 6y ago
        Next you'll tell me you are surprised that a 9gag and an HN regular gets laid! ;) In all seriousness, I casually browse it for 30-40 minutes a day. Helps put my mind at ease here and there (although it too became heavily politicized and not as much fun as it was years ago).
      - justinlloyd 6y ago
        You leave him alone, he is a renowned Field Marshal, though as OP, also known to suck a lot of #!@% P.S. 9gag in-joke, you wouldn't get it.
  - yumraj 6y ago
    I went from 2.039 million to 2.043 million, so about 5,000 new domains.

gbh444g 6y ago

TBH, at this point we may as well start using whitelists: 1st party domains and known 3rd party CDNs for static content and maybe media, 1st party scripts for frequently visited sites and a special button hidden in a safe place to enable 3rd party scripts for those who want to live dangerously.

ignoramous 6y ago
That's uMatrix (at least in-browser)... Sadly it appears to have been discontinued by gorhill. IP whitelisting isn't fool-proof though. For example, I run an anti-censorship DoH resolver "IP fronted" by Cloudflare specifically to be a hard target to block at IP layer. I mean, you can block Cloudflare's IPs but that'd mean you'd end up blocking other websites too.
- hosteur 6y ago
  UMatrix is vital. Did anyone pick up the project after gorhill?
  - ignoramous 6y ago
    Don't think u/gorhill would be inclined at the idea of handing over the reins after the uBlock.org fiasco. Forks may survive, but the extension looks to be all but dead.
eli 6y ago
Known 3rd party CDNs are used for all kinds of content from CSS needed to make a site work to ads and tracking scripts.
CivBase 6y ago
That's basically what I do with uMatrix.
- nashashmi 6y ago
  Yup. I was going to post this too. Why keep 34000 site Peter Lowe’s ad blocker when a 50 site white list will do the same. But umatrix is complex. Ublock is good enough. I just wished I could whitelist all cdns

Merman_Mike 6y ago

I wish there were more details about how this list of domains was compiled. This is the explanation on the Gitlab: > We were testing an AI that could show some basic emotions about internet content, and turns out it was very precise at getting “annoyed” by ads and “unsolicited” third party connections… > From that, I forked our own project and tweaked it in a specific way to basically only focus on ads, trackers, etc. and act like a web crawler, turns out to be very effective!

lightswitch05 6y ago
I agree. I think its very likely 'import a ton of other block lists and then apply some sort of logic to whitelist potential problem domains', but I would love to be wrong.
ComodoHacker 6y ago
Explanation is what our current AI tech struggles with most.

caterama 6y ago

This really makes me wish there was a way to include files in /etc/hosts files...

sebow 6y ago
HostsMan is a good utility for windows, which from what i recall lets you use lists and even add update URLs; Apparently a less-fledged foss alternative is SwitchHosts! for linux, which also seem to have this crucial feature
mech422 6y ago
Just generate /etc/hosts from a cron job... Pretty easy to concat, sort, and filter however you like in a short script.
- coldpie 6y ago
  You can also set up a systemd service to monitor files in a directory and refresh the /etc/hosts when they are changed.
justinlloyd 6y ago
If you are okay with a GUI on Windows, Mac or Linux, I can recommend Hozz. https://blog.zhangruipeng.me/Hozz/ https://blog.zhangruipeng.me/Hozz/ No longer under development but works just fine.
TheQuantumAlpha 6y ago
I made an installer...

deleted 6y ago

[deleted]

smashah 6y ago

Awesome work! For anyone interested in using The Quantum ad-list in their adguard setups, I've added this to my self-updating ad gaurd block list generator (smashblock - which is also based on hblock). https://github.com/smashah/smashblock https://github.com/smashah/smashblock

deleted 6y ago

[deleted]

sciencesama 6y ago

Is there a way we can block part of the subdomain, like a proxy firewall. Used to see this thing called mcafee web gateway. That used to block all ads,banner ads and even instagram ads !! Is there one such open source project out there ?? Where we can download and add the fanboy type block list to block the ad crap !!

_wldu 6y ago
DNS and IP blocks are like hitting a tiny finishing nail with a 10 pound hammer. Sure, you smash it, but you smash everything around it too. Try Squid. It's a dynamic higher layer HTTP/web proxy that can block specific URL content, etc. https://en.wikipedia.org/wiki/Squid_(software) https://en.wikipedia.org/wiki/Squid_(software)

secondcoming 6y ago

Interesting. My company is on there, and one of our domains listed is not used in any for ad serving. It's the URL of our customer portal.

TheQuantumAlpha 6y ago

I see... Thanks!!

TheQuantumAlpha 6y ago

Thus, some other ad-list holders are panicking that they may become irrelevant, therefore trying to silence us... They threaten to copystrike us as we have similar domains they have... Ads are ads, right?

TheQuantumAlpha 6y ago

https://bitbucket.org/The_Quantum_Alpha/the-quantum-ad-list/src/master/ https://bitbucket.org/The_Quantum_Alpha/the-quantum-ad-list/...

Havoc 6y ago

Having used it for a while, on thing it seems to pick up that the standard pihole lists don't is >clients.l.google.com >clients4.google.com Thousands of requests though so pushed up the overall block rate by a good 5%