5 ms·

TuM'Fatig – Ads blocking with OpenBSD unbound(8)

59 points by protomyth 4y ago

INTPenis 4y ago
This is really cool, thanks for the write up. I did the same thing in my homelab but much simpler, my shell script only fetched from a few sources and converted it into unbound format so it was much shorter. And I didn't care about statistics, but the idea was the same. Also, why use ftp instead of curl?[1] 1. https://curl.se/docs/manual.html#time-conditions https://curl.se/docs/manual.html#time-conditions
- protomyth 4y ago
  My guess is that curl is a port/package while ftp comes with the base system.
  - INTPenis 4y ago
    True but I also run OpenBSD at home and curl is part of my common packages to install list in ansible. How I bootstrap all my nodes.
    - protomyth 4y ago
      As others have mentioned, ftp that ships with OpenBSD is a bit more robust then its name suggests. https://man.openbsd.org/ftp https://man.openbsd.org/ftp
- nixcraft 4y ago
  The ftp client is, by default, installed in the OpenBSD base system. Even though the name is ftp, it works with HTTP, HTTPS, FTP/FTPS and so on.
  - merpkz 4y ago
    is it like poor man's curl?
sgt 4y ago
What is TuM'Fatig - klingon or something?
- pookee 4y ago
  It's french slang, = Tu me fatigues = "You make me tired" (tired of ads I assume)
  - Off 4y ago
    A more correct translation would be "you wear me out".
  - deleted 4y ago
    [deleted]
  - claviola 4y ago
    This is but one post on his blog, though.
- make3 4y ago
  means, you ~ irritate me
justsomehnguy 4y ago
> Grab and parse the Top 10 milion domains (based on Open PageRank data) so that they can be used by dnsperf(1). Be vary of using such sources. Go to the source [0], make "Display: 100" and scroll to "1,351 to 1,450 of 10,000,000 websites". You will find a bunch of bullshit domains clearly used for something nefarious (and some no longer exists), but to believe what those sites are as popular as spreadsheets.google.com? NO WAI Tiny sample for the lazy: 1,361 ecoquipo.ga 5.92/10 ga 1,362 h4uvpev4tx.ga 5.92/10 ga 1,363 lfsni36qvn.ga 5.92/10 ga 1,364 dashseo.ga 5.92/10 ga 1,365 velocityconf.com 5.92/10 com 1,366 filezilla-project.org 5.92/10 org 1,367 bx2vnequhc.ga 5.92/10 ga 1,368 incblizzard.ga 5.92/10 ga 1,369 anisima.ru 5.92/10 ru 1,370 ou14ib22tf.ga 5.92/10 ga 1,371 i12tfukfft.ga 5.92/10 ga 1,372 blankmedia.ga 5.92/10 ga 1,373 nicelia.ga 5.92/10 ga 1,374 finlandtoday.fi 5.92/10 fi 1,375 spreadsheets.google.com 5.92/10 com [0] https://www.domcop.com/top-10-million-domains https://www.domcop.com/top-10-million-domains
gh02t 4y ago
So I've always been curious, does anybody have a writeup of how DNS servers like unbound etc are able to handle evaluating these massive blocklists? Like the algorithms and data structures they use to evaluate queries against these lists efficiently. My pfsense+pfblocker unbound instance has something like 400K hosts blocked and I've always been amazed that it doesn't even break a sweat scanning that many hosts with every query.
- dsr_ 4y ago
  Here's Tony Finch's discussion of the data structures he's working on for ISC's BIND: https://dotat.at/prog/qp/README.html https://dotat.at/prog/qp/README.html
- justsomehnguy 4y ago
  > My pfsense+pfblocker unbound instance has something like 400K hosts blocked But it is not 400k hosts to search: com facebook www ad google ad ads notadswepromise googletagmanager tags ads to block ad.facebook.com you only need to scan 10-100k of .com domains in your block list and only a couple of records in facebook.com. Don't forget, DNS is a hierarchical system. And we, humans, actually record, use and remember all the DNS records in the wrong, reverse direction ;-) > 400K hosts I bet 30% of those hosts are don't resolve anymore and another 30% resolve to domain parking/404 hosts. Eg: see my comment about "TOP 10M SITES" https://news.ycombinator.com/item?id=33150554 https://news.ycombinator.com/item?id=33150554
  - LinuxBender 4y ago
    I bet 30% of those hosts are don't resolve anymore I've seen some evidence of this. I periodically dig all the blocked domains and noticed that some of them come and go so I assume people cycle them in and out to evade blocking. A handful of the domains just change the A record so I just block the apex domain which makes it a little harder to keep the block lists optimized but does reduce the size a little. I should add that blocking the apex of some domains includes risk of dropping legit traffic but if I see more than {n} badware domains on the apex, I write it off as compromised or at very least poorly implemented and poorly secured.
LinuxBender 4y ago
As a fun experiment one could set this up on a handful of VM's and leave the Unbound TLS port open or DoH if compiled with libnghttp2 to the world in addition to the graphs. I think it would be interesting to see how many clients each VM could handle. Maybe enable the rate limits in Unbound first. I'm half tempted to do this with a few of mine but I do not have the fancy graphs, just shell scripts to parse the logs which I guess could be written out as text files for nginx to serve. Unfortunately for me the Unbound package on Alpine is not compiled for DoH, only DoT and I am too lazy to recompile things any more.
liotier 4y ago
In spite of my entire satisfaction with it, I had to ditch my OPNsense router... Its unbound ad filtering DNS is among the features I miss most from it.
- dbtc 4y ago
  Why did you have to ditch?
  - liotier 4y ago
    Bus factor. I had a very nice home setup (three 24 ports switches and a 6-port router) with multiple VLAN, GPON terminated on my own device (emulating the ISP's hardware), bonded Ethernet links, fully routed IPv6, sane firewalling, ad-filtered DNS, reverse proxy etc. - but I was the only person in my home (six kids + adults) to know how to maintain it. Anything happens while I'm away and Internet access stops. I borked an update and was offline for a couple of days - that was once too many for the users. So we are back to the ISP's router, no VLAN and soon some Wireguard overlay for my hosting. I'm sad about this: replacing infrastructure I control with the ISP invading my home, replacing tailor-made with generic consumer products, replacing native Internet with yet another overlay network... But MTTR rules my world.
m3047 4y ago
As somebody who spends some time on this, identifying common infrastructure and targeting that is the best bang for the buck: all kinds of stuff lives in the sewer that you never thought of. OTOH it does require some whitelisting, because all kinds of stuff lives in the sewer.
kincl 4y ago
I just realized that OpenBSD unbound was likely named as a counter to ISC BIND. Similar to Sun Java and the Eclipse Foundation.
- somat 4y ago
  It is not really openbsd's unbound, it is nlnet labs unbound. openbsd just ships it with their operating system. https://nlnetlabs.nl/projects/unbound/about/ https://nlnetlabs.nl/projects/unbound/about/ but yes it was made to avoid a bind monoculture. and comes in two parts nsd is the authoritive name server. and unbound is a recursive name server.
somat 4y ago
This is what I do. I was trying to figure out what pi-hole did one easy thing was it black-holed domains, so I got the same list put it on my dns relay box and there you go same thing on openbsd. https://pgl.yoyo.org/adservers/ https://pgl.yoyo.org/adservers/ basicly all it is is include: /var/unbound/etc/yoyo.conf where yoyo.conf is a dns blacklist
- pgl 4y ago
  Glad you're finding the list useful!
pgl 4y ago
Just for the record, but my list is available natively in a format for use with Unbound: - https://pgl.yoyo.org/as/serverlist.php?hostformat=unbound https://pgl.yoyo.org/as/serverlist.php?hostformat=unbound Remove the header with &showintro=0 and for plain text, use &mimetype=plaintext: - https://pgl.yoyo.org/as/serverlist.php?hostformat=unbound&mimetype=plaintext&showintro=0 https://pgl.yoyo.org/as/serverlist.php?hostformat=unbound&mi...

59 points by protomyth 4y ago

This is really cool, thanks for the write up. I did the same thing in my homelab but much simpler, my shell script only fetched from a few sources and converted it into unbound format so it was much shorter. And I didn't care about statistics, but the idea was the same. Also, why use ftp instead of curl?[1] 1. https://curl.se/docs/manual.html#time-conditions https://curl.se/docs/manual.html#time-conditions

protomyth 4y ago
My guess is that curl is a port/package while ftp comes with the base system.
- INTPenis 4y ago
  True but I also run OpenBSD at home and curl is part of my common packages to install list in ansible. How I bootstrap all my nodes.
  - protomyth 4y ago
    As others have mentioned, ftp that ships with OpenBSD is a bit more robust then its name suggests. https://man.openbsd.org/ftp https://man.openbsd.org/ftp
nixcraft 4y ago
The ftp client is, by default, installed in the OpenBSD base system. Even though the name is ftp, it works with HTTP, HTTPS, FTP/FTPS and so on.
- merpkz 4y ago
  is it like poor man's curl?

sgt 4y ago

What is TuM'Fatig - klingon or something?

pookee 4y ago
It's french slang, = Tu me fatigues = "You make me tired" (tired of ads I assume)
- Off 4y ago
  A more correct translation would be "you wear me out".
- deleted 4y ago
  [deleted]
- claviola 4y ago
  This is but one post on his blog, though.
make3 4y ago
means, you ~ irritate me

justsomehnguy 4y ago

> Grab and parse the Top 10 milion domains (based on Open PageRank data) so that they can be used by dnsperf(1). Be vary of using such sources. Go to the source [0], make "Display: 100" and scroll to "1,351 to 1,450 of 10,000,000 websites". You will find a bunch of bullshit domains clearly used for something nefarious (and some no longer exists), but to believe what those sites are as popular as spreadsheets.google.com? NO WAI Tiny sample for the lazy: 1,361 ecoquipo.ga 5.92/10 ga 1,362 h4uvpev4tx.ga 5.92/10 ga 1,363 lfsni36qvn.ga 5.92/10 ga 1,364 dashseo.ga 5.92/10 ga 1,365 velocityconf.com 5.92/10 com 1,366 filezilla-project.org 5.92/10 org 1,367 bx2vnequhc.ga 5.92/10 ga 1,368 incblizzard.ga 5.92/10 ga 1,369 anisima.ru 5.92/10 ru 1,370 ou14ib22tf.ga 5.92/10 ga 1,371 i12tfukfft.ga 5.92/10 ga 1,372 blankmedia.ga 5.92/10 ga 1,373 nicelia.ga 5.92/10 ga 1,374 finlandtoday.fi 5.92/10 fi 1,375 spreadsheets.google.com 5.92/10 com [0] https://www.domcop.com/top-10-million-domains https://www.domcop.com/top-10-million-domains

gh02t 4y ago

So I've always been curious, does anybody have a writeup of how DNS servers like unbound etc are able to handle evaluating these massive blocklists? Like the algorithms and data structures they use to evaluate queries against these lists efficiently. My pfsense+pfblocker unbound instance has something like 400K hosts blocked and I've always been amazed that it doesn't even break a sweat scanning that many hosts with every query.

dsr_ 4y ago
Here's Tony Finch's discussion of the data structures he's working on for ISC's BIND: https://dotat.at/prog/qp/README.html https://dotat.at/prog/qp/README.html
justsomehnguy 4y ago
> My pfsense+pfblocker unbound instance has something like 400K hosts blocked But it is not 400k hosts to search: com facebook www ad google ad ads notadswepromise googletagmanager tags ads to block ad.facebook.com you only need to scan 10-100k of .com domains in your block list and only a couple of records in facebook.com. Don't forget, DNS is a hierarchical system. And we, humans, actually record, use and remember all the DNS records in the wrong, reverse direction ;-) > 400K hosts I bet 30% of those hosts are don't resolve anymore and another 30% resolve to domain parking/404 hosts. Eg: see my comment about "TOP 10M SITES" https://news.ycombinator.com/item?id=33150554 https://news.ycombinator.com/item?id=33150554
- LinuxBender 4y ago
  I bet 30% of those hosts are don't resolve anymore I've seen some evidence of this. I periodically dig all the blocked domains and noticed that some of them come and go so I assume people cycle them in and out to evade blocking. A handful of the domains just change the A record so I just block the apex domain which makes it a little harder to keep the block lists optimized but does reduce the size a little. I should add that blocking the apex of some domains includes risk of dropping legit traffic but if I see more than {n} badware domains on the apex, I write it off as compromised or at very least poorly implemented and poorly secured.

LinuxBender 4y ago

As a fun experiment one could set this up on a handful of VM's and leave the Unbound TLS port open or DoH if compiled with libnghttp2 to the world in addition to the graphs. I think it would be interesting to see how many clients each VM could handle. Maybe enable the rate limits in Unbound first. I'm half tempted to do this with a few of mine but I do not have the fancy graphs, just shell scripts to parse the logs which I guess could be written out as text files for nginx to serve. Unfortunately for me the Unbound package on Alpine is not compiled for DoH, only DoT and I am too lazy to recompile things any more.

liotier 4y ago

In spite of my entire satisfaction with it, I had to ditch my OPNsense router... Its unbound ad filtering DNS is among the features I miss most from it.

dbtc 4y ago
Why did you have to ditch?
- liotier 4y ago
  Bus factor. I had a very nice home setup (three 24 ports switches and a 6-port router) with multiple VLAN, GPON terminated on my own device (emulating the ISP's hardware), bonded Ethernet links, fully routed IPv6, sane firewalling, ad-filtered DNS, reverse proxy etc. - but I was the only person in my home (six kids + adults) to know how to maintain it. Anything happens while I'm away and Internet access stops. I borked an update and was offline for a couple of days - that was once too many for the users. So we are back to the ISP's router, no VLAN and soon some Wireguard overlay for my hosting. I'm sad about this: replacing infrastructure I control with the ISP invading my home, replacing tailor-made with generic consumer products, replacing native Internet with yet another overlay network... But MTTR rules my world.

m3047 4y ago

As somebody who spends some time on this, identifying common infrastructure and targeting that is the best bang for the buck: all kinds of stuff lives in the sewer that you never thought of. OTOH it does require some whitelisting, because all kinds of stuff lives in the sewer.

kincl 4y ago

I just realized that OpenBSD unbound was likely named as a counter to ISC BIND. Similar to Sun Java and the Eclipse Foundation.

somat 4y ago
It is not really openbsd's unbound, it is nlnet labs unbound. openbsd just ships it with their operating system. https://nlnetlabs.nl/projects/unbound/about/ https://nlnetlabs.nl/projects/unbound/about/ but yes it was made to avoid a bind monoculture. and comes in two parts nsd is the authoritive name server. and unbound is a recursive name server.

somat 4y ago

This is what I do. I was trying to figure out what pi-hole did one easy thing was it black-holed domains, so I got the same list put it on my dns relay box and there you go same thing on openbsd. https://pgl.yoyo.org/adservers/ https://pgl.yoyo.org/adservers/ basicly all it is is include: /var/unbound/etc/yoyo.conf where yoyo.conf is a dns blacklist

pgl 4y ago
Glad you're finding the list useful!

pgl 4y ago

Just for the record, but my list is available natively in a format for use with Unbound: - https://pgl.yoyo.org/as/serverlist.php?hostformat=unbound https://pgl.yoyo.org/as/serverlist.php?hostformat=unbound Remove the header with &showintro=0 and for plain text, use &mimetype=plaintext: - https://pgl.yoyo.org/as/serverlist.php?hostformat=unbound&mimetype=plaintext&showintro=0 https://pgl.yoyo.org/as/serverlist.php?hostformat=unbound&mi...