4 ms·

To ensure there are no nefarious redirects in such a massive list, this one-liner comes in handy[1]: grep -v '#' steves_hosts | awk '{print $1}'| sort | un

• hda111 3y ago
  What is a nefarious redirect? Does it look for entries without 127.0.0.1?
  • miles 3y ago
    A redirect to some unexpected IP address. Here's what the output looks like: 190625 0.0.0.0 3 127.0.0.1 1 255.255.255.255 3 ::1 1 fe80::1%lo0 2 ff00::0 1 ff02::1 1 ff02::2 1 ff02::3
  • 8organicbits 3y ago
    There's a concern that a malicious list could point domain names to a malicious IP address. I don't think its a big concern with https:// https:// since the cert will be invalid, but it's still a concern.
    • DaiPlusPlus 3y ago
      I suspect it’s very likely that somewhere in the world is a domain-validation server, used by a trusted CA, which has this very anti-advertising hosts file installed onto it.
      • 8organicbits 3y ago
        Why? A CA would be unable to issue certificates for advertising sites with that configuration.
        • DaiPlusPlus 3y ago
          It could be a lesser-known CA, perhaps the national CA of a small country (under 2m people) that normally only issues less than tens-of-thousands of certs and only exists for regulatory reasons (e.g. the country requires all of its own gov services to use its internal CA, while all commercial/popular services use CA based in another country, usually LetsEncrypt)
• JdeBP 3y ago
  Someone explicitly called out and removed the Useless Use of cat but yet left the Useless Uses of grep, sort, and uniq in, I see. (-: * https://porkmail.org/era/unix/award#grep https://porkmail.org/era/unix/award#grep awk '!/#/ { seen[$1]++; } END { for (k in seen) { print seen[k],k; } }' steves_hosts
  • tedivm 3y ago
    I'd rather have the readability of the first command than the "efficiency" of your example.
  • miles 3y ago
    Guilty as charged!
  • arp242 3y ago
    I love awk as much as the next person and I've written some pretty large scripts with it, but the original grep/sort/uniq is a lot easier to understand in two seconds. I don't think it's "useless" at all. Also your awk isn't exactly identical as it's not sorted, but in this case that probably doesn't matter.
    • j0hnyl 3y ago
      Same, I think I use grep | sort | uniq almost on the daily.
    • dredmorbius 3y ago
      One of the advantage of the array[item]++ / for (item in array) construct is that it avoids possibly expensive sorts (at the cost of generating potentially large arrays), and tends to be quite fast. That's even if you end up sorting the output by frequency, as the summary listing is typically shorter than the overall input data: {a[$1]++} END {for (i in array) printf("%6i %s\n", a[i], i) | "sort -k1nr | cat -n";} The accumulator/loop idiom is ... fairly readily recognisable to someone familiar with awk.
• rovr138 3y ago
  find . -name hosts -exec grep -v '^#' {} \; | awk '{print $1}'| sort | uniq -c 43029 204 # 4289934 0.0.0.0 10763 127.0.0.1 18 255.255.255.255 54 ::1 1 analytics.shein.co.uk 1 analytics.shein.com 1 auxilium.ftb.team 1 bstats.org 1 fe00::0 17 fe80::1%lo0 33 ff00::0 17 ff02::1 17 ff02::2 17 ff02::3 1 mcmc.dev 1 mcstats.org 1 metrics.shmeeb.net 1 openeye.openmods.info And looking at those last ones, looks like some files have a different format. For example, the data/minecraft-hosts/hosts file, ~/D/P/hosts   master  grep -irn 'openeye.openmods.info' . ./alternates/fakenews/hosts:174307:0.0.0.0 openeye.openmods.info ... ./hosts:174306:0.0.0.0 openeye.openmods.info ./data/minecraft-hosts/hosts:9:openeye.openmods.info ~/D/P/hosts   master  grep -irn 'metrics.shmeeb.net' . ... ./data/minecraft-hosts/hosts:8:metrics.shmeeb.net