3 ms·

The parent's suggestion to filter on size seems on the surface like it would work. What makes you think it's harder than that?

pabs3 5y ago
I'm not sure how filtering based on size relates to filtering out domains for sale, spam domains etc. Lots of legit domains have small size or large size and lots of spam domains similarly vary in size. Filtering out unwanted domains (sale, spam etc) is a problem for a bunch of regexes, bayesian classification or machine learning. Edit: I think I misread the original post quite badly and I don't understand the proposed feature.
- squarefoot 5y ago
  My bad, I didn't specify what sizes I was referring to; I meant the page size, including graphics, number of links etc. If there's a way to extract a rough estimate of the page "weight", it could be used to filter out empty (as in clearly expired) pages without downloading them. I'm not a web dev, so I'm not sure if that is possible.
  - pabs3 5y ago
    I think the best way to provide what you want would be to provide pre-download and post-download hooks, so you could write some code to prevent some downloads and detect downloaded files are spammy. Then you could write your size heuristic as a plugin and others could use regexes or machine learning to do something similar.
    - pabs3 5y ago
      Thinking it over again, maybe JSON input/output is enough, then you can filter that and change the list of pages to download, or delete downloaded files.
  - EricE 5y ago
    Unfortunately a lot of adfarm sites that hoover up abandoned/popular domains scrape content from other sites and aggregate it to keep Google engaged at driving clicks to their ad-laced pages. On the surface a Yelp like system to rate domains as legit vs. click bait seems logical until you realize the scammers would just work at gaming that system too :p This is where a universal ID would really help - but the other ways something like that could be used make me even more uncomfortable so here we are with no real good solution :(

pabs3 5y ago

I'm not sure how filtering based on size relates to filtering out domains for sale, spam domains etc. Lots of legit domains have small size or large size and lots of spam domains similarly vary in size. Filtering out unwanted domains (sale, spam etc) is a problem for a bunch of regexes, bayesian classification or machine learning. Edit: I think I misread the original post quite badly and I don't understand the proposed feature.

squarefoot 5y ago
My bad, I didn't specify what sizes I was referring to; I meant the page size, including graphics, number of links etc. If there's a way to extract a rough estimate of the page "weight", it could be used to filter out empty (as in clearly expired) pages without downloading them. I'm not a web dev, so I'm not sure if that is possible.
- pabs3 5y ago
  I think the best way to provide what you want would be to provide pre-download and post-download hooks, so you could write some code to prevent some downloads and detect downloaded files are spammy. Then you could write your size heuristic as a plugin and others could use regexes or machine learning to do something similar.
  - pabs3 5y ago
    Thinking it over again, maybe JSON input/output is enough, then you can filter that and change the list of pages to download, or delete downloaded files.
- EricE 5y ago
  Unfortunately a lot of adfarm sites that hoover up abandoned/popular domains scrape content from other sites and aggregate it to keep Google engaged at driving clicks to their ad-laced pages. On the surface a Yelp like system to rate domains as legit vs. click bait seems logical until you realize the scammers would just work at gaming that system too :p This is where a universal ID would really help - but the other ways something like that could be used make me even more uncomfortable so here we are with no real good solution :(