4 ms·
My bad, I didn't specify what sizes I was referring to; I meant the page size, including graphics, number of links etc. If there's a way to extract a rough esti
by squarefoot 5y ago
My bad, I didn't specify what sizes I was referring to; I meant the page size, including graphics, number of links etc. If there's a way to extract a rough estimate of the page "weight", it could be used to filter out empty (as in clearly expired) pages without downloading them. I'm not a web dev, so I'm not sure if that is possible.
- pabs3 5y agoI think the best way to provide what you want would be to provide pre-download and post-download hooks, so you could write some code to prevent some downloads and detect downloaded files are spammy. Then you could write your size heuristic as a plugin and others could use regexes or machine learning to do something similar.
- pabs3 5y agoThinking it over again, maybe JSON input/output is enough, then you can filter that and change the list of pages to download, or delete downloaded files.
- EricE 5y agoUnfortunately a lot of adfarm sites that hoover up abandoned/popular domains scrape content from other sites and aggregate it to keep Google engaged at driving clicks to their ad-laced pages. On the surface a Yelp like system to rate domains as legit vs. click bait seems logical until you realize the scammers would just work at gaming that system too :p This is where a universal ID would really help - but the other ways something like that could be used make me even more uncomfortable so here we are with no real good solution :(