3 ms·

It would be handy to include an option to walk back in time without start downloading until a certain site size is met, to exclude 404s, domain for sale placeho

It would be handy to include an option to walk back in time without start downloading until a certain site size is met, to exclude 404s, domain for sale placeholders etc, which aren't uncommon among old sites, so that the Archive.org precious bandwidth isn't wasted.

pabs3 5y ago
Already has an option to include/exclude 404s. Please file issues for the other requests, except the filter for domains for sale, that is a hard problem that probably isn't in-scope for this tool.
- tempestn 5y ago
  The parent's suggestion to filter on size seems on the surface like it would work. What makes you think it's harder than that?
  - pabs3 5y ago
    I'm not sure how filtering based on size relates to filtering out domains for sale, spam domains etc. Lots of legit domains have small size or large size and lots of spam domains similarly vary in size. Filtering out unwanted domains (sale, spam etc) is a problem for a bunch of regexes, bayesian classification or machine learning. Edit: I think I misread the original post quite badly and I don't understand the proposed feature.
    - squarefoot 5y ago
      My bad, I didn't specify what sizes I was referring to; I meant the page size, including graphics, number of links etc. If there's a way to extract a rough estimate of the page "weight", it could be used to filter out empty (as in clearly expired) pages without downloading them. I'm not a web dev, so I'm not sure if that is possible.
      - pabs3 5y ago
        I think the best way to provide what you want would be to provide pre-download and post-download hooks, so you could write some code to prevent some downloads and detect downloaded files are spammy. Then you could write your size heuristic as a plugin and others could use regexes or machine learning to do something similar.
        pabs3 5y ago
        Thinking it over again, maybe JSON input/output is enough, then you can filter that and change the list of pages to download, or delete downloaded files.
      - EricE 5y ago
        Unfortunately a lot of adfarm sites that hoover up abandoned/popular domains scrape content from other sites and aggregate it to keep Google engaged at driving clicks to their ad-laced pages. On the surface a Yelp like system to rate domains as legit vs. click bait seems logical until you realize the scammers would just work at gaming that system too :p This is where a universal ID would really help - but the other ways something like that could be used make me even more uncomfortable so here we are with no real good solution :(

pabs3 5y ago

Already has an option to include/exclude 404s. Please file issues for the other requests, except the filter for domains for sale, that is a hard problem that probably isn't in-scope for this tool.

tempestn 5y ago
The parent's suggestion to filter on size seems on the surface like it would work. What makes you think it's harder than that?
- pabs3 5y ago
  I'm not sure how filtering based on size relates to filtering out domains for sale, spam domains etc. Lots of legit domains have small size or large size and lots of spam domains similarly vary in size. Filtering out unwanted domains (sale, spam etc) is a problem for a bunch of regexes, bayesian classification or machine learning. Edit: I think I misread the original post quite badly and I don't understand the proposed feature.
  - squarefoot 5y ago
    My bad, I didn't specify what sizes I was referring to; I meant the page size, including graphics, number of links etc. If there's a way to extract a rough estimate of the page "weight", it could be used to filter out empty (as in clearly expired) pages without downloading them. I'm not a web dev, so I'm not sure if that is possible.
    - pabs3 5y ago
      I think the best way to provide what you want would be to provide pre-download and post-download hooks, so you could write some code to prevent some downloads and detect downloaded files are spammy. Then you could write your size heuristic as a plugin and others could use regexes or machine learning to do something similar.
      - pabs3 5y ago
        Thinking it over again, maybe JSON input/output is enough, then you can filter that and change the list of pages to download, or delete downloaded files.
    - EricE 5y ago
      Unfortunately a lot of adfarm sites that hoover up abandoned/popular domains scrape content from other sites and aggregate it to keep Google engaged at driving clicks to their ad-laced pages. On the surface a Yelp like system to rate domains as legit vs. click bait seems logical until you realize the scammers would just work at gaming that system too :p This is where a universal ID would really help - but the other ways something like that could be used make me even more uncomfortable so here we are with no real good solution :(