3 ms·

Well, yes, for a single download. Not for crawling - a repeated pattern of download, inspect for links, download some more etc.

• tyingq 11y ago
  Not sure I understand your point. Wget does recursive crawling, and supports timestamp comparison. Perhaps you are noting that it wouldn't prioritize new pages over changed pages? Or that it doesn't support something like "don't check for changes at all unless the download is more than X days old?". Generally, yes, there is some point at which wget would fall short of a purpose built tool. I think that point, though, is farther out than you're suggesting.
  • johansch 11y ago
    In this very specific example: It doesn't keep the crawler state (which links have been visited, which links are discovered but not yet visited) persistently. (You made it this specific by picking this particular example; my more general observation is that this is a common thing in command line/shell script constructs. They remain simple only until you start to care about such things.)
    • tyingq 11y ago
      >>It doesn't keep the crawler state It does though. Just not in the way you're expecting. The timestamp support would keep it from re-downloading anything already downloaded (though it would do an HTTP head for the comparison). Or, it also has a "no-clobber" feature that would keep it from even trying to download. Yes, both approaches are more limited than a specific state datastore, but there is state.
  • deleted 11y ago
    [deleted]