3 ms·
In this very specific example: It doesn't keep the crawler state (which links have been visited, which links are discovered but not yet visited) persistently.
by johansch 11y ago
In this very specific example:
It doesn't keep the crawler state (which links have been visited, which links are discovered but not yet visited) persistently.
(You made it this specific by picking this particular example; my more general observation is that this is a common thing in command line/shell script constructs. They remain simple only until you start to care about such things.)
- tyingq 11y ago>>It doesn't keep the crawler state It does though. Just not in the way you're expecting. The timestamp support would keep it from re-downloading anything already downloaded (though it would do an HTTP head for the comparison). Or, it also has a "no-clobber" feature that would keep it from even trying to download. Yes, both approaches are more limited than a specific state datastore, but there is state.