4 ms·
Doesn’t DNS server only see the hostname instead of the URL? The approach would allow to only scrap home pages then I guess. Also, what about pages behind login
by pythux 7y ago
Doesn’t DNS server only see the hostname instead of the URL? The approach would allow to only scrap home pages then I guess. Also, what about pages behind login?
- kick 7y agoCorrect for the first two, I hadn't thought about that. Could be done by uploading history once every hour but that's annoying. Pages behind a login aren't ideal for this under any approach because they tend to be dynamic.