2 ms·
Very interesting service! A number of questions... What User-Agent do you use? Do you crawl non-textual resources? Do you save all headers from the crawled r
by gojomo 18y ago
Very interesting service! A number of questions...
What User-Agent do you use?
Do you crawl non-textual resources?
Do you save all headers from the crawled responses?
Do you perform any processing on the returned content (like de-chunking or de-compressing) or can it be retrieved verbatim?
If two customers request the same URL/site be crawled, are their requests merged so the site is only crawled once?
Do you save the exact time of the request (not trusting the returned 'Date' header)?