Y
HN Search
Hacker News Search
new
|
comments
|
top
|
jobs
x-header
searching PlanetScale…
1.
▲
2.
▲
3.
▲
4.
▲
5.
▲
6.
▲
28 ms
·
1.
▲
by
x-header
14y ago
This is not how to crawl webpages. He started with the Alexa list. Those are not necessarily domain names of servers serving webpages. I would guess that some of the request to cease crawling came from some of these listings. Working fro