Y
HN Search
Hacker News Search
new
|
comments
|
top
|
jobs
tconaugh
searching PlanetScale…
1.
▲
2.
▲
3.
▲
4.
▲
5.
▲
6.
▲
3 ms
·
1.
▲
by
tconaugh
9y ago
They use distributed web crawlers to crawl 100s of billions of web pages. Probably one of the following options: 1) Built their own crawlers. 2) Using an Apache Nutch/Heritrix cluster in a colo facility. 3) Use 3rd party services like