3 ms·
From a naive point of view it's easy. But it has to scale and launching a new process for every request works only for a pet project. So a crawler based on a h
by frik 11y ago
From a naive point of view it's easy. But it has to scale and launching a new process for every request works only for a pet project.
So a crawler based on a headless browser that consumes little memory and runs for weeks is a major achivement.
- manigandham 11y agoWhy doesn't that scale? You might be underestimating Google's resources.