3 ms·
> Right now it is extremely difficult to build your own web crawler that would compete with Google. Nah, not a small task but you can break it down into well u
by 6510 4y ago
> Right now it is extremely difficult to build your own web crawler that would compete with Google.
Nah, not a small task but you can break it down into well understood problems that have known solution.
The hard part is ranking everything.
- kolinko 4y agoAre you speaking from theory or experience? If from experience - how did you get around multiple sites disallowing crawlers other than google or bing?