2 ms·
Haha, nice! Crawler traps are a quite old phenomenon. Been around since before Google. Dunno about the others, but my crawler has a set depth it will crawl. It
by marginalia_nu 4y ago
Haha, nice! Crawler traps are a quite old phenomenon. Been around since before Google.
Dunno about the others, but my crawler has a set depth it will crawl. It'll BFS for like 1000-10000 documents depending on some factors.