3 ms·
most naive code I see is like the following const page = await crawler.browser.newPage(); await page.goto(url); await page.waitForSelector("a[href]
by bryanrasmussen 9y ago
most naive code I see is like the following
const page = await crawler.browser.newPage();
await page.goto(url);
await page.waitForSelector("a[href]");
const hrefs = await page.evaluate(
() => Array.from(document.body.querySelectorAll('a[href]'), ({ href }) => href)
);
and then you do something with hrefs.
However if you have a page that loads with 4 links defined, does its script and then ends up 100+ links you miss the 100+ links. I notice people often failing to account for this in their crawlers, so I wondered if this one was.