5 ms·
I don't know about this project specifically, but typically with headless Chrome, you let it run the JS and then read the DOM.
by trevyn 9y ago
I don't know about this project specifically, but typically with headless Chrome, you let it run the JS and then read the DOM.
- bryanrasmussen 9y agomost naive code I see is like the following const page = await crawler.browser.newPage(); await page.goto(url); await page.waitForSelector("a[href]"); const hrefs = await page.evaluate( () => Array.from(document.body.querySelectorAll('a[href]'), ({ href }) => href) ); and then you do something with hrefs. However if you have a page that loads with 4 links defined, does its script and then ends up 100+ links you miss the 100+ links. I notice people often failing to account for this in their crawlers, so I wondered if this one was.