3 ms·
To add to the sibling posts: Crawling the Web. Although you might think that you can crawl a web site by poking through the HTML looking for <a> tags and openi
by dboreham 9y ago
To add to the sibling posts:
Crawling the Web. Although you might think that you can crawl a web site by poking through the HTML looking for <a> tags and opening the URLs they contain; that stopped working well a long time ago. Modern sites are essentially complex GUI applications that run inside...a browser. You need either a browser, or some browser-equivalent thing therefore in order to run them and properly crawl them.
- camoby 9y agoIsn’t this why these ‘headless’ solution have (or are actually built on) JavaScrict engines like V8 with the ability to also simulate any user interaction?