3 ms·
Another issue, is that it is more and more difficult to create a web crawler as you need to render js to be sure to really see all content and you can't know be
by polote 6y ago
Another issue, is that it is more and more difficult to create a web crawler as you need to render js to be sure to really see all content and you can't know beforehand if the pages needs Javascript rendering. As it cost much more to render js than parsing a html, only Google and a few other big Corps can afford it
- walshemj 6y agoAnd Google does it in two stages a first pass that's then handed off to a second stage for full rendering
- ex3ndr 6y agoIs it? we just have multiple instances of puppeteer that makes screenshots and dumps rendered HTML for parsing. Very simple service with autoscaling it is very affordable.
- MereInterest 6y agoDefinitely more difficult than a single curl/wget request. You now need to parse the results, run a Turing-complete language, launch other requests based on output from that language, then wait for all results to be compiled into the final HTML, which may not even be possible for a continuously updated page.
- ex3ndr 6y agoThat's all literally 10-20 lines of code and actually much shorter than downloading html manually. Puppeteer can wait for final HTML.
- MereInterest 6y agoSolving the halting problem is now 10-20 lines of code? Because that would be needed in order to determine if a page is fully compiled to HTML.
- ex3ndr 6y agoThis is not a problem, this feature is built in in Chrome and is literally one line: https://pptr.dev/#?product=Puppeteer&version=v3.3.0&show=api-pagewaitfornavigationoptions https://pptr.dev/#?product=Puppeteer&version=v3.3.0&show=api...