5 ms·
Is it? we just have multiple instances of puppeteer that makes screenshots and dumps rendered HTML for parsing. Very simple service with autoscaling it is very
by ex3ndr 6y ago
Is it? we just have multiple instances of puppeteer that makes screenshots and dumps rendered HTML for parsing. Very simple service with autoscaling it is very affordable.
- MereInterest 6y agoDefinitely more difficult than a single curl/wget request. You now need to parse the results, run a Turing-complete language, launch other requests based on output from that language, then wait for all results to be compiled into the final HTML, which may not even be possible for a continuously updated page.
- ex3ndr 6y agoThat's all literally 10-20 lines of code and actually much shorter than downloading html manually. Puppeteer can wait for final HTML.
- MereInterest 6y agoSolving the halting problem is now 10-20 lines of code? Because that would be needed in order to determine if a page is fully compiled to HTML.
- ex3ndr 6y agoThis is not a problem, this feature is built in in Chrome and is literally one line: https://pptr.dev/#?product=Puppeteer&version=v3.3.0&show=api-pagewaitfornavigationoptions https://pptr.dev/#?product=Puppeteer&version=v3.3.0&show=api...