5 ms·

I run a ton of Puppeteer jobs (300k in the last month), currently on EC2 and Digital Ocean VM's, mostly due to the subtle difficulties of running Puppeteer on L

by tnolet 8y ago

I run a ton of Puppeteer jobs (300k in the last month), currently on EC2 and Digital Ocean VM's, mostly due to the subtle difficulties of running Puppeteer on Lambda. Will certainly have a look at this project and contribute where possible. My main concerns are not so much cold start time, as for my use case this is not really a huge issue, but mainly the performance of Chrome on AWS Lambda boxes. The rendering, navigation etc. needs to be snappy.

thesandlord 8y ago
Google App Engine and Google Cloud Functions got native support for Puppeteer a few months ago as well. Let me know what you think if you try it out. https://news.ycombinator.com/item?id=17795626 https://news.ycombinator.com/item?id=17795626 (I work for Google Cloud)
- alixaxel 8y ago
  The performance of puppeteer is super bad on CGF (you can read more about it here https://github.com/GoogleChrome/puppeteer/issues/3120 https://github.com/GoogleChrome/puppeteer/issues/3120). It would actually be great to have someone really improve this situation instead of dismissing it as a weird IO problem.
  - thesandlord 8y ago
    Did some research internally, this is being tracked but still no root cause AFAIK :(
- tnolet 8y ago
  Would love to use GCF but the performance is terrible (as mentioned) and I need more geographic locations than GCF offers.
alixaxel 8y ago
I also run hundreds of thousands of puppeteer sessions every month, all on Lambda and so far I'm pretty happy with it, from scalability itself to session performance. Granted, there are some issues with rendering (fonts, emojis and whatnot) but meanwhile there are solutions available that could be explored. Feel free to try it out and share your specific challenges on GitHub, I'll do my best to come up with solutions for them.
- russian_bot 8y ago
  hi, out of curiosity, what is it that you do that demands so many sessions? Just webscraping?

by tnolet 8y ago

thesandlord 8y ago

Google App Engine and Google Cloud Functions got native support for Puppeteer a few months ago as well. Let me know what you think if you try it out. https://news.ycombinator.com/item?id=17795626 https://news.ycombinator.com/item?id=17795626 (I work for Google Cloud)

alixaxel 8y ago
The performance of puppeteer is super bad on CGF (you can read more about it here https://github.com/GoogleChrome/puppeteer/issues/3120 https://github.com/GoogleChrome/puppeteer/issues/3120). It would actually be great to have someone really improve this situation instead of dismissing it as a weird IO problem.
- thesandlord 8y ago
  Did some research internally, this is being tracked but still no root cause AFAIK :(
tnolet 8y ago
Would love to use GCF but the performance is terrible (as mentioned) and I need more geographic locations than GCF offers.

alixaxel 8y ago

I also run hundreds of thousands of puppeteer sessions every month, all on Lambda and so far I'm pretty happy with it, from scalability itself to session performance. Granted, there are some issues with rendering (fonts, emojis and whatnot) but meanwhile there are solutions available that could be explored. Feel free to try it out and share your specific challenges on GitHub, I'll do my best to come up with solutions for them.

russian_bot 8y ago
hi, out of curiosity, what is it that you do that demands so many sessions? Just webscraping?