8 ms·
Headless Chrome support in Cloud Functions and App Engine
- k__ 8y agoDoes this mean that HC is preinstalled in the runtime? Becauee as far as I know you can already run HC with other FaaS solutions, but having this out of the box would be really nice.
- tnolet 8y agoRunning Headless Chrome / Chromium is a bit of a hassle on AWS Lambda and other FAAS providers. Chrome requires some specific bindings/binaries to work. I think the Chrome guys and girls convinced their Cloud coworkers to provides these in the underlying Linux machines that run the Cloud Functions.
- mylesborins 8y agoexactly this. The base operating system comes with the system libraries necessary to support headless chrome out of the box. (disclaimer: I work for Google Cloud)
- giancarlostoro 8y agoWhat's to really stop other providers from doing the same thing though? :)
- tener 8y agoProject priorities, I guess.
- manigandham 8y agoNothing. It's about developer convenience, not technical possibility.
- clarkevans 8y agoAny chance of also supporting Firefox?
- skunkworker 8y agoIt actually is easier than people think, after chrome added the ability to disable shm (which isn't available on lambda) you can statically compile it into a single binary and run it with chromedriver & capybara easily. Unfortunately Google Cloud Functions don't natively support golang yet so for my business this is a non-starter.
- philliphaydon 8y agoI just found this https://github.com/adieuadieu/serverless-chrome/blob/master/README.md https://github.com/adieuadieu/serverless-chrome/blob/master/... Don’t know if it works. Going to try this week.
- emilburzo 8y agoIt works and quite well, it's been pretty much set-and-forget for me for quite some months now.
- atebyagrue 8y agoSeconded. It works great. I've been using this in a lambda web collection project for the past year. Initial load times for headless chrome can be a bit slow, but iterative uses after it has launched are pretty fast.
- isuckatcoding 8y agoI wonder how feature/pricing compete with Browserless?
- mrskitch 8y agoOur small instance ($30/month) is roughly similar to their $44.38 instance on App Engine. If you were to run a full 10 concurrent sessions constantly, which a small browserless instance can max-out at, this would cost roughly $427 a month in Functions. So depends on your use-case
- fefb 8y agoFor reference, how did you get $427? Thanks in advance
- mrskitch 8y agoRunning 10 concurrent Google Functions at 1GB/1CPU. Assumes 30 days in a month
- fefb 8y agoI am getting a different price from GCP calculator. Almost $2000. In your example, 10functions por Second, so each function is taking 1000ms to finish. 2592000s in a month * 10invocation = 25920000 invocations month, running with a 1GB function and 500kb of network bandwidth (out). https://cloud.google.com/products/calculator/#id=555e1af7-c858-4573-9d8a-a7d87475c236 https://cloud.google.com/products/calculator/#id=555e1af7-c8...
- mrskitch 8y agoAh, I wasn’t calculating their other fees like invocations and networking costs. Crazy it’s almost an order-of-magnitude
- isuckatcoding 8y agoThanks makes sense it would be based on use case. Not sure who is petty enough to downvote my comment.
- tnolet 8y agoI've been screwing around with running Headless Chrome & Puppeteer on Lambda/Serverless/FAAS solutions. It's all a bit of a mixed bag. You CAN run Headless Chrome on AWS Lambda, but the cost involved is pretty crazy as you need ~1500Mb in RAM to comfortably run any code with Chrome. Google Cloud of course has "inside knowledge" and I would love to switch to them for my SaaS https://checklyhq.com https://checklyhq.com, were it not that Google Cloud Functions is just offered in four (!) regions...
- mrskitch 8y agoGreat to see you here Tim, love that chrome extension! We should chat a bit more sometime. I’d love to back checkly’s infrastructure.
- bergie 8y agoNice to see this concept get into the big cloud platforms. We built something similar couple of years ago, primarily to get a sandbox for some compute jobs we were running on Heroku: https://github.com/flowhub/jsjob https://github.com/flowhub/jsjob
- mostlystatic 8y agoI tried to use Headless Chrome on Cloud Functions for a project I'm working on, but even on the fastest instances loading pages was sometimes really slow (pages timing out after waiting for 60s). It seems sometimes JS execution was taking a long time, so I guess that was preventing requests from being made. In a single CPU cloud function you have network requests, JavaScript execution, rendering, and the Node process controlling the browser all competing for resources. That being said, it was super simple to get started!
- leesalminen 8y agoHow many requests per minute were you running? I was planning on using Headless Chrome on Cloud Functions to take in some arbitrary HTML and output it to a PDF. In my case I can’t see >2 rpm happening anytime soon.
- mrskitch 8y agoHave you seen https://docs.browserless.io/docs/pdf.html https://docs.browserless.io/docs/pdf.html? It’s in a docker image too if that’s all you need.
- leesalminen 8y agoJust saw your other comment! The /function call looks interesting. Would I be able to use puppeteer’s request intercept API with it? A bit more on my use case: currently using pdf.js for rendering HTML reports as a PDF. It’s been a pain. I’d like to essentially take the HTML of some (say, just a <table>) or all (full HTML doc) of the current page and send it off. The main gotcha I can think of is relative URLs but intercept request would resolve that.
- mrskitch 8y agoYes: anything available on the `page` object is available in the /function route, so request interception is available!
- 8y ago
- eknkc 8y agoTried HC just yesterday on cloud functions. For some reason, it runs extremely slow. Did some comparisons to AWS lambda with similar memory / cpu sizes and basic “load page and screenshot” jobs would take 2x - 3x more time on google cloud functions. I’ll dig deeper soon but this is a bad start.
- leesalminen 8y agoIs it still within say 5 seconds? Do you get the impression there’s some config tuning to do or something you can’t control?
- rmdashrfroot 8y agoI wrote a collection of Dockerfiles for images running Python 2.7 or Python 3.6 + Selenium with either Chrome or Firefox and using Xvfb for the X display (necessary for running Selenium headlessly). https://github.com/seanpianka/docker-python-xvfb-selenium-chrome-firefox https://github.com/seanpianka/docker-python-xvfb-selenium-ch... Using this, in conjunction with AWS Step Functions, Lambda, and ECS, it became merely cents a month to run a headless scraper task in the cloud.
- lars_francke 8y agoCan you elaborate a bit? Sounds interesting. I had never heard of AWS Step Functions before. What does your workflow look like?
- tjbiddle 8y agoNot OP, but to elaborate on AWS Step Functionss: In short - this gives you the ability to pass the output of one lambda function to the input of another lambda function. An example of one that I've written to regularly create a new copy of our Production RDS database in Ireland as a Staging RDS database in Oregon. 1. Cloudwatch Event starts Step Function on the 15th 2. Copy last Production snapshot from Ireland to Oregon 3. Restore this snapshot as a new RDS instance (It will fail until the snapshot is available and retry with exponential backoff - this is a step function feature) 4. In parallel: - Add tags to the instance (Once it's available) - Delete the snapshot copy (When finished restoring) - Modify the new instance with security groups and subnets - In parallel: - Run a SQL query to anonymize all of PII columns for GDPR compliance as data has now left the EU. - Call out to the Cloudflare API to update our DNS entry with the new RDS endpoint. - Delete the old Staging database instance
- ocdnix 8y agoDo you run into problems with Lambda's 5-minute maximum execution time for those kinds of operations? I'd like to do something similar to this for both RDS and DynamoDB, but the execution time will often surpass 5 minutes, meaning I'd have to run a Step Functions worker on EC2 or ECS. That opens up a whole bunch of complexity with managing the worker code and its deployment, which I'd rather avoid if possible.
- mrskitch 8y agoIf Google cloud ain’t your jam then checkout browserless (https://browserless.io/ https://browserless.io/). It can be considerably cheaper under certain situations, and we’ve been up and running for almost a year. Happy to answer questions if anyone has any. EDIT: We’ve got stuff on GH: https://github.com/joelgriffith/browserless https://github.com/joelgriffith/browserless, and startup is under 100ms most of the time. Fonts and other things “just work” as well, plus there’s a slew of REST APIs for common stuff as well. Selenium webdriver support landing soon!
- jotto 8y ago...and if browserless ain't your jam, checkout https://www.prerender.cloud/ https://www.prerender.cloud/ cheap, because we optimized for just 3 things: pre-rendering, screenshots, or PDFs for $0.000365 per API request curl https://service.prerender.cloud/screenshot/https://google.com/ > out.jpg curl https://service.prerender.cloud/pdf/https://google.com/ > out.pdf curl https://service.prerender.cloud/https://google.com/ > out.html
- jotto 8y agonote: prerender.cloud just keeps a pool of instances running, so there is no startup latency, compared to Google Cloud functions
- Operyl 8y agoPricing is in tiers, not per request as you seem to imply?
- deleted 8y ago[deleted]
- jotto 8y agoCorrect, only the final tier is variable rate; apologies for the confusion. Under 20,000 monthly requests = $9 flat rate ($0.00045) Under 100,000 monthly requests = $40 flat rate ($0.0004) >= 100,000 monthly requests = variable rate @ $0.00036/req
- schappim 8y agoI wish Google would support Ruby! If you do too checkout the petition over at https://www.serverless-ruby.org https://www.serverless-ruby.org
- mrskitch 8y agoSomething somewhat related: I think more Cloud providers need to start doing Docker-functions. It’s always going to be a waiting game for runtimes and upgrades. Docker is portable and can run just about anything, so why not support that? https://zeit.co/blog/serverless-docker https://zeit.co/blog/serverless-docker is an example of what I mean
- dantiberian 8y agoGoogle Cloud has Docker based Cloud Functions ("Serverless Containers") in private alpha at the moment. More info at https://cloud.google.com/blog/products/gcp/cloud-functions-serverless-platform-is-generally-available https://cloud.google.com/blog/products/gcp/cloud-functions-s..., and https://youtu.be/Y1sRy0Q2qig?t=33m54s https://youtu.be/Y1sRy0Q2qig?t=33m54s.
- hugelgupf 8y agoSee the "Serverless Containers" section on https://www.google.com/amp/s/gweb-cloudblog-publish.appspot.com/products/gcp/cloud-functions-serverless-platform-is-generally-available/amp/ https://www.google.com/amp/s/gweb-cloudblog-publish.appspot.... It's still in alpha, so Google is not advertising it wide. It was announced about a week before Zeit's Serverless docker at GCP Next. Disclaimer: I work at Google and I've had some involvement in the tech behind this.
- exikyut 8y agoThat's an interesting domain: 'http://gweb-cloudblog-publish.appspot.com/' http://gweb-cloudblog-publish.appspot.com/' (it just redirects to cloud.google.com)
- acdha 8y ago
- wslh 8y agoNot perfect but HtmlUnit anyone? I used it for scraping in the past with mixed experiences.
- ksahin 8y agoHtmlUnit API is really cool. It's fine for most use cases. But obviously the Javascript support is not perfect. [Shameless plug] I wrote a book about web scraping where I talk about HtmlUnit & headless chrome with Java: https://www.javawebscrapinghandbook.com https://www.javawebscrapinghandbook.com
- guiomie 8y agoOk, I was just looking into this 1 week ago and was gonna spin up a VM to do headless. Now I get to keep my firebase project in cloud functions only. Much cleaner architecture.
- kwerk 8y agoI’ve tested HC on GCF and GAE standard with the IO launch. Sadly they’re 3-10x slower than App Engine Flex (same vm size on App Engine Flex vs Standard). Even a screenshot of google.com takes 6+ seconds on GCF / GAE Standard vs 2 seconds for Flex. I hope they fix this as spinning to zero is important for me but the latency is too high right now.
- iamjustlooking 8y agoAmong other things with puppeteer we do screenshot generation using GKE on Google Cloud @ https://screenshots.cloud/ https://screenshots.cloud/ scaling up and down running instances depending on demand. We keep browser instances running constantly as the startup time is significant. I will be interested to see what the startup time is for puppeteer on this, will definitely be giving it a try.
- exikyut 8y agoNice reference immediately above "used by us" :) One completely unrelated thing. On Chrome 68.0.3440.84, I noticed the large icons (particularly the Kubernetes one) looked "weird", with jagged edges that didn't make any sense. Some poking with the devtools revealed that 'backface-visibility: hidden' seems to be disabling antialiasing. Suggest opening the following in new tabs so you can flip back and forth between them: - As is right now: https://i.imgur.com/nYzsukI.jpg https://i.imgur.com/nYzsukI.jpg - Nicer-looking: https://i.imgur.com/GNlvx7Z.jpg https://i.imgur.com/GNlvx7Z.jpg I noticed disabling this has an effect on the animation at the top (the edges of the moving webpage slides don't have constantly-moving jaggies). There may well be a valid reason you have this enabled, perhaps for added performance. Or perhaps React added it in for you? :P
- iamjustlooking 8y agoThanks for spending time to let me know because I don't think I would have noticed it otherwise! I can't see it on retina but I can on my non retina display. We'll have to make the CSS rule more specific. As to why it's there I believe Firefox 57 or around that version had an issue with the sliding animation on the top of the page causing images to tear or not render at all when they scrolled in. This bug must have been solved recently because disabling backface-visibility on the image doesn't cause the same tearing.
- exikyut 8y agoI was very curious what was causing the non-antialiasing, it was fun. And you can repro :) cool. Makes a lot of sense you can't see it on retina. Interesting FF bug you hit. CSS3 GPU-accelerated animations are incredibly complex... heh, adding the rule fixed Firefox ~57, but now Chrome 68 is glitching out because the rule is there. I wonder if Google realizes yet. Ponders complexity of creating minimal testcase, versus waiting for someone else to notice :P
- antoncohen 8y agoI think it would be pretty cool to use Cloud Functions as a Selenium Grid, sort of like Zalenium (https://github.com/zalando/zalenium https://github.com/zalando/zalenium) does with Kubernetes. If you could parallelize end-to-end tests enough, you could get massive burstable capacity to run parallel tests.
- pwaai 8y agothat's it...im moving to GCP sorry but Rekognition rekt it for any type of computer vision on AWS. Great infrastructure...after all I do have an AWS Solution Architect Associate certification....which means jack shit Great move by GCP, I'm also very pleased with Firebase and it's integration with cloud functions.... BUT my biggest reservation still in 2018 when it comes to serverless is the cold start up time... I built a token based API on AWS Lambda and registering, signing up took forever when the app was not at peak. that was 2014 tho.
- aviv 8y agoWe use GCP heavily. It's really great. Latency between Google Cloud Functions and other GCP products improved significantly in the past week, as did start up and execution time. However we only use GCFs for background tasks and not for any web API/micro services, etc. Still too much latency for that use case.
- fefb 8y agoThe Cold Start of AWS lambda is better now. You can get under 1000ms. However, Google Function needs to improve the Cold start.
- benatkin 8y agoI'm not buying it, Google Cloud just moved to node 8 earlier this year as the post says, but now it's node 10. It's just not good tech, it's unnecessary lock-in. Docker on anything is better, this is similar: https://zeit.co/blog/serverless-docker https://zeit.co/blog/serverless-docker
- chrisabrams 8y agoIt’s wonderful tech that has saved us thousands a month. The lock in is very little as it’s running a JS function. We’ve written our functions so that the export to the GC function merely passes in arguments to another function that is required. We could move to AWS, Zeit, or anywhere else with little friction.
- nojvek 8y agoI don’t know what this means for browserless.io but I hope he still retains a strong niche and has a desirable product.
- techsin101 8y agoHow Google is behaving recently... I can feel it becoming Oracle. I want to stay 10 miles far from it. Learned my lesson with Google maps.
- defied 8y agoMy company provides a similar service, with both Chrome and FireFox headless support for Automated Testing/Screenshots: https://testingbot.com/support/getting-started/headless.html https://testingbot.com/support/getting-started/headless.html We run each test in a new VM, running on our own private cloud (dedicated servers). Note: we use the Selenium protocol for this, not yet Puppeteer.
- dakom 8y agoCan someone please explain what the advantage of running a snapshot service via GCF would be vs. AppEngine Standard (w/ node)?
- wereHamster 8y agoLower costs. With AppEngine you pay for a whole instance, regardless of how much utilisation it gets. With cloud functions you only pay for the time the function is being executed. If your code isn't being executed often, GCF are much cheaper.
- dakom 8y agoOh I see... AppEngine is also "pay for what you use" but it seems it's rounded to 15 mins rather than CloudFunctions which is 100ms. Thanks!
- jancurn 8y agoIt's nice to see serverless platforms adding support for headless Chrome. But there's still one problem with AWS Lambda / Cloud Functions / Zeit Now - the run time is limited to a few minutes. If you want to run any longer job, e.g. a web crawler, you need to either spin up the instances yourself or use platform like Apify, which allows running arbitrary-long jobs, provides pre-built Docker images for headless Chrome or XVFB, and provides SDK to simplify state persistence, access to proxies etc. For example, a simple actor to convert HTML to PDF looks like this: https://www.apify.com/jancurn/url-to-pdf https://www.apify.com/jancurn/url-to-pdf More info: https://www.apify.com/docs/actor https://www.apify.com/docs/actor https://www.apify.com/docs/sdk/apify-runtime-js/latest https://www.apify.com/docs/sdk/apify-runtime-js/latest https://www.apify.com/library?type=acts https://www.apify.com/library?type=acts Disclaimer: I'm a co-founder of Apify
- patd 8y agoI'm currently using a headless Chrome for my latest project www.blockedby.com (still in alpha stage, looking for feedback) I've been looking at a non-local solution. I'm using Python and this article hints that Puppeteer is not the only way to invoke this. But I don't see any documentation on the DevTools protocol. Anyone knows if it's supported ? Or any providers that do ?
- anyzen 8y agoI'm not sure if this is what you are looking for, but I have been using headless Chrome through Selenium (using nightwatch.js) for some time now and it works great. You will probably need to rewrite your tests though, but they will support FF too. YMMV.
- transreal 8y agoYou should just need to get a handle on the host and port to use Devtools protocol to talk to the launcher headless chrome instance. If you use node.js to start it, you can get the port like this: https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#browserwsendpoint https://github.com/GoogleChrome/puppeteer/blob/master/docs/a... Then you can use PyChromeDevtools to connect to that host/port: https://github.com/marty90/PyChromeDevTools/blob/master/README.md https://github.com/marty90/PyChromeDevTools/blob/master/READ...