5 ms·

Show HN: Chromda – Serverless Website Screenshots

88 points by lfarzati 7y ago

appwiz 7y ago
Very nice! You should publish it to the Serverless App Repo (https://aws.amazon.com/serverless/serverlessrepo/ https://aws.amazon.com/serverless/serverlessrepo/)
whoisjuan 7y ago
This is very impressive (the way that is implemented on top AWS services). I would be curious to see how accurate it is when it comes to rendering. I have a Screenshot as a Service side project (https://getscreenshot.rasterwise.com https://getscreenshot.rasterwise.com) that is also build with Puppeteer, but accuracy is a completely different animal. Out of the box Puppeteer is really not that accurate so you need fine tune it to address a huge amount of rendering edge cases. This is the area where most services come short imo. I have put a lot of effort into optimizing my service accuracy but it’s a never-ending task. There always seems to be one website that doesn’t render the way it’s supposed to be rendered. I really don’t understand why headless browsers seem to be developed with UI rendering as a low priority. Simulating a viewport is not equal to simulating an actual screen.
- cjr 7y ago
  I hear you. I run Urlbox (https://urlbox.io https://urlbox.io) and accuracy is one of the hardest parts, as well as scaling hundreds or thousands of instances of chrome since it’s a real memory hog. I also managed to get a version of the urlbox renderer working on lambda but it fails on some websites due to the /tmp space limits, for example trying to do a full page screenshot of a heavy page like dailymail would use up all of the available disk space in the lambda environment with chrome’s media cache.
  - whoisjuan 7y ago
    Love what you have done with URLBOX dude! That's my goal standard when it comes to improving accuracy for GetScreenshot. I have tested with your website preview and I think both of our solutions nail accuracy, but you have been longer in this game so you probably have way more edge cases covered. I still find myself trying to address new edge cases every certain time.
  - artur_makly 7y ago
    We managed to screenshot DailyMail.co.uk very well ..no coding required: https://app.visualsitemaps.com/share/6a79e25354f16408731b86006944dcbe https://app.visualsitemaps.com/share/6a79e25354f16408731b860...
- bhl 7y ago
  Isn’t there also a memory limit with using Puppeteer? Last time I checked, there was a hard coded memory value for screenshots within Chrome.
arvinaminpour 7y ago
Was looking at building something similar on top of Lambda but I'm so happy that I found this! Quickly browsed through the code and I don't think it supports local pages but might throw a PR your way to add support if you don't mind. Thanks!
pritambarhate 7y ago
One thing from the readme that concerns me: S3_ACL "public-read" It will be great if you can explain why and how is this used in the ReadMe file.
- grenoire 7y ago
  Searching the constant will yield one use in the code, further Googling yields https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview...
- lfarzati 7y ago
  This is for making the uploaded images in S3 accessible by anyone. The rationale is that the screenshots can be viewed by browsers without any sort of authorization. But since this is just an environment setting, you can change it if your use case is different.
sharma_pradeep 7y ago
Is there any javascript library to do the same? It would be useful to have a library as a dependency and just initialize it in the code and then you have screenshot in meta of each page
guillem_lefait 7y ago
Isn't a problem with lambda to have a potential CPU waiting time of 30 seconds ?
- SwiftyBug 7y ago
  I'm not sure I get what you meant. One AWS Lambda function can run for up to 15 minutes.
  - throw03172019 7y ago
    I think they meant run time which means they are concerned about how much it will cost since lambda charges by run time.
    - guillem_lefait 7y ago
      Yeah, that's correct, my question was about costs.
tlarkworthy 7y ago
Oh cool. I did a similar thing for snapshotting Unity games https://corepox.net/devlog/visual-game-worlds-hosted-on-functions https://corepox.net/devlog/visual-game-worlds-hosted-on-func.... The joy of Chrome is that it has a webGL software implementation, so you don't need to spin up GPUs to get a graphical runtime of a 3D game engine.
the_resistence 7y ago
Definitely a noob at npm, when I run 'npm install', it freezes at: [ ......] / extract:aws-sdk: sill extract aws-sdk@2.595.0 extracted to /home/foo/serverless_apps/chromda/node_modules/.staging/... Any tips or tricks would be much appreciated?
- the_resistence 7y ago
  Never mind. No idea why but installed after multiple tries.

88 points by lfarzati 7y ago

Very nice! You should publish it to the Serverless App Repo (https://aws.amazon.com/serverless/serverlessrepo/ https://aws.amazon.com/serverless/serverlessrepo/)

whoisjuan 7y ago

This is very impressive (the way that is implemented on top AWS services). I would be curious to see how accurate it is when it comes to rendering. I have a Screenshot as a Service side project (https://getscreenshot.rasterwise.com https://getscreenshot.rasterwise.com) that is also build with Puppeteer, but accuracy is a completely different animal. Out of the box Puppeteer is really not that accurate so you need fine tune it to address a huge amount of rendering edge cases. This is the area where most services come short imo. I have put a lot of effort into optimizing my service accuracy but it’s a never-ending task. There always seems to be one website that doesn’t render the way it’s supposed to be rendered. I really don’t understand why headless browsers seem to be developed with UI rendering as a low priority. Simulating a viewport is not equal to simulating an actual screen.

cjr 7y ago
I hear you. I run Urlbox (https://urlbox.io https://urlbox.io) and accuracy is one of the hardest parts, as well as scaling hundreds or thousands of instances of chrome since it’s a real memory hog. I also managed to get a version of the urlbox renderer working on lambda but it fails on some websites due to the /tmp space limits, for example trying to do a full page screenshot of a heavy page like dailymail would use up all of the available disk space in the lambda environment with chrome’s media cache.
- whoisjuan 7y ago
  Love what you have done with URLBOX dude! That's my goal standard when it comes to improving accuracy for GetScreenshot. I have tested with your website preview and I think both of our solutions nail accuracy, but you have been longer in this game so you probably have way more edge cases covered. I still find myself trying to address new edge cases every certain time.
- artur_makly 7y ago
  We managed to screenshot DailyMail.co.uk very well ..no coding required: https://app.visualsitemaps.com/share/6a79e25354f16408731b86006944dcbe https://app.visualsitemaps.com/share/6a79e25354f16408731b860...
bhl 7y ago
Isn’t there also a memory limit with using Puppeteer? Last time I checked, there was a hard coded memory value for screenshots within Chrome.

arvinaminpour 7y ago

Was looking at building something similar on top of Lambda but I'm so happy that I found this! Quickly browsed through the code and I don't think it supports local pages but might throw a PR your way to add support if you don't mind. Thanks!

pritambarhate 7y ago

One thing from the readme that concerns me: S3_ACL "public-read" It will be great if you can explain why and how is this used in the ReadMe file.

grenoire 7y ago
Searching the constant will yield one use in the code, further Googling yields https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview...
lfarzati 7y ago
This is for making the uploaded images in S3 accessible by anyone. The rationale is that the screenshots can be viewed by browsers without any sort of authorization. But since this is just an environment setting, you can change it if your use case is different.

sharma_pradeep 7y ago

Is there any javascript library to do the same? It would be useful to have a library as a dependency and just initialize it in the code and then you have screenshot in meta of each page

guillem_lefait 7y ago

Isn't a problem with lambda to have a potential CPU waiting time of 30 seconds ?

SwiftyBug 7y ago
I'm not sure I get what you meant. One AWS Lambda function can run for up to 15 minutes.
- throw03172019 7y ago
  I think they meant run time which means they are concerned about how much it will cost since lambda charges by run time.
  - guillem_lefait 7y ago
    Yeah, that's correct, my question was about costs.

tlarkworthy 7y ago

Oh cool. I did a similar thing for snapshotting Unity games https://corepox.net/devlog/visual-game-worlds-hosted-on-functions https://corepox.net/devlog/visual-game-worlds-hosted-on-func.... The joy of Chrome is that it has a webGL software implementation, so you don't need to spin up GPUs to get a graphical runtime of a 3D game engine.

the_resistence 7y ago

Definitely a noob at npm, when I run 'npm install', it freezes at: [ ......] / extract:aws-sdk: sill extract aws-sdk@2.595.0 extracted to /home/foo/serverless_apps/chromda/node_modules/.staging/... Any tips or tricks would be much appreciated?

the_resistence 7y ago
Never mind. No idea why but installed after multiple tries.