8 ms·
Show HN: Ready-to-use API to convert any web page to PDF using headless Chrome
- conradk 9y agoSaving a webpage to a PDF is literally one command line away: chromium --headless --disable-gpu --print-to-pdf=google.pdf http://google.com/ http://google.com/ What does Apify add in this case?
- granda 9y agoAre there any downsides to me using this to build my own archiver?
- nikisweeting 9y agoMany projects are already using this for archiving purposes. Check out https://github.com/pirate/bookmark-archiver https://github.com/pirate/bookmark-archiver It's robust because the modern web is pretty much built for Chrome, although it can be resource-intensive if you're archiving many sites.
- xfer 9y agoconverting to pdf is lossy, i wouldn't use this for archiving purposes, normally i use webrecorder for archiving.
- jancurn 9y agoYou can call the API from anywhere, from resource-constrained servers, Docker containers that cannot run headless Chrome, JavaScript on a website etc. Also, we'll keep adding new features to this act to make it worthwhile to use it, e.g. retries on failures, posting of the file to some URL etc.
- nickjj 9y agoAlso if you can't access this command for whatever reason, another option is to open the print dialog box in Chrome and set the destination to "Save as PDF" and it will work. You'll even get to see a preview. It's very useful for 1 off saves where you want to consume a really long post offline in a PDF viewer.
- Scarbutt 9y agoSure, but the idea is to do it programatically.
- scaryclam 9y agoThanks, I didn't know that was a thing. I was checking out wkhtmltox earlier this week as well. Looks like there are a lot of options available for this. I suspect Apify's using one of them.
- jancurn 9y agoActually we're simply using Puppeteer - see the source code at the bottom of https://www.apify.com/jancurn/url-to-pdf https://www.apify.com/jancurn/url-to-pdf
- zulln 9y agoAny special reason to why you use this over Chrome?
- nikisweeting 9y agoPuppeteer is a scripting library for Chrome, built by the Chrome team.
- xilni 9y agoThis is exactly what I do bundled into a nice function added to my .zshrc chromepdf() { chrome --headless --disable-gpu --print-to-pdf="$1" $2 }
- zulln 9y agoWhat do you use this for?
- nsomaru 9y agoSave a webpage as a PDF
- zulln 9y agoDid not mean to sound negative, but what is the use-case of saving web pages as PDFs? I understand building in the functionality in something else, but here it sounds like you manually type/paste in URLs on a regular basis. Edit, I see now that I replied to the wrong comment. It was meant to they who made an alias to it.
- sgolestane 9y agoIs there anyway to make chrome wait until the page loads?
- lashkari 9y agoLooks like it prints automatically once Page.loadEventFired is triggered. Alternatively, you can run Chrome headless with the remote debugging API (--remote-debugging-port=9222) and send a Page.printToPDF (https://chromedevtools.github.io/devtools-protocol/tot/Page/#method-printToPDF https://chromedevtools.github.io/devtools-protocol/tot/Page/...) after some delay.
- Keats 9y agoIs there a way to add a delay to let JS render?
- jancurn 9y agoNow there is - I've just added the "sleepMillis" input option.
- Natsu 9y agoThere's also wkhtmltopdf that more or less does the same thing using webkit.
- jotto 9y agoI've been working on something similar: https://www.prerender.cloud/docs/api https://www.prerender.cloud/docs/api // URL to screenshot service.prerender.cloud/screenshot/https://www.google.com/ // URL to pdf service.prerender.cloud/pdf/https://www.google.com/ // URL to html (prerender) service.prerender.cloud/https://www.google.com/
- FabioFleitas 9y agoHeads up that I'm getting a "Too many requests for the month, sign up for an account at https://www.prerender.cloud/" https://www.prerender.cloud/" when trying to go to any of those links.
- jotto 9y agoThank you - I had an overaggressive rate limiter for non auth'd accounts, it's _improved_ now.
- rahulroy9202 9y agoNope. Still getting - Too many requests for the month, sign up for an account at https://www.prerender.cloud/ https://www.prerender.cloud/
- nikisweeting 9y agoThere are quite a few of these Chrome headless-as-a-service already out there, this is the third time I've seen one posted on HN.
- visarga 9y agoBy the way, is there an opposite service that converts PDF's into plain HTML for reading? I know about https://www.arxiv-vanity.com/papers/ https://www.arxiv-vanity.com/papers/ but it only works on arXiv PDFs.
- rpedela 9y agoBest one I have found. https://github.com/coolwanglu/pdf2htmlEX https://github.com/coolwanglu/pdf2htmlEX
- jancurn 9y agoThere's another Apify act that extracts text from a PDF using the pdf-text-extract NPM package - see https://www.apify.com/juansgaitan/pdf-scraping https://www.apify.com/juansgaitan/pdf-scraping If there's any library or tool that can convert PDF to HTML, it will only take a few minutes to setup such an API on Apify.
- jancurn 9y agoHere we go - another Apify act that uses pdf2htmlEX to convert PDF to HTML: https://www.apify.com/jancurn/pdf-to-html https://www.apify.com/jancurn/pdf-to-html
- jugjug 9y agoOff-topic, but Apify as a service looks really good. I was spinning up a dedicated VM on AWS with Docker installed only to get a simple webscraper running. Apify solves this elegantly and removes an significant pain in my workflow.
- dmmalam 9y agoAlso check out https://urlbox.io/ https://urlbox.io/. YC alum, super helpful.
- laktek 9y agoI built Screen.rip, which also supports PDF generation. https://screen.rip/#pdf https://screen.rip/#pdf Screen.rip gives you more control over the generated PDF beyond Puppeteer's options (like it can wait for certain elements to appear, inject CSS or switch to screen stylesheet instead of the print stylesheet).
- colordrops 9y agoAs long as GPU support is not functional in headless, "any web page" is a misnomer. A large enough percentage of sites use GPU acceleration so that headless mode is useless. This needs to be addressed by the Chrome team.
- kyberias 9y agoWhat do you mean "use GPU acceleration"? Are you saying that a large percentage of sites use WebGL? Using GPU acceleration for web page rendering is just a browser performance optimization. They can render the same page without GPU only slower.
- colordrops 9y agoI'm more specifically talking about WebGL. We'd love to use headless chrome at our company but we can't. But even for things like CSS transforms, we do a lot of really heavy 3D work and software emulation won't cut it.
- nikisweeting 9y agoCSS transforms work just fine without the GPU. We use it extensively for screenshot testing our CSS transform-based animations on https://oddslingers.com https://oddslingers.com.
- colordrops 9y agoWe are using a ton of them to create 3D heads up display camera overlays. They are too slow with GL software rendering.
- nikisweeting 9y agoNot sure what you mean? Usually people use either CSS 3D Transforms or WebGL, but not both.
- tehlike 9y agoyou could rpobably launch this service free, and someone will probably create a docker image, and make it one click.
- panda888888 9y agoDoes this work if the page is behind a password/SSO wall? And is it possible to print multiple Chrome tabs? Printing pages to PDF is pretty straightforward. It's the above two issues were I've run into problems. Anyone know of a good solution to the second one?
- gbrits 9y agoAssuming you want this done automatically, what's the advantage of 'printing' multiple tabs to PDF in a headless browser, over just sequentially loading and printing the pages you want done?
- jancurn 9y agoNot yet, but we can quite easily add these features. Just let me know at jan@apify.com what would you need
- laktek 9y agoYou can give https://screen.rip/#pdf https://screen.rip/#pdf a try. It supports capturing pages behind auth.
- nikisweeting 9y agoChrome headless does, just specify the --user-data-dir parameter to give it a profile to use (a profile where you're authenticated to the site you want to snapshot). e.g. chrome --disable-gpu --headless --user-data-dir=/Users/username/Library/Application\ Support/Google/Chrome/Default https://example.com/paywall/article.html https://example.com/paywall/article.html Alternatively, use puppeteer to script the auth process. https://github.com/GoogleChrome/puppeteer https://github.com/GoogleChrome/puppeteer
- Robdel12 9y agoQuestion: does this create accessible PDFs? That would be a really nice _possible_ work around for screen reader users having issues with a website.
- jancurn 9y agoI believe it's possible, by adding something like "scale: 1.5" to "pdfOptions" you might render an accessible PDF
- ak39 9y agoAny info how this compares to commercial html to pdf renderers like PrinceXML?
- defenestration 9y agoI would like to know that as well. We are using wkhtmltopdf for around 2 years. Had many issues with incompatible css. I considered PrinceXML. Looked solid, but a bit expensive and also lacking some css support. I'm considering switching to Puppeteer / Chrome.
- jancurn 9y agoIf you'd like to add specific features, please let me know at jan@apify.com, I'm sure we'll figure it out
- schneidmaster 9y agoIn my experience, Prince is great for static HTML + CSS rendering, but its JavaScript engine is pretty lackluster -- I couldn't get it to work with rendering React components, for example. So it depends a lot on your use case and if you can server-side render everything. It's also pretty pricey[1] -- not that I mind paying for quality software but that sticker could be prohibitive for a lot of folks. [1] https://www.princexml.com/purchase https://www.princexml.com/purchase
- jamespaden 9y agoI'm a developer at https://docraptor.com https://docraptor.com. We're an official Prince partner with a SaaS pricing model, but we've got a separate JavaScript engine for that very reason.
- waddlesworth 9y agoMy main concern is support for the CSS3 Paged Media Module. I have been using PDFReactor for this reason, as wkhtml2pdf and weasyprint have had problems, as will any webkit or chromium based renderer because these layout systems simply do not support paged printing.
- sebazzz 9y agoWe are not too happy with our EvoPDF license so in the basis this is a good option. However, I do not think this allows adding headers, footers, page numbers etc.
- jeppebemad 9y agoIs there a similar API around that accepts HTML instead of a URL? I’ve build one for my project, but I would prefer to delegate this to an external service.
- jancurn 9y agoWe can easily add this feature. So you'd like to pass a single HTML file?
- inferiorhuman 9y agoSo something like wkhtmltopdf?
- chrismorgan 9y agoBear in mind that you’ll need to either embed all your resources, or only use CORS-enabled resources, or fake the origin for your HTML document so that it can access non-CORS-enabled resources on a particular domain. Encoding your HTML as a data: URI might work for this service as-is (provided you use no non-CORS-enabled resources). Haven’t tried it.
- nikisweeting 9y agoIf you're interested in running your own personal Way-Back machine that uses Chrome headless for archiving (among other methods), check out Bookmark Archiver. https://github.com/pirate/bookmark-archiver https://github.com/pirate/bookmark-archiver
- phmagic 9y agoI love this service! I think ease of adoption you can allow pre-made scripts to be shared so the non-technical can easily set up work flows that go right into their email. For the technical folks, I think it would be great to have examples of things you can do with Apify that is a hassle to do with your local chrome headless. Great job!