7 ms·
I’d like to know, what are people on HN using headless browsers for?
by RyanShook 8y ago
I’d like to know, what are people on HN using headless browsers for?
- stevenjohns 8y agoPersonally, for testing.
- mrskitch 8y agoYou’d actually be surprised by the applications, I definitely was. Some I’ve seen: - Screenshots - PDF generation - Functional tests - Regression tests - Scraping SPAs - Crazy stuff like printing user-generated content for personalization products..
- leetbulb 8y agoAmong some of these, I've also built an internal microservice for analyzing URL redirect chains. Input a URL and it will provide me with all of the redirects it goes through and information for each one including method (html meta, 301, etc), timing, etc.
- kwerk 8y agoI’ve done some hacky work redirect analysis but just the final url. Thought of open sourcing?
- skinnymuch 8y agoAwesome! What did you use to go through the redirects? I’ve had issues with some browser automation only giving the final url.
- stcredzero 8y agoI am using them to run node-electron hosted WebRTC proxies, to enable UDP-like networking for a real time browser game.
- jotto 8y agohttps://www.prerender.cloud/ https://www.prerender.cloud/ - server-side rendering API for single-page apps
- fake-name 8y agoPersonally? Dealing with cloudflare and other bullshit WAFs that break the internet.
- PetahNZ 8y agoRendering high res vectors (SVG)
- ikeboy 8y agoI scrape lots of data from services while logged in. Scraping is done with curl, but every so often cookies expire, so I use selenium to login, save cookies, then go back to curl.
- smallbigfish 8y agoIsn't curl fit to also do the login part and save the cookies?
- ikeboy 8y agoTons of stuff get loaded during login, hard to go through each one and code the proper urls and POST params etc, quicker to write login code.
- manquer 8y agoRecording of webrtc video conferencing sessions , pdf generation, spa test suites
- avip 8y agoHeadless browsers are the de-facto standard for scrapping websites which have some protection layer. Ad-fraud is also a thing.
- greenleafjacob 8y agoI'm using splash [1] for scraping websites for personal automation purposes. [1] https://splash.readthedocs.io/en/stable/ https://splash.readthedocs.io/en/stable/
- heipei 8y agoI'm using Chrome Headless to analyze user-submitted pages on https://urlscan.io https://urlscan.io I will save a screenshot, network request info, request content (for JS/CSS/HTML), cookies, log messages, JavaScript global variables etc. Annotate and correlate it all and you have a nice summary of what a certain website "does".
- orliesaurus 8y agoBuilt an DOM 2 png image api (hivis.io)
- williamxd3 8y agointeracting with an app which doesn't provides a public API and has too much frontend code to be worth reverse engineering, I just hack into the app's webpack bundled code and use it
- tnolet 8y agoI'm using Puppeteer for website / transaction monitoring at https://checklyhq.com https://checklyhq.com. People script their own checks and we run them for you on a cron like schedule and alert when things go off the rails.
- deejbee 8y agoAutomating logging into and data capture for financial/banking sites, building a new finance aggregation service.
- mskullcap 8y agoPDF generation for us. Had been relying on phantomjs for years, but it has been a bear. Attempted to write something with PDFKit to avoid the overhead of hosting headless chrome, but the effort was too great for our small team. Using chrome we are able to build high fidelity reports with the same web components we use in the application's UI.