4 ms·

Can someone do an ELI5 on why headless browsers are useful? I've use them before to make screenshots of webpages, and I have noticed that many NPM packages com

Can someone do an ELI5 on why headless browsers are useful? I've use them before to make screenshots of webpages, and I have noticed that many NPM packages come (or came) with PhantomJS as a dependency, but I have no idea why one would need that.

cbhl 9y ago
Automated integration/regression tests is a pretty big use case. You spin up a copy of your server, point the headless browser at it, run some JavaScript to simulate some user interactions, and verify the page contains the right strings / screenshot doesn't deviate too much from a golden image. I guess the ELI5 way of saying that would be: "Making sure your website still works." People also headless browsers for content generation inside larger systems. (I saw one team make videos by creating CSS animations, and then capturing screenshots from the headless browser at 30 fps.)
- nkkollaw 9y ago
  Got it. Thanks!
  - tyingq 9y ago
    Scraping for data is another big use case. Like a price comparison web site. Or the company LinkedIn is unhappy about because they are scraping profile data.
- nashashmi 9y ago
  So an ELI5 of everything you said would be: Putting the website through user actions in a firefox browser and recording/capturing/documenting the results headlessly. Headless version allows you to go through hundreds of results in the background.
  - cbhl 9y ago
    Maybe. I got stuck trying to imagine whether a five-year-old would understand what "headless" means in a software context (as opposed to, "headless dragon" or something).
paps 9y ago
Where I work (https://phantombuster.com https://phantombuster.com) we use them for scraping the web. They're a hassle-free way of getting the data. No need to worry about CORS, sessions, cookies, CSRF and other modern web stuff. Just simulate a human and you’re in.
- AndyNemmity 9y ago
  Yeah, I used to work for a company doing similar things. Was more expensive that way, but there were tons of sites you could only get the data being in a proper browser.
dboreham 9y ago
To add to the sibling posts: Crawling the Web. Although you might think that you can crawl a web site by poking through the HTML looking for <a> tags and opening the URLs they contain; that stopped working well a long time ago. Modern sites are essentially complex GUI applications that run inside...a browser. You need either a browser, or some browser-equivalent thing therefore in order to run them and properly crawl them.
- camoby 9y ago
  Isn’t this why these ‘headless’ solution have (or are actually built on) JavaScrict engines like V8 with the ability to also simulate any user interaction?
jeeyoungk 9y ago
phantomjs is a lot of work to maintain, and the developer officially deprecated it, recommending people to use chrome headless mode instead.

cbhl 9y ago

Automated integration/regression tests is a pretty big use case. You spin up a copy of your server, point the headless browser at it, run some JavaScript to simulate some user interactions, and verify the page contains the right strings / screenshot doesn't deviate too much from a golden image. I guess the ELI5 way of saying that would be: "Making sure your website still works." People also headless browsers for content generation inside larger systems. (I saw one team make videos by creating CSS animations, and then capturing screenshots from the headless browser at 30 fps.)

nkkollaw 9y ago
Got it. Thanks!
- tyingq 9y ago
  Scraping for data is another big use case. Like a price comparison web site. Or the company LinkedIn is unhappy about because they are scraping profile data.
nashashmi 9y ago
So an ELI5 of everything you said would be: Putting the website through user actions in a firefox browser and recording/capturing/documenting the results headlessly. Headless version allows you to go through hundreds of results in the background.
- cbhl 9y ago
  Maybe. I got stuck trying to imagine whether a five-year-old would understand what "headless" means in a software context (as opposed to, "headless dragon" or something).

paps 9y ago

Where I work (https://phantombuster.com https://phantombuster.com) we use them for scraping the web. They're a hassle-free way of getting the data. No need to worry about CORS, sessions, cookies, CSRF and other modern web stuff. Just simulate a human and you’re in.

AndyNemmity 9y ago
Yeah, I used to work for a company doing similar things. Was more expensive that way, but there were tons of sites you could only get the data being in a proper browser.

dboreham 9y ago

To add to the sibling posts: Crawling the Web. Although you might think that you can crawl a web site by poking through the HTML looking for <a> tags and opening the URLs they contain; that stopped working well a long time ago. Modern sites are essentially complex GUI applications that run inside...a browser. You need either a browser, or some browser-equivalent thing therefore in order to run them and properly crawl them.

camoby 9y ago
Isn’t this why these ‘headless’ solution have (or are actually built on) JavaScrict engines like V8 with the ability to also simulate any user interaction?

jeeyoungk 9y ago

phantomjs is a lot of work to maintain, and the developer officially deprecated it, recommending people to use chrome headless mode instead.