5 ms·

Hey folks! I'm Joel, entrepreveloper behind this browserless.io service, happy to answer questions or chat!

wand3r 8y ago
Thanks for the article I ran into an issue a few minutes ago and phantomjs isn’t going to cut it. I am more of a hobbyist so this was a nice set of pointers and a good intro— id never heard of puppeteer. I was going to use nightmareJS or selenium but reckon ill try this out. Also, the screenshot tip is nice, i was driving myself crazy trying to figure out where the bug in my code was— there wasn’t one but the page was just not rendering. Screenshot is a nice sanity check
bmpafa 8y ago
head(less)s up: the links in the footer of your docs are busted. it looks like they all point to an `/en/` route but the actual URLs include no such tier.
- mrskitch 8y ago
  Ouch, thanks for the heads up!
gitgud 8y ago
Entrepreveloper, I like that.
avip 8y ago
Great write-up! Hands-on, code included, concise. Thanks for posting.
int0x80 8y ago
Hi. What is this exactly? Skimming through the site didnt give me a clear detailed explanation. Is it a programming api to chrome? As in: host:~$ ./browserless-app example.com # will fetch an interact using js/DOM with the web @ url Is this supposed to be used as a proxy between some random browser, a remote chrome browser and the http response? As in: <IE/edge/safary script> page-load: ws.connect(ws://mybless-service/); this.html = ws.recv()</script> Or is it more oriented to have a remote service which you can use to send commands to process and control browsing sessions? login to foobar.org; update this data. fetch something; return data/status too peer. I'm really lost. A sugestion. I cant check 4 layers of subproject dependencies in orther to have a clear picture of what the thing does. An how it does it. I dont know if this is a node server running standalone (can I autodeploy this service in my server? Is this a saas?), or something you use in the client js. Or something you can access from the client side that internally connects to a websocket. Or even a mix of all this. A clear arquitectural design diagram helps IMO in this things: - Layer A runs on yours/ours/both linux/$X server using Y framework - Layer B Is a websocket service (standalone/saas) for clients to consume. - Layer C is a browser client js API wrapper for the service. - This is useful for this <full use case and run session of the example> Dont get me wrong, this is not a hate comment. Is just a personal suggestion to help people not deeply invested in JS/node ecosystem. Maybe its me just being thick this morning and not getting it. Thanks, it looks very cool anyway.
- mrskitch 8y ago
  Yeah, definitely a product that you don't realize you need...until you find the issue. As of now it's a pool of headless Chrome instances that you can control remotely and run tasks through. Screenshots, PDFs, scraping... whatever you need. The reason that folks use this is because it's _really scary_ to run Chrome alongside your app's infra, so separating it out (much like a DB) is a good idea. In the near-future we'll offer functions and other REST endpoints for doing common and user-defined tasks. This make it even more contained so your app can keep running happily without babysitting Chrome. Great question!
  - nojvek 8y ago
    Hi Joel, I had fun collaborating with you last year on a puppeteer docker thread in GitHub. I’m super happy browserless is picking up. A bit jealous of your success but still rooting for you! Containerized Puppeteer is amazing. Hope Google acquires browserless.io some day.
  - int0x80 8y ago
    Thanks very much for the clarification. Indeed it looks very useful.
dschnurr 8y ago
Hey Joel–awesome service and post. Your tip about parallelizing via browser instances instead of tabs resonated with me as I've had a lot of headaches with one bad request taking down a full instance. That said, it does seem like the constant launch/close of puppeteer browser instances does have some sort of memory leak – Here's what happened when I switched from parallelizing via pages to browsers: https://imgur.com/a/rCz3oow https://imgur.com/a/rCz3oow Have you seen any similar behavior in your system? I initially thought I was forgetting to close the browser instance but triple checked my code and it does seem like I'm calling close() everywhere.
- mtrunkat 8y ago
  We have found 2 related issues: - Sometimes Promise returned by page.close() never resolves so it's good to call Promise.race() on that together with a Promise that resolves after some timeout period (30s?) - Sometimes Chrome process doesn't get killed so we are also manually killing remaining Chrome process after browser.close()
  - dschnurr 8y ago
    Good tips. How are you killing remaining zombie Chrome processes? Just polling the processlist and running "kill" commands?
- mrskitch 8y ago
  Are you using something like dumb-init or equivalent? That's a definite requirement as _not_ having it results in zombie processes hanging around (which do retain some memory footprint). Read more about that here: https://github.com/Yelp/dumb-init#why-you-need-an-init-system https://github.com/Yelp/dumb-init#why-you-need-an-init-syste...
  - dschnurr 8y ago
    I'm using dumb-init for the main node process in my docker container (node in-turn spawns chrome instances via puppeteer as requests come in). Is there a way to use dumb-init for each individual chrome instance as well? Or are you suggesting we make chrome the main container process and start new containers for each session?
testcross 8y ago
Hey. What is the error rate you face? Like, how often it is not possible to render a page because the navigation fails, chrome crashes, a strange exception is raised by puppeteer, ...
detaro 8y ago
Small feedback about the website: It should be easier to leave the docs subdomain and look at the main page. Took me way to long to get to your pricing page.
- mrskitch 8y ago
  Appreciate the feedback! I'm undergoing a massive redesign of the site that should be much friendlier. What you're seeing right now is the work of one person under tight deadlines :)
martin-adams 8y ago
Great article. Do you have any prevention of overloading a single domain if the number of concurrent requests is increased?

Thanks for the article I ran into an issue a few minutes ago and phantomjs isn’t going to cut it. I am more of a hobbyist so this was a nice set of pointers and a good intro— id never heard of puppeteer. I was going to use nightmareJS or selenium but reckon ill try this out. Also, the screenshot tip is nice, i was driving myself crazy trying to figure out where the bug in my code was— there wasn’t one but the page was just not rendering. Screenshot is a nice sanity check

bmpafa 8y ago

head(less)s up: the links in the footer of your docs are busted. it looks like they all point to an `/en/` route but the actual URLs include no such tier.

mrskitch 8y ago
Ouch, thanks for the heads up!

gitgud 8y ago

Entrepreveloper, I like that.

avip 8y ago

Great write-up! Hands-on, code included, concise. Thanks for posting.

int0x80 8y ago

Hi. What is this exactly? Skimming through the site didnt give me a clear detailed explanation. Is it a programming api to chrome? As in: host:~$ ./browserless-app example.com # will fetch an interact using js/DOM with the web @ url Is this supposed to be used as a proxy between some random browser, a remote chrome browser and the http response? As in: <IE/edge/safary script> page-load: ws.connect(ws://mybless-service/); this.html = ws.recv()</script> Or is it more oriented to have a remote service which you can use to send commands to process and control browsing sessions? login to foobar.org; update this data. fetch something; return data/status too peer. I'm really lost. A sugestion. I cant check 4 layers of subproject dependencies in orther to have a clear picture of what the thing does. An how it does it. I dont know if this is a node server running standalone (can I autodeploy this service in my server? Is this a saas?), or something you use in the client js. Or something you can access from the client side that internally connects to a websocket. Or even a mix of all this. A clear arquitectural design diagram helps IMO in this things: - Layer A runs on yours/ours/both linux/$X server using Y framework - Layer B Is a websocket service (standalone/saas) for clients to consume. - Layer C is a browser client js API wrapper for the service. - This is useful for this <full use case and run session of the example> Dont get me wrong, this is not a hate comment. Is just a personal suggestion to help people not deeply invested in JS/node ecosystem. Maybe its me just being thick this morning and not getting it. Thanks, it looks very cool anyway.

mrskitch 8y ago
Yeah, definitely a product that you don't realize you need...until you find the issue. As of now it's a pool of headless Chrome instances that you can control remotely and run tasks through. Screenshots, PDFs, scraping... whatever you need. The reason that folks use this is because it's _really scary_ to run Chrome alongside your app's infra, so separating it out (much like a DB) is a good idea. In the near-future we'll offer functions and other REST endpoints for doing common and user-defined tasks. This make it even more contained so your app can keep running happily without babysitting Chrome. Great question!
- nojvek 8y ago
  Hi Joel, I had fun collaborating with you last year on a puppeteer docker thread in GitHub. I’m super happy browserless is picking up. A bit jealous of your success but still rooting for you! Containerized Puppeteer is amazing. Hope Google acquires browserless.io some day.
- int0x80 8y ago
  Thanks very much for the clarification. Indeed it looks very useful.

dschnurr 8y ago

Hey Joel–awesome service and post. Your tip about parallelizing via browser instances instead of tabs resonated with me as I've had a lot of headaches with one bad request taking down a full instance. That said, it does seem like the constant launch/close of puppeteer browser instances does have some sort of memory leak – Here's what happened when I switched from parallelizing via pages to browsers: https://imgur.com/a/rCz3oow https://imgur.com/a/rCz3oow Have you seen any similar behavior in your system? I initially thought I was forgetting to close the browser instance but triple checked my code and it does seem like I'm calling close() everywhere.

mtrunkat 8y ago
We have found 2 related issues: - Sometimes Promise returned by page.close() never resolves so it's good to call Promise.race() on that together with a Promise that resolves after some timeout period (30s?) - Sometimes Chrome process doesn't get killed so we are also manually killing remaining Chrome process after browser.close()
- dschnurr 8y ago
  Good tips. How are you killing remaining zombie Chrome processes? Just polling the processlist and running "kill" commands?
mrskitch 8y ago
Are you using something like dumb-init or equivalent? That's a definite requirement as _not_ having it results in zombie processes hanging around (which do retain some memory footprint). Read more about that here: https://github.com/Yelp/dumb-init#why-you-need-an-init-system https://github.com/Yelp/dumb-init#why-you-need-an-init-syste...
- dschnurr 8y ago
  I'm using dumb-init for the main node process in my docker container (node in-turn spawns chrome instances via puppeteer as requests come in). Is there a way to use dumb-init for each individual chrome instance as well? Or are you suggesting we make chrome the main container process and start new containers for each session?

testcross 8y ago

Hey. What is the error rate you face? Like, how often it is not possible to render a page because the navigation fails, chrome crashes, a strange exception is raised by puppeteer, ...

detaro 8y ago

Small feedback about the website: It should be easier to leave the docs subdomain and look at the main page. Took me way to long to get to your pricing page.

mrskitch 8y ago
Appreciate the feedback! I'm undergoing a massive redesign of the site that should be much friendlier. What you're seeing right now is the work of one person under tight deadlines :)

martin-adams 8y ago

Great article. Do you have any prevention of overloading a single domain if the number of concurrent requests is increased?