16 ms·

Creating an open-source solution to the headaches of headless browsers

252 points by mrskitch 7y ago

deleted 7y ago
[deleted]
sneak 7y ago
> There's always been this thought that I've had, that advertising and "paid" attention is really in no one's best interest. You're likely to get users who really aren't going to get any value out of your software, so your churn increases, and you've also just paid for that user that's churned. These things get harder to tease out since it's almost impossible to ask "show me all the churned users this period acquired from advertising channels." Maybe that's possible, but you'd have to do a lot wrangling together to get it all working. Correct me if I’m wrong, but isn’t precisely that kind of analytics simply table stakes for any modern crm/marketing/customer intelligence suite in 2019? It seems like that is absolutely a solved problem.
- mrskitch 7y ago
  Yeah, it's a pretty contrived example on my part. My sentiment here is that there's so many inputs to modeling behavior, and trying to find signal in the noise, that at this scale your time is likely spent better elsewhere. Unless you have the revenue stream to do it and do it well, then the effort can be a time sink
  - sneak 7y ago
    How many variables do you need to track, though? $X spend, Y signups at $Z each, AA% churn, $BB retained MRR for an estimated $CC CLV based on an $X/Y*(1-AA%) CAC - why does it need to be much more complicated than that when you don’t have millions of users? (Seriously though, I’m asking, not poking fun. You probably know more about this stuff than I do, having actually done it. It seems really simple to figure, to me. What am I missing?)
    - mrskitch 7y ago
      Stripe honestly does a lot of this for you. You can hack all that together with what you've got, the problem is that there's some seasonality to business as well as other market effects that make it tough to determine _why_ something is trending one way or the other. For instance: we had a week last year where we had a flood of cancellations at once and there was nothing I could attribute it to. Looking back it was just a coincidence, however it consumed a lot of my time (writing emails and looking at analytics) to figure out why instead of just pushing ahead. I was likely over-reacting, but I have noticed there's a lot of people spending a lot of time doing analytics and doing research on trends in stead of just executing. And, especially early on, you should just be executing and not thinking too much about trends.
    - matthiaswh 7y ago
      It's more a matter of being messy rather than complicated. You're trying to track users who often visit your site multiple times before purchasing, coming from different sources, on different devices, and somehow tie that attribution to the purchase. None of the data inputs are consistent nor tied together. Analytics tells you differently than your ad platform, your payment processor tells you yet another thing, and your email marketing tool, and your CRM. (And if you want the serious tools for data monitoring and reporting have moved to focusing on enterprises...) You have to somehow factor in refunds, free trials, prorated billing, early cancellations. You have multiple ad campaigns running ad variations. Don't forget a/b testing. Finally you see some data point that hints something might be working, but you know you have to account for all the other factors involved. Did I make any website edits that day? Did the ad network change their algorithm slightly? Was there a holiday affecting traffic? When did I insert that new ad again? Wait, I know I changed my ad bid at some point... Did I get an influx of traffic from another source? Was it just a fluke? If you want good, real data, it's messy. And far from a solved problem.
      - mrskitch 7y ago
        Wow, you said this so much better than I could, thanks for chiming in
MuffinFlavored 7y ago
What $288k/yr headache is there around `docker pull buildkite/puppeteer`? https://hub.docker.com/r/buildkite/puppeteer https://hub.docker.com/r/buildkite/puppeteer
- maxaf 7y ago
  I dove into the comments to ask the same question. I just did something similar to this with a few lines in my docker-compose.yml, which eventually turned into a few lines in a Helm chart. Why would I pay someone money for something as trivial as this? Now, I won’t write another rant on the subject of Eternal September, and how our field is increasingly populated by charlatans who can’t do simple things on their own. But yes: that’s exactly how I feel.
  - avip 7y ago
    Disclaimer: Neither I nor my colleagues or anyone I'm aware of are, were, or ever will, use said service. Did you divide (time that took you to complete this trivial task + time to document it, version it, explain it to coworkers, maintain it next time chrome decides to break something) * (your $/h cost) / 0.00008 before doing that?
    - rozenmd 7y ago
      Here's a build vs buy calculator to save you from doing the maths: https://baremetrics.com/startup-calculator https://baremetrics.com/startup-calculator
      - viraptor 7y ago
        This is a dangerous calculator. It misses the cost of deployment, integration, internal support, etc. (This can be significantly larger for the purchased solution then an internal one as well)
- servercobra 7y ago
  People who would rather pay $30/mo to have someone manage it than run it themselves.
rozenmd 7y ago
Probably worth mentioning that's $24k MRR, not how much it costs...
- NiekvdMaas 7y ago
  Still impressive for a 1-man show who says he spends just 1% if him time on it.
- Jefro118 7y ago
  Editor here, my apologies. I did of course mean to communicate MRR and didn't really notice how the title might be confusing when I wrote it but I see it now. I'll make it clearer.
iamEAP 7y ago
I recently learned (by trial and error) some of the headaches associated with running headless browsers at scale that Joel mentions here; wish I'd heard of this service earlier. I ended up finding other solutions to fill in the gaps: Puppeteer Cluster is one I'd recommend (https://github.com/thomasdondorf/puppeteer-cluster https://github.com/thomasdondorf/puppeteer-cluster) I especially like the "host it yourself" commercial license model, here; while automating browser _actions_ over a network works well enough, _detailed scraping_ over a network can quickly become inefficient (as many requests for elements or element attributes may incur individual round-trips). In some cases, colocating your browser instance with your scraping logic becomes a necessity.
- mrskitch 7y ago
  We hear about puppeteer-cluster _a lot_, and we hear the same thing from folks (that's it's great). browserless.io essentially does "clustering" at an infrastructure level, whereas puppeteer-cluster does it at the application level. Both essentially solve the same problem, just in different ways.
mrskitch 7y ago
(Author here): lots of comments about why would someone pay for this. I think the answer is simple if you step out of your developer shoes. There's a lot of complexity in managing and even building the thing from the start -- and then you have to support it. If you're working in a large org then there's a chance that you can just DIY it, however for small-medium businesses this isn't practical and it's a waste of time (their most precious resource). I like to think of it as a managed database. Sure, you can freely download postgres in a container and you're up and going, however there's a lot more costs to it than just that. Having a fully-managed database saves you time and other non-tangibles that it can be worth the cost. Just depends on your circumstances.
- quickthrower2 7y ago
  I looked closely at Browserless a while ago, for a 15 - dev company it would be a good tool. I would have recommended it but circumstances (particularly a lot of effort put into a container setup that could already run these kinds of tests) means it made sense to keep everything working the same way in our k8s. But definitely this is a good offering for small businesses. The only downside to using a lot of IaaS type stuff like this is almost every month something goes down - Github or Azure or Appveyor etc, and this is another thing that could go down, even for a few minutes, and basically the CI is stuffed, or pingdom goes heywire. But this is a more general problem with cloudifying all the things, not with this particular service.
- lwb 7y ago
  Am I reading your website wrong, or is your hosted solution cheaper than the on-prem license (360/yr vs 500)? I would think it would be much more expensive to run it for them, right?
  - mrskitch 7y ago
    Nope, you've got it right. We're increasing prices next, however your sentiment is still true. Once we have the enterprise-flavor out the inverse will be true.
    - 7y ago
jotto 7y ago
For a similar headless Chrome project launched around the same time, but with a price-per-api-request model, see https://www.prerender.cloud/ https://www.prerender.cloud/ (PDFs, screenshots, pre-rendering). MRR is about the same.
- trpc 7y ago
  Rendora is true FOSS, free and self-hosted with very lightweight usage of resources. https://github.com/rendora/rendora https://github.com/rendora/rendora
  - kresten 7y ago
    Last commit 12 months ago. https://rendora.co/ https://rendora.co/ Seems to be gone.
pearjuice 7y ago
What's with the snarky comments? The guy build a successful business with happy customers. He didn't make it in 2% of his time, he built it over the years with experience from other projects and solved an issue people are willing to pay money for. Maintenance now might be on low time but doesn't mean it's easy to solve a problem and consistently ship the solution. I honestly think most of you have a misunderstanding about serviced software and the demand for it. The fact that you can do it with a Docker script and some time doesn't mean it works properly for all use cases or everyone can and should manage that infrastructure themselves. Some people happily pay for it so they don't have to worry whether their instance is still running and can call support instead of firing a terminal and looking through obscure Stackoverflow answers. He even open sourced his entire business. Instead of throwing apples, try to learn something or at least appreciate the effort.
- mrskitch 7y ago
  I felt similar, thanks for echoing the sentiment.
trpc 7y ago
Seriously I admire Joel, he is all over the internet everyday for 2 years now marketing his product. Browserless technically is a big nothing burger pretending to be a SaaS, it's written as open but it is not open source. You can't even use it commercially even it if it's self hosted https://github.com/browserless/chrome#licensing https://github.com/browserless/chrome#licensing. This is a rare case of a successful rudimentary fake FOSS project that makes real money. Real BOSS! EDIT: Why the hate, Joel? I've been following your success over the past year and I sincerely admire it even though it's outright parasite-tier big nothing burger of a project, but who cares 10 or 20 years from now, you got the money! EDIT2: Okay Joel, I really get it you hate me, but you've to understand that it's one my favorite hobbies in my free time is to track online indie parasites like you and I've been following your excessive self-promotion on HN, Reddit, IndiHackers for a long time now. I don't know why, but it's just a hobby.
- jayfk 7y ago
  How is GPL3 a “fake FOSS” project?
  - trpc 7y ago
    there is no such thing neither in GPL nor in FOSS to be GPL in noncommercial uses and at the same time invalidate the license in commercial or closed uses. This is outright gaslighting, not to mention that the entire project is literally a big nothing burger like I said before, it's even less nothing burger than paid "get Geo/IP" apis for instance which needs at least some operations effort even despite becoming increasingly trivial in the serverless age.
    - trpc 7y ago
      I am still waiting for Joel and the other downvoters (if any beside his other accounts) to list where exactly in the GPL or any other FOSS license to be invalid for some cases including the commercial ones. Literally countless FOSS projects, real projects that required real knowledge and expertise unlike Browserless tried to get away with this but couldn't, why don't you use something like SSPL or go proprietary instead of gaslighting your users?
encoderer 7y ago
Congrats on the continued success. It can stun people when they are reminded that the skills they sell to others can be applied for their own prosperity and I think you see that here. If you’re a developer with a day job there has never been a better time to get started building and selling your own software. It’s not glamorous but it is rewarding.
- mrskitch 7y ago
  I spent ~4 hours on my 10th wedding anniversary debugging a production issue. It's not a fun thing to talk about, and doesn't get a lot of attention, but the truth of the matter is that when things are bad _they are bad_. I can see now why folks say that this isn't for everyone.
msmithstubbs 7y ago
Solo founder: I built a tool and it solves problems for people to the tune of $24k/month Almost everyone: That's great! Well done. HN commenters: Pffft.
- james_s_tayler 7y ago
  That's great! Well done.
- daveguy 7y ago
  (there's a lot of pffft here because so many have been trying to do this with $0k MRR and no MVP)
- dang 7y ago
  That's not even close to a fair summary of this thread.
hoten 7y ago
I work on Lighthouse (user of the Chrome DevTools protocol) and work with the DevTools team (obviously the main user of the protocol) - and when I first saw browserless I was blown away. So cool! Good job with your success. What was the hardest part re: working with the protocol?
- mrskitch 7y ago
  The protocol is pretty easy, I think coordinating the necessary “enable” calls is a bit cumbersome. Also the legacy JSON protocol is harder to support, but I understand why. Hardest part is debugging crash issues and why they happened. You either just get a generic “Page crashed!” error (which I think is puppeteers handler message), or “browser disconnected!”. That and chromes logs are just crazy noisy, which I haven’t gotten a lot out of. Those are probably the biggest, thanks for asking!
ausjke 7y ago
how is this different from apify (https://apify.com/ https://apify.com/), apify seems can do what browserless does and is also open-source, means you can self-host it freely.
bobblywobbles 7y ago
Looks interesting, however I can't view your webpage - I am getting this error: "The character encoding of the plain text document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the file needs to be declared in the transfer protocol or file needs to use a byte order mark as an encoding signature."
- mrskitch 7y ago
  Interesting, I’ll take a look and see if our markup is encoded improperly.
jrockway 7y ago
I wrote something like this a couple months ago and thought about selling it. I ultimately decided that the price where you make it cost-prohibitive to mine cryptocurrency is too high for someone that just wants to render PDFs without dealing with the burstiness of running several copies of Chrome on their production infrastructure. I was also concerned about the underlying browser changing how things render when upgraded; I didn't want to run an outdated browser, but I also didn't want to tell users "hey we updated Chrome, better check the output of your batch job and make an emergency fix to your HTML". How are you dealing with these issues?
- RussianCow 7y ago
  How often do browsers break backwards compatibility? I don't think I can recall a single time I've had working code break due to a browser upgrade, with the exception of non-standard features (which is maybe what you're referring to, but then it's a known risk and you should already be aware of it). Edit: And if a client really does need version X of Chrome, you could give them the ability to pay extra to pin the version indefinitely.
jessaustin 7y ago
Does this handle something like Distil [0]? Or is that type of scraping not the focus of this product? [0] https://www.distilnetworks.com/block-bot-detection/ https://www.distilnetworks.com/block-bot-detection/
- botwriter 7y ago
  I second this!
- stickfigure 7y ago
  No, it won't. However, Distil is not hard to work around if you automate a real browser in headful mode.
  - jessaustin 7y ago
    Could you point me to some reasonably straightforward ways to do that? Thanks!
    - stickfigure 7y ago
      I don't think there's any HOWTO posted online; I just worked it out by trial and error. Use a real version of Chrome (not Chromium) and headful mode. Mask the navigator.webdriver property. Pace your requests and take care to use "good" IP addresses. Keep in mind that as soon as Distil sees something obviously automated (like a headless browser) the source IP address is "burned" for some number of days.
      - jessaustin 7y ago
        Thanks!
pmarreck 7y ago
Scripting headless browsers for testing is an antipattern and should literally be avoided as much as humanly possible
- RussianCow 7y ago
  Care to elaborate?
  - pmarreck 7y ago
    I have never had a good experience incorporating a headless browser test into a test suite. In literally every case it added so much complexity, suite run time and uncertainty that I realized it was better to just do the test via unit tests (which test the logic directly) and/or integration tests (which test the HTTP output of the controllers) if it was at all possible to rework the logic to operate in that fashion. 1) Increased load times and test run times due to browser complexity and memory consumption 2) Impossible to run concurrently without additional instances, each of which takes up massive memory 3) Tests are slow and often nondeterministic (literally THE WORST property a test suite can have), with many cases where things like "sleep()" delays are put in to circumvent some opaque browser latency issue, which is just gross 4) Even after suffering all of the above, you're still only testing ONE engine (say, WebKit) instead of all of the popular ones (Blink (Chrome), Gecko (Firefox), EdgeHTML (eh, Blink, I guess, now?), etc) I did not enumerate all the disadvantages, but these should be enough to support my position. The number of browser driver driven tests in your test suite should be as close to "zero" as possible. Does this discourage the use of SPA's? A lot, yes. But when necessary, I manage to do a separate frontend JS test suite via jsdom which does not require firing up a headless browser, and my build process runs both the frontend and backend test suites and only deploys if they both pass.
    - RussianCow 7y ago
      Thanks! That's interesting, because at work we're struggling with browser-specific regressions and were looking at headless browser testing to help solve that. I agree with all the drawbacks you listed (except #4, since some headless browser solutions let you use multiple browsers), but unit tests don't do anything to help with differences in browsers. Do you just do manual QA and hope for the best? Or is this not as big an issue for you as it is for our company? (We still have to support IE 11, so that's where the majority of browser-specific issues manifest.)
r_singh 7y ago
I'm using chrome-aws-lambda on Lambda and it works like a dream. Luckily for my use case I don't need images, fonts, etc. There's also GCF for those on Google Cloud. I have used Browserless' trial and felt like the 2+ GB instances were kind of expensive because they require reservation unlike Lambda where you get 400,000 GB-seconds and 1M request per month for free.
Thorrez 7y ago
I'm confused about the license[1]. It seems to not be actually open source. The Open Source Definition says[2]: > The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research. But this seems to be doing that exact restriction. Additionally the license seems like it contains a loophole: >If you are creating an open source application under a license compatible with the GNU GPL license v3, you may use browserless under the terms of the GPLv3. If I make an open source application, I can use browserless under the terms of the GPLv3. That means I can redistribute browserless under the GPLv3. That means people can take the browserless code I redistribute and use that for commercial products (as long as they don't distribute a non-GPLv3 binary form of the commercial products containing browserless, because that would break the GPLv3). [1] https://github.com/browserless/chrome#licensing https://github.com/browserless/chrome#licensing [2] https://opensource.org/osd#fields-of-endeavor https://opensource.org/osd#fields-of-endeavor
- Reelin 7y ago
  Just checked the GitHub license page (https://github.com/browserless/chrome/blob/master/LICENSE.md https://github.com/browserless/chrome/blob/master/LICENSE.md). > This work is dual-licensed under GPL-3.0 OR the browserless commercial license. You can choose between one of them if you use this work. So it's clearly GPLv3 (no loophole required), which AFAIK does allow closed source proprietary use within a company so long as the program isn't redistributed externally (perhaps the developer didn't understand that?). It seems that the licensing section in the readme should have it's wording adjusted somewhat. In fact, I think you're even in the clear to run a proprietary cloud service using GPLv3 code which is why the AGPL (among others) exists. Some recent drama (https://techcrunch.com/2019/05/30/lack-of-leadership-in-open-source-results-in-source-available-licenses/ https://techcrunch.com/2019/05/30/lack-of-leadership-in-open...) for reference. (Oddly, the header underneath that states "GPL-3.0-or-later" which is a bit inconsistent.)
mrskitch 7y ago
Thanks again for the questions. Please do email me if I happen to overlook anything: joel at browserless dot io
perl4ever 7y ago
"How cool would it be if you could just fire up your browser, do the work you want it to, and press a button and now it just magically does that someplace for you without ever having to write code?" Like, recording macros has been a thing forever, but how are you going to magically generalize them, without better-than-human AGI?

252 points by mrskitch 7y ago

deleted 7y ago

[deleted]

> There's always been this thought that I've had, that advertising and "paid" attention is really in no one's best interest. You're likely to get users who really aren't going to get any value out of your software, so your churn increases, and you've also just paid for that user that's churned. These things get harder to tease out since it's almost impossible to ask "show me all the churned users this period acquired from advertising channels." Maybe that's possible, but you'd have to do a lot wrangling together to get it all working. Correct me if I’m wrong, but isn’t precisely that kind of analytics simply table stakes for any modern crm/marketing/customer intelligence suite in 2019? It seems like that is absolutely a solved problem.

mrskitch 7y ago
Yeah, it's a pretty contrived example on my part. My sentiment here is that there's so many inputs to modeling behavior, and trying to find signal in the noise, that at this scale your time is likely spent better elsewhere. Unless you have the revenue stream to do it and do it well, then the effort can be a time sink
- sneak 7y ago
  How many variables do you need to track, though? $X spend, Y signups at $Z each, AA% churn, $BB retained MRR for an estimated $CC CLV based on an $X/Y*(1-AA%) CAC - why does it need to be much more complicated than that when you don’t have millions of users? (Seriously though, I’m asking, not poking fun. You probably know more about this stuff than I do, having actually done it. It seems really simple to figure, to me. What am I missing?)
  - mrskitch 7y ago
    Stripe honestly does a lot of this for you. You can hack all that together with what you've got, the problem is that there's some seasonality to business as well as other market effects that make it tough to determine _why_ something is trending one way or the other. For instance: we had a week last year where we had a flood of cancellations at once and there was nothing I could attribute it to. Looking back it was just a coincidence, however it consumed a lot of my time (writing emails and looking at analytics) to figure out why instead of just pushing ahead. I was likely over-reacting, but I have noticed there's a lot of people spending a lot of time doing analytics and doing research on trends in stead of just executing. And, especially early on, you should just be executing and not thinking too much about trends.
  - matthiaswh 7y ago
    It's more a matter of being messy rather than complicated. You're trying to track users who often visit your site multiple times before purchasing, coming from different sources, on different devices, and somehow tie that attribution to the purchase. None of the data inputs are consistent nor tied together. Analytics tells you differently than your ad platform, your payment processor tells you yet another thing, and your email marketing tool, and your CRM. (And if you want the serious tools for data monitoring and reporting have moved to focusing on enterprises...) You have to somehow factor in refunds, free trials, prorated billing, early cancellations. You have multiple ad campaigns running ad variations. Don't forget a/b testing. Finally you see some data point that hints something might be working, but you know you have to account for all the other factors involved. Did I make any website edits that day? Did the ad network change their algorithm slightly? Was there a holiday affecting traffic? When did I insert that new ad again? Wait, I know I changed my ad bid at some point... Did I get an influx of traffic from another source? Was it just a fluke? If you want good, real data, it's messy. And far from a solved problem.
    - mrskitch 7y ago
      Wow, you said this so much better than I could, thanks for chiming in

MuffinFlavored 7y ago

What $288k/yr headache is there around `docker pull buildkite/puppeteer`? https://hub.docker.com/r/buildkite/puppeteer https://hub.docker.com/r/buildkite/puppeteer

maxaf 7y ago
I dove into the comments to ask the same question. I just did something similar to this with a few lines in my docker-compose.yml, which eventually turned into a few lines in a Helm chart. Why would I pay someone money for something as trivial as this? Now, I won’t write another rant on the subject of Eternal September, and how our field is increasingly populated by charlatans who can’t do simple things on their own. But yes: that’s exactly how I feel.
- avip 7y ago
  Disclaimer: Neither I nor my colleagues or anyone I'm aware of are, were, or ever will, use said service. Did you divide (time that took you to complete this trivial task + time to document it, version it, explain it to coworkers, maintain it next time chrome decides to break something) * (your $/h cost) / 0.00008 before doing that?
  - rozenmd 7y ago
    Here's a build vs buy calculator to save you from doing the maths: https://baremetrics.com/startup-calculator https://baremetrics.com/startup-calculator
    - viraptor 7y ago
      This is a dangerous calculator. It misses the cost of deployment, integration, internal support, etc. (This can be significantly larger for the purchased solution then an internal one as well)
servercobra 7y ago
People who would rather pay $30/mo to have someone manage it than run it themselves.

rozenmd 7y ago

Probably worth mentioning that's $24k MRR, not how much it costs...

NiekvdMaas 7y ago
Still impressive for a 1-man show who says he spends just 1% if him time on it.
Jefro118 7y ago
Editor here, my apologies. I did of course mean to communicate MRR and didn't really notice how the title might be confusing when I wrote it but I see it now. I'll make it clearer.

iamEAP 7y ago

I recently learned (by trial and error) some of the headaches associated with running headless browsers at scale that Joel mentions here; wish I'd heard of this service earlier. I ended up finding other solutions to fill in the gaps: Puppeteer Cluster is one I'd recommend (https://github.com/thomasdondorf/puppeteer-cluster https://github.com/thomasdondorf/puppeteer-cluster) I especially like the "host it yourself" commercial license model, here; while automating browser _actions_ over a network works well enough, _detailed scraping_ over a network can quickly become inefficient (as many requests for elements or element attributes may incur individual round-trips). In some cases, colocating your browser instance with your scraping logic becomes a necessity.

mrskitch 7y ago
We hear about puppeteer-cluster _a lot_, and we hear the same thing from folks (that's it's great). browserless.io essentially does "clustering" at an infrastructure level, whereas puppeteer-cluster does it at the application level. Both essentially solve the same problem, just in different ways.

mrskitch 7y ago

(Author here): lots of comments about why would someone pay for this. I think the answer is simple if you step out of your developer shoes. There's a lot of complexity in managing and even building the thing from the start -- and then you have to support it. If you're working in a large org then there's a chance that you can just DIY it, however for small-medium businesses this isn't practical and it's a waste of time (their most precious resource). I like to think of it as a managed database. Sure, you can freely download postgres in a container and you're up and going, however there's a lot more costs to it than just that. Having a fully-managed database saves you time and other non-tangibles that it can be worth the cost. Just depends on your circumstances.

quickthrower2 7y ago
I looked closely at Browserless a while ago, for a 15 - dev company it would be a good tool. I would have recommended it but circumstances (particularly a lot of effort put into a container setup that could already run these kinds of tests) means it made sense to keep everything working the same way in our k8s. But definitely this is a good offering for small businesses. The only downside to using a lot of IaaS type stuff like this is almost every month something goes down - Github or Azure or Appveyor etc, and this is another thing that could go down, even for a few minutes, and basically the CI is stuffed, or pingdom goes heywire. But this is a more general problem with cloudifying all the things, not with this particular service.
lwb 7y ago
Am I reading your website wrong, or is your hosted solution cheaper than the on-prem license (360/yr vs 500)? I would think it would be much more expensive to run it for them, right?
- mrskitch 7y ago
  Nope, you've got it right. We're increasing prices next, however your sentiment is still true. Once we have the enterprise-flavor out the inverse will be true.
  - 7y ago

jotto 7y ago

For a similar headless Chrome project launched around the same time, but with a price-per-api-request model, see https://www.prerender.cloud/ https://www.prerender.cloud/ (PDFs, screenshots, pre-rendering). MRR is about the same.

trpc 7y ago
Rendora is true FOSS, free and self-hosted with very lightweight usage of resources. https://github.com/rendora/rendora https://github.com/rendora/rendora
- kresten 7y ago
  Last commit 12 months ago. https://rendora.co/ https://rendora.co/ Seems to be gone.

pearjuice 7y ago

What's with the snarky comments? The guy build a successful business with happy customers. He didn't make it in 2% of his time, he built it over the years with experience from other projects and solved an issue people are willing to pay money for. Maintenance now might be on low time but doesn't mean it's easy to solve a problem and consistently ship the solution. I honestly think most of you have a misunderstanding about serviced software and the demand for it. The fact that you can do it with a Docker script and some time doesn't mean it works properly for all use cases or everyone can and should manage that infrastructure themselves. Some people happily pay for it so they don't have to worry whether their instance is still running and can call support instead of firing a terminal and looking through obscure Stackoverflow answers. He even open sourced his entire business. Instead of throwing apples, try to learn something or at least appreciate the effort.

mrskitch 7y ago
I felt similar, thanks for echoing the sentiment.

trpc 7y ago

Seriously I admire Joel, he is all over the internet everyday for 2 years now marketing his product. Browserless technically is a big nothing burger pretending to be a SaaS, it's written as open but it is not open source. You can't even use it commercially even it if it's self hosted https://github.com/browserless/chrome#licensing https://github.com/browserless/chrome#licensing. This is a rare case of a successful rudimentary fake FOSS project that makes real money. Real BOSS! EDIT: Why the hate, Joel? I've been following your success over the past year and I sincerely admire it even though it's outright parasite-tier big nothing burger of a project, but who cares 10 or 20 years from now, you got the money! EDIT2: Okay Joel, I really get it you hate me, but you've to understand that it's one my favorite hobbies in my free time is to track online indie parasites like you and I've been following your excessive self-promotion on HN, Reddit, IndiHackers for a long time now. I don't know why, but it's just a hobby.

jayfk 7y ago
How is GPL3 a “fake FOSS” project?
- trpc 7y ago
  there is no such thing neither in GPL nor in FOSS to be GPL in noncommercial uses and at the same time invalidate the license in commercial or closed uses. This is outright gaslighting, not to mention that the entire project is literally a big nothing burger like I said before, it's even less nothing burger than paid "get Geo/IP" apis for instance which needs at least some operations effort even despite becoming increasingly trivial in the serverless age.
  - trpc 7y ago
    I am still waiting for Joel and the other downvoters (if any beside his other accounts) to list where exactly in the GPL or any other FOSS license to be invalid for some cases including the commercial ones. Literally countless FOSS projects, real projects that required real knowledge and expertise unlike Browserless tried to get away with this but couldn't, why don't you use something like SSPL or go proprietary instead of gaslighting your users?

encoderer 7y ago

Congrats on the continued success. It can stun people when they are reminded that the skills they sell to others can be applied for their own prosperity and I think you see that here. If you’re a developer with a day job there has never been a better time to get started building and selling your own software. It’s not glamorous but it is rewarding.

mrskitch 7y ago
I spent ~4 hours on my 10th wedding anniversary debugging a production issue. It's not a fun thing to talk about, and doesn't get a lot of attention, but the truth of the matter is that when things are bad _they are bad_. I can see now why folks say that this isn't for everyone.

msmithstubbs 7y ago

Solo founder: I built a tool and it solves problems for people to the tune of $24k/month Almost everyone: That's great! Well done. HN commenters: Pffft.

james_s_tayler 7y ago
That's great! Well done.
daveguy 7y ago
(there's a lot of pffft here because so many have been trying to do this with $0k MRR and no MVP)
dang 7y ago
That's not even close to a fair summary of this thread.

hoten 7y ago

I work on Lighthouse (user of the Chrome DevTools protocol) and work with the DevTools team (obviously the main user of the protocol) - and when I first saw browserless I was blown away. So cool! Good job with your success. What was the hardest part re: working with the protocol?

mrskitch 7y ago
The protocol is pretty easy, I think coordinating the necessary “enable” calls is a bit cumbersome. Also the legacy JSON protocol is harder to support, but I understand why. Hardest part is debugging crash issues and why they happened. You either just get a generic “Page crashed!” error (which I think is puppeteers handler message), or “browser disconnected!”. That and chromes logs are just crazy noisy, which I haven’t gotten a lot out of. Those are probably the biggest, thanks for asking!

ausjke 7y ago

how is this different from apify (https://apify.com/ https://apify.com/), apify seems can do what browserless does and is also open-source, means you can self-host it freely.

bobblywobbles 7y ago

Looks interesting, however I can't view your webpage - I am getting this error: "The character encoding of the plain text document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the file needs to be declared in the transfer protocol or file needs to use a byte order mark as an encoding signature."

mrskitch 7y ago
Interesting, I’ll take a look and see if our markup is encoded improperly.

jrockway 7y ago

I wrote something like this a couple months ago and thought about selling it. I ultimately decided that the price where you make it cost-prohibitive to mine cryptocurrency is too high for someone that just wants to render PDFs without dealing with the burstiness of running several copies of Chrome on their production infrastructure. I was also concerned about the underlying browser changing how things render when upgraded; I didn't want to run an outdated browser, but I also didn't want to tell users "hey we updated Chrome, better check the output of your batch job and make an emergency fix to your HTML". How are you dealing with these issues?

RussianCow 7y ago
How often do browsers break backwards compatibility? I don't think I can recall a single time I've had working code break due to a browser upgrade, with the exception of non-standard features (which is maybe what you're referring to, but then it's a known risk and you should already be aware of it). Edit: And if a client really does need version X of Chrome, you could give them the ability to pay extra to pin the version indefinitely.

jessaustin 7y ago

Does this handle something like Distil [0]? Or is that type of scraping not the focus of this product? [0] https://www.distilnetworks.com/block-bot-detection/ https://www.distilnetworks.com/block-bot-detection/

botwriter 7y ago
I second this!
stickfigure 7y ago
No, it won't. However, Distil is not hard to work around if you automate a real browser in headful mode.
- jessaustin 7y ago
  Could you point me to some reasonably straightforward ways to do that? Thanks!
  - stickfigure 7y ago
    I don't think there's any HOWTO posted online; I just worked it out by trial and error. Use a real version of Chrome (not Chromium) and headful mode. Mask the navigator.webdriver property. Pace your requests and take care to use "good" IP addresses. Keep in mind that as soon as Distil sees something obviously automated (like a headless browser) the source IP address is "burned" for some number of days.
    - jessaustin 7y ago
      Thanks!

pmarreck 7y ago

Scripting headless browsers for testing is an antipattern and should literally be avoided as much as humanly possible

RussianCow 7y ago
Care to elaborate?
- pmarreck 7y ago
  I have never had a good experience incorporating a headless browser test into a test suite. In literally every case it added so much complexity, suite run time and uncertainty that I realized it was better to just do the test via unit tests (which test the logic directly) and/or integration tests (which test the HTTP output of the controllers) if it was at all possible to rework the logic to operate in that fashion. 1) Increased load times and test run times due to browser complexity and memory consumption 2) Impossible to run concurrently without additional instances, each of which takes up massive memory 3) Tests are slow and often nondeterministic (literally THE WORST property a test suite can have), with many cases where things like "sleep()" delays are put in to circumvent some opaque browser latency issue, which is just gross 4) Even after suffering all of the above, you're still only testing ONE engine (say, WebKit) instead of all of the popular ones (Blink (Chrome), Gecko (Firefox), EdgeHTML (eh, Blink, I guess, now?), etc) I did not enumerate all the disadvantages, but these should be enough to support my position. The number of browser driver driven tests in your test suite should be as close to "zero" as possible. Does this discourage the use of SPA's? A lot, yes. But when necessary, I manage to do a separate frontend JS test suite via jsdom which does not require firing up a headless browser, and my build process runs both the frontend and backend test suites and only deploys if they both pass.
  - RussianCow 7y ago
    Thanks! That's interesting, because at work we're struggling with browser-specific regressions and were looking at headless browser testing to help solve that. I agree with all the drawbacks you listed (except #4, since some headless browser solutions let you use multiple browsers), but unit tests don't do anything to help with differences in browsers. Do you just do manual QA and hope for the best? Or is this not as big an issue for you as it is for our company? (We still have to support IE 11, so that's where the majority of browser-specific issues manifest.)

r_singh 7y ago

I'm using chrome-aws-lambda on Lambda and it works like a dream. Luckily for my use case I don't need images, fonts, etc. There's also GCF for those on Google Cloud. I have used Browserless' trial and felt like the 2+ GB instances were kind of expensive because they require reservation unlike Lambda where you get 400,000 GB-seconds and 1M request per month for free.

Thorrez 7y ago

I'm confused about the license[1]. It seems to not be actually open source. The Open Source Definition says[2]: > The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research. But this seems to be doing that exact restriction. Additionally the license seems like it contains a loophole: >If you are creating an open source application under a license compatible with the GNU GPL license v3, you may use browserless under the terms of the GPLv3. If I make an open source application, I can use browserless under the terms of the GPLv3. That means I can redistribute browserless under the GPLv3. That means people can take the browserless code I redistribute and use that for commercial products (as long as they don't distribute a non-GPLv3 binary form of the commercial products containing browserless, because that would break the GPLv3). [1] https://github.com/browserless/chrome#licensing https://github.com/browserless/chrome#licensing [2] https://opensource.org/osd#fields-of-endeavor https://opensource.org/osd#fields-of-endeavor

Reelin 7y ago
Just checked the GitHub license page (https://github.com/browserless/chrome/blob/master/LICENSE.md https://github.com/browserless/chrome/blob/master/LICENSE.md). > This work is dual-licensed under GPL-3.0 OR the browserless commercial license. You can choose between one of them if you use this work. So it's clearly GPLv3 (no loophole required), which AFAIK does allow closed source proprietary use within a company so long as the program isn't redistributed externally (perhaps the developer didn't understand that?). It seems that the licensing section in the readme should have it's wording adjusted somewhat. In fact, I think you're even in the clear to run a proprietary cloud service using GPLv3 code which is why the AGPL (among others) exists. Some recent drama (https://techcrunch.com/2019/05/30/lack-of-leadership-in-open-source-results-in-source-available-licenses/ https://techcrunch.com/2019/05/30/lack-of-leadership-in-open...) for reference. (Oddly, the header underneath that states "GPL-3.0-or-later" which is a bit inconsistent.)

mrskitch 7y ago

Thanks again for the questions. Please do email me if I happen to overlook anything: joel at browserless dot io

perl4ever 7y ago

"How cool would it be if you could just fire up your browser, do the work you want it to, and press a button and now it just magically does that someplace for you without ever having to write code?" Like, recording macros has been a thing forever, but how are you going to magically generalize them, without better-than-human AGI?