18 ms·

PhantomJS: Stepping down as maintainer

854 points by wfunction 9y ago

d--b 9y ago
Wow, I'm very impressed. At this stage it is a very wise decision to step down and to focus onto something else, rather than to hold on to a project that will eventually disappear. It takes a lot of courage to move on from a project that had to be maintained for several years and that had such a reach. We can only be thankful for all the good work that went into PhantomJS, and wish the maintainers the best of luck in their next endeavors. Cheers!
- moxious 9y ago
  He says in his message it's been a slog for some time, looks like a good time to be done with it. Open source is great, we all like it, but demanding unpaid jobs can get old, too.
  - yebyen 9y ago
    And it makes sense. You want Chrome in your tests if your users are using Chrome. Very few (if any) of your users will ever visit your app with a headless PhantomJS browser, so it's not a platform that you should go out of your way to support. I've been using Selenium Driver with Chrome in xvfb for my headless testing needs, and I've used PhantomJS for some automation things in the past where it was great, but since I switched I really haven't looked back! I had things breaking subtly that I couldn't fix, and they did not manifest as problems in the Chrome browser or Selenium. I still don't know what was wrong, I just know that my Rails app won't pass its JavaScript functional tests if I use PhantomJS. When I did the evaluation of 3 test drivers, I found that the one with the actual browser in it was the one that worked most reliably.
    - cookiecaper 9y ago
      You can use PhantomJS as the backend on a Selenium script, and this news clearly demonstrates the utility of a higher-level API than using PhantomJS directly. If your tests are in Selenium, changing backends is generally a small matter. I started writing in Selenium instead of CasperJS (a PhantomJS frontend) because PhantomJS experienced intermittent bugs on the page I was trying to access. I think you're right that "real browsers" are still much more reliable for complex use cases, but the low profile of PhantomJS is definitely nice when it works.
    - holtalanm 9y ago
      I used Phantom once upon a time, but I eventually switched to just using xvfb as well within a docker instance. Headless Chrome and Firefox with xvfb.
    - thom_nic 9y ago
      I actually started using webdriverIO + chromedriver after fighting too much with casperJS - while webdriver (and Selenium) seem to have much more momentum there are still some things I really miss that PhantomJS gave when using Capybara. I was a very happy Capy+Phantom user when testing my rails apps. Things like reading the HTTP response code, detecting 404s on assets and catching JS errors in the console are all not possible with Selenium/ Webdriver, and I relied heavily on those capabilities in my Capybara tests. While headless chrome might be able to replace PhantomJS for many use cases that doesn't necessarily mean the APIs will be comparable. In fact I'd more likely expect the Chrome folks to say "the webdriver API is it, because it's a standard." [1] Sadly IMO it's lacking compared to what PhantomJS was capable of. [1] https://www.w3.org/TR/webdriver/ https://www.w3.org/TR/webdriver/
      - erik 9y ago
        When I looked into it, I was surprised by how hard it is to set up basic smoke tests for web development, and that Selenium / Webdriver can't do this.
    - moogly 9y ago
      Thanks so much to the PhantomJS maintainer for his hard work over the years! To me, it feels like his decision is the correct one here. After we realised that we hadn't seen a "one-browser" bug for 2 years in our massive angularjs app, we got rid of all browsers but PhantomJS in our karma suite. PhantomJS' slowness, lagging-behind in webstandards and just my general gut feeling (these facts above made me question the point in running JavaScript tests in an actual browser at all) made me port our karma test suite to jest w/ jsdom and haven't been happier since we, years prior, got rid of our gnarly Selenium test suite that caught 0 bugs but was the major cause for maintenance headache.
      - forgotpwtomain 9y ago
        > After we realised that we hadn't seen a "one-browser" bug for 2 years in our massive angularjs app Really? When I was writing JavaScript I thought I'd hit one ~ once a month. As a matter of fact I hit a v8 bug just yesterday which apparently doesn't support /Etc/GMT[-+][0-9]{0,2} timezones.
      - deleted 9y ago
        [deleted]
- Touche 9y ago
  Some context, he's essentially been maintaining a web browser (which is a project on the level of an operating system) on his own. Phantom 2 switched to QTWebKit which I'm sure was a tremendous amount of work. Probably at the end of that he was hoping things would get "easier", and it sounds like it hasn't. It's just too much work for one person and if companies aren't will to pay people to do it, I'd quit too.
frik 9y ago
That's quite sad.
- etatoby 9y ago
  It might be for the best. One of the many companies using PhantomJS to make money could go ahead and employ Vitaly to work on the project full time.
redsummer 9y ago
I'm curious - What is the utility of headless browsers? Are there people who earn money by getting it to automatically fill out forms, enter competitions etc?
- kasparsklavins 9y ago
  Automated tests.
  - askmike 9y ago
    As well as scraping more JS heavy stuff.
- kowdermeister 9y ago
  Server side rendering, testing, automation just to name a few.
- praptak 9y ago
  Web scraping for sites that discourage simple robots by checking for JS or serving content via JS. Also, the darker stuff like click fraud and all the other kinds of fraud where you pretend there are humans doing something when in fact there's just a bot.
- ajholyoake 9y ago
  I have used them in the past to convert graphs and reports to JPGs and PDFs so that I can automatically email them to people in the company who are unable (or unwilling) to use a web page.
- d--b 9y ago
  I use it to automate turning webpages in pdf.
- avaer 9y ago
  Testing, rendering, scraping, streaming, botting, and many more. Think "what could I accomplish with a browser, with the slow human replaced by a fast program" and let your imagination run wild. The space is interesting enough that people have jumped through a lot of hoops to make it work in the past; this makes it one less hoop. Oh, and if you ever wondered why web captchas are a thing, one of the reasons is headless browsers.
malinens 9y ago
My colleague from automatic testing says that phantomjs actually is much more stable than chrome...
- onion2k 9y ago
  That's true, but in my experience the instability of Chrome comes from opening and closing it's windows repeatedly as a large test suite often does - occasionally it doesn't seem to like opening a new instance while another is closing. Headless mode should resolve that problem.
  - holtalanm 9y ago
    There was also the issue a few years ago (last time I wrote automated scripts) where the Chrome driver for Selenium would go too fast for the browser to keep up, causing false failures. I had to implement a "wait between actions" feature to handle it, while Phantom had no such problems. I'm assuming this will not be an issue with headless Chrome, since I think half of the problem was due to graphical rendering.
askmike 9y ago
Though a collaboration between the two projects might not be out of the question: https://groups.google.com/d/msg/phantomjs-dev/S-mEBwuSgKQ/PQdE1A5FAwAJ https://groups.google.com/d/msg/phantomjs-dev/S-mEBwuSgKQ/PQ...
- make3 9y ago
  google should hire him
deleted 9y ago
[deleted]
z3t4 9y ago
> I even bought the Mac for that! I did too, then found out you also needed a dev license for users being able to run your app. Supporting Mac/OSX is damn expensive if your app is free.
- Cthulhu_ 9y ago
  For iOS development I believe they got rid of that requirement, you can develop iOS apps (at least) using a personal dev account; only when you want to go to the app store do they ask for the license fee.
  - prashnts 9y ago
    Unfortunately they limit app capabilities (such as iCloud, keychain access), so you can't use those even for your personal account.
  - tokenizerrr 9y ago
    How do I do this without buying dedicated hardware?
    - j_s 9y ago
      Illegaly (virtualization) or in the cloud.
Hernanpm 9y ago
Besides testing, my team uses phantomjs to convert pages to PDF and also to convert javascript generated charts to images. This is sad news, I'm don't think chrome eventually will add support for.
- masklinn 9y ago
  You can use WebDriver to take screenshots. OTOH I don't think there's any way to do PDF generation without fucking around with injecting window.print() and trying to go from there.
- egeozcan 9y ago
  There's wkhtmltopdf[1] when you need the PDF functionality. It is licensed with LGPL, so you can make it interop with your commercial product. [1]: https://wkhtmltopdf.org/ https://wkhtmltopdf.org/
  - throwaway2016a 9y ago
    Second wkhtmltopdf ... I love PhantonJS but wkhtmltopdf is a better tool for PDFs.
    - Hernanpm 9y ago
      did you have luck with SPA to PDF? I recall I tried it but with no luck for good javascript support.
      - throwaway2016a 9y ago
        SPA as in Single Page Application? We are using it for reports and we are tailoring our HTML specifically to the reports so we haven't had that issue. If I needed Javascript it may not be the best solution. I like that it generates full PDFs (with real text objects) though not just a static image. I'm not sure if you can generate a PDF like that with PhantomJS. I haven't tried.
  - Globz 9y ago
chatmasta 9y ago
That sucks for any scraping use case. I have to imagine google has built in some way to detect headless browser mode serverside, even if only they can access it.
- deleted 9y ago
  [deleted]
- wayn3 9y ago
  i use phantomjs with the chrome webdriver, or whatever it may be called, already, all the time. if they wanted to do that, theyd already do it. this change doesnt seem to be all that big.
hpaavola 9y ago
Also Firefox will get headless mode in few releases https://bugzilla.mozilla.org/show_bug.cgi?id=1338004 https://bugzilla.mozilla.org/show_bug.cgi?id=1338004
- imr_ 9y ago
  thank god. I was afraid we will be back in the grasp of the google's moloch.
  - tyingq 9y ago
    Curious what "moloch" means in this context. I can google it, of course, but I get "Biblical name relating to a Canaanite god associated with child sacrifice". Which doesn't help me much. End users are children that Google is sacrificing? Or?
    - Pent 9y ago
      They are probably referring to http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ http://slatestarcodex.com/2014/07/30/meditations-on-moloch/
    - fnovd 9y ago
      https://www.poetryfoundation.org/poems-and-poets/poems/detail/49303 https://www.poetryfoundation.org/poems-and-poets/poems/detai... Heading II
    - philh 9y ago
      To give a shorter answer: Scott Alexander (at the slatestarcodex link), through the poem, associates Moloch with negative-sum games, where no one comes out better than they went in. In extreme cases they force us to sacrifice the things we love in order to survive. You throw your children to Moloch to help you defeat enemies; otherwise, you die. Your enemies do the same thing. It would be better if nobody sacrificed their children, but nobody is in a position to bring that outcome to pass. In this context, I would interpret "google's Moloch" along the lines of: Google is net-bad for the world, because of privacy issues and problems with centralisation and so on. Using Google's software (and services) makes them more powerful, so people don't want to use Google's software. But because everyone else is using Google's software, the world is optimized for Google users in a way that it isn't optimized for non-Google users, and so it's difficult to escape. And so Google grows yet stronger, and it becomes more difficult to escape. (To clarify: this is my interpretation of grandparent's use of the phrase. It's not my own position, and there's a decent chance that I'm completely off-base and it's got nothing to do with grandparent's position either.)
tal_berzniz 9y ago
Thank you for PhantomJS! Been using it for testing, generating PDFs and screenshot.
- amelius 9y ago
  Yes, thanks PhantomJS! I'm wondering, are there any examples out there for generating screenshots on headless Chrome?
  - throwaway2016a 9y ago
    From the docs it sounds like it is largely compatible with Selenium https://chromium.googlesource.com/chromium/src/+/lkgr/headless/README.md https://chromium.googlesource.com/chromium/src/+/lkgr/headle... If that is true, the takeScreenshot() should work as normal hopefully.
    - veidr 9y ago
      I can confirm that. Here's an example of how we take screenshots when automated tests fail, using Selenium + Chrome: https://gist.github.com/masonmark/2332c1238a2fa70b5e4fcfffdcd76ca3 https://gist.github.com/masonmark/2332c1238a2fa70b5e4fcfffdc...
avaer 9y ago
Open source software has to be one of the least efficient markets out there. If you sum up the very real value PhantomJS has delivered to very real companies over the last several years, napkin math tells me we wouldn't have the project being abandoned for being a "bloody hell" to work on.
- Cthulhu_ 9y ago
  The main problem for developers is that getting your company to pay for something like developer tools can be very hard and a long project. Ideally every team gets a budget and a credit card to do as they see fit, but in practice a lot of (especially bigger) companies have a whole acquisition process. It's not uncommon for the hours spent in getting a license for e.g. an editor to be far more expensive than the product itself. I believe this is often highlighted again when Sublime Text is in the news again. This as opposed to open source tooling which has no such hurdles.
  - taude 9y ago
    Why do you think open source tooling has no such hurdles? At these same "larger" companies, they're going to have an open source review board, and legal involved for each open source product you want to use. Just because you can download and use it on your local machine, doesn't mean you're not violating your corp policy and procedure.
    - hackits 9y ago
      Whats more surprising is the company I work for does have a support agreement with Oracle, although when we've run into problems that do need a technical expert from Oracle such as crash dumps and stack traces we're told to figure it out our-self. Kind of defeats the whole reason to have a closed source system when you're still on the hook and charging out $500 per hour to clients.
      - jonny_eh 9y ago
        and then you can't search stackoverflow for help, compared to using something like postgres
JepZ 9y ago
I am happy for the guy as he seems to be able to let go without letting anybody down (which seems to be important to him). At the same time, it is sad when people have such a pressure for something they probably started as a fun project.
seleniumuser 9y ago
How does this news fit in with Selenium?
- fbnlsr 9y ago
  You'll use Headless Chrome instead of PhantomJS as your driver.
- cookiecaper 9y ago
  Selenium won't really be impacted, as it's higher-level than PhantomJS. This is also a great example of why it's smart to use Selenium or something like it for scripting the tests. You can easily swap out for another backend in Selenium, but if you wrote tests in pure PhantomJS, you're now stuck with a codebase that depends on unmaintained software.
tempodox 9y ago
Wow, that was a quick reaction. Thanks to the maintainers for all the good work! To me, it's always a sad occasion to see diversity diminished. Nothing against Chromium, but I hope it won't be the one browser to rule them all. It's always good to have alternatives.
- gus_massa 9y ago
  They know since June about the project. This is an interesting conversation: https://groups.google.com/forum/#!topic/phantomjs-dev/S-mEBwuSgKQ https://groups.google.com/forum/#!topic/phantomjs-dev/S-mEBw... ( posted by askmike in https://news.ycombinator.com/item?id=14105613 https://news.ycombinator.com/item?id=14105613 )
mrfusion 9y ago
Naive question here. What makes headless mode so difficult?
- tpaschalis 9y ago
  It's the same order of magnitude of work, as it is maintaining a fully-fledged browser. It's sad to see open-source projects shut down, but being a sole developer is a lot different than having the resources of a giant like Google, for example.
- pavlov 9y ago
  It's a very good question. One might imagine that the browser renders everything into a buffer at some point, and you could simply ask the engine to give you a pointer to that data. The reality is very different. WebKit/Blink rendering is intimately tied with the graphics system of each platform, in particular through the use of native widgets and native window system compositors. For example, on the Mac, a lot of compositing within the browser window is done using Core Animation layers. This is a really good idea for performance, because it leverages the work done by Apple to improve their GUI performance. The downside is that capturing the output becomes very tricky when the browser doesn't do the final compositing. Previously this didn't really matter because 99.99% of browser rendering is for end users and they don't need to capture the output (or if they do, they would just use platform GUI functionality like screen capture). An increasing demand for headless rendering has effectively forced browser engine teams to rethink some of the internal APIs so that a pipeline can be built to capture the final rendering.
ohitsdom 9y ago
PhantomJS is a great tool, I implemented it as a PDF report generating system. But will Chromium be able to replace it in this regard? Will Chromium have paging features? Will it be able to repeat table headers when a table body content extends to the next page?
- jffry 9y ago
  I can't answer all of your questions, but this thread may interest you: https://news.ycombinator.com/item?id=14102248 https://news.ycombinator.com/item?id=14102248
  - ohitsdom 9y ago
    Huge help, thank you!
Doctor_Fegg 9y ago
> Chrome is faster and more stable than PhantomJS. And it doesn't eat memory like crazy. Wait, can someone tell me where to download this doesn't-eat-memory-like-crazy version of Chrome? Activity Monitor is showing me 2GB of Chrome processes right now and that's even with The Great Suspender having paused almost all my tabs.
- yebyen 9y ago
  I saw a trick where you can run Chrome and give it less memory, and it uses less memory. This is done using cgroups. The blog post is somewhat old[1] and not in sync with the version that is in Git[2], you might find a way to do this without Docker (I was using an old version of Docker and kernel that couldn't get it to work. But I need the old version of Docker for reasons.) Chrome will aggressively consume any memory you give it (up to a point?) to "make your browsing experience better" somehow. You're not wrong. But there is modern technology that can make it better. If you have a fast SSD, then Chrome can still use Swap to make your experience better. The later version in the Dockerfile linked on Git also leverages swapaccount with the seccomp setting. This may be one great use of Docker for people that wouldn't yet have been convinced to use Docker for any serious reason. [1]: https://blog.jessfraz.com/post/docker-containers-on-the-desktop/#5-chrome https://blog.jessfraz.com/post/docker-containers-on-the-desk... [2]: https://github.com/jessfraz/dockerfiles/blob/master/chrome/stable/Dockerfile https://github.com/jessfraz/dockerfiles/blob/master/chrome/s...
  - tayo42 9y ago
    You can just use cgroups on their own without docker
    - tyingq 9y ago
      I would guess ulimit -v might work as well, so you may not even need cgroups.
      - yebyen 9y ago
        Hah! Something new is also something old. Thanks for that. But won't you run into issues with child processes, of which Chrome tends to spawn a zillion? (I'm reading that each one gets its own limit under ulimit...) And potentially get your browser killed when it hits the limit? I haven't tested this and I don't really know how ulimit works, but I think it's a less effective solution than the cgroups for at least one reason.
est 9y ago
This is sad, phantomjs is better stripped than Chromium headless, if you ever try to install Chromium on servers without X, it requires shit ton of dependencies, while phantomjs was properly modified requires only minimal library.
- akx 9y ago
  I don't really see the problem. After all, you just install those deps once in your Dockerfile, right? ;)
  - yeukhon 9y ago
    Not everyone is using Docker.
  - est 9y ago
    chromium-browser requires these chromium-browser chromium-browser-l10n chromium-codecs-ffmpeg-extra cpp cpp-4.8 fontconfig fontconfig-config fonts-dejavu-core hicolor-icon-theme libasound2 libasound2-data libatk1.0-0 libatk1.0-data libatomic1 libavahi-client3 libavahi-common-data libavahi-common3 libcairo2 libcloog-isl4 libcups2 libdatrie1 libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libfile-basedir-perl libfile-desktopentry-perl libfile-mimeinfo-perl libfontconfig1 libfontenc1 libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-common libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa libgmp10 libgnome-keyring-common libgnome-keyring0 libgraphite2-3 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libharfbuzz0b libice6 libisl10 libjasper1 libjbig0 libjpeg-turbo8 libjpeg8 libllvm3.4 libmpc3 libmpfr4 libnspr4 libnss3 libnss3-nssdb libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0 libpciaccess0 libpixman-1-0 libsm6 libspeechd2 libthai-data libthai0 libtiff5 libtxc-dxtn-s2tc0 libx11-xcb1 libxaw7 libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-render0 libxcb-shape0 libxcb-shm0 libxcb-sync1 libxcomposite1 libxcursor1 libxdamage1 libxfixes3 libxft2 libxi6 libxinerama1 libxmu6 libxpm4 libxrandr2 libxrender1 libxshmfence1 libxss1 libxt6 libxtst6 libxv1 libxxf86dga1 libxxf86vm1 x11-common x11-utils x11-xserver-utils xdg-utils phantomjs: fontconfig-config fonts-dejavu-core libfontconfig1 libjpeg-turbo8 libjpeg8
    - 9y ago
bigato 9y ago
Question for those of you more involved with such headless tasks. Do you think that chromium and firefox supporting headless will induce a surge in bots crawling the open web from now on?
- uitgewis 9y ago
  Not a whole lot more than they are at the moment.
- heipei 9y ago
  Right off the bat: No. The reason for that is that crawling using a proper browser (i.e. Chrome) is a lot more resource-intensive than using a dedicated tool which only gets the top resource and maybe tries to parse some additional resources. With these kinds of tools you're limited by available bandwidth and IO speeds if you want to store things. If you're looking at a browser, you'll be limited more by things like memory consumption and CPU time, so you'd need a bigger box or more of them to drive the same amount of traffic. There is also not the same amount of ready made applications which take care of crawling, storing and maybe even indexing your data, so not something you can do without actually implementing a lot of things yourself. Of course, that is only talking about wide-scale scanning. If you're only looking to scrape a single target, for whatever reason, then having an instrumented headless browser will greatly simplify things. Headless Chrome should be more efficient than running it in a (virtual) framebuffer. Plus the whole setup for a powerful crawler is reduced to "install Chrome, start it, point $crawler at the API endpoint". My guess is that we might see turnkey crawling / automation tools appear where you supply a list of URLs and the library + Chrome does the rest. Then, browser-based large scale scanning will be within everyone's reach, only limited by their resources. Background: I created https://urlscan.io https://urlscan.io which will simply visit a website and record HTTP interactions (annotating the resources with some helpful meta data). I've been preaching the power of headless instrumented browsers for the better part of a year now ;)
gildas 9y ago
Thanks Vitaly for your work! SEO4Ajax would certainly not exist without PhantomJS. It helped us to deliver the service efficiently at the time. Unfortunately, we had quite a few compatibility issues with it leading us to migrate to Chrome (with xvfb) one year ago. Since then, we must confess that we are very happy of this choice. Chrome is indeed very stable, fast and more importantly for us, always up-to-date.
nashashmi 9y ago
Anybody know how to port code from using PhantomJS to headless Chrome? I have been using CasperJS that wraps to PhantomJS. PhantomJS had its own set of commands. headless chrome will have to be different one way or another.
- aantix 9y ago
  Looks like it's already usable? "Use headless chromium with capybara and selenium webdriver - today!" -- March 30th http://blog.faraday.io/headless-chromium-with-capybara-and-selenium-webdriver/ http://blog.faraday.io/headless-chromium-with-capybara-and-s...
- JimTheDev 9y ago
  I wrote a bit about how you can get started here using Node: https://objectpartners.com/2017/04/13/how-to-install-and-use-headless-chrome-on-osx/ https://objectpartners.com/2017/04/13/how-to-install-and-use...
lambduh 9y ago
It's both sad to see an incredibly useful project be sunsetted and exciting that it's no longer needed. I remember a project that used phantomjs to scrape an old government camping site to build a compatibility layer on top. Thank you, if you're reading.
Globz 9y ago
PhantomJS enabled us at the time to bootstrap a big project at work where at the end workflow the app had to turn HTML orders to PDF on the fly, eventually we moved to WKHTMLTOPDF (https://wkhtmltopdf.org/ https://wkhtmltopdf.org/) which is much less hungry with resources but nonetheless PhantomJS played a huge role during the early days of the project and was easy to setup. If I remember correctly the only down side was to find the correct format for our HTML template so PhantomJS would render proper page break and repeat the header for super long orders. I can understand why stepping down is the right decision, maintaining such project by himself is an amazing feat on its own and even more when it proves to be useful for so many companies. Sadly when it becomes your second job you might always be on the lookout for a clean exit and such opportunity just became a reality. Good luck in your future projects Vitaly!
- neebz 9y ago
  We have used PhantomJS and WKHTMLTOPDF both. Phantomjs hogs a lot of resources but is very good when you want print large PDFs (500+ pages). WKHTMLTOPDF struggles with larger HTMLs.
  - Globz 9y ago
    Good to know, this is not our use case at the moment, we mostly generate PDF of 2-3 pages long.
  - chrisber 9y ago
    checkout https://github.com/arachnys/athenapdf https://github.com/arachnys/athenapdf. Its based on electron.
eric_b 9y ago
This is a great move. I appreciate what Phantom (and the maintainer) were trying to do, but I have always loathed PhantomJS. It has never worked well. In fact, I'd been away from it for some time, but just last night needed to install it to run some tests and it caused massive frustration. I pulled a Node repo that ran tests using Karma (why people use Karma is a complete mystery to me). I pulled the repo, ran `npm install` and then `npm test`. Sure enough Karma explodes out of the gate. Phantom can't start. I'm on Windows 8.1. I debug for an hour, eventually finding a magic custom binary Ariya created. I then have to copy this binary to the `/node_modules/karma-phantom-launcher/node_modules/phantomjs2-ext/bin` directory. All this to run some Jasmine specs. If Chrome headless support is really as good as "works just like Chrome without the GUI" then I will be one happy camper.
- joshuaduffy 9y ago
  I personally use Chrome at the moment as on larger projects PhantomJS as non forgiving with syntax errors and tests will simply fail.
- deleted 9y ago
  [deleted]
bluepeter 9y ago
Where's Ariya in all this? He seems to have also completely abandoned Phantom...?
amiga-workbench 9y ago
Thanks for the work, PhantomJS made generating screenshots to show different UI states in bulk effortless. It has saved me days of effort over the last year.
kyriakos 9y ago
With chrome headless we still need an api like phantomjs or slimerjs to have the same functionality.
m4k 9y ago
Have been using PhantomJS since couple of years for data scrapping. It's a really good project.
aaossa 9y ago
I think is a wise decision too. As an OSS collaborator is hard to explain how important/demanding this work is. I really understand his feeling, and I hope that more people like him could collaborate in OSS projects. Thanks for everything!
sillero 9y ago
Thank you for your work Vitaly, it's truly inspiring, you've made a difference for the community. Good luck on your next project.
franciscop 9y ago
IMO it'd be a really smart move for Google to hire Vitaly to help with the launching of this feature and things around it. He has done a great job with PhantomJS. Even an acquisition[1] of PhantomJS would totally make sense, then let him keep working on it but based on headless Chrome and with real resources. [1] Careful how you spin it for this route, learn from TJ/Express https://medium.com/@tjholowaychuk/strongloop-express-40b8bcb8e5af https://medium.com/@tjholowaychuk/strongloop-express-40b8bcb...
- why-el 9y ago
  This is an excellent idea.
  - cr0sh 9y ago
    Yeah - I like that idea too. I played around with PhantomJS for something at my current job - it ultimately didn't work out for us (and we went a different route), but it was interesting and fun to learn about.
- krzyk 9y ago
  Could you elaborate on the TJ/Express? The post you provided looks like the last one of series of events.
  - stevenspasbo 9y ago
    There was a small controversy when StrongLoop purchased the Express repository from TJ, the creator of the project.
  - franciscop 9y ago
    Yes, there was some controversy in the Node.js world because he sold the project of Express to StrongLoop just as he was leaving for Go. I think it was totally a problem on how things were communicated and that he had really a good intention. What didn't help much though is that StrongLoop then left the project unattended so the main person contributing to it (Doug Wilson) wasn't even acknowledged. So if an open source project is being sold: - Make sure everyone knows it's because it'll be better handled and not because of the money. - At least get a decent amount of money! According to TJ it was half a month worth of money. - If you are the company buying it don't leave it unattended afterwards. Anyway, now TJ is surely remembered with a lot of gratitude in the Node.js community and I'm so happy he helped it grow so much! This is all better explained here (and read the comments for TJ opinion): http://thefullstack.xyz/history-express-javascript-framework/ http://thefullstack.xyz/history-express-javascript-framework...
runnr_az 9y ago
Hey dude... thanks from a grateful dev in Scottsdale, AZ. Your hard work enabled a lot of really cool stuff for us! Good luck in your future adventures!
tannhaeuser 9y ago
Many thanks to the maintainer for his work. I think this isn't unexpected, and actually encourage other unpaid maintainers to follow. Reason I'm thinking this is that the current state of voluntary support is unsustainable anyway, and by letting it go we maybe could make the market for dev tools economically viable again.
shaggyfrog 9y ago
But can he invert a binary tree? https://twitter.com/mxcl/status/608682016205344768 https://twitter.com/mxcl/status/608682016205344768 Edit: some people don't get sarcasm
- draw_down 9y ago
  Yeah, I had the same thought. Just because you have a ton of experience that suggests you'd be good at doing a job doesn't mean a tech company would do something silly like hire you for that job.
  - nurettin 9y ago
    Furthermore, you may simply be uninterested in the sort of tasks google might give you.
    - ionwake 9y ago
      Like inverting binary trees
      - true_religion 9y ago
        I want to say it'd never come up (because it's never come up for me) but I just remembered Google is a search engine. It may very well be that they have inverted binary trees on every desk. -- Yes... I am aware inverted indexes are not binary trees.
- namelezz 9y ago
  I am about to ask the same question. LOL
- deleted 9y ago
  [deleted]
- hoenn 9y ago
  Missing the forest for the binary trees
nrook 9y ago
Is there any way to donate to the PhantomJS project? This seems like a good time to throw some money their way, in thanks for what the maintainers (mostly Vitaly over the past few months, at least, it looks like) have done.
deleted 9y ago
[deleted]
elchief 9y ago
I found PhantomJS pretty unusable for my use-case. It always spits logs to stdout and there's no way to change that. Filed a bug and he said "fix it yourself"...sorry I don't know Qt.
- cheeze 9y ago
  Welcome to open source software. Somebody has to fix it. Better than it being closed source and you being told to go away.
- scottydelta 9y ago
  As a developer, I wonder how hard can it be to disable console logging on any project if the source code is provided.
- __s 9y ago
  You don't need to know Qt to comment out printf
stretchwithme 9y ago
I appreciate all the hard work. In my last job, I used PhantomJS with highcharts to provide a web service for generating charts. And used it with the poltergeist gem for headless testing.
tomphoolery 9y ago
Good riddance! PhantomJS is a non-stop firehose of random errors and productivity breakdowns. It's also a way better JS driver than anything else out there. I'm glad Google is following in their footsteps and integrating Phantom's features directly into Chrome, where it will be supported by a large team and (hopefully) headless use of the Blink engine will be standardized so your test integrity doesn't depend on a patch version upgrade of your underlying JS implementation. So, cheers to you Vitaly and anyone else who's helped make Phantom & Poltergeist into my favorite Capybara web driver!
sitepodmatt 9y ago
You deserve a nobel peace prize. I recall fondly informally introducing colleagues to chrome dev tools, injecting jquery via a booklet, and querying the dom like xml xpath. Then taking this headless (server-side) with almost minimal wrapping due to your work. Hail you and damn regexp.. :)
logn 9y ago
I maintain a web driver too (Selenium-based) and have been wondering about how it would compare to Selenium ChromeDriver headless. Mine is built in Java and uses Java's embedded WebKit. If anyone has feedback this is where I'm discussing it, https://github.com/MachinePublishers/jBrowserDriver/issues/273 https://github.com/MachinePublishers/jBrowserDriver/issues/2...
kyledrake 9y ago
PhantomJS was the first thing we used for Neocities screenshots and I've always had a special affection for this project for that reason. Neocities really wouldn't have been possible without the ability to do screenshots.
rogerwang 9y ago
NW.js v0.23 will support headless with Chromium 59. I've been collecting feature requests and sharing the plan. Please see https://github.com/nwjs/nw.js/issues/769#issuecomment-259867271 https://github.com/nwjs/nw.js/issues/769#issuecomment-259867... and https://github.com/nwjs/nw.js/issues/769#issuecomment-294064358 https://github.com/nwjs/nw.js/issues/769#issuecomment-294064...
novaleaf 9y ago
I run phantomjscloud.com, I guess this is the writing on the wall and I better start building another [chrome] back end soon. Probably a name change is in order too!

854 points by wfunction 9y ago

d--b 9y ago

Wow, I'm very impressed. At this stage it is a very wise decision to step down and to focus onto something else, rather than to hold on to a project that will eventually disappear. It takes a lot of courage to move on from a project that had to be maintained for several years and that had such a reach. We can only be thankful for all the good work that went into PhantomJS, and wish the maintainers the best of luck in their next endeavors. Cheers!

moxious 9y ago
He says in his message it's been a slog for some time, looks like a good time to be done with it. Open source is great, we all like it, but demanding unpaid jobs can get old, too.
- yebyen 9y ago
  And it makes sense. You want Chrome in your tests if your users are using Chrome. Very few (if any) of your users will ever visit your app with a headless PhantomJS browser, so it's not a platform that you should go out of your way to support. I've been using Selenium Driver with Chrome in xvfb for my headless testing needs, and I've used PhantomJS for some automation things in the past where it was great, but since I switched I really haven't looked back! I had things breaking subtly that I couldn't fix, and they did not manifest as problems in the Chrome browser or Selenium. I still don't know what was wrong, I just know that my Rails app won't pass its JavaScript functional tests if I use PhantomJS. When I did the evaluation of 3 test drivers, I found that the one with the actual browser in it was the one that worked most reliably.
  - cookiecaper 9y ago
    You can use PhantomJS as the backend on a Selenium script, and this news clearly demonstrates the utility of a higher-level API than using PhantomJS directly. If your tests are in Selenium, changing backends is generally a small matter. I started writing in Selenium instead of CasperJS (a PhantomJS frontend) because PhantomJS experienced intermittent bugs on the page I was trying to access. I think you're right that "real browsers" are still much more reliable for complex use cases, but the low profile of PhantomJS is definitely nice when it works.
  - holtalanm 9y ago
    I used Phantom once upon a time, but I eventually switched to just using xvfb as well within a docker instance. Headless Chrome and Firefox with xvfb.
  - thom_nic 9y ago
    I actually started using webdriverIO + chromedriver after fighting too much with casperJS - while webdriver (and Selenium) seem to have much more momentum there are still some things I really miss that PhantomJS gave when using Capybara. I was a very happy Capy+Phantom user when testing my rails apps. Things like reading the HTTP response code, detecting 404s on assets and catching JS errors in the console are all not possible with Selenium/ Webdriver, and I relied heavily on those capabilities in my Capybara tests. While headless chrome might be able to replace PhantomJS for many use cases that doesn't necessarily mean the APIs will be comparable. In fact I'd more likely expect the Chrome folks to say "the webdriver API is it, because it's a standard." [1] Sadly IMO it's lacking compared to what PhantomJS was capable of. [1] https://www.w3.org/TR/webdriver/ https://www.w3.org/TR/webdriver/
    - erik 9y ago
      When I looked into it, I was surprised by how hard it is to set up basic smoke tests for web development, and that Selenium / Webdriver can't do this.
  - moogly 9y ago
    Thanks so much to the PhantomJS maintainer for his hard work over the years! To me, it feels like his decision is the correct one here. After we realised that we hadn't seen a "one-browser" bug for 2 years in our massive angularjs app, we got rid of all browsers but PhantomJS in our karma suite. PhantomJS' slowness, lagging-behind in webstandards and just my general gut feeling (these facts above made me question the point in running JavaScript tests in an actual browser at all) made me port our karma test suite to jest w/ jsdom and haven't been happier since we, years prior, got rid of our gnarly Selenium test suite that caught 0 bugs but was the major cause for maintenance headache.
    - forgotpwtomain 9y ago
      > After we realised that we hadn't seen a "one-browser" bug for 2 years in our massive angularjs app Really? When I was writing JavaScript I thought I'd hit one ~ once a month. As a matter of fact I hit a v8 bug just yesterday which apparently doesn't support /Etc/GMT[-+][0-9]{0,2} timezones.
    - deleted 9y ago
      [deleted]
Touche 9y ago
Some context, he's essentially been maintaining a web browser (which is a project on the level of an operating system) on his own. Phantom 2 switched to QTWebKit which I'm sure was a tremendous amount of work. Probably at the end of that he was hoping things would get "easier", and it sounds like it hasn't. It's just too much work for one person and if companies aren't will to pay people to do it, I'd quit too.

frik 9y ago

That's quite sad.

etatoby 9y ago
It might be for the best. One of the many companies using PhantomJS to make money could go ahead and employ Vitaly to work on the project full time.

redsummer 9y ago

I'm curious - What is the utility of headless browsers? Are there people who earn money by getting it to automatically fill out forms, enter competitions etc?

kasparsklavins 9y ago
Automated tests.
- askmike 9y ago
  As well as scraping more JS heavy stuff.
kowdermeister 9y ago
Server side rendering, testing, automation just to name a few.
praptak 9y ago
Web scraping for sites that discourage simple robots by checking for JS or serving content via JS. Also, the darker stuff like click fraud and all the other kinds of fraud where you pretend there are humans doing something when in fact there's just a bot.
ajholyoake 9y ago
I have used them in the past to convert graphs and reports to JPGs and PDFs so that I can automatically email them to people in the company who are unable (or unwilling) to use a web page.
d--b 9y ago
I use it to automate turning webpages in pdf.
avaer 9y ago
Testing, rendering, scraping, streaming, botting, and many more. Think "what could I accomplish with a browser, with the slow human replaced by a fast program" and let your imagination run wild. The space is interesting enough that people have jumped through a lot of hoops to make it work in the past; this makes it one less hoop. Oh, and if you ever wondered why web captchas are a thing, one of the reasons is headless browsers.

malinens 9y ago

My colleague from automatic testing says that phantomjs actually is much more stable than chrome...

onion2k 9y ago
That's true, but in my experience the instability of Chrome comes from opening and closing it's windows repeatedly as a large test suite often does - occasionally it doesn't seem to like opening a new instance while another is closing. Headless mode should resolve that problem.
- holtalanm 9y ago
  There was also the issue a few years ago (last time I wrote automated scripts) where the Chrome driver for Selenium would go too fast for the browser to keep up, causing false failures. I had to implement a "wait between actions" feature to handle it, while Phantom had no such problems. I'm assuming this will not be an issue with headless Chrome, since I think half of the problem was due to graphical rendering.

askmike 9y ago

Though a collaboration between the two projects might not be out of the question: https://groups.google.com/d/msg/phantomjs-dev/S-mEBwuSgKQ/PQdE1A5FAwAJ https://groups.google.com/d/msg/phantomjs-dev/S-mEBwuSgKQ/PQ...

make3 9y ago
google should hire him

deleted 9y ago

[deleted]

z3t4 9y ago

> I even bought the Mac for that! I did too, then found out you also needed a dev license for users being able to run your app. Supporting Mac/OSX is damn expensive if your app is free.

Cthulhu_ 9y ago
For iOS development I believe they got rid of that requirement, you can develop iOS apps (at least) using a personal dev account; only when you want to go to the app store do they ask for the license fee.
- prashnts 9y ago
  Unfortunately they limit app capabilities (such as iCloud, keychain access), so you can't use those even for your personal account.
- tokenizerrr 9y ago
  How do I do this without buying dedicated hardware?
  - j_s 9y ago
    Illegaly (virtualization) or in the cloud.

Hernanpm 9y ago

Besides testing, my team uses phantomjs to convert pages to PDF and also to convert javascript generated charts to images. This is sad news, I'm don't think chrome eventually will add support for.

masklinn 9y ago
You can use WebDriver to take screenshots. OTOH I don't think there's any way to do PDF generation without fucking around with injecting window.print() and trying to go from there.
egeozcan 9y ago
There's wkhtmltopdf[1] when you need the PDF functionality. It is licensed with LGPL, so you can make it interop with your commercial product. [1]: https://wkhtmltopdf.org/ https://wkhtmltopdf.org/
- throwaway2016a 9y ago
  Second wkhtmltopdf ... I love PhantonJS but wkhtmltopdf is a better tool for PDFs.
  - Hernanpm 9y ago
    did you have luck with SPA to PDF? I recall I tried it but with no luck for good javascript support.
    - throwaway2016a 9y ago
      SPA as in Single Page Application? We are using it for reports and we are tailoring our HTML specifically to the reports so we haven't had that issue. If I needed Javascript it may not be the best solution. I like that it generates full PDFs (with real text objects) though not just a static image. I'm not sure if you can generate a PDF like that with PhantomJS. I haven't tried.
- Globz 9y ago

chatmasta 9y ago

That sucks for any scraping use case. I have to imagine google has built in some way to detect headless browser mode serverside, even if only they can access it.

deleted 9y ago
[deleted]
wayn3 9y ago
i use phantomjs with the chrome webdriver, or whatever it may be called, already, all the time. if they wanted to do that, theyd already do it. this change doesnt seem to be all that big.

hpaavola 9y ago

Also Firefox will get headless mode in few releases https://bugzilla.mozilla.org/show_bug.cgi?id=1338004 https://bugzilla.mozilla.org/show_bug.cgi?id=1338004

imr_ 9y ago
thank god. I was afraid we will be back in the grasp of the google's moloch.
- tyingq 9y ago
  Curious what "moloch" means in this context. I can google it, of course, but I get "Biblical name relating to a Canaanite god associated with child sacrifice". Which doesn't help me much. End users are children that Google is sacrificing? Or?
  - Pent 9y ago
    They are probably referring to http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ http://slatestarcodex.com/2014/07/30/meditations-on-moloch/
  - fnovd 9y ago
    https://www.poetryfoundation.org/poems-and-poets/poems/detail/49303 https://www.poetryfoundation.org/poems-and-poets/poems/detai... Heading II
  - philh 9y ago
    To give a shorter answer: Scott Alexander (at the slatestarcodex link), through the poem, associates Moloch with negative-sum games, where no one comes out better than they went in. In extreme cases they force us to sacrifice the things we love in order to survive. You throw your children to Moloch to help you defeat enemies; otherwise, you die. Your enemies do the same thing. It would be better if nobody sacrificed their children, but nobody is in a position to bring that outcome to pass. In this context, I would interpret "google's Moloch" along the lines of: Google is net-bad for the world, because of privacy issues and problems with centralisation and so on. Using Google's software (and services) makes them more powerful, so people don't want to use Google's software. But because everyone else is using Google's software, the world is optimized for Google users in a way that it isn't optimized for non-Google users, and so it's difficult to escape. And so Google grows yet stronger, and it becomes more difficult to escape. (To clarify: this is my interpretation of grandparent's use of the phrase. It's not my own position, and there's a decent chance that I'm completely off-base and it's got nothing to do with grandparent's position either.)

tal_berzniz 9y ago

Thank you for PhantomJS! Been using it for testing, generating PDFs and screenshot.

amelius 9y ago
Yes, thanks PhantomJS! I'm wondering, are there any examples out there for generating screenshots on headless Chrome?
- throwaway2016a 9y ago
  From the docs it sounds like it is largely compatible with Selenium https://chromium.googlesource.com/chromium/src/+/lkgr/headless/README.md https://chromium.googlesource.com/chromium/src/+/lkgr/headle... If that is true, the takeScreenshot() should work as normal hopefully.
  - veidr 9y ago
    I can confirm that. Here's an example of how we take screenshots when automated tests fail, using Selenium + Chrome: https://gist.github.com/masonmark/2332c1238a2fa70b5e4fcfffdcd76ca3 https://gist.github.com/masonmark/2332c1238a2fa70b5e4fcfffdc...

avaer 9y ago

Open source software has to be one of the least efficient markets out there. If you sum up the very real value PhantomJS has delivered to very real companies over the last several years, napkin math tells me we wouldn't have the project being abandoned for being a "bloody hell" to work on.

Cthulhu_ 9y ago
The main problem for developers is that getting your company to pay for something like developer tools can be very hard and a long project. Ideally every team gets a budget and a credit card to do as they see fit, but in practice a lot of (especially bigger) companies have a whole acquisition process. It's not uncommon for the hours spent in getting a license for e.g. an editor to be far more expensive than the product itself. I believe this is often highlighted again when Sublime Text is in the news again. This as opposed to open source tooling which has no such hurdles.
- taude 9y ago
  Why do you think open source tooling has no such hurdles? At these same "larger" companies, they're going to have an open source review board, and legal involved for each open source product you want to use. Just because you can download and use it on your local machine, doesn't mean you're not violating your corp policy and procedure.
  - hackits 9y ago
    Whats more surprising is the company I work for does have a support agreement with Oracle, although when we've run into problems that do need a technical expert from Oracle such as crash dumps and stack traces we're told to figure it out our-self. Kind of defeats the whole reason to have a closed source system when you're still on the hook and charging out $500 per hour to clients.
    - jonny_eh 9y ago
      and then you can't search stackoverflow for help, compared to using something like postgres

JepZ 9y ago

I am happy for the guy as he seems to be able to let go without letting anybody down (which seems to be important to him). At the same time, it is sad when people have such a pressure for something they probably started as a fun project.

seleniumuser 9y ago

How does this news fit in with Selenium?

fbnlsr 9y ago
You'll use Headless Chrome instead of PhantomJS as your driver.
cookiecaper 9y ago
Selenium won't really be impacted, as it's higher-level than PhantomJS. This is also a great example of why it's smart to use Selenium or something like it for scripting the tests. You can easily swap out for another backend in Selenium, but if you wrote tests in pure PhantomJS, you're now stuck with a codebase that depends on unmaintained software.

tempodox 9y ago

Wow, that was a quick reaction. Thanks to the maintainers for all the good work! To me, it's always a sad occasion to see diversity diminished. Nothing against Chromium, but I hope it won't be the one browser to rule them all. It's always good to have alternatives.

gus_massa 9y ago
They know since June about the project. This is an interesting conversation: https://groups.google.com/forum/#!topic/phantomjs-dev/S-mEBwuSgKQ https://groups.google.com/forum/#!topic/phantomjs-dev/S-mEBw... ( posted by askmike in https://news.ycombinator.com/item?id=14105613 https://news.ycombinator.com/item?id=14105613 )

mrfusion 9y ago

Naive question here. What makes headless mode so difficult?

tpaschalis 9y ago
It's the same order of magnitude of work, as it is maintaining a fully-fledged browser. It's sad to see open-source projects shut down, but being a sole developer is a lot different than having the resources of a giant like Google, for example.
pavlov 9y ago
It's a very good question. One might imagine that the browser renders everything into a buffer at some point, and you could simply ask the engine to give you a pointer to that data. The reality is very different. WebKit/Blink rendering is intimately tied with the graphics system of each platform, in particular through the use of native widgets and native window system compositors. For example, on the Mac, a lot of compositing within the browser window is done using Core Animation layers. This is a really good idea for performance, because it leverages the work done by Apple to improve their GUI performance. The downside is that capturing the output becomes very tricky when the browser doesn't do the final compositing. Previously this didn't really matter because 99.99% of browser rendering is for end users and they don't need to capture the output (or if they do, they would just use platform GUI functionality like screen capture). An increasing demand for headless rendering has effectively forced browser engine teams to rethink some of the internal APIs so that a pipeline can be built to capture the final rendering.

ohitsdom 9y ago

PhantomJS is a great tool, I implemented it as a PDF report generating system. But will Chromium be able to replace it in this regard? Will Chromium have paging features? Will it be able to repeat table headers when a table body content extends to the next page?

jffry 9y ago
I can't answer all of your questions, but this thread may interest you: https://news.ycombinator.com/item?id=14102248 https://news.ycombinator.com/item?id=14102248
- ohitsdom 9y ago
  Huge help, thank you!

Doctor_Fegg 9y ago

> Chrome is faster and more stable than PhantomJS. And it doesn't eat memory like crazy. Wait, can someone tell me where to download this doesn't-eat-memory-like-crazy version of Chrome? Activity Monitor is showing me 2GB of Chrome processes right now and that's even with The Great Suspender having paused almost all my tabs.

yebyen 9y ago
I saw a trick where you can run Chrome and give it less memory, and it uses less memory. This is done using cgroups. The blog post is somewhat old[1] and not in sync with the version that is in Git[2], you might find a way to do this without Docker (I was using an old version of Docker and kernel that couldn't get it to work. But I need the old version of Docker for reasons.) Chrome will aggressively consume any memory you give it (up to a point?) to "make your browsing experience better" somehow. You're not wrong. But there is modern technology that can make it better. If you have a fast SSD, then Chrome can still use Swap to make your experience better. The later version in the Dockerfile linked on Git also leverages swapaccount with the seccomp setting. This may be one great use of Docker for people that wouldn't yet have been convinced to use Docker for any serious reason. [1]: https://blog.jessfraz.com/post/docker-containers-on-the-desktop/#5-chrome https://blog.jessfraz.com/post/docker-containers-on-the-desk... [2]: https://github.com/jessfraz/dockerfiles/blob/master/chrome/stable/Dockerfile https://github.com/jessfraz/dockerfiles/blob/master/chrome/s...
- tayo42 9y ago
  You can just use cgroups on their own without docker
  - tyingq 9y ago
    I would guess ulimit -v might work as well, so you may not even need cgroups.
    - yebyen 9y ago
      Hah! Something new is also something old. Thanks for that. But won't you run into issues with child processes, of which Chrome tends to spawn a zillion? (I'm reading that each one gets its own limit under ulimit...) And potentially get your browser killed when it hits the limit? I haven't tested this and I don't really know how ulimit works, but I think it's a less effective solution than the cgroups for at least one reason.

est 9y ago

This is sad, phantomjs is better stripped than Chromium headless, if you ever try to install Chromium on servers without X, it requires shit ton of dependencies, while phantomjs was properly modified requires only minimal library.

akx 9y ago
I don't really see the problem. After all, you just install those deps once in your Dockerfile, right? ;)
- yeukhon 9y ago
  Not everyone is using Docker.
- est 9y ago
  chromium-browser requires these chromium-browser chromium-browser-l10n chromium-codecs-ffmpeg-extra cpp cpp-4.8 fontconfig fontconfig-config fonts-dejavu-core hicolor-icon-theme libasound2 libasound2-data libatk1.0-0 libatk1.0-data libatomic1 libavahi-client3 libavahi-common-data libavahi-common3 libcairo2 libcloog-isl4 libcups2 libdatrie1 libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libfile-basedir-perl libfile-desktopentry-perl libfile-mimeinfo-perl libfontconfig1 libfontenc1 libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-common libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa libgmp10 libgnome-keyring-common libgnome-keyring0 libgraphite2-3 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libharfbuzz0b libice6 libisl10 libjasper1 libjbig0 libjpeg-turbo8 libjpeg8 libllvm3.4 libmpc3 libmpfr4 libnspr4 libnss3 libnss3-nssdb libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0 libpciaccess0 libpixman-1-0 libsm6 libspeechd2 libthai-data libthai0 libtiff5 libtxc-dxtn-s2tc0 libx11-xcb1 libxaw7 libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-render0 libxcb-shape0 libxcb-shm0 libxcb-sync1 libxcomposite1 libxcursor1 libxdamage1 libxfixes3 libxft2 libxi6 libxinerama1 libxmu6 libxpm4 libxrandr2 libxrender1 libxshmfence1 libxss1 libxt6 libxtst6 libxv1 libxxf86dga1 libxxf86vm1 x11-common x11-utils x11-xserver-utils xdg-utils phantomjs: fontconfig-config fonts-dejavu-core libfontconfig1 libjpeg-turbo8 libjpeg8
  - 9y ago

bigato 9y ago

Question for those of you more involved with such headless tasks. Do you think that chromium and firefox supporting headless will induce a surge in bots crawling the open web from now on?

uitgewis 9y ago
Not a whole lot more than they are at the moment.
heipei 9y ago
Right off the bat: No. The reason for that is that crawling using a proper browser (i.e. Chrome) is a lot more resource-intensive than using a dedicated tool which only gets the top resource and maybe tries to parse some additional resources. With these kinds of tools you're limited by available bandwidth and IO speeds if you want to store things. If you're looking at a browser, you'll be limited more by things like memory consumption and CPU time, so you'd need a bigger box or more of them to drive the same amount of traffic. There is also not the same amount of ready made applications which take care of crawling, storing and maybe even indexing your data, so not something you can do without actually implementing a lot of things yourself. Of course, that is only talking about wide-scale scanning. If you're only looking to scrape a single target, for whatever reason, then having an instrumented headless browser will greatly simplify things. Headless Chrome should be more efficient than running it in a (virtual) framebuffer. Plus the whole setup for a powerful crawler is reduced to "install Chrome, start it, point $crawler at the API endpoint". My guess is that we might see turnkey crawling / automation tools appear where you supply a list of URLs and the library + Chrome does the rest. Then, browser-based large scale scanning will be within everyone's reach, only limited by their resources. Background: I created https://urlscan.io https://urlscan.io which will simply visit a website and record HTTP interactions (annotating the resources with some helpful meta data). I've been preaching the power of headless instrumented browsers for the better part of a year now ;)

gildas 9y ago

Thanks Vitaly for your work! SEO4Ajax would certainly not exist without PhantomJS. It helped us to deliver the service efficiently at the time. Unfortunately, we had quite a few compatibility issues with it leading us to migrate to Chrome (with xvfb) one year ago. Since then, we must confess that we are very happy of this choice. Chrome is indeed very stable, fast and more importantly for us, always up-to-date.

nashashmi 9y ago

Anybody know how to port code from using PhantomJS to headless Chrome? I have been using CasperJS that wraps to PhantomJS. PhantomJS had its own set of commands. headless chrome will have to be different one way or another.

aantix 9y ago
Looks like it's already usable? "Use headless chromium with capybara and selenium webdriver - today!" -- March 30th http://blog.faraday.io/headless-chromium-with-capybara-and-selenium-webdriver/ http://blog.faraday.io/headless-chromium-with-capybara-and-s...
JimTheDev 9y ago
I wrote a bit about how you can get started here using Node: https://objectpartners.com/2017/04/13/how-to-install-and-use-headless-chrome-on-osx/ https://objectpartners.com/2017/04/13/how-to-install-and-use...

lambduh 9y ago

It's both sad to see an incredibly useful project be sunsetted and exciting that it's no longer needed. I remember a project that used phantomjs to scrape an old government camping site to build a compatibility layer on top. Thank you, if you're reading.

Globz 9y ago

PhantomJS enabled us at the time to bootstrap a big project at work where at the end workflow the app had to turn HTML orders to PDF on the fly, eventually we moved to WKHTMLTOPDF (https://wkhtmltopdf.org/ https://wkhtmltopdf.org/) which is much less hungry with resources but nonetheless PhantomJS played a huge role during the early days of the project and was easy to setup. If I remember correctly the only down side was to find the correct format for our HTML template so PhantomJS would render proper page break and repeat the header for super long orders. I can understand why stepping down is the right decision, maintaining such project by himself is an amazing feat on its own and even more when it proves to be useful for so many companies. Sadly when it becomes your second job you might always be on the lookout for a clean exit and such opportunity just became a reality. Good luck in your future projects Vitaly!

neebz 9y ago
We have used PhantomJS and WKHTMLTOPDF both. Phantomjs hogs a lot of resources but is very good when you want print large PDFs (500+ pages). WKHTMLTOPDF struggles with larger HTMLs.
- Globz 9y ago
  Good to know, this is not our use case at the moment, we mostly generate PDF of 2-3 pages long.
- chrisber 9y ago
  checkout https://github.com/arachnys/athenapdf https://github.com/arachnys/athenapdf. Its based on electron.

eric_b 9y ago

This is a great move. I appreciate what Phantom (and the maintainer) were trying to do, but I have always loathed PhantomJS. It has never worked well. In fact, I'd been away from it for some time, but just last night needed to install it to run some tests and it caused massive frustration. I pulled a Node repo that ran tests using Karma (why people use Karma is a complete mystery to me). I pulled the repo, ran `npm install` and then `npm test`. Sure enough Karma explodes out of the gate. Phantom can't start. I'm on Windows 8.1. I debug for an hour, eventually finding a magic custom binary Ariya created. I then have to copy this binary to the `/node_modules/karma-phantom-launcher/node_modules/phantomjs2-ext/bin` directory. All this to run some Jasmine specs. If Chrome headless support is really as good as "works just like Chrome without the GUI" then I will be one happy camper.

joshuaduffy 9y ago
I personally use Chrome at the moment as on larger projects PhantomJS as non forgiving with syntax errors and tests will simply fail.
deleted 9y ago
[deleted]

bluepeter 9y ago

Where's Ariya in all this? He seems to have also completely abandoned Phantom...?

amiga-workbench 9y ago

Thanks for the work, PhantomJS made generating screenshots to show different UI states in bulk effortless. It has saved me days of effort over the last year.

kyriakos 9y ago

With chrome headless we still need an api like phantomjs or slimerjs to have the same functionality.

m4k 9y ago

Have been using PhantomJS since couple of years for data scrapping. It's a really good project.

aaossa 9y ago

I think is a wise decision too. As an OSS collaborator is hard to explain how important/demanding this work is. I really understand his feeling, and I hope that more people like him could collaborate in OSS projects. Thanks for everything!

sillero 9y ago

Thank you for your work Vitaly, it's truly inspiring, you've made a difference for the community. Good luck on your next project.

franciscop 9y ago

IMO it'd be a really smart move for Google to hire Vitaly to help with the launching of this feature and things around it. He has done a great job with PhantomJS. Even an acquisition[1] of PhantomJS would totally make sense, then let him keep working on it but based on headless Chrome and with real resources. [1] Careful how you spin it for this route, learn from TJ/Express https://medium.com/@tjholowaychuk/strongloop-express-40b8bcb8e5af https://medium.com/@tjholowaychuk/strongloop-express-40b8bcb...

why-el 9y ago
This is an excellent idea.
- cr0sh 9y ago
  Yeah - I like that idea too. I played around with PhantomJS for something at my current job - it ultimately didn't work out for us (and we went a different route), but it was interesting and fun to learn about.
krzyk 9y ago
Could you elaborate on the TJ/Express? The post you provided looks like the last one of series of events.
- stevenspasbo 9y ago
  There was a small controversy when StrongLoop purchased the Express repository from TJ, the creator of the project.
- franciscop 9y ago
  Yes, there was some controversy in the Node.js world because he sold the project of Express to StrongLoop just as he was leaving for Go. I think it was totally a problem on how things were communicated and that he had really a good intention. What didn't help much though is that StrongLoop then left the project unattended so the main person contributing to it (Doug Wilson) wasn't even acknowledged. So if an open source project is being sold: - Make sure everyone knows it's because it'll be better handled and not because of the money. - At least get a decent amount of money! According to TJ it was half a month worth of money. - If you are the company buying it don't leave it unattended afterwards. Anyway, now TJ is surely remembered with a lot of gratitude in the Node.js community and I'm so happy he helped it grow so much! This is all better explained here (and read the comments for TJ opinion): http://thefullstack.xyz/history-express-javascript-framework/ http://thefullstack.xyz/history-express-javascript-framework...

runnr_az 9y ago

Hey dude... thanks from a grateful dev in Scottsdale, AZ. Your hard work enabled a lot of really cool stuff for us! Good luck in your future adventures!

tannhaeuser 9y ago

Many thanks to the maintainer for his work. I think this isn't unexpected, and actually encourage other unpaid maintainers to follow. Reason I'm thinking this is that the current state of voluntary support is unsustainable anyway, and by letting it go we maybe could make the market for dev tools economically viable again.

shaggyfrog 9y ago

But can he invert a binary tree? https://twitter.com/mxcl/status/608682016205344768 https://twitter.com/mxcl/status/608682016205344768 Edit: some people don't get sarcasm

draw_down 9y ago
Yeah, I had the same thought. Just because you have a ton of experience that suggests you'd be good at doing a job doesn't mean a tech company would do something silly like hire you for that job.
- nurettin 9y ago
  Furthermore, you may simply be uninterested in the sort of tasks google might give you.
  - ionwake 9y ago
    Like inverting binary trees
    - true_religion 9y ago
      I want to say it'd never come up (because it's never come up for me) but I just remembered Google is a search engine. It may very well be that they have inverted binary trees on every desk. -- Yes... I am aware inverted indexes are not binary trees.
namelezz 9y ago
I am about to ask the same question. LOL
deleted 9y ago
[deleted]
hoenn 9y ago
Missing the forest for the binary trees

nrook 9y ago

Is there any way to donate to the PhantomJS project? This seems like a good time to throw some money their way, in thanks for what the maintainers (mostly Vitaly over the past few months, at least, it looks like) have done.

deleted 9y ago

[deleted]

elchief 9y ago

I found PhantomJS pretty unusable for my use-case. It always spits logs to stdout and there's no way to change that. Filed a bug and he said "fix it yourself"...sorry I don't know Qt.

cheeze 9y ago
Welcome to open source software. Somebody has to fix it. Better than it being closed source and you being told to go away.
scottydelta 9y ago
As a developer, I wonder how hard can it be to disable console logging on any project if the source code is provided.
__s 9y ago
You don't need to know Qt to comment out printf

stretchwithme 9y ago

I appreciate all the hard work. In my last job, I used PhantomJS with highcharts to provide a web service for generating charts. And used it with the poltergeist gem for headless testing.

tomphoolery 9y ago

Good riddance! PhantomJS is a non-stop firehose of random errors and productivity breakdowns. It's also a way better JS driver than anything else out there. I'm glad Google is following in their footsteps and integrating Phantom's features directly into Chrome, where it will be supported by a large team and (hopefully) headless use of the Blink engine will be standardized so your test integrity doesn't depend on a patch version upgrade of your underlying JS implementation. So, cheers to you Vitaly and anyone else who's helped make Phantom & Poltergeist into my favorite Capybara web driver!

sitepodmatt 9y ago

You deserve a nobel peace prize. I recall fondly informally introducing colleagues to chrome dev tools, injecting jquery via a booklet, and querying the dom like xml xpath. Then taking this headless (server-side) with almost minimal wrapping due to your work. Hail you and damn regexp.. :)

logn 9y ago

I maintain a web driver too (Selenium-based) and have been wondering about how it would compare to Selenium ChromeDriver headless. Mine is built in Java and uses Java's embedded WebKit. If anyone has feedback this is where I'm discussing it, https://github.com/MachinePublishers/jBrowserDriver/issues/273 https://github.com/MachinePublishers/jBrowserDriver/issues/2...

kyledrake 9y ago

PhantomJS was the first thing we used for Neocities screenshots and I've always had a special affection for this project for that reason. Neocities really wouldn't have been possible without the ability to do screenshots.

rogerwang 9y ago

NW.js v0.23 will support headless with Chromium 59. I've been collecting feature requests and sharing the plan. Please see https://github.com/nwjs/nw.js/issues/769#issuecomment-259867271 https://github.com/nwjs/nw.js/issues/769#issuecomment-259867... and https://github.com/nwjs/nw.js/issues/769#issuecomment-294064358 https://github.com/nwjs/nw.js/issues/769#issuecomment-294064...

novaleaf 9y ago

I run phantomjscloud.com, I guess this is the writing on the wall and I better start building another [chrome] back end soon. Probably a name change is in order too!