8 ms·

Node.js Compiler – Compiling your Node.js application into a single executable

115 points by pmq20 9y ago

voltagex_ 9y ago
Cool tidbit found via this: there's a FUSE driver for SquashFS! https://github.com/pmq20/squashfuse/ https://github.com/pmq20/squashfuse/
frik 9y ago
First install the prerequisites: SquashFS Tools 4.3 Python 2.6 or 2.7 Visual Studio 2015 Update 3, all editions including the Community edition (remember to select "Common Tools for Visual C++ 2015" feature during installation). Why not just Node.js and Git for Windows as requirements? Python 2 and a specific VS version, really? Is using npm command to get further packages too uncool?
- minitech 9y ago
  Because the project is based on SquashFS, and the standard Node build tool is GYP, which requires Python 2?
- forgotpw1123 9y ago
  This is risky to say but... NPM is a bad package manager. I could blab about why forever but tons of other people already have. I've never used a popular language with a package manager as bad as npm.
  - sillysaurus3 9y ago
    It's not risky to say. It's risky to do a shallow critique with no solid analysis. I would personally enjoy hearing a critique of why npm is bad in comparison to other package managers. As someone who has spent a lot of time with npm (not even as a webdev), it appears to solve all of my common use cases while also dealing impressively well with various versions of specific dependencies. (Like, A depends on B@0.0.1, C depends on B@0.0.2, etc.)
    - forgotpw1123 9y ago
      There's a ton of annoying little stuff (like doing nothing to prevent path names too long for your filesystem) that could be forgiven for being young, but the big problems are in its architecture. Node module syntax is non standard and can't be tree shaken, with a little more forethought it could have been designed to support static analysis and they wouldn't have needed to reinvent the module syntax for ES6. You can't reliably tell if a piece of code is using a given package until you run it. The dependency resolution sucks. It's better now that they finally made it so nested dependencies aren't duplicated, but why the hell do I have to run a manual dedupe command to do so? What that tells me is that npm doesn't actually track dependencies, its hardly more than a fancy file downloading tool. The repository curation sucks. I'm not sure if you've used Python's package manager, or Java's, or C#'s, or Perls, but npm has zero package curation and is filled to the brim with garbage compared to the others. I have published packages for C# and Java and the semantics are well defined and reasonable, you even cryptographcally sign your code and the system checks to make sure you upload documentation etc... In the case of java, which has the best system I've seen, they even use something similar to DNS validation to make sure you control package namespace. The above point is the absolute worst thing about npm and drawfs the other problems. Let me give an example by pasting the package publishing guidelines for Maven(Java) vs npm: Publishing an npm package: >Use npm publish to publish the package. >Note that everything in the directory will be included unless it is ignored by a local .gitignore or .npmignore file as described in npm-developers. >Also make sure there isn't already a package with the same name, owned by somebody else. >Test: Go to https://npmjs.com/package/<package> https://npmjs.com/package/<package>. You should see the information for your new package. I had to condense the Java guidelines but it's roughly this: create JIRA account apply for access review requirements verify namespace ownership generate crypto keys package coordinates/versioning project description URL and license info developer info SCM info package deployment generate binaries revolve package dependencies crypto sign package generate documentation online verification Sounds like a lot of work right? It takes a few hours to setup and verify your domain/keys the first time but after that all of these steps are automated. The signing/building/doc generation/ deployment is totally automated. Npm is a toy in comparison. from their docs: Why Do We Have Requirements? In order to ensure a minimum level of quality of the components available in the Central Repository, we have established a number of requirements your deployment components have to meet. This allows your users to find all the relevant details about the components from the metadata provided in the Central Repository. The following sections will detail these requirements. Left-pad is a perfect example of why npm sucks. Nuget, Maven, Phython, and CPAN would have never let something that stupid happen.
  - abritinthebay 9y ago
    Most of those critiques boil down to "it works differently to how I'm used to" or "I don't know how to use it so I'm going to criticize it poorly". NPM has issues but most criticism of it is really complaints about bad dev practices (like not pinning versions correctly).
    - microcolonel 9y ago
      One other one is that it has hilariously poor performance, and bizarre failure cases. For some forms of network failures it'll just say "callback called too many times" and crash. There are literally no worse major package managers. A good package manager can be misused and still work reasonably well; a bad one doesn't even work all that well when used correctly.
      - int_19h 9y ago
        Or how about this for error reporting: https://github.com/npm/npm/issues/9633 https://github.com/npm/npm/issues/9633 I especially like the part where it says "This is most likely not a problem with npm itself", even though it clearly is a problem with npm itself (as evidenced by non-deterministic nature of the bug).
        abritinthebay 9y ago
        That message is their generic message because ANY error ended up being reported in github as an issue with NPM. Again, bad dev practice.
    - forgotpw1123 9y ago
      More like it doesn't work as well as any other package manage I've used :)
      - allover 9y ago
        It works better than Pip and the whole setup.py mess. (Pip doesn't even have a dependency resolver: https://github.com/pypa/pip/issues/988 https://github.com/pypa/pip/issues/988)
  - kowdermeister 9y ago
    You clearly haven't played with PIP then :)
- johncolanduoni 9y ago
  How many projects have you seen that are agnostic about VS versions? VS doesn't maintain a stable C++ ABI between versions, not to mention the fact that VS2017 has broken a fair number of projects. It's not like pinning a clang/gcc version.
icf80 9y ago
50 MB executable?
- est 9y ago
  There are quite a few minimalize Blink efforts. Like this one: https://github.com/bloomberg/chromium.bb https://github.com/bloomberg/chromium.bb https://github.com/weolar/miniblink49 https://github.com/weolar/miniblink49 The latter is around 9MB. The author is working on a electron wrapper.
  - ComodoHacker 9y ago
    Does this project include Blink? Looks like not.
    - est 9y ago
      It contains a stripped version of Blink with most of functions.
      - dchest 9y ago
        Blink is a rendering engine, V8 is a JavaScript engine. Node.js contains V8, but not Blink.
        est 9y ago
        It's a naming thing, it does include v8 to execute js. Blink is a fork of webkit, by your logic, most softwares based on webkit can not execute javascript???
        dchest 9y ago
        No, Blink doesn't include V8, it's a separate project. WebKit consists of WebCore — layout engine, and JavaScriptCore — JavaScript engine. Google forked WebCore part of WebKit to create Blink. Before the fork it used a combination of WebCore and V8. This information is readily available on, e.g. Wikipedia, so I'm not sure why you even try to argue about it. Node.js doesn't include Blink, it only includes V8. Node.js Compiler doesn't include Blink into the build result — it includes Node.js.
slgg 9y ago
Just wanted to mention a similar project, which was recently announced by Zeit, called "pkg": https://github.com/zeit/pkg https://github.com/zeit/pkg
vortico 9y ago
52 MB and 32 MB for macOS and Windows, respectively, for anyone looking for the important numbers.
- _pmf_ 9y ago
  Not too bad; with Qt, you can get to about 10 MB - 20 MB for a dynamically linked application.
- TheRealPomax 9y ago
  how does that compare to the numbers for pkg, that Zeit just released and was on HN a few days ago? https://github.com/zeit/pkg https://github.com/zeit/pkg
  - akras14 9y ago
    About the same 30MB for a hello world application.
- zmix 9y ago
  Hmmm... why the heck did Mozilla kill xulrunner? It's a few MB less ;-)
sandGorgon 9y ago
squashfs ? maybe docker is a better alternative - especially with the new 1.13 experimental feature "--squash" https://blog.docker.com/2017/01/whats-new-in-docker-1-13/ https://blog.docker.com/2017/01/whats-new-in-docker-1-13/
notsohuman 9y ago
it would be awesome if someone creates a compiler for javaScript.
- notsohuman 9y ago
  I mean an actual compiler rather which is much more optimized than node environment rather than just making things into executables
  - ComodoHacker 9y ago
    V8 includes JIT compiler. Could it really be "much more" optimized?
    - notsohuman 9y ago
      I agree V8 does an awesome job buts it compilation phase is not done separate before running. Lot could done to gain the speed in those areas.
    - mbel 9y ago
      Perhaps. JIT compiler has to care deeply for compilation time. For AOT compiler compilation time is a little bit less pressing problem which leaves some space for optimization passes that take too long for JIT.
    - perbu 9y ago
      If you look at Hiphop vs. HHVM there is certainly a significant performance upside in taking advantage of clangs or gccs ability to produce efficient code. I'm not sure why Facebook decided to stop maintaining hiphop. Perhaps manageability wasn't great.
    - notsohuman 9y ago
      there is always something more that can be done
  - perbu 9y ago
    This would be pretty useful. Something like hiphop which could transpile into C or C++ would mean we (IncludeOS) could build tooling to run a node application on directly in a vm without a underlying OS.
drinchev 9y ago
I'm really happy with the effort of "compiling" NodeJS to native binary. What I'm looking forward to is lightweight electron that will allow us to use the native browser-engine for each platform. QT/GTK for Linux, WebkitView for macOS, Edge for Windows. Hope the effort of node-compiler, zeit/pkg will be beneficial for the future of desktop apps.
- dchest 9y ago
  Here's proof of concept from yesterday: https://news.ycombinator.com/item?id=14260755 https://news.ycombinator.com/item?id=14260755
- statictype 9y ago
  One major advantage of Electron is not having to deal with browser compatibility issues - you write your code against exactly one runtime (sure, you may need platform specific flags for things - but no risk of the runtime changing/breaking after you deploy) Leveraging existing OS engine will reduce file size but at the cost of not having a stable and known platform that your code will run on.
  - drinchev 9y ago
    These days fixing those issues should not be a big deal. Having all those polyfills / babel / webpack tools you can be quite safe about your app. Nevertheless having a nice test suite also makes a benefit.
- tracker1 9y ago
  I can't help but think that electron could eventually become similar to Adobe Air (except actually open), where the main runtime is/can be installed separately, so multiple apps can share a runtime.
  - h1d 9y ago
    Please don't ever do this. No one likes installing separate system library of some sort, even multiple versions of it besides downloading the app. Besides what's the point when most people would only use just a few Electron apps anyway. App developers also have to ask users to download versions between this and that system library only causing confusions for users.
franciscop 9y ago
I am wondering if using some of the front-end techniques for back-end would be beneficial. Tree-shaking, minimization and gzip would probably help with the binary file size. But as we go there, maybe some more advanced things like the closure[1] compiler or prepack[2] would make it faster. [1] http://www.syntaxsuccess.com/viewarticle/closure-compiler-vs-uglifyjs http://www.syntaxsuccess.com/viewarticle/closure-compiler-vs... [2] https://prepack.io/ https://prepack.io/
- mbel 9y ago
  > But as we go there, maybe some more advanced things like the closure[1] compiler or prepack[2] would make it faster. And then one could probably go even deeper in search of performance and instead of compiling JS to JS, compile it to C or even straight to native CPU assembly.
  - andars 9y ago
    AOT compilation of js is probably not worthwhile.
- douche 9y ago
  Yeah. Like all those compilers and runtimes for the back-end languages we were already using before JavaScript ate everyone's brains, that served as the inspiration for all of this madness on the front-end side.
- h1d 9y ago
  Back end is about stability (= don't do fancy). So unless it increases performance like maybe prepack does, I don't see much point.
vmasto 9y ago
How is this different than zeit's recently released pkg? https://github.com/zeit/pkg https://github.com/zeit/pkg
- mstade 9y ago
  The readme says this: > Pkg hacked fs.* API's dynamically in order to access in-package files, whereas Node.js Compiler leaves them alone and instead works on a deeper level via libsquash. Pkg uses JSON to store in-package files while Node.js Compiler uses the more sophisticated and widely used SquashFS as its data structure.
  - michaelmior 9y ago
    More sophisticated is not a great way to suggest something ks better. That said, I could image this probably does give better performance if you have a lot of I/O with files in the package.
    - jazoom 9y ago
      Maybe they weren't saying that is necessarily better?
      - mstade 9y ago
        Yeah I didn't read it that way either, just a matter of factly "here are the differences" – that section probably could be expanded and clarified.
z3t4 9y ago
Is there any reason to compile your JavaScript except obfuscating the code !? It will most likely run slower, because of no runtime optimizations.
- ricardobeat 9y ago
  There is no real compilation going on here, V8 is bundled in the executable.
ricardobeat 9y ago
The comparison table fails to mention one huge advantage zeit/pkg has: it is able to generate executables for all three platforms from any one of them, no need to setup three dev or build environments.

115 points by pmq20 9y ago

Cool tidbit found via this: there's a FUSE driver for SquashFS! https://github.com/pmq20/squashfuse/ https://github.com/pmq20/squashfuse/

frik 9y ago

First install the prerequisites: SquashFS Tools 4.3 Python 2.6 or 2.7 Visual Studio 2015 Update 3, all editions including the Community edition (remember to select "Common Tools for Visual C++ 2015" feature during installation). Why not just Node.js and Git for Windows as requirements? Python 2 and a specific VS version, really? Is using npm command to get further packages too uncool?

minitech 9y ago
Because the project is based on SquashFS, and the standard Node build tool is GYP, which requires Python 2?
forgotpw1123 9y ago
This is risky to say but... NPM is a bad package manager. I could blab about why forever but tons of other people already have. I've never used a popular language with a package manager as bad as npm.
- sillysaurus3 9y ago
  It's not risky to say. It's risky to do a shallow critique with no solid analysis. I would personally enjoy hearing a critique of why npm is bad in comparison to other package managers. As someone who has spent a lot of time with npm (not even as a webdev), it appears to solve all of my common use cases while also dealing impressively well with various versions of specific dependencies. (Like, A depends on B@0.0.1, C depends on B@0.0.2, etc.)
  - forgotpw1123 9y ago
    There's a ton of annoying little stuff (like doing nothing to prevent path names too long for your filesystem) that could be forgiven for being young, but the big problems are in its architecture. Node module syntax is non standard and can't be tree shaken, with a little more forethought it could have been designed to support static analysis and they wouldn't have needed to reinvent the module syntax for ES6. You can't reliably tell if a piece of code is using a given package until you run it. The dependency resolution sucks. It's better now that they finally made it so nested dependencies aren't duplicated, but why the hell do I have to run a manual dedupe command to do so? What that tells me is that npm doesn't actually track dependencies, its hardly more than a fancy file downloading tool. The repository curation sucks. I'm not sure if you've used Python's package manager, or Java's, or C#'s, or Perls, but npm has zero package curation and is filled to the brim with garbage compared to the others. I have published packages for C# and Java and the semantics are well defined and reasonable, you even cryptographcally sign your code and the system checks to make sure you upload documentation etc... In the case of java, which has the best system I've seen, they even use something similar to DNS validation to make sure you control package namespace. The above point is the absolute worst thing about npm and drawfs the other problems. Let me give an example by pasting the package publishing guidelines for Maven(Java) vs npm: Publishing an npm package: >Use npm publish to publish the package. >Note that everything in the directory will be included unless it is ignored by a local .gitignore or .npmignore file as described in npm-developers. >Also make sure there isn't already a package with the same name, owned by somebody else. >Test: Go to https://npmjs.com/package/<package> https://npmjs.com/package/<package>. You should see the information for your new package. I had to condense the Java guidelines but it's roughly this: create JIRA account apply for access review requirements verify namespace ownership generate crypto keys package coordinates/versioning project description URL and license info developer info SCM info package deployment generate binaries revolve package dependencies crypto sign package generate documentation online verification Sounds like a lot of work right? It takes a few hours to setup and verify your domain/keys the first time but after that all of these steps are automated. The signing/building/doc generation/ deployment is totally automated. Npm is a toy in comparison. from their docs: Why Do We Have Requirements? In order to ensure a minimum level of quality of the components available in the Central Repository, we have established a number of requirements your deployment components have to meet. This allows your users to find all the relevant details about the components from the metadata provided in the Central Repository. The following sections will detail these requirements. Left-pad is a perfect example of why npm sucks. Nuget, Maven, Phython, and CPAN would have never let something that stupid happen.
- abritinthebay 9y ago
  Most of those critiques boil down to "it works differently to how I'm used to" or "I don't know how to use it so I'm going to criticize it poorly". NPM has issues but most criticism of it is really complaints about bad dev practices (like not pinning versions correctly).
  - microcolonel 9y ago
    One other one is that it has hilariously poor performance, and bizarre failure cases. For some forms of network failures it'll just say "callback called too many times" and crash. There are literally no worse major package managers. A good package manager can be misused and still work reasonably well; a bad one doesn't even work all that well when used correctly.
    - int_19h 9y ago
      Or how about this for error reporting: https://github.com/npm/npm/issues/9633 https://github.com/npm/npm/issues/9633 I especially like the part where it says "This is most likely not a problem with npm itself", even though it clearly is a problem with npm itself (as evidenced by non-deterministic nature of the bug).
      - abritinthebay 9y ago
        That message is their generic message because ANY error ended up being reported in github as an issue with NPM. Again, bad dev practice.
  - forgotpw1123 9y ago
    More like it doesn't work as well as any other package manage I've used :)
    - allover 9y ago
      It works better than Pip and the whole setup.py mess. (Pip doesn't even have a dependency resolver: https://github.com/pypa/pip/issues/988 https://github.com/pypa/pip/issues/988)
- kowdermeister 9y ago
  You clearly haven't played with PIP then :)
johncolanduoni 9y ago
How many projects have you seen that are agnostic about VS versions? VS doesn't maintain a stable C++ ABI between versions, not to mention the fact that VS2017 has broken a fair number of projects. It's not like pinning a clang/gcc version.

icf80 9y ago

50 MB executable?

est 9y ago
There are quite a few minimalize Blink efforts. Like this one: https://github.com/bloomberg/chromium.bb https://github.com/bloomberg/chromium.bb https://github.com/weolar/miniblink49 https://github.com/weolar/miniblink49 The latter is around 9MB. The author is working on a electron wrapper.
- ComodoHacker 9y ago
  Does this project include Blink? Looks like not.
  - est 9y ago
    It contains a stripped version of Blink with most of functions.
    - dchest 9y ago
      Blink is a rendering engine, V8 is a JavaScript engine. Node.js contains V8, but not Blink.
      - est 9y ago
        It's a naming thing, it does include v8 to execute js. Blink is a fork of webkit, by your logic, most softwares based on webkit can not execute javascript???
        dchest 9y ago
        No, Blink doesn't include V8, it's a separate project. WebKit consists of WebCore — layout engine, and JavaScriptCore — JavaScript engine. Google forked WebCore part of WebKit to create Blink. Before the fork it used a combination of WebCore and V8. This information is readily available on, e.g. Wikipedia, so I'm not sure why you even try to argue about it. Node.js doesn't include Blink, it only includes V8. Node.js Compiler doesn't include Blink into the build result — it includes Node.js.

slgg 9y ago

Just wanted to mention a similar project, which was recently announced by Zeit, called "pkg": https://github.com/zeit/pkg https://github.com/zeit/pkg

vortico 9y ago

52 MB and 32 MB for macOS and Windows, respectively, for anyone looking for the important numbers.

_pmf_ 9y ago
Not too bad; with Qt, you can get to about 10 MB - 20 MB for a dynamically linked application.
TheRealPomax 9y ago
how does that compare to the numbers for pkg, that Zeit just released and was on HN a few days ago? https://github.com/zeit/pkg https://github.com/zeit/pkg
- akras14 9y ago
  About the same 30MB for a hello world application.
zmix 9y ago
Hmmm... why the heck did Mozilla kill xulrunner? It's a few MB less ;-)

sandGorgon 9y ago

squashfs ? maybe docker is a better alternative - especially with the new 1.13 experimental feature "--squash" https://blog.docker.com/2017/01/whats-new-in-docker-1-13/ https://blog.docker.com/2017/01/whats-new-in-docker-1-13/

notsohuman 9y ago

it would be awesome if someone creates a compiler for javaScript.

notsohuman 9y ago
I mean an actual compiler rather which is much more optimized than node environment rather than just making things into executables
- ComodoHacker 9y ago
  V8 includes JIT compiler. Could it really be "much more" optimized?
  - notsohuman 9y ago
    I agree V8 does an awesome job buts it compilation phase is not done separate before running. Lot could done to gain the speed in those areas.
  - mbel 9y ago
    Perhaps. JIT compiler has to care deeply for compilation time. For AOT compiler compilation time is a little bit less pressing problem which leaves some space for optimization passes that take too long for JIT.
  - perbu 9y ago
    If you look at Hiphop vs. HHVM there is certainly a significant performance upside in taking advantage of clangs or gccs ability to produce efficient code. I'm not sure why Facebook decided to stop maintaining hiphop. Perhaps manageability wasn't great.
  - notsohuman 9y ago
    there is always something more that can be done
- perbu 9y ago
  This would be pretty useful. Something like hiphop which could transpile into C or C++ would mean we (IncludeOS) could build tooling to run a node application on directly in a vm without a underlying OS.

drinchev 9y ago

I'm really happy with the effort of "compiling" NodeJS to native binary. What I'm looking forward to is lightweight electron that will allow us to use the native browser-engine for each platform. QT/GTK for Linux, WebkitView for macOS, Edge for Windows. Hope the effort of node-compiler, zeit/pkg will be beneficial for the future of desktop apps.

dchest 9y ago
Here's proof of concept from yesterday: https://news.ycombinator.com/item?id=14260755 https://news.ycombinator.com/item?id=14260755
statictype 9y ago
One major advantage of Electron is not having to deal with browser compatibility issues - you write your code against exactly one runtime (sure, you may need platform specific flags for things - but no risk of the runtime changing/breaking after you deploy) Leveraging existing OS engine will reduce file size but at the cost of not having a stable and known platform that your code will run on.
- drinchev 9y ago
  These days fixing those issues should not be a big deal. Having all those polyfills / babel / webpack tools you can be quite safe about your app. Nevertheless having a nice test suite also makes a benefit.
tracker1 9y ago
I can't help but think that electron could eventually become similar to Adobe Air (except actually open), where the main runtime is/can be installed separately, so multiple apps can share a runtime.
- h1d 9y ago
  Please don't ever do this. No one likes installing separate system library of some sort, even multiple versions of it besides downloading the app. Besides what's the point when most people would only use just a few Electron apps anyway. App developers also have to ask users to download versions between this and that system library only causing confusions for users.

franciscop 9y ago

I am wondering if using some of the front-end techniques for back-end would be beneficial. Tree-shaking, minimization and gzip would probably help with the binary file size. But as we go there, maybe some more advanced things like the closure[1] compiler or prepack[2] would make it faster. [1] http://www.syntaxsuccess.com/viewarticle/closure-compiler-vs-uglifyjs http://www.syntaxsuccess.com/viewarticle/closure-compiler-vs... [2] https://prepack.io/ https://prepack.io/

mbel 9y ago
> But as we go there, maybe some more advanced things like the closure[1] compiler or prepack[2] would make it faster. And then one could probably go even deeper in search of performance and instead of compiling JS to JS, compile it to C or even straight to native CPU assembly.
- andars 9y ago
  AOT compilation of js is probably not worthwhile.
douche 9y ago
Yeah. Like all those compilers and runtimes for the back-end languages we were already using before JavaScript ate everyone's brains, that served as the inspiration for all of this madness on the front-end side.
h1d 9y ago
Back end is about stability (= don't do fancy). So unless it increases performance like maybe prepack does, I don't see much point.

vmasto 9y ago

How is this different than zeit's recently released pkg? https://github.com/zeit/pkg https://github.com/zeit/pkg

mstade 9y ago
The readme says this: > Pkg hacked fs.* API's dynamically in order to access in-package files, whereas Node.js Compiler leaves them alone and instead works on a deeper level via libsquash. Pkg uses JSON to store in-package files while Node.js Compiler uses the more sophisticated and widely used SquashFS as its data structure.
- michaelmior 9y ago
  More sophisticated is not a great way to suggest something ks better. That said, I could image this probably does give better performance if you have a lot of I/O with files in the package.
  - jazoom 9y ago
    Maybe they weren't saying that is necessarily better?
    - mstade 9y ago
      Yeah I didn't read it that way either, just a matter of factly "here are the differences" – that section probably could be expanded and clarified.

z3t4 9y ago

Is there any reason to compile your JavaScript except obfuscating the code !? It will most likely run slower, because of no runtime optimizations.

ricardobeat 9y ago
There is no real compilation going on here, V8 is bundled in the executable.

ricardobeat 9y ago

The comparison table fails to mention one huge advantage zeit/pkg has: it is able to generate executables for all three platforms from any one of them, no need to setup three dev or build environments.