4 ms·

Nice write up. It still boggles the mind when coming from a Java background how things change but also get worse. Having zip's (.jar/.war) which had the source

Nice write up. It still boggles the mind when coming from a Java background how things change but also get worse. Having zip's (.jar/.war) which had the source and compiled artefacts available and not cluttering the OS with unzip items. I've run into huge issues dealing with deployments to AWS Lambda nodejs, where the previous developer could not work out how to make a 'slip' deployment and just zipped the whole folder node_modules and all and shipped that, this included all testing tools including an entire browser exe etc. 150mb file expanded to 350mb with 300k in files. And 95%+ of it was boiler-pate to test the api endpoint logic... I've still not come across the simplest way to have dependences on modules with testing and still be able to deploy a slim zip. There is more and more push to do container layers for the node_modules, but still many of these has 100'000 objects which are not needed causing lambda stutter(startup sleepyness).

5350-uiop-1130 4y ago
yeah that's the biggest issue with node/npm. 90% of your code lives on the registry, and you probably not even sure what 99% of it actually does. would be nice if you could somehow reverse the compilation of your build back into a source folder, instead of having node_modules - but i think that's probably not possible.
- korijn 4y ago
  Isn't that exactly what bundling does?
  - 5350-uiop-1130 4y ago
    i think bundling generates your build from the source, but i mean generating a source from a build. so what i mean is more like reverse bundling. so that you don't have to pull node_modules every time and install. i;m not totally sure but i dont think you can turn a minified bundled js files back into their original node_module source files...
    - andrewaylett 4y ago
      That very much depends on the build -- if you've a source map, then it may very well be possible. And while I know not everyone publishes them, I strongly recommend doing so. But honestly, ship your sources with your generated files. Please.
KronisLV 4y ago
> just zipped the whole folder node_modules and all and shipped that I mean, in a sense, something similar is what you should do if you're aiming for maximum reproducibility, of course, excluding the things that you don't need. For a comparison, if you wanted to build a Docker container, it's going to include the node_modules folder in it, instead of installing dependencies on startup. So, in a word, somewhat similar to a fat .jar file (e.g. that has all dependencies, vs assuming that the app server will provide some). My problem is that sometimes it's not awfully clear how to exclude all of the stuff that you don't need. For example, my homepage is made with Ruby on Rails, for which I have a basic container image. But to actually bundle all of the resources, I do need Python, Node (with Yarn in my case) and also a bunch of gems, so I also make a separate dev container image. I can install all of the dependencies I need for packaging and run them in a multi-stage image, carrying over the packaged files to something based on the basic image for just running it (e.g. without Python and Node), but I still need the gems to be present and there's no obvious way to carry over only the ones I need for running a Rails app. Even without containers, this very same pain is present in many tech stacks, even if you wanted to run software directly on the servers to which you deploy it - many stacks out there aren't really big on fully self-contained executable packages (or even ones that depend on a particular version of a runtime, like JVM or .NET, but have everything else included). Ergo, working with Ruby, Python, Node and many others can be a bit more troublesome than it should be. Personally, containers alleviate some of those problems to a degree, at the expense of exposing that complexity to you during build time.
- ItsTooMuch 4y ago
  Hmm, why don't you specify your development-time dependencies as devDependencies in your package.json? Then you can simply install only the production ones in your final step Docker image - and only use the devDependencies in your build step. I'm doing it, works like a charm. It'd be really weird and wasteful to ship my TypeScript and Eslint and whatever to cloud/Lambda/...
  - KronisLV 4y ago
    > Hmm, why don't you specify your development-time dependencies as devDependencies in your package.json? Then you can simply install only the production ones in your final step Docker image - and only use the devDependencies in your build step. This is pretty close to the solution and would work if I ran just Node.js server side. Then I could easily have separate sets of dependencies for development and deployment/running, which is what most folks should do. Yet in certain tech stacks, things aren't as easy and there is reliance on system folders instead of local ones. For example, my application that's running on the server should have absolutely nothing to do with Node, since it should run Ruby alone and use the bundled files, however it should have all of the Ruby Gems (basically packages) available. Of course, I also need Node and possibly something else during build time, for generating my assets. For example, my intermediate container images might look like: FROM ruby_with_node_and_python AS builder # Copy only what's needed for installing dependencies and other garbage (e.g. changed code files won't invalidate Docker cache) COPY ./src/.browserslistrc ./src/.ruby-version ./src/babel.config.js ./src/config.ru ./src/Gemfile ./src/Gemfile.lock ./src/yarn.lock ./src/package.json ./src/postcss.config.js ./src/Rakefile /app # install front end dependencies RUN yarn install # add code and run asset creation COPY ./src /app RUN bundle exec rake assets:clobber RAILS_ENV=production RUN bundle exec rake assets:precompile RAILS_ENV=production # install back end dependencies RUN bundle install # run tests (local SQLite, for example) RUN rails db:migrate RAILS_ENV=test RUN rails test RAILS_ENV=test # clean up files after build (so copying in later images doesn't drag garbage along; this could be a separate intermediate image too) RUN rm -rf src/node_modules # clean up files after test (so copying in later images doesn't drag garbage along; this could be a separate intermediate image too) RUN rm -rf src/log/*.log src/tmp/* And then my container images for deployment would look like this: FROM ruby_with_nothing_else AS runner # copy over application files COPY --from=builder /app /app Except that this doesn't work! Why? Because Ruby Gems aren't installed in the application directory, but instead sit in a variety of directories on the actual file system. So instead I'd need something like: FROM ruby_with_package_manager AS runner # we actually could not carry over the precompiled gems etc. because I'm stupid, so instead we now run the install again :( # well it's not like you need front end files here, but this doesn't really mean much COPY ./src/.browserslistrc ./src/.ruby-version ./src/babel.config.js ./src/config.ru ./src/Gemfile ./src/Gemfile.lock ./src/yarn.lock ./src/package.json ./src/postcss.config.js ./src/Rakefile /app # install back end dependencies RUN bundle install # copy over application files COPY --from=builder /app /app So while npm/yarn allows you to avoid some of the issues with `npm install --production` to exclude the devDependencies, other tech stacks aren't quite there yet. In addition, it feels kind of crazy to ship package managers like npm/yarn/pip/composer/bundle/maven or anything else in containers that should be immutable, so in practice you'd end up with more stages, even if you knew how to carry over the packages from one image to another in the technology stacks where it's not entirely clear how to do that. I actually tried something similar to this, but it simply didn't work, especially when native extensions were needed for some of the Gems: https://stackoverflow.com/a/33778136 https://stackoverflow.com/a/33778136
moltar 4y ago
You can use esbuild to produce a single file for the Lambda handler. That’s the approach taken by AWS CDK via NodejsFunction construct.

5350-uiop-1130 4y ago

yeah that's the biggest issue with node/npm. 90% of your code lives on the registry, and you probably not even sure what 99% of it actually does. would be nice if you could somehow reverse the compilation of your build back into a source folder, instead of having node_modules - but i think that's probably not possible.

korijn 4y ago
Isn't that exactly what bundling does?
- 5350-uiop-1130 4y ago
  i think bundling generates your build from the source, but i mean generating a source from a build. so what i mean is more like reverse bundling. so that you don't have to pull node_modules every time and install. i;m not totally sure but i dont think you can turn a minified bundled js files back into their original node_module source files...
  - andrewaylett 4y ago
    That very much depends on the build -- if you've a source map, then it may very well be possible. And while I know not everyone publishes them, I strongly recommend doing so. But honestly, ship your sources with your generated files. Please.

KronisLV 4y ago

> just zipped the whole folder node_modules and all and shipped that I mean, in a sense, something similar is what you should do if you're aiming for maximum reproducibility, of course, excluding the things that you don't need. For a comparison, if you wanted to build a Docker container, it's going to include the node_modules folder in it, instead of installing dependencies on startup. So, in a word, somewhat similar to a fat .jar file (e.g. that has all dependencies, vs assuming that the app server will provide some). My problem is that sometimes it's not awfully clear how to exclude all of the stuff that you don't need. For example, my homepage is made with Ruby on Rails, for which I have a basic container image. But to actually bundle all of the resources, I do need Python, Node (with Yarn in my case) and also a bunch of gems, so I also make a separate dev container image. I can install all of the dependencies I need for packaging and run them in a multi-stage image, carrying over the packaged files to something based on the basic image for just running it (e.g. without Python and Node), but I still need the gems to be present and there's no obvious way to carry over only the ones I need for running a Rails app. Even without containers, this very same pain is present in many tech stacks, even if you wanted to run software directly on the servers to which you deploy it - many stacks out there aren't really big on fully self-contained executable packages (or even ones that depend on a particular version of a runtime, like JVM or .NET, but have everything else included). Ergo, working with Ruby, Python, Node and many others can be a bit more troublesome than it should be. Personally, containers alleviate some of those problems to a degree, at the expense of exposing that complexity to you during build time.

ItsTooMuch 4y ago
Hmm, why don't you specify your development-time dependencies as devDependencies in your package.json? Then you can simply install only the production ones in your final step Docker image - and only use the devDependencies in your build step. I'm doing it, works like a charm. It'd be really weird and wasteful to ship my TypeScript and Eslint and whatever to cloud/Lambda/...
- KronisLV 4y ago
  > Hmm, why don't you specify your development-time dependencies as devDependencies in your package.json? Then you can simply install only the production ones in your final step Docker image - and only use the devDependencies in your build step. This is pretty close to the solution and would work if I ran just Node.js server side. Then I could easily have separate sets of dependencies for development and deployment/running, which is what most folks should do. Yet in certain tech stacks, things aren't as easy and there is reliance on system folders instead of local ones. For example, my application that's running on the server should have absolutely nothing to do with Node, since it should run Ruby alone and use the bundled files, however it should have all of the Ruby Gems (basically packages) available. Of course, I also need Node and possibly something else during build time, for generating my assets. For example, my intermediate container images might look like: FROM ruby_with_node_and_python AS builder # Copy only what's needed for installing dependencies and other garbage (e.g. changed code files won't invalidate Docker cache) COPY ./src/.browserslistrc ./src/.ruby-version ./src/babel.config.js ./src/config.ru ./src/Gemfile ./src/Gemfile.lock ./src/yarn.lock ./src/package.json ./src/postcss.config.js ./src/Rakefile /app # install front end dependencies RUN yarn install # add code and run asset creation COPY ./src /app RUN bundle exec rake assets:clobber RAILS_ENV=production RUN bundle exec rake assets:precompile RAILS_ENV=production # install back end dependencies RUN bundle install # run tests (local SQLite, for example) RUN rails db:migrate RAILS_ENV=test RUN rails test RAILS_ENV=test # clean up files after build (so copying in later images doesn't drag garbage along; this could be a separate intermediate image too) RUN rm -rf src/node_modules # clean up files after test (so copying in later images doesn't drag garbage along; this could be a separate intermediate image too) RUN rm -rf src/log/*.log src/tmp/* And then my container images for deployment would look like this: FROM ruby_with_nothing_else AS runner # copy over application files COPY --from=builder /app /app Except that this doesn't work! Why? Because Ruby Gems aren't installed in the application directory, but instead sit in a variety of directories on the actual file system. So instead I'd need something like: FROM ruby_with_package_manager AS runner # we actually could not carry over the precompiled gems etc. because I'm stupid, so instead we now run the install again :( # well it's not like you need front end files here, but this doesn't really mean much COPY ./src/.browserslistrc ./src/.ruby-version ./src/babel.config.js ./src/config.ru ./src/Gemfile ./src/Gemfile.lock ./src/yarn.lock ./src/package.json ./src/postcss.config.js ./src/Rakefile /app # install back end dependencies RUN bundle install # copy over application files COPY --from=builder /app /app So while npm/yarn allows you to avoid some of the issues with `npm install --production` to exclude the devDependencies, other tech stacks aren't quite there yet. In addition, it feels kind of crazy to ship package managers like npm/yarn/pip/composer/bundle/maven or anything else in containers that should be immutable, so in practice you'd end up with more stages, even if you knew how to carry over the packages from one image to another in the technology stacks where it's not entirely clear how to do that. I actually tried something similar to this, but it simply didn't work, especially when native extensions were needed for some of the Gems: https://stackoverflow.com/a/33778136 https://stackoverflow.com/a/33778136

moltar 4y ago

You can use esbuild to produce a single file for the Lambda handler. That’s the approach taken by AWS CDK via NodejsFunction construct.