7 ms·

Building Node.js applications without dependencies

50 points by puuush 3y ago

deleted 3y ago
[deleted]
robbiejs 3y ago
The article is interesting, though a little messy in its structure. I also don't get what the idea is of the JS testing: "I just want to know if some content is or isn’t rendered". Isn't the easiest way to do this to make a request to the URL via the browser and see for yourself? Or am I being unprofessional?
- tobyhinloopen 3y ago
  > The article is interesting, though a little messy in its structure. Thanks :) I tend to write messy. I'll give future blogs some more revisions to reduce the mess. > Isn't the easiest way to do this to make a request to the URL via the browser and see for yourself? In the applications I'm usually working with, there are many pages, and in many cases they do things differently based on the current context (permissions, roles, state of the business, date) I generally test every route at least twice; one happy flow, and one unhappy flow (trying to submit a form to trigger validation errors, or trying to visit a page you should have no access to), but generally there are many tests per route. I have routes that have literally 1000s of lines of tests for every "real world" case imaginable. Additionally, every bug that's found gets reproduced with a test before it's fixed. The whole web application is usually just gathering params and converting them to domain logic calls, and taking the return value and converting these to a HTML document as a response. The domain logic is fully tested, and the web controller is tested separately by some very basic tests (a simple GET to see if the form fields are present, a POST to see if errors are rendered, and to see if there's no error in the template). Occasionally I will replace the domain logic modules by mocks if the domain logic modules are complex or slow, to keep tests isolated. Because the domain logic modules are tested thoroughly AND the web controllers that invoke the domain logic modules are also tested, most pages (that are backed by modules representing the domain logic) are tested multiple times in different ways. It is not uncommon for me to have widely more LOC for tests than actual code. There's currently 2300 tests in my largest NodeJS project and the whole suite runs in about 30 seconds, but most modules can be tested in less than a second. It is only the "main" module that's slow because it's actually starting the whole application and being tested by actual over-the-network HTTP requests using fetch while the application is also doing real API calls in the background to external services which are mocked in other tests.
  - robbiejs 3y ago
    Thanks! 1000 of lines to test a single route? What logic is on the page? I just find it hard to wrap my mind around. No matter how fancy a web page presents itself, isn't it always in essence a form that triggers a thing, like a post to a db?
    - tobyhinloopen 3y ago
      Think of some kind of submission form with lots of conditional fields. There are a lot of these for business to business communications. It's not just the form but also the data it needs to be loading and saving
user3113 3y ago
We've made a starter template with minimal dependencies available at https://github.com/stormkit-io/monorepo-template-react https://github.com/stormkit-io/monorepo-template-react. Instead of opting for frameworks like next.js, you have the flexibility to use this template, which is platform-agnostic.We have another template built with htmx, outlined in detail at https://stormkit.io/blog/building-dynamic-web-applications-with-ssr-and-htmx https://stormkit.io/blog/building-dynamic-web-applications-w.... You can find the corresponding template at https://github.com/stormkit-io/vite-handlerbar-htmx https://github.com/stormkit-io/vite-handlerbar-htmx. I mainy work with Ruby and Go. I have some Nodejs projects like discord bot and whenever I update my dependencies something breaks. I find that managing dependencies in ruby and go is comparatively smoother for me.
lhnz 3y ago
I have a question for you all: When you're building something and you notice one of your dependencies has a bug or is missing a key feature that you need, do you (a) PR the fix into the dependency and then try to "harrass" the maintainer into merging it for you, (b) publish your own fork of the dependency with the necessary fix, (c) inline the source code for the dependency into your project, effectively taking it on as if it's your own code, (d) completely rewrite the dependency either as a separate package you control or built directly into your own project, or (e) code around the problem / do a hack? I find that often maintainers are so over-worked that it's practically impossible to get a merge in a timely manner, and this leads me to rely on a fork until the PR eventually gets merged. However, I think creating a new package under either your own ownership or the company you work for is often really bad as it can become a kind of hidden technical debt. Nowadays, I definitely consider inlining as a way to capture ownership of the technical debt in a way that is highly visible, but this can add 100s or 1000s of lines of code to a project and if eventually the upstream project moves on you don't get the benefits of their changes without removing the inlined code and untangling any changes that were made to it. The only other approach I've seen is the 'hack' approach, in which you try to dodge the bug or semantic issue. Honestly, that might be the right thing to do in some situations, but it isn't very hygienic within a long-term project (unless you carefully maintain a TODO list of things that need 'correct' solutions).
- alexaholic 3y ago
  On a reasonably large project, you accumulate sufficient dependencies that you end up doing all of that.
- lategloriousgnu 3y ago
  We patch our dependencies. This adds a diff to your repo, not an entire fork of the dependency. A package manager like pnpm will install the package as usual, then apply your patch over the top. https://pnpm.io/cli/patch https://pnpm.io/cli/patch
  - lhnz 3y ago
    Yes, good shout -- I do this sometimes, too. The only thing is that it doesn't work if what you're releasing is a shared package or installable and is not part of an application.
gryzzly 3y ago
I’ve been writing all my side projects without dependencies or only with ones that have no dependencies themselves if I need to do a proof-of-concept or idea and don’t want to write out the whole thing. It’s been so so educating, so fun, I learned a million things. Why companies overwhelmingly choose and allow to rely on 3rd party source code, that needs its own lifecycle support, especially for own domain logic, is not clear to me at all. Sure some specialist tooling is high quality and you should then contribute to it (if it’s in your critical path), but the advantage that is lost by not betting on the platform and properly learning it is huge.
hellsten 3y ago
I’ve tried something similar on the frontend side: I decided to build a UI for Ollama.ai using only HTML, CSS, and JS (Single-Page Application). The goal is to learn something new and have zero runtime dependencies on other projects and NPM modules. Only Node and Parcel.js (https://parceljs.org/ https://parceljs.org/) are needed during development for serving files, bundling, etc. The only runtime dependency is a modern browser. Here's what I have found so far: - JavaScript (vanilla) is a viable alternative to React.js - HTML entities (UTF-8) are an alternative to, for example, font-awesome - Parcel.js is great for bundling cross-browser compatible Javascript apps: simple to install and no configuration needed. By default Parcel.js supports every browser having 0.25% or more of the total amount of active web users. - The HTML template tag and JavaScript work well as a replacement for template libraries like Pug - ChatGPT4 is great at writing the skeleton code for a project Future plans include experimenting with `node:test` for testing. I would need to add external libraries to fully support the following features, but I think I will continue on the zero-dependencies path: - Sanitizing HTML - Code highlighting - Markdown rendering The code can be found here: https://github.com/christianhellsten/ollama-html-ui/ https://github.com/christianhellsten/ollama-html-ui/
a13n 3y ago
Was curious if author considers node libs like `http` a dependency and the answer is no they don't. Looks like they're defining a dependency as an npm module, which seems pretty fair.
- tobyhinloopen 3y ago
  (author here) Anything that's available to you, without configuration, after installing nodejs and starting the application. That includes the whole NodeJS standard library and maybe standard (or extremely common) utilities available on the server (curl, git, ls, find, sed, etc). I still have to find a way to store data and I really don't want to create my own database, but I haven't found a database in the standard library.
  - marcus_holmes 3y ago
    This is analogous to Go. The interface for databases is in the std lib, but that actual implementation of that interface is not. If you want to use Postgres in Go, you have to pick a non-std package (or write your own). I'd consider this an acceptable import ;)
- eyelidlessness 3y ago
  Your comment got me curious if it’s at all possible to build a Node web service without importing any Node modules (ie with just the global/module-local namespace). Glancing at the globals docs, I think you’d be limited to WASM, binary extensions (like N-API), or maaaaaaybe doing something horrifically hacky with CJS require.
  - tomashubelbauer 3y ago
    You can also use HTTP ESM imports which is how I like to build dependencies for my personal projects. In Node I think you still need a special loader for this, in Deno, this is a native feature.
  - DecoPerson 3y ago
    If you’re OK importing node’s internal modules [0], which I believe are loaded anyway, you could achieve a lot, though it definitely wouldn’t be worth it. No developer is an island. [1] [0] https://github.com/nodejs/node/tree/main/lib/internal https://github.com/nodejs/node/tree/main/lib/internal [1] https://youtu.be/TY9xsT6S75A?si=uqPUuxKNl_1vqMhm https://youtu.be/TY9xsT6S75A?si=uqPUuxKNl_1vqMhm
notpachet 3y ago
The author says: > Normally, I use express as a webserver, and I use jest for testing. ... Naturally, without dependencies, I can’t use any of these things. But the examples are still using Jest from the look of it.
- svieira 3y ago
  I think he just elided the imports of node's new-ish built-in `node:test` module: https://nodejs.org/api/test.html https://nodejs.org/api/test.html
  - tobyhinloopen 3y ago
    (author here) Yes, that's correct. It seems like I forget to mention that I would be using node's test module. It was in the script at some point but deleted it in a later revision, it seems.
    - notpachet 3y ago
      Ohh, my bad. I didn't realize that the native module offers a describe/it syntax. That's pretty cool!
      - tobyhinloopen 3y ago
        Me neither until I wrote my own testing library for this blog and threw it away when I noticed the built-in module!
willsmith72 3y ago
Somehow in my career I've never managed to develop the hatred for dependencies and upgrades that so many seem to have. What's the big deal?
- ozim 3y ago
  Because with big enough project you have 2 ways: You update dependencies as a full time job and it bites you because something breaks big way. But most likely you have some help to fix it and as you are on top you might need to put work to fix it but yeah still full time job. You neglect updating dependencies and it bites you and you don’t even expect it just because you have to update something you did not expect will need it and you are now scrambling to do anything because you are messed up. Besides I like writing my own code more than dealing with some lib shenanigans.
  - dspillett 3y ago
    Or you neglect updating them and get bitten by a missed security update, either just the rush to update (and potentially deal with breaking changes and other issues that come from not having done so in a long while) worse because it actually affects your projects instances in the wild.
- anonzzzies 3y ago
  I used to not have issues with Java or .net, but with npm it’s very painful; I need to pin everything as updates, even minor ones, break whole slews of things. And that’s with dependencies of dependencies of dependencies. If you are not a consulting outfit (I suspect consultancy outfits love the js ecosystem; every day something breaks so that’s another few $150/hr hours), but have a company with 100+ applications installed that actively make or help you make your bottom line, you want them to run in peace for years or decades. If no new features or security issues arrive, you don’t want to spend $0.01 on them as what’s the point of that? And that’s where this new fangled crap falls down: we have Django and .net apps running that are 10 years old, php and java ones 15-20 years old that all survived without much or any work. We have apps in js/typescript from 1 year ago that have bucketloads of deprecated, no longer supported, mandatory update warnings. It’s a pita so we are looking to move back to php or/and Django. I can see that in a startup with a dedicated team, this would be no issue; you are on top of all and working daily. Note as well that for some products (the more regulated of our products; financial/health/gov) have strict dependencies we all have to vet; in our experience of the past 4 years, it’s not a hallucination that npms change so often without any useful new features or fixes; just some useless dep updates that sometimes break things and weren’t needed, at all (you do not always need the latest version of everything ya know); I suspect (but maybe someone here knows) is that npms often update because it gives the authors more kudos somehow (as in; if posting many videos on TikTok ups your engagement; does that work for npm updates and new useless npms too?).

50 points by puuush 3y ago

deleted 3y ago

[deleted]

The article is interesting, though a little messy in its structure. I also don't get what the idea is of the JS testing: "I just want to know if some content is or isn’t rendered". Isn't the easiest way to do this to make a request to the URL via the browser and see for yourself? Or am I being unprofessional?

tobyhinloopen 3y ago
> The article is interesting, though a little messy in its structure. Thanks :) I tend to write messy. I'll give future blogs some more revisions to reduce the mess. > Isn't the easiest way to do this to make a request to the URL via the browser and see for yourself? In the applications I'm usually working with, there are many pages, and in many cases they do things differently based on the current context (permissions, roles, state of the business, date) I generally test every route at least twice; one happy flow, and one unhappy flow (trying to submit a form to trigger validation errors, or trying to visit a page you should have no access to), but generally there are many tests per route. I have routes that have literally 1000s of lines of tests for every "real world" case imaginable. Additionally, every bug that's found gets reproduced with a test before it's fixed. The whole web application is usually just gathering params and converting them to domain logic calls, and taking the return value and converting these to a HTML document as a response. The domain logic is fully tested, and the web controller is tested separately by some very basic tests (a simple GET to see if the form fields are present, a POST to see if errors are rendered, and to see if there's no error in the template). Occasionally I will replace the domain logic modules by mocks if the domain logic modules are complex or slow, to keep tests isolated. Because the domain logic modules are tested thoroughly AND the web controllers that invoke the domain logic modules are also tested, most pages (that are backed by modules representing the domain logic) are tested multiple times in different ways. It is not uncommon for me to have widely more LOC for tests than actual code. There's currently 2300 tests in my largest NodeJS project and the whole suite runs in about 30 seconds, but most modules can be tested in less than a second. It is only the "main" module that's slow because it's actually starting the whole application and being tested by actual over-the-network HTTP requests using fetch while the application is also doing real API calls in the background to external services which are mocked in other tests.
- robbiejs 3y ago
  Thanks! 1000 of lines to test a single route? What logic is on the page? I just find it hard to wrap my mind around. No matter how fancy a web page presents itself, isn't it always in essence a form that triggers a thing, like a post to a db?
  - tobyhinloopen 3y ago
    Think of some kind of submission form with lots of conditional fields. There are a lot of these for business to business communications. It's not just the form but also the data it needs to be loading and saving

user3113 3y ago

We've made a starter template with minimal dependencies available at https://github.com/stormkit-io/monorepo-template-react https://github.com/stormkit-io/monorepo-template-react. Instead of opting for frameworks like next.js, you have the flexibility to use this template, which is platform-agnostic.We have another template built with htmx, outlined in detail at https://stormkit.io/blog/building-dynamic-web-applications-with-ssr-and-htmx https://stormkit.io/blog/building-dynamic-web-applications-w.... You can find the corresponding template at https://github.com/stormkit-io/vite-handlerbar-htmx https://github.com/stormkit-io/vite-handlerbar-htmx. I mainy work with Ruby and Go. I have some Nodejs projects like discord bot and whenever I update my dependencies something breaks. I find that managing dependencies in ruby and go is comparatively smoother for me.

lhnz 3y ago

I have a question for you all: When you're building something and you notice one of your dependencies has a bug or is missing a key feature that you need, do you (a) PR the fix into the dependency and then try to "harrass" the maintainer into merging it for you, (b) publish your own fork of the dependency with the necessary fix, (c) inline the source code for the dependency into your project, effectively taking it on as if it's your own code, (d) completely rewrite the dependency either as a separate package you control or built directly into your own project, or (e) code around the problem / do a hack? I find that often maintainers are so over-worked that it's practically impossible to get a merge in a timely manner, and this leads me to rely on a fork until the PR eventually gets merged. However, I think creating a new package under either your own ownership or the company you work for is often really bad as it can become a kind of hidden technical debt. Nowadays, I definitely consider inlining as a way to capture ownership of the technical debt in a way that is highly visible, but this can add 100s or 1000s of lines of code to a project and if eventually the upstream project moves on you don't get the benefits of their changes without removing the inlined code and untangling any changes that were made to it. The only other approach I've seen is the 'hack' approach, in which you try to dodge the bug or semantic issue. Honestly, that might be the right thing to do in some situations, but it isn't very hygienic within a long-term project (unless you carefully maintain a TODO list of things that need 'correct' solutions).

alexaholic 3y ago
On a reasonably large project, you accumulate sufficient dependencies that you end up doing all of that.
lategloriousgnu 3y ago
We patch our dependencies. This adds a diff to your repo, not an entire fork of the dependency. A package manager like pnpm will install the package as usual, then apply your patch over the top. https://pnpm.io/cli/patch https://pnpm.io/cli/patch
- lhnz 3y ago
  Yes, good shout -- I do this sometimes, too. The only thing is that it doesn't work if what you're releasing is a shared package or installable and is not part of an application.

gryzzly 3y ago

I’ve been writing all my side projects without dependencies or only with ones that have no dependencies themselves if I need to do a proof-of-concept or idea and don’t want to write out the whole thing. It’s been so so educating, so fun, I learned a million things. Why companies overwhelmingly choose and allow to rely on 3rd party source code, that needs its own lifecycle support, especially for own domain logic, is not clear to me at all. Sure some specialist tooling is high quality and you should then contribute to it (if it’s in your critical path), but the advantage that is lost by not betting on the platform and properly learning it is huge.

hellsten 3y ago

I’ve tried something similar on the frontend side: I decided to build a UI for Ollama.ai using only HTML, CSS, and JS (Single-Page Application). The goal is to learn something new and have zero runtime dependencies on other projects and NPM modules. Only Node and Parcel.js (https://parceljs.org/ https://parceljs.org/) are needed during development for serving files, bundling, etc. The only runtime dependency is a modern browser. Here's what I have found so far: - JavaScript (vanilla) is a viable alternative to React.js - HTML entities (UTF-8) are an alternative to, for example, font-awesome - Parcel.js is great for bundling cross-browser compatible Javascript apps: simple to install and no configuration needed. By default Parcel.js supports every browser having 0.25% or more of the total amount of active web users. - The HTML template tag and JavaScript work well as a replacement for template libraries like Pug - ChatGPT4 is great at writing the skeleton code for a project Future plans include experimenting with `node:test` for testing. I would need to add external libraries to fully support the following features, but I think I will continue on the zero-dependencies path: - Sanitizing HTML - Code highlighting - Markdown rendering The code can be found here: https://github.com/christianhellsten/ollama-html-ui/ https://github.com/christianhellsten/ollama-html-ui/

a13n 3y ago

Was curious if author considers node libs like `http` a dependency and the answer is no they don't. Looks like they're defining a dependency as an npm module, which seems pretty fair.

tobyhinloopen 3y ago
(author here) Anything that's available to you, without configuration, after installing nodejs and starting the application. That includes the whole NodeJS standard library and maybe standard (or extremely common) utilities available on the server (curl, git, ls, find, sed, etc). I still have to find a way to store data and I really don't want to create my own database, but I haven't found a database in the standard library.
- marcus_holmes 3y ago
  This is analogous to Go. The interface for databases is in the std lib, but that actual implementation of that interface is not. If you want to use Postgres in Go, you have to pick a non-std package (or write your own). I'd consider this an acceptable import ;)
eyelidlessness 3y ago
Your comment got me curious if it’s at all possible to build a Node web service without importing any Node modules (ie with just the global/module-local namespace). Glancing at the globals docs, I think you’d be limited to WASM, binary extensions (like N-API), or maaaaaaybe doing something horrifically hacky with CJS require.
- tomashubelbauer 3y ago
  You can also use HTTP ESM imports which is how I like to build dependencies for my personal projects. In Node I think you still need a special loader for this, in Deno, this is a native feature.
- DecoPerson 3y ago
  If you’re OK importing node’s internal modules [0], which I believe are loaded anyway, you could achieve a lot, though it definitely wouldn’t be worth it. No developer is an island. [1] [0] https://github.com/nodejs/node/tree/main/lib/internal https://github.com/nodejs/node/tree/main/lib/internal [1] https://youtu.be/TY9xsT6S75A?si=uqPUuxKNl_1vqMhm https://youtu.be/TY9xsT6S75A?si=uqPUuxKNl_1vqMhm

notpachet 3y ago

The author says: > Normally, I use express as a webserver, and I use jest for testing. ... Naturally, without dependencies, I can’t use any of these things. But the examples are still using Jest from the look of it.

svieira 3y ago
I think he just elided the imports of node's new-ish built-in `node:test` module: https://nodejs.org/api/test.html https://nodejs.org/api/test.html
- tobyhinloopen 3y ago
  (author here) Yes, that's correct. It seems like I forget to mention that I would be using node's test module. It was in the script at some point but deleted it in a later revision, it seems.
  - notpachet 3y ago
    Ohh, my bad. I didn't realize that the native module offers a describe/it syntax. That's pretty cool!
    - tobyhinloopen 3y ago
      Me neither until I wrote my own testing library for this blog and threw it away when I noticed the built-in module!

willsmith72 3y ago

Somehow in my career I've never managed to develop the hatred for dependencies and upgrades that so many seem to have. What's the big deal?

ozim 3y ago
Because with big enough project you have 2 ways: You update dependencies as a full time job and it bites you because something breaks big way. But most likely you have some help to fix it and as you are on top you might need to put work to fix it but yeah still full time job. You neglect updating dependencies and it bites you and you don’t even expect it just because you have to update something you did not expect will need it and you are now scrambling to do anything because you are messed up. Besides I like writing my own code more than dealing with some lib shenanigans.
- dspillett 3y ago
  Or you neglect updating them and get bitten by a missed security update, either just the rush to update (and potentially deal with breaking changes and other issues that come from not having done so in a long while) worse because it actually affects your projects instances in the wild.
anonzzzies 3y ago
I used to not have issues with Java or .net, but with npm it’s very painful; I need to pin everything as updates, even minor ones, break whole slews of things. And that’s with dependencies of dependencies of dependencies. If you are not a consulting outfit (I suspect consultancy outfits love the js ecosystem; every day something breaks so that’s another few $150/hr hours), but have a company with 100+ applications installed that actively make or help you make your bottom line, you want them to run in peace for years or decades. If no new features or security issues arrive, you don’t want to spend $0.01 on them as what’s the point of that? And that’s where this new fangled crap falls down: we have Django and .net apps running that are 10 years old, php and java ones 15-20 years old that all survived without much or any work. We have apps in js/typescript from 1 year ago that have bucketloads of deprecated, no longer supported, mandatory update warnings. It’s a pita so we are looking to move back to php or/and Django. I can see that in a startup with a dedicated team, this would be no issue; you are on top of all and working daily. Note as well that for some products (the more regulated of our products; financial/health/gov) have strict dependencies we all have to vet; in our experience of the past 4 years, it’s not a hallucination that npms change so often without any useful new features or fixes; just some useless dep updates that sometimes break things and weren’t needed, at all (you do not always need the latest version of everything ya know); I suspect (but maybe someone here knows) is that npms often update because it gives the authors more kudos somehow (as in; if posting many videos on TikTok ups your engagement; does that work for npm updates and new useless npms too?).