12 ms·

Vitest vs. Jest

44 points by ritzaco 2y ago

thecopy 2y ago
Im interested if anyone has experience using node's native test runner in a non-trivial project? We are not using the module mocks from Jest in our project and a quick PoC showed significant (3x) speed improvement, but i am worried about the battle-tested:ness.
- ilkkao 2y ago
  I just started to use nodejs test runner in a new project. All configuration is basically: node --experimental-transform-types --experimental-test-module-mocks --env-file test/test.env --test It then automatically runs all my test/*.tests.ts files. So nice to get rid of all that extra config. As this is still a tiny project, I can tolerate some API changes in these experimental features.
- loevborg 2y ago
  Yes we're using the built-in test runner and it's much better than jest. Jest is full of unneeded magic (= complexity), whereas node:test is straightforward and has a better design. Highly recommended. You can turn off process isolation for extra speedup!
  - thecopy 2y ago
    How large is your project? Without process isolation we got 6x speedup vs. jest! But it is difficult to ensure a clean slate between our test suites (~40 suites which all use a test DB) We are also using some helpers to mock, e.g https://github.com/marchaos/jest-mock-extended https://github.com/marchaos/jest-mock-extended - its such helper libraries im concerned about.
    - loevborg 2y ago
      A young project and a small team (3 months, 3 developers). We're working with a relational db, and we're DELETE FROM all tables that we use at the beginning of each test. That's fast enough so far. Process isolation doesn't help you with state in out-of-process dependencies anyway.
  - jitl 2y ago
    Process isolation is an absolute non-negotiable must for any seriously sized project (>50 developers) I’ve worked on
    - loevborg 2y ago
      If you start the project with good hygiene and clear guidelines (no mutable global state), it's 100% doable. AFAIK JUnit and other test runners don't do any process isolation for unit tests. Why is JavaScript so different?
      - jitl 2y ago
        Many major 3rd party libraries in Javascript either have their own global state, like React, or you interact with platform global state, like NodeJS `process` global (eg `process.on(signal, handler)`) or browser `window.navigator` or `window.document` globals. I have also heard plenty of horror stories about Java unit testing pain and suffering from developer tools engineers supporting Java, after all that's why Bazel is so focused on "hermetic" everything. If I could just make some rules that all engineers would follow exactly, I'd simply make a rule "write no bugs" and then I wouldn't need a test framework at all.
- WorldMaker 2y ago
  I've been very happy with node:test (and deno:test), but also I migrated from mostly mocha as jest I found difficult to configure, especially without unnecessary bundling configurations as the ESM support among other things was broken for far too long (but mocha delivered it well). Most of my successful uses of jest were out-of-the-box configs in create-react-app and the Angular tools to strip uses of karma and replace them with jest. (Side rant: Seriously whoever thought that karma was a good idea for a test runner? "I love V8 so much I'm using Node's V8 to spin up another headless Chromium's V8 so I can V8 from inside V8." lolsob)
- jitl 2y ago
  My experience using built in node:test and other rapidly evolving things is frustration around getting improvements. I test one of my projects with the oldest supported NodeJS version, and simply can’t use node:test there because the old node:test in Node 18 is missing a lot of features. It’s similar to building for Apple’s platforms - great stuff happens every new major version, but there’s no way to use new libraries with older OS versions. Gotta wait for all your users (and yourself) to update your whole runtime to get those new features. Maybe if you are on Node 23 and you absolutely have everything you need, then node:test is okay, but I DO NOT recommend it.
OptionOfT 2y ago
Vitest does not support CommonJS mocking as of today. I switched to Vitest because Jest was giving me flaky coverage results: https://github.com/nodejs/node/issues/51251 https://github.com/nodejs/node/issues/51251 And while Vitest' coverage is stable, I had to add an indirection layer to change how I mock out calls to GitHub's API (@github/octokit). Oh, side note: vite-node's --watch functionality does not restart the process. It does hot reloading which is different enough that it requires some rework at how you set up your app structure and callbacks.
- pcb-rework 2y ago
  The problem is jest is run by megacorp Meta, so it won't be fixed in any reasonable timeframe because they're all about new features to make "impact". Its FOSS contributors are typically slow and working primarily to keep their jobs. Also, with node > 20.9.0 being broken and node = 20.9.0 not being a viable option by being insecure, so it's worth having a look at alternatives like bun or deno because it does seem to have lost its way too. I'm currently pinned to 20.9.0 for system dependencies, but use bun for new dev.
  - cowsandmilk 2y ago
    As described in the article, Jest is no longer run by Meta and its primary maintainers don’t work there. I also would argue Jest development slowed down after Meta transferred the project to OpenJS, so…
    - pcb-rework 2y ago
      > Please don't comment on whether someone read an article. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html A rotten apple doesn't roll far from the tree. The vast majority of Meta projects and former projects are garbage. If you need to ask why, because it's experience.
lscharen 2y ago
I just recently did a website conversion that used vite and took a look at both vitest and playwright. Ended up going with playwright for the “batteries included” browser support and because we use .Net Core for the backend. dotnet doesn’t always get the best support from pure JS/TS projects, so it seemed like a reasonable hedge.
- mst 2y ago
  I was under the impression that playwright was (a very good) how your code behaves when actually running in a browser testing library, and vitest was a how your code behaves on its own testing library. But I'd tend to have relatively independent test suites for my frontend model/state layer and the "how it behaves live in a browser" stuff because that way if I'm not currently iterating on my view components I can test the rest of my frontend code changes without having to faff with headless browser instances. (obviously you'd then want to run both before committing, and again under CI before merge, but still) Depending on how thin your model/state layer is (and if you have a Sufficiently Rich Backend then it can probably be very thin without that being an issue) your situation may not match mine, of course. But y'know, free data point, worth exactly what you paid :D
mythz 2y ago
I prefer using bun:test [1] where possible, super fast to run, all built-in no need to install any packages, supports watch mode, also TypeScript tests just works out of the box without additional deps or steps. You'll also get all the benefits of bun which is a breath of fresh air over npm. [1] https://bun.sh/docs/cli/test https://bun.sh/docs/cli/test
- loevborg 2y ago
  Yes, same with node:test https://nodejs.org/api/test.html https://nodejs.org/api/test.html It's an excellent test runner and much, much faster than the others, especially if you set the "isolation" flag to "none".
- fastball 2y ago
  Testing with Bun would only make sense if you're running it with Bun (or targeting Safari), right?
  - mythz 2y ago
    It's my preferred js runtime for developing/testing *any* new TypeScript/JS libs since I prefer writing JS libraries with TypeScript and Bun had the best UX for running TS out of the box (also big fan of their built-in SQLite and $ shell support). It only doesn't make sense if you're testing a library requiring node, otherwise it's my preferred choice for testing my js libs. Even when testing libraries requiring node in which case I'll run the tests with node:test but still use bun's built-in bundler [1] to build & bundle the typescript code base. Basically Bun's my first choice, but will still use node if I have to. [1] https://bun.sh/docs/bundler https://bun.sh/docs/bundler
  - WorldMaker 2y ago
    If your JS tests are dependent on specifics in JavaScriptCore or V8 are you really writing unit tests? If you want the extra confidence, run it with bun:test and node:test to cover JSC and V8. But JS should be JS and for unit tests feeling a need to test on multiple JS engines is a possible code smell.
draw_down 2y ago
[dead]
deleted 2y ago
[deleted]
petesergeant 2y ago
It’s fair to say I will never use Jest again where possible, instead preferring vitest
catlifeonmars 2y ago
What is meant by “interactive snapshot testing”? If this is the ability to generate and update snapshots in watch mode, Vitest supports this, even though the feature matrix in the article indicates that it does not. I know because I used this feature last month.
- ndimares 2y ago
  Hi, I work at Speakeasy. Thanks for this. I'll look into this and correct the post. Thanks :)
  - hadskasfs 2y ago
    Also "Multi-project runner" is supported, Vitest has workspaces feature.
brianzelip 2y ago
Good write up that helps you get started if you haven't used either lib.
britches11 2y ago
We switched from Jest to Vitest this summer as part of the effort to migrate a large node repository to ESM. Jest presented a lot of issues after switching to ESM. Vitest was compatible with our existing jest code, minus a couple altered method signatures. It was also compatible with custom matchers we had implemented with jest-extend. Most of the migration effort was spent on configuration for our needs. Vitest is quite flexible so it was just a matter of nailing down what worked for us.
streptomycin 2y ago
FWIW I tried switching a fairly large project from Jest to Vitest, hoping for performance improvements. Instead I got some inscrutable errors about circular dependencies. More generally this is one of the really annoying things about Node.js. You might have some circular dependency that works everywhere, but it fails when you use some specific bundler or test runner. Cause they all handle things slightly differently.
- mst 2y ago
  Circular dependencies almost always end up causing trouble *eventually* because the body of one of the modules involved has to be evaluated first (when the other participants in the cycle aren't a fully operational battlestation yet). I've used p3rl.org/circular::require on perl projects before now to root out such dependencies and kill them with fire, and "figure out how to get equivalent technology in TS/JS codebases" is on my list of things to do. Fundamentally, a circular dependency that currently (apparently) "works everywhere" should still be treated as an unexploded footbomb. Excising them can absolutely be annoying and inconvenient, but IME not nearly as much as having the damn thing go off when a future maintainer makes an innocuous change an indeterminate amount of time later. tl;dr: HUMBUG (and sorry I don't have a better answer yet)
craftoman 2y ago
Why not use Node’s built-in testing platform with node --test for server-side purposes or small projects? You can run typescript too. The only drawback is the lack of an expect function, but you can easily built your own or use mine, it’s called tinyexpect.

44 points by ritzaco 2y ago

Im interested if anyone has experience using node's native test runner in a non-trivial project? We are not using the module mocks from Jest in our project and a quick PoC showed significant (3x) speed improvement, but i am worried about the battle-tested:ness.

ilkkao 2y ago
I just started to use nodejs test runner in a new project. All configuration is basically: node --experimental-transform-types --experimental-test-module-mocks --env-file test/test.env --test It then automatically runs all my test/*.tests.ts files. So nice to get rid of all that extra config. As this is still a tiny project, I can tolerate some API changes in these experimental features.
loevborg 2y ago
Yes we're using the built-in test runner and it's much better than jest. Jest is full of unneeded magic (= complexity), whereas node:test is straightforward and has a better design. Highly recommended. You can turn off process isolation for extra speedup!
- thecopy 2y ago
  How large is your project? Without process isolation we got 6x speedup vs. jest! But it is difficult to ensure a clean slate between our test suites (~40 suites which all use a test DB) We are also using some helpers to mock, e.g https://github.com/marchaos/jest-mock-extended https://github.com/marchaos/jest-mock-extended - its such helper libraries im concerned about.
  - loevborg 2y ago
    A young project and a small team (3 months, 3 developers). We're working with a relational db, and we're DELETE FROM all tables that we use at the beginning of each test. That's fast enough so far. Process isolation doesn't help you with state in out-of-process dependencies anyway.
- jitl 2y ago
  Process isolation is an absolute non-negotiable must for any seriously sized project (>50 developers) I’ve worked on
  - loevborg 2y ago
    If you start the project with good hygiene and clear guidelines (no mutable global state), it's 100% doable. AFAIK JUnit and other test runners don't do any process isolation for unit tests. Why is JavaScript so different?
    - jitl 2y ago
      Many major 3rd party libraries in Javascript either have their own global state, like React, or you interact with platform global state, like NodeJS `process` global (eg `process.on(signal, handler)`) or browser `window.navigator` or `window.document` globals. I have also heard plenty of horror stories about Java unit testing pain and suffering from developer tools engineers supporting Java, after all that's why Bazel is so focused on "hermetic" everything. If I could just make some rules that all engineers would follow exactly, I'd simply make a rule "write no bugs" and then I wouldn't need a test framework at all.
WorldMaker 2y ago
I've been very happy with node:test (and deno:test), but also I migrated from mostly mocha as jest I found difficult to configure, especially without unnecessary bundling configurations as the ESM support among other things was broken for far too long (but mocha delivered it well). Most of my successful uses of jest were out-of-the-box configs in create-react-app and the Angular tools to strip uses of karma and replace them with jest. (Side rant: Seriously whoever thought that karma was a good idea for a test runner? "I love V8 so much I'm using Node's V8 to spin up another headless Chromium's V8 so I can V8 from inside V8." lolsob)
jitl 2y ago
My experience using built in node:test and other rapidly evolving things is frustration around getting improvements. I test one of my projects with the oldest supported NodeJS version, and simply can’t use node:test there because the old node:test in Node 18 is missing a lot of features. It’s similar to building for Apple’s platforms - great stuff happens every new major version, but there’s no way to use new libraries with older OS versions. Gotta wait for all your users (and yourself) to update your whole runtime to get those new features. Maybe if you are on Node 23 and you absolutely have everything you need, then node:test is okay, but I DO NOT recommend it.

OptionOfT 2y ago

Vitest does not support CommonJS mocking as of today. I switched to Vitest because Jest was giving me flaky coverage results: https://github.com/nodejs/node/issues/51251 https://github.com/nodejs/node/issues/51251 And while Vitest' coverage is stable, I had to add an indirection layer to change how I mock out calls to GitHub's API (@github/octokit). Oh, side note: vite-node's --watch functionality does not restart the process. It does hot reloading which is different enough that it requires some rework at how you set up your app structure and callbacks.

pcb-rework 2y ago
The problem is jest is run by megacorp Meta, so it won't be fixed in any reasonable timeframe because they're all about new features to make "impact". Its FOSS contributors are typically slow and working primarily to keep their jobs. Also, with node > 20.9.0 being broken and node = 20.9.0 not being a viable option by being insecure, so it's worth having a look at alternatives like bun or deno because it does seem to have lost its way too. I'm currently pinned to 20.9.0 for system dependencies, but use bun for new dev.
- cowsandmilk 2y ago
  As described in the article, Jest is no longer run by Meta and its primary maintainers don’t work there. I also would argue Jest development slowed down after Meta transferred the project to OpenJS, so…
  - pcb-rework 2y ago
    > Please don't comment on whether someone read an article. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html A rotten apple doesn't roll far from the tree. The vast majority of Meta projects and former projects are garbage. If you need to ask why, because it's experience.

lscharen 2y ago

I just recently did a website conversion that used vite and took a look at both vitest and playwright. Ended up going with playwright for the “batteries included” browser support and because we use .Net Core for the backend. dotnet doesn’t always get the best support from pure JS/TS projects, so it seemed like a reasonable hedge.

mst 2y ago
I was under the impression that playwright was (a very good) how your code behaves when actually running in a browser testing library, and vitest was a how your code behaves on its own testing library. But I'd tend to have relatively independent test suites for my frontend model/state layer and the "how it behaves live in a browser" stuff because that way if I'm not currently iterating on my view components I can test the rest of my frontend code changes without having to faff with headless browser instances. (obviously you'd then want to run both before committing, and again under CI before merge, but still) Depending on how thin your model/state layer is (and if you have a Sufficiently Rich Backend then it can probably be very thin without that being an issue) your situation may not match mine, of course. But y'know, free data point, worth exactly what you paid :D

mythz 2y ago

I prefer using bun:test [1] where possible, super fast to run, all built-in no need to install any packages, supports watch mode, also TypeScript tests just works out of the box without additional deps or steps. You'll also get all the benefits of bun which is a breath of fresh air over npm. [1] https://bun.sh/docs/cli/test https://bun.sh/docs/cli/test

loevborg 2y ago
Yes, same with node:test https://nodejs.org/api/test.html https://nodejs.org/api/test.html It's an excellent test runner and much, much faster than the others, especially if you set the "isolation" flag to "none".
fastball 2y ago
Testing with Bun would only make sense if you're running it with Bun (or targeting Safari), right?
- mythz 2y ago
  It's my preferred js runtime for developing/testing *any* new TypeScript/JS libs since I prefer writing JS libraries with TypeScript and Bun had the best UX for running TS out of the box (also big fan of their built-in SQLite and $ shell support). It only doesn't make sense if you're testing a library requiring node, otherwise it's my preferred choice for testing my js libs. Even when testing libraries requiring node in which case I'll run the tests with node:test but still use bun's built-in bundler [1] to build & bundle the typescript code base. Basically Bun's my first choice, but will still use node if I have to. [1] https://bun.sh/docs/bundler https://bun.sh/docs/bundler
- WorldMaker 2y ago
  If your JS tests are dependent on specifics in JavaScriptCore or V8 are you really writing unit tests? If you want the extra confidence, run it with bun:test and node:test to cover JSC and V8. But JS should be JS and for unit tests feeling a need to test on multiple JS engines is a possible code smell.

draw_down 2y ago

[dead]

deleted 2y ago

[deleted]

petesergeant 2y ago

It’s fair to say I will never use Jest again where possible, instead preferring vitest

catlifeonmars 2y ago

What is meant by “interactive snapshot testing”? If this is the ability to generate and update snapshots in watch mode, Vitest supports this, even though the feature matrix in the article indicates that it does not. I know because I used this feature last month.

ndimares 2y ago
Hi, I work at Speakeasy. Thanks for this. I'll look into this and correct the post. Thanks :)
- hadskasfs 2y ago
  Also "Multi-project runner" is supported, Vitest has workspaces feature.

brianzelip 2y ago

Good write up that helps you get started if you haven't used either lib.

britches11 2y ago

We switched from Jest to Vitest this summer as part of the effort to migrate a large node repository to ESM. Jest presented a lot of issues after switching to ESM. Vitest was compatible with our existing jest code, minus a couple altered method signatures. It was also compatible with custom matchers we had implemented with jest-extend. Most of the migration effort was spent on configuration for our needs. Vitest is quite flexible so it was just a matter of nailing down what worked for us.

streptomycin 2y ago

FWIW I tried switching a fairly large project from Jest to Vitest, hoping for performance improvements. Instead I got some inscrutable errors about circular dependencies. More generally this is one of the really annoying things about Node.js. You might have some circular dependency that works everywhere, but it fails when you use some specific bundler or test runner. Cause they all handle things slightly differently.

mst 2y ago
Circular dependencies almost always end up causing trouble *eventually* because the body of one of the modules involved has to be evaluated first (when the other participants in the cycle aren't a fully operational battlestation yet). I've used p3rl.org/circular::require on perl projects before now to root out such dependencies and kill them with fire, and "figure out how to get equivalent technology in TS/JS codebases" is on my list of things to do. Fundamentally, a circular dependency that currently (apparently) "works everywhere" should still be treated as an unexploded footbomb. Excising them can absolutely be annoying and inconvenient, but IME not nearly as much as having the damn thing go off when a future maintainer makes an innocuous change an indeterminate amount of time later. tl;dr: HUMBUG (and sorry I don't have a better answer yet)

craftoman 2y ago

Why not use Node’s built-in testing platform with node --test for server-side purposes or small projects? You can run typescript too. The only drawback is the lack of an expect function, but you can easily built your own or use mine, it’s called tinyexpect.