4 ms·

Folio: A customizable test framework to build your own test frameworks

41 points by tilt 5y ago

AriaMinaei 5y ago
On a tangent, are there any test frameworks focused on runtime performance? With esbuild speeding up everything else, jest has become the main bottleneck in my dev workflow.
- mhagemeister 5y ago
  There is uvu https://github.com/lukeed/uvu https://github.com/lukeed/uvu which has runtime performance as the number one goal. Maybe that is something for you.
- brundolf 5y ago
  Is it the overhead from Jest itself, or your tests? At the end of the day, a JS test runner has to run a bunch of tests written in JS. That can't be optimized away, and I'd assume it's always going to dominate performance.
  - eyelidlessness 5y ago
    Yes, Jest is slow. Performance-focused test frameworks like ava and uvu prominently display benchmarks demonstrating this. Jest’s slowness, in my experience, is primarily rooted in three issues: - The transformations it performs to support auto-mocking (AFAIK still its flagship feature) are constant overhead. - Its process model is per-module, rather than per-test, limiting concurrency. A hypothetical test run with one test module with 100 tests is ~100x slower than 100 modules with 1 test each. - Its process model is per-module (yes, I’m repeating this factor because it has another major impact on perf), but Jest destroys its dependency cache between tests, causing memory leaks to accumulate. Memory leaks in Node especially are common and trivial, where long-running libraries (like, say, a logger) set up some config-driven singleton on require/import. I’ve sunk literally weeks if not months into trying to work around issues like these on teams that insisted on keeping either Jest or other tools with aforementioned memory leaks. And while I value performance, I primarily spent that time trying to mitigate problems (false positives/negatives in test runs, intermittent failures) caused by the poor isolation model.
    - eyelidlessness 5y ago
      I can’t edit this but I’ve scrolled through my threads several times and a mistake I made keeps bothering me: > A hypothetical test run with one test module with 100 tests is ~100x slower than 100 modules with 1 test each. This was really bad math. For CPU bound cases, it’s ~N cores times slower give or take HT and otherwise ~X times slower given whatever other overhead/contention.
alephnan 5y ago
Feature wise, I’m not sure what’s new here. Popular BDD style JS test runners/frameworks like Jasmine, Jest (IIRC delegates to Jasmine by default, though that’s swappable), Ava (custom built test runner), can all be configured, more or more. Some of the tools require you to bring-your-own-TypeScript, some don’t offer parallelization, but those are just general features that are not coupled to this pitch of being configurable. With relatively straightforward “refactoring” and functional programming patterns, the aforementioned mainstream test runners/frameworks can provide the same ergonomics as Folio. Folio then is just an opinionated interface for specifying parameterized configurations on top of a test runner. Functional programming with higher order functions is powerful and not necessarily require more than a screen’s worth of custom code either. I’m not sure the ergonomic abstractions offered by Folio is worth adopting a new test runner versus defining your own abstraction on top of existing battle hardened test runners ( test frameworks are already the abstraction on top of test runners, with test runners be analogous to React just being a library. Jest sits on top of Jasmine by default, but you can swap that out ). This is like adopting a new Linux distro that decided to roll its own non-Linux kernel. Edit: ah, I see, this is a project out of Microsoft versus some rando JS developer bloating the ecosystem to pad their resumes and spend more time wordsmithing/branding than actually building something novel. From an org perspective, I empathize a bit with the not-invented-here syndrome then, especially since Facebook leads Jest development Edit 2: I think it would be more appealing to market this as a TypeScript-as-a-first-class-citizen test framework that has the backing of a reputable org with resources. TypeScript flavors of Ava or Jest are after-market forks with an order magnitude less support / community / maintenance, meaning you’re on your own when you run into problems, plus having to do the due diligence of auditing the source code for malicious behavior ( I was responsible for defining the JS test setup for a project last year, and defending the trustworthiness of the open source tooling to the rest of the team was the main decision rather than the ergonomics of the test framework )
- eyelidlessness 5y ago
  I’m quite surprised by this response. I find the Folio pitch very compelling, I’m a TypeScript fanatic, and the TS accommodations are pretty low down the list for me in this case. For me the isolation story is by far the most interesting. Of course you can wrap an existing test framework to provide different kinds of isolation, to some extent. There are some things you can’t do with most if not all of the BDD-style JS test frameworks I can recall, for instance any kind of `aroundEach` where a test’s setup, execution and teardown can run in a shared call stack. This is useful (for instance) in situations where you might use CLS to wrap a test in a database transaction. At least walking up to this, that appears to be solved by their interface. Another thing it solves is providing matrix testing primitives where you can branch at the framework level depending on arbitrary environment differences. This is a godsend for library authors, and also for anyone integrating with a variety of external dependencies. Again, of course that could be solved by wrapping some more naive framework. But having the primitives first class and built in means: - You’re not reinventing the wheel. - There’s an idiomatic way to do it. That’s huge! This is stuff I’ve one-offed a zillion times because it never felt quite right. Now it’s a potentially shared problem, and people can coalesce around the solution.

41 points by tilt 5y ago

On a tangent, are there any test frameworks focused on runtime performance? With esbuild speeding up everything else, jest has become the main bottleneck in my dev workflow.

mhagemeister 5y ago
There is uvu https://github.com/lukeed/uvu https://github.com/lukeed/uvu which has runtime performance as the number one goal. Maybe that is something for you.
brundolf 5y ago
Is it the overhead from Jest itself, or your tests? At the end of the day, a JS test runner has to run a bunch of tests written in JS. That can't be optimized away, and I'd assume it's always going to dominate performance.
- eyelidlessness 5y ago
  Yes, Jest is slow. Performance-focused test frameworks like ava and uvu prominently display benchmarks demonstrating this. Jest’s slowness, in my experience, is primarily rooted in three issues: - The transformations it performs to support auto-mocking (AFAIK still its flagship feature) are constant overhead. - Its process model is per-module, rather than per-test, limiting concurrency. A hypothetical test run with one test module with 100 tests is ~100x slower than 100 modules with 1 test each. - Its process model is per-module (yes, I’m repeating this factor because it has another major impact on perf), but Jest destroys its dependency cache between tests, causing memory leaks to accumulate. Memory leaks in Node especially are common and trivial, where long-running libraries (like, say, a logger) set up some config-driven singleton on require/import. I’ve sunk literally weeks if not months into trying to work around issues like these on teams that insisted on keeping either Jest or other tools with aforementioned memory leaks. And while I value performance, I primarily spent that time trying to mitigate problems (false positives/negatives in test runs, intermittent failures) caused by the poor isolation model.
  - eyelidlessness 5y ago
    I can’t edit this but I’ve scrolled through my threads several times and a mistake I made keeps bothering me: > A hypothetical test run with one test module with 100 tests is ~100x slower than 100 modules with 1 test each. This was really bad math. For CPU bound cases, it’s ~N cores times slower give or take HT and otherwise ~X times slower given whatever other overhead/contention.

alephnan 5y ago

Feature wise, I’m not sure what’s new here. Popular BDD style JS test runners/frameworks like Jasmine, Jest (IIRC delegates to Jasmine by default, though that’s swappable), Ava (custom built test runner), can all be configured, more or more. Some of the tools require you to bring-your-own-TypeScript, some don’t offer parallelization, but those are just general features that are not coupled to this pitch of being configurable. With relatively straightforward “refactoring” and functional programming patterns, the aforementioned mainstream test runners/frameworks can provide the same ergonomics as Folio. Folio then is just an opinionated interface for specifying parameterized configurations on top of a test runner. Functional programming with higher order functions is powerful and not necessarily require more than a screen’s worth of custom code either. I’m not sure the ergonomic abstractions offered by Folio is worth adopting a new test runner versus defining your own abstraction on top of existing battle hardened test runners ( test frameworks are already the abstraction on top of test runners, with test runners be analogous to React just being a library. Jest sits on top of Jasmine by default, but you can swap that out ). This is like adopting a new Linux distro that decided to roll its own non-Linux kernel. Edit: ah, I see, this is a project out of Microsoft versus some rando JS developer bloating the ecosystem to pad their resumes and spend more time wordsmithing/branding than actually building something novel. From an org perspective, I empathize a bit with the not-invented-here syndrome then, especially since Facebook leads Jest development Edit 2: I think it would be more appealing to market this as a TypeScript-as-a-first-class-citizen test framework that has the backing of a reputable org with resources. TypeScript flavors of Ava or Jest are after-market forks with an order magnitude less support / community / maintenance, meaning you’re on your own when you run into problems, plus having to do the due diligence of auditing the source code for malicious behavior ( I was responsible for defining the JS test setup for a project last year, and defending the trustworthiness of the open source tooling to the rest of the team was the main decision rather than the ergonomics of the test framework )

eyelidlessness 5y ago
I’m quite surprised by this response. I find the Folio pitch very compelling, I’m a TypeScript fanatic, and the TS accommodations are pretty low down the list for me in this case. For me the isolation story is by far the most interesting. Of course you can wrap an existing test framework to provide different kinds of isolation, to some extent. There are some things you can’t do with most if not all of the BDD-style JS test frameworks I can recall, for instance any kind of `aroundEach` where a test’s setup, execution and teardown can run in a shared call stack. This is useful (for instance) in situations where you might use CLS to wrap a test in a database transaction. At least walking up to this, that appears to be solved by their interface. Another thing it solves is providing matrix testing primitives where you can branch at the framework level depending on arbitrary environment differences. This is a godsend for library authors, and also for anyone integrating with a variety of external dependencies. Again, of course that could be solved by wrapping some more naive framework. But having the primitives first class and built in means: - You’re not reinventing the wheel. - There’s an idiomatic way to do it. That’s huge! This is stuff I’ve one-offed a zillion times because it never felt quite right. Now it’s a potentially shared problem, and people can coalesce around the solution.