4 ms·

> I don't care if they've passed before and the files haven't changed. This is a common interjection from some people... which means you think that your tests

by brabel 4y ago

> I don't care if they've passed before and the files haven't changed. This is a common interjection from some people... which means you think that your tests are not deterministic, otherwise it would be completely pointless to run them again without inputs changing. I actually admit that from end-to-end tests, this is usually true despite our best efforts to the contrary... but for unit tests, I really think tests should be 100% deterministic and only ever run when something they rely on changes. The Unison programming language goes even further[1] and it NEVER executes a test again once the code under test has been "committed" into its image. [1] https://www.unison-lang.org/ https://www.unison-lang.org/

bin_bash 4y ago
> completely pointless to run them again without inputs changing What if I installed some dependency outside of my project? What if I'm trying to test performance with `time make test`? What if it failed due to a transitory error—or I'm simply trying to discover if it is transitory? What if I made a change to the way the tests are run in the Makefile or a different file not in ./src? Like running `npm install`? What if I want to set an environment variable or pass an argument to the test runner?
- tirpen 4y ago
  Just run make -B test to force a rerun?
  - bin_bash 4y ago
    First, the engineer needs to understand how the makefile is written and that it doesn't rerun without changes. Is it not rerunning because of npm? because of jest? If the engineer isn't the one that wrote the makefile: they won't. Next, you're assuming a JS engineer would know what the flag is (or even that such a flag exists) to force reruns. I've literally never used this flag in my (JS) career so I wouldn't expect anyone to know this. Don't send engineers down some debugging rabbit hole just to save a few seconds when no changes happen. If you want this functionality, just use `jest --onlyChanged` in your own workflow and don't screw with everyone else's.
    - gpderetta 4y ago
      On non-toy projects it will save minutes if not much more. It also encourage good test discipline l. The engineer will know because it is their job to know. (Yes I'm a huge fan of deterministic tests and running only what's necessary).
- chriswarbo 4y ago
  > What if I installed some dependency outside of my project? > What if I made a change to the way the tests are run in the Makefile or a different file not in ./src? Like running `npm install`? > What if I want to set an environment variable or pass an argument to the test runner? Those are all inputs. Unfortunately, Make isn't very good at tracking inputs (or outputs, for that matter: e.g. if we change an input back to some previous value, make will still rebuild rather than using the old result). These days I find Nix to be a much more sane alternative to Make.
bravetraveler 4y ago
Keep in mind I'm about as ignorant of Make as one can be -- I use it to build things, but I've never put things into it I have a feeling meeting this may scope creep the thing into being more environment aware. Interpretation of the things it's sourcing [and their paths], and less-than-obvious things like environment variables This just feels like one of those things trying to be too helpful that inevitably gets in my way
marcosdumay 4y ago
> which means you think that your tests are not deterministic This would be relevant if the command was saving some test report somewhere. But then the target would just depend on the report, and there would be no need to add guard files. Looks like that `make test` just does the normal thing that is run the tests and print the results to the screen. If so, people would want to repeat it any number of times.
fn-mote 4y ago
> [...] which means you think that your tests are not deterministic, otherwise it would be completely pointless to run them again without inputs changing. Re-running seems reasonable to me. It's make, not Bazel. In the article I'm looking at targets with a "find" shell command on the right hand side. I don't think the dependencies of those files are carefully mapped somewhere else in the makefile.

by brabel 4y ago

bin_bash 4y ago

> completely pointless to run them again without inputs changing What if I installed some dependency outside of my project? What if I'm trying to test performance with `time make test`? What if it failed due to a transitory error—or I'm simply trying to discover if it is transitory? What if I made a change to the way the tests are run in the Makefile or a different file not in ./src? Like running `npm install`? What if I want to set an environment variable or pass an argument to the test runner?

tirpen 4y ago
Just run make -B test to force a rerun?
- bin_bash 4y ago
  First, the engineer needs to understand how the makefile is written and that it doesn't rerun without changes. Is it not rerunning because of npm? because of jest? If the engineer isn't the one that wrote the makefile: they won't. Next, you're assuming a JS engineer would know what the flag is (or even that such a flag exists) to force reruns. I've literally never used this flag in my (JS) career so I wouldn't expect anyone to know this. Don't send engineers down some debugging rabbit hole just to save a few seconds when no changes happen. If you want this functionality, just use `jest --onlyChanged` in your own workflow and don't screw with everyone else's.
  - gpderetta 4y ago
    On non-toy projects it will save minutes if not much more. It also encourage good test discipline l. The engineer will know because it is their job to know. (Yes I'm a huge fan of deterministic tests and running only what's necessary).
chriswarbo 4y ago
> What if I installed some dependency outside of my project? > What if I made a change to the way the tests are run in the Makefile or a different file not in ./src? Like running `npm install`? > What if I want to set an environment variable or pass an argument to the test runner? Those are all inputs. Unfortunately, Make isn't very good at tracking inputs (or outputs, for that matter: e.g. if we change an input back to some previous value, make will still rebuild rather than using the old result). These days I find Nix to be a much more sane alternative to Make.

bravetraveler 4y ago

Keep in mind I'm about as ignorant of Make as one can be -- I use it to build things, but I've never put things into it I have a feeling meeting this may scope creep the thing into being more environment aware. Interpretation of the things it's sourcing [and their paths], and less-than-obvious things like environment variables This just feels like one of those things trying to be too helpful that inevitably gets in my way

marcosdumay 4y ago

> which means you think that your tests are not deterministic This would be relevant if the command was saving some test report somewhere. But then the target would just depend on the report, and there would be no need to add guard files. Looks like that `make test` just does the normal thing that is run the tests and print the results to the screen. If so, people would want to repeat it any number of times.

fn-mote 4y ago

> [...] which means you think that your tests are not deterministic, otherwise it would be completely pointless to run them again without inputs changing. Re-running seems reasonable to me. It's make, not Bazel. In the article I'm looking at targets with a "find" shell command on the right hand side. I don't think the dependencies of those files are carefully mapped somewhere else in the makefile.