4 ms·

Thanks for posting about this! I'm the main author of nextest, and it represents my best foot forward for how Rust testing should be done. Happy to answer quest

Thanks for posting about this! I'm the main author of nextest, and it represents my best foot forward for how Rust testing should be done. Happy to answer questions though I might be a bit intermittent.

mrec 3mo ago
Have there been any discussions about upstreaming this into cargo proper? Are there any significant downsides to nextest compared to its predecessor?
- sunshowers 3mo ago
  The How it works [1] and Why process-per-test? [2] pages should answer your questions. [1] https://nexte.st/docs/design/how-it-works/ https://nexte.st/docs/design/how-it-works/ [2] https://nexte.st/docs/design/why-process-per-test/ https://nexte.st/docs/design/why-process-per-test/
  - mrec 3mo ago
    Ah, I see. You're aiming to become the hashbrown of testing.
    - sunshowers 3mo ago
      Oh gosh, were we to be so lucky :) just aiming to solve problems my coworkers and users see, and doing it with care, is all.
landr0id 3mo ago
Big fan of nextest and this is my first time seeing this site. I'll be real I feel a bit ridiculous commenting this but you might want to consider rephrasing this: >Treat tests as cattle, not pets. Detect and terminate slow tests. Not sure saying, "hey, treat your tests as an animal you can kill at will" paints the right image.
- sunshowers 3mo ago
  That's fair! I'll find a way to rephrase it. edit: Updated to "Detect and handle slow tests". Thanks again!
- trollbridge 3mo ago
  This is from the Kubernetes saying of "treat servers like cattle, not pets". Of course, some people like me keep cattle as pets, but then again I also name my servers, even the virtual or containerised ones.
  - sunshowers 3mo ago
    Yeah that was indeed the inspiration (though I'm pretty sure it predates Kubernetes!) but the juxtaposition with "terminate" is unfortunate.
    - wojciii 3mo ago
      I liked the way it was phrased. You can't make everybody happy. :)
  - seanhunter 3mo ago
    It’s a horrible saying in that context also.
    - embedding-shape 3mo ago
      I mean, I like animals too, but in context it does make sense. The context was to treat them as "obtainable yet ultimately killable entities you keep as a group, not individuals", which cattle pretty much is. Unless you consider keeping cattle as draft animals, but I think that stopped being the main purpose a long time ago. It got the point across, at a time where most people basically acquired servers, kept them until they died, and he was trying to push a development workflow where you constantly close("kill")/bring up new servers.
      - 3mo ago
gdcbe 3mo ago
Thank you very much for developing nextest. It is what allows our projects like rama [1] to run thousands and thousands of tests in a blink of an eye! Keep it up! [1] https://ramaproxy.org https://ramaproxy.org
coding-wizard 3mo ago
Hey, I love nextest. But, perhaps because of the one-process per test approach, endpoint security solutions like CrowdStrike Falcon or Palo Alto Cortex tend to make computers hang whenever tests kick-in. I would love if you were able to introduce a workaround, because none of those companies will fix their stuff. I am guessing a possible mitigation would be to have stagger the first invocation of any large test binary, but I haven't had a chance to dig deep into this issue.
- arw0n 3mo ago
  Wouldn't running the tests in a container solve that issue? Or is that another thing that gets flagged?
- sunshowers 3mo ago
  Nextest does run tests in lexicographic (binary, test name) order by default, so first executions tend to be staggered. Unfortunately this is a known problem that is much larger than nextest, and it is why Windows introduced Dev Drive with a mode to disable AV/EDR on those drives entirely. I wish I had a better answer here — it is frustrating how poorly behaved the EDR stuff tends to be. Maybe I should try and network a bit to get in touch with the people working on this stuff. (Planning to be at DEF CON this year, so come find me if you work on this stuff and will be there!)
cogman10 3mo ago
First, I like how all this works, it's great. Can you run tests serially in the (horrible) case when tests need to build on one another? How are you querying for the tests? Is that just built into rust's test stuff? Would it be possible to fork the test process? It'd be pretty interesting if you could spawn a test process, and then fork it for each test to save both on memory and any static state stored within the test.
- sunshowers 3mo ago
  > Can you run tests serially in the (horrible) case when tests need to build on one another? Yes: https://nexte.st/docs/configuration/test-groups/ https://nexte.st/docs/configuration/test-groups/ (edit: though tests that build on each other is a bit harder — test groups are meant for when tests need access to a shared resource) > How are you querying for the tests? Is that just built into rust's test stuff? Just running --list against test binaries. > Would it be possible to fork the test process? It'd be pretty interesting if you could spawn a test process, and then fork it for each test to save both on memory and any static state stored within the test. This is possible in principle, but nextest doesn't really inject itself into tests like that (injection can cause reliability issues in practice, and a big focus of nextest is reliability). Forking is also not possible in multithreaded programs.
dpc_01234 3mo ago
Thank you! It's great.
thejosh 3mo ago
nextest is one of those things I never have to think about, because it always "just works" and I have no dramas with it. <3

mrec 3mo ago

Have there been any discussions about upstreaming this into cargo proper? Are there any significant downsides to nextest compared to its predecessor?

sunshowers 3mo ago
The How it works [1] and Why process-per-test? [2] pages should answer your questions. [1] https://nexte.st/docs/design/how-it-works/ https://nexte.st/docs/design/how-it-works/ [2] https://nexte.st/docs/design/why-process-per-test/ https://nexte.st/docs/design/why-process-per-test/
- mrec 3mo ago
  Ah, I see. You're aiming to become the hashbrown of testing.
  - sunshowers 3mo ago
    Oh gosh, were we to be so lucky :) just aiming to solve problems my coworkers and users see, and doing it with care, is all.

landr0id 3mo ago

Big fan of nextest and this is my first time seeing this site. I'll be real I feel a bit ridiculous commenting this but you might want to consider rephrasing this: >Treat tests as cattle, not pets. Detect and terminate slow tests. Not sure saying, "hey, treat your tests as an animal you can kill at will" paints the right image.

sunshowers 3mo ago
That's fair! I'll find a way to rephrase it. edit: Updated to "Detect and handle slow tests". Thanks again!
trollbridge 3mo ago
This is from the Kubernetes saying of "treat servers like cattle, not pets". Of course, some people like me keep cattle as pets, but then again I also name my servers, even the virtual or containerised ones.
- sunshowers 3mo ago
  Yeah that was indeed the inspiration (though I'm pretty sure it predates Kubernetes!) but the juxtaposition with "terminate" is unfortunate.
  - wojciii 3mo ago
    I liked the way it was phrased. You can't make everybody happy. :)
- seanhunter 3mo ago
  It’s a horrible saying in that context also.
  - embedding-shape 3mo ago
    I mean, I like animals too, but in context it does make sense. The context was to treat them as "obtainable yet ultimately killable entities you keep as a group, not individuals", which cattle pretty much is. Unless you consider keeping cattle as draft animals, but I think that stopped being the main purpose a long time ago. It got the point across, at a time where most people basically acquired servers, kept them until they died, and he was trying to push a development workflow where you constantly close("kill")/bring up new servers.
    - 3mo ago

gdcbe 3mo ago

Thank you very much for developing nextest. It is what allows our projects like rama [1] to run thousands and thousands of tests in a blink of an eye! Keep it up! [1] https://ramaproxy.org https://ramaproxy.org

coding-wizard 3mo ago

Hey, I love nextest. But, perhaps because of the one-process per test approach, endpoint security solutions like CrowdStrike Falcon or Palo Alto Cortex tend to make computers hang whenever tests kick-in. I would love if you were able to introduce a workaround, because none of those companies will fix their stuff. I am guessing a possible mitigation would be to have stagger the first invocation of any large test binary, but I haven't had a chance to dig deep into this issue.

arw0n 3mo ago
Wouldn't running the tests in a container solve that issue? Or is that another thing that gets flagged?
sunshowers 3mo ago
Nextest does run tests in lexicographic (binary, test name) order by default, so first executions tend to be staggered. Unfortunately this is a known problem that is much larger than nextest, and it is why Windows introduced Dev Drive with a mode to disable AV/EDR on those drives entirely. I wish I had a better answer here — it is frustrating how poorly behaved the EDR stuff tends to be. Maybe I should try and network a bit to get in touch with the people working on this stuff. (Planning to be at DEF CON this year, so come find me if you work on this stuff and will be there!)

cogman10 3mo ago

First, I like how all this works, it's great. Can you run tests serially in the (horrible) case when tests need to build on one another? How are you querying for the tests? Is that just built into rust's test stuff? Would it be possible to fork the test process? It'd be pretty interesting if you could spawn a test process, and then fork it for each test to save both on memory and any static state stored within the test.

sunshowers 3mo ago
> Can you run tests serially in the (horrible) case when tests need to build on one another? Yes: https://nexte.st/docs/configuration/test-groups/ https://nexte.st/docs/configuration/test-groups/ (edit: though tests that build on each other is a bit harder — test groups are meant for when tests need access to a shared resource) > How are you querying for the tests? Is that just built into rust's test stuff? Just running --list against test binaries. > Would it be possible to fork the test process? It'd be pretty interesting if you could spawn a test process, and then fork it for each test to save both on memory and any static state stored within the test. This is possible in principle, but nextest doesn't really inject itself into tests like that (injection can cause reliability issues in practice, and a big focus of nextest is reliability). Forking is also not possible in multithreaded programs.

dpc_01234 3mo ago

Thank you! It's great.

thejosh 3mo ago

nextest is one of those things I never have to think about, because it always "just works" and I have no dramas with it. <3