4 ms·

I'm curious if anyone knows the test tool they're using here, in particular for lines like: $machine->waitForWindow("^Desktop "); It looks incredibly useful

I'm curious if anyone knows the test tool they're using here, in particular for lines like: $machine->waitForWindow("^Desktop "); It looks incredibly useful. Thanks!

O_H_E 8y ago
Well there seems to be something called nix-shell, but I never heard of it before. It seams pretty useful. The link to plasma tests is provided in the article. https://github.com/NixOS/nixpkgs/blob/49a6964a4250d98644da61f24dcc11ee0b28c4f9/nixos/tests/plasma5.nix https://github.com/NixOS/nixpkgs/blob/49a6964a4250d98644da61...
- chris_st 8y ago
  Thanks!
- ris 8y ago
  This isn't nix-shell. nix-shell is a tool for spawning ephemeral software environments composed of nix packages, something a little like python's virtualenvs. Digging a little bit, nixos' tests appear to be written in perl and are executed in a context that has access to a bunch of useful primitives for asserting these sorts of things. https://nixos.org/nixos/manual/#sec-writing-nixos-tests https://nixos.org/nixos/manual/#sec-writing-nixos-tests
grhmc 8y ago
The tooling used by NixOS is custom to NixOS. You can check out the test directory for many more examples. I'll be writing more soon about how these tests work and how neat they are. They really help NixOS quickly release really high quality updates!
noneucat 8y ago
The tests are written in Perl and are invoked by Nix's build tools. You can skim the API at https://nixos.org/nixos/manual/#sec-nixos-tests https://nixos.org/nixos/manual/#sec-nixos-tests In this case, the test makes use of both xdotool and Nix's own testing library.
aseipp 8y ago
It's our custom tool. NixOS has testing infrastructure built on top of QEMU, and you can use Perl scripts like this in order to automate aspects of testing inside that environment by talking to the QEMU monitor -- for example, you can not only wait for a window, but click a button in that window, then also wait for a service to start afterwords, for example. (If you were testing GUI functionality). Tests are written in a mix of Nix and Perl automation scripts like this; the Perl scripts control QEMU, but the Nix code describes what is put into the disk images. It's fairly bespoke and tuned to our own needs. The source code for the Perl driver is here: https://github.com/NixOS/nixpkgs/blob/f148c5c4a1ffa353836bfd18a94d457acf83bd36/nixos/lib/test-driver/Machine.pm https://github.com/NixOS/nixpkgs/blob/f148c5c4a1ffa353836bfd... It can even do multi-machine test setups as another comment pointed out, and it's pretty fun for writing functional demonstrations; for example I've been meaning to use these tools to write examples of doing PostgreSQL replication and failover, etc.
- chris_st 8y ago
  Thanks very much for this information (you and @grhmc). PLEASE consider releasing some form of this tool, or at least documenting the snot out of it... it looks incredible. We'd really use this at work.
  - grhmc 8y ago
    I think it would be really difficult to get such a good experience with something that isn't NixOS. You might look at things like: - http://open.qa/ http://open.qa/ - TestKitchen But I must recommend, indeed, taking a good hard took at NixOS! :)
  - dezgeg 8y ago
    The internals of the test framework are pretty heavily tied to Nix and NixOS, so a release independent of Nix(OS) isn't probably going to happen. With that said, the test infrastructure itself is exposed as library functions that any Nix user can use. And even though the test VMs itself will run NixOS, the host machine can be any Linux distro with Nix installed.

I'm curious if anyone knows the test tool they're using here, in particular for lines like: $machine->waitForWindow("^Desktop "); It looks incredibly useful. Thanks!

O_H_E 8y ago

Well there seems to be something called nix-shell, but I never heard of it before. It seams pretty useful. The link to plasma tests is provided in the article. https://github.com/NixOS/nixpkgs/blob/49a6964a4250d98644da61f24dcc11ee0b28c4f9/nixos/tests/plasma5.nix https://github.com/NixOS/nixpkgs/blob/49a6964a4250d98644da61...

chris_st 8y ago
Thanks!
ris 8y ago
This isn't nix-shell. nix-shell is a tool for spawning ephemeral software environments composed of nix packages, something a little like python's virtualenvs. Digging a little bit, nixos' tests appear to be written in perl and are executed in a context that has access to a bunch of useful primitives for asserting these sorts of things. https://nixos.org/nixos/manual/#sec-writing-nixos-tests https://nixos.org/nixos/manual/#sec-writing-nixos-tests

grhmc 8y ago

The tooling used by NixOS is custom to NixOS. You can check out the test directory for many more examples. I'll be writing more soon about how these tests work and how neat they are. They really help NixOS quickly release really high quality updates!

noneucat 8y ago

The tests are written in Perl and are invoked by Nix's build tools. You can skim the API at https://nixos.org/nixos/manual/#sec-nixos-tests https://nixos.org/nixos/manual/#sec-nixos-tests In this case, the test makes use of both xdotool and Nix's own testing library.

aseipp 8y ago

It's our custom tool. NixOS has testing infrastructure built on top of QEMU, and you can use Perl scripts like this in order to automate aspects of testing inside that environment by talking to the QEMU monitor -- for example, you can not only wait for a window, but click a button in that window, then also wait for a service to start afterwords, for example. (If you were testing GUI functionality). Tests are written in a mix of Nix and Perl automation scripts like this; the Perl scripts control QEMU, but the Nix code describes what is put into the disk images. It's fairly bespoke and tuned to our own needs. The source code for the Perl driver is here: https://github.com/NixOS/nixpkgs/blob/f148c5c4a1ffa353836bfd18a94d457acf83bd36/nixos/lib/test-driver/Machine.pm https://github.com/NixOS/nixpkgs/blob/f148c5c4a1ffa353836bfd... It can even do multi-machine test setups as another comment pointed out, and it's pretty fun for writing functional demonstrations; for example I've been meaning to use these tools to write examples of doing PostgreSQL replication and failover, etc.

chris_st 8y ago
Thanks very much for this information (you and @grhmc). PLEASE consider releasing some form of this tool, or at least documenting the snot out of it... it looks incredible. We'd really use this at work.
- grhmc 8y ago
  I think it would be really difficult to get such a good experience with something that isn't NixOS. You might look at things like: - http://open.qa/ http://open.qa/ - TestKitchen But I must recommend, indeed, taking a good hard took at NixOS! :)
- dezgeg 8y ago
  The internals of the test framework are pretty heavily tied to Nix and NixOS, so a release independent of Nix(OS) isn't probably going to happen. With that said, the test infrastructure itself is exposed as library functions that any Nix user can use. And even though the test VMs itself will run NixOS, the host machine can be any Linux distro with Nix installed.