5 ms·

Goss – Quick and Easy server validation

31 points by mountainview 8y ago

pageandrew 8y ago
This is cool. I like the idea quite a bit. This definitely solves a problem that I've thought about quite a bit, and have taken a stab at with one-off scripts. What I don't quite like is the fact that as one's expectations become more complex, one has to start encoding that logic into YAML with this custom assertion/pattern matching DSL as shown in the README. I think this might be more easily adopted if it could be integrated into an existing testing framework, allowing to developers to use our familiar patterns of assertions, and use the tools that we're already familiar with.
drinchev 8y ago
I used goss for testing ansible roles. It is simple enough to put in my CI and to configure. Having a .yaml syntax similar to Ansible makes it the appropriate choice for the task.
devn0ll 8y ago
Honest question: Why? Why do things twice? Ansible already promises me to install a specific version of a program, check a certain port (wait_for), make sure a file is owned by someone, etc, etc. If Ansible can't make it happen: it will quit. Why add goss to the equation?
- dkoeji89oe 8y ago
  For the same reason you write unit or integration tests.
  - devn0ll 8y ago
    For software development those things make sense. Not so much for Operations(?) I'm already declaring things with Ansible. Unless I got the syntax wrong ansible will finish a playbook and the server will be as I ordered ansible to make it. So why add another check?
    - nineteen999 8y ago
      Belts and braces approach. I use Ansible but I also use testinfra (which is similar to serverspec and I suppose goss as well) to validate things that are not explicitly covered by Ansible, and even in some cases, some things that are. The tests can be run independently at any later date, helping to ensure that admins haven't messed with important files, upgraded random RPM's or that the servers haven't suffered any other type of configuration drift. But then I build/run emergency services infrastructure with 99.999% availability targets. Once you are writing tests alongside your Ansible playbooks and committing them to source control, it doesn't really take a lot longer and eventually you have a test suite you can run across your entire environment at a moments notice.
      - devn0ll 8y ago
        "The tests can be run independently at any later date, helping to ensure that admins haven't messed with important files" We all run puppet / ansible multiple times a day on our infra right? Checking for config drift. I'm used to a banking sector where we have extremely stringent demands. And still I see no added value for something like infratest/goss. Seriously: running ansible on your infra checking for config drift.. is that not exactly the same as running goss? Plus: Ansible returns changes to what they should be at the same time?
        nineteen999 8y ago
        > We all run puppet / ansible multiple times a day on our infra right? Checking for config drift. No, not necessarily. I think a lot depends on the rate of change in your environment and the size of the team managing it, and the number, complexity and types of devel/test/production environments being managed. In an emergency services network like mine changes are rather the exception than the norm and running ansible/puppet in a loop every 30 mins is a waste of resources. Also, changes that break when pushed to production could result in fire engines or ambulances not being dispatched. Not good. A devops team I worked for ran up a full vagrant VM on every commit to the puppet repository, then ran puppet inside the VM and ran the full gamut of testinfra tests as well. The whole process took 20-30 minutes at times and if the new code you just pushed broke the tests right at the end it would be 20-30 minutes before you found out. Of course you were supposed to mitigate this by running the tests against your new code in a VM before you pushed them. So that is the other extreme. Personally I found that to be overkill, although it didn't stop broken changes being implemented to production sometimes. > Ansible returns changes to what they should be at the same time? Well yes so does puppet, but that assumes that your puppet plays/ansible playbooks etc. are all written to be idempotent. The default modules generally work in that manner, but both allow you to write plays/playbooks that aren't idempotent and that can break things. Infrastructure tests allow you to seperate out (or augment) the validation that running puppet/ansible gives you. Because the tests are generally of a read-only nature there is a less chance of accidentally changing the state of the servers when you run the tests. You could run ansible or puppet in 'dry-run' mode I suppose, and examine the output for errors, but testinfra or serverspec give you a much nicer interface IMHO and are more lightweight and execute much faster than a full ansible/puppet run.
- leg100 8y ago
  The idea is that you don't test the things that ansible has already declared it will do (install pkg, start service), but test the side effects, i.e. is the started service listening on the expected port, is the process of the service daemon running, etc.
  - devn0ll 8y ago
    Do you have more examples? Because: "is the started service listening on the expected port, is the process of the service daemon running" can all be done with straight Ansible as well. I'm just having a hard time understanding the real added value to a goss / ansible combo. Plus: If you do this after installing with ansible. It's basically a check right? Like: Nagios or other monitoring. I'd rather that just use THAT and spend time in creating checks in those systems instead of spending time in goss?
hb3b 8y ago
Why this vs. InSpec?

31 points by mountainview 8y ago

This is cool. I like the idea quite a bit. This definitely solves a problem that I've thought about quite a bit, and have taken a stab at with one-off scripts. What I don't quite like is the fact that as one's expectations become more complex, one has to start encoding that logic into YAML with this custom assertion/pattern matching DSL as shown in the README. I think this might be more easily adopted if it could be integrated into an existing testing framework, allowing to developers to use our familiar patterns of assertions, and use the tools that we're already familiar with.

drinchev 8y ago

I used goss for testing ansible roles. It is simple enough to put in my CI and to configure. Having a .yaml syntax similar to Ansible makes it the appropriate choice for the task.

devn0ll 8y ago

Honest question: Why? Why do things twice? Ansible already promises me to install a specific version of a program, check a certain port (wait_for), make sure a file is owned by someone, etc, etc. If Ansible can't make it happen: it will quit. Why add goss to the equation?

dkoeji89oe 8y ago
For the same reason you write unit or integration tests.
- devn0ll 8y ago
  For software development those things make sense. Not so much for Operations(?) I'm already declaring things with Ansible. Unless I got the syntax wrong ansible will finish a playbook and the server will be as I ordered ansible to make it. So why add another check?
  - nineteen999 8y ago
    Belts and braces approach. I use Ansible but I also use testinfra (which is similar to serverspec and I suppose goss as well) to validate things that are not explicitly covered by Ansible, and even in some cases, some things that are. The tests can be run independently at any later date, helping to ensure that admins haven't messed with important files, upgraded random RPM's or that the servers haven't suffered any other type of configuration drift. But then I build/run emergency services infrastructure with 99.999% availability targets. Once you are writing tests alongside your Ansible playbooks and committing them to source control, it doesn't really take a lot longer and eventually you have a test suite you can run across your entire environment at a moments notice.
    - devn0ll 8y ago
      "The tests can be run independently at any later date, helping to ensure that admins haven't messed with important files" We all run puppet / ansible multiple times a day on our infra right? Checking for config drift. I'm used to a banking sector where we have extremely stringent demands. And still I see no added value for something like infratest/goss. Seriously: running ansible on your infra checking for config drift.. is that not exactly the same as running goss? Plus: Ansible returns changes to what they should be at the same time?
      - nineteen999 8y ago
        > We all run puppet / ansible multiple times a day on our infra right? Checking for config drift. No, not necessarily. I think a lot depends on the rate of change in your environment and the size of the team managing it, and the number, complexity and types of devel/test/production environments being managed. In an emergency services network like mine changes are rather the exception than the norm and running ansible/puppet in a loop every 30 mins is a waste of resources. Also, changes that break when pushed to production could result in fire engines or ambulances not being dispatched. Not good. A devops team I worked for ran up a full vagrant VM on every commit to the puppet repository, then ran puppet inside the VM and ran the full gamut of testinfra tests as well. The whole process took 20-30 minutes at times and if the new code you just pushed broke the tests right at the end it would be 20-30 minutes before you found out. Of course you were supposed to mitigate this by running the tests against your new code in a VM before you pushed them. So that is the other extreme. Personally I found that to be overkill, although it didn't stop broken changes being implemented to production sometimes. > Ansible returns changes to what they should be at the same time? Well yes so does puppet, but that assumes that your puppet plays/ansible playbooks etc. are all written to be idempotent. The default modules generally work in that manner, but both allow you to write plays/playbooks that aren't idempotent and that can break things. Infrastructure tests allow you to seperate out (or augment) the validation that running puppet/ansible gives you. Because the tests are generally of a read-only nature there is a less chance of accidentally changing the state of the servers when you run the tests. You could run ansible or puppet in 'dry-run' mode I suppose, and examine the output for errors, but testinfra or serverspec give you a much nicer interface IMHO and are more lightweight and execute much faster than a full ansible/puppet run.
leg100 8y ago
The idea is that you don't test the things that ansible has already declared it will do (install pkg, start service), but test the side effects, i.e. is the started service listening on the expected port, is the process of the service daemon running, etc.
- devn0ll 8y ago
  Do you have more examples? Because: "is the started service listening on the expected port, is the process of the service daemon running" can all be done with straight Ansible as well. I'm just having a hard time understanding the real added value to a goss / ansible combo. Plus: If you do this after installing with ansible. It's basically a check right? Like: Nagios or other monitoring. I'd rather that just use THAT and spend time in creating checks in those systems instead of spending time in goss?

hb3b 8y ago

Why this vs. InSpec?