4 ms·

Where do people recommend logging infrastructure changes (e.g. database changes, server configuration) for all things potentially not in code?

ab_io 8y ago
Choose a documentation source-of-truth (Confluence, etc) and keep a changelog[1]. I've also seen a dedicated email group (i.e. change-mngt@company.com) used for tracking non-code changes. [1]: https://keepachangelog.com/en/1.0.0/ https://keepachangelog.com/en/1.0.0/
bmurphy1976 8y ago
We need more context. Server configuration should definitely be in code/source control these days. Database changes, well, what kind of changes are you talking about?
- ryeguy_24 8y ago
  These days, as we have multiple departments and services, it’s hard to track down any issues we identify without knowing what kind of changes have happened to potentially cause it. As we don’t have a single development platform (multiple acquisitionss) this is hard for us at least until we have unified systems (will take time). I’m referring to any infrastructure or operational change to the stack. For example, deployments/roll outs, scaling servers up or down, port changes, updates to software on key servers, changes to database table structure, manual changes to database data. Any changes at all to the production stack.
mattbillenstein 8y ago
It's all in code now... At least how I do it.
sandstrom 8y ago
Terraform checked into version control is pretty common
sciurus 8y ago
Whether something is "in code" or not is orthogonal to the problem of tracking infrastructure changes. E.G. You can commit a change to Terraform, but that doesn't tell someone when "terraform apply" occurred. What you should use depends on what tools you have available. We currently use Datadog, so we use their API for publishing events. We can then search for them in the event stream or overlay them on dashboards. You could look at the datastores Grafana supports for annotations and choose whichever of them you're comfortable operating. http://docs.grafana.org/reference/annotations/ http://docs.grafana.org/reference/annotations/
hunter_n 8y ago
You might want to take a look at Architectural Decision Records in addition to Change logs. A lightweight format for tracking changes to infra that can be checked into source control. One example - https://github.com/joelparkerhenderson/architecture_decision_record https://github.com/joelparkerhenderson/architecture_decision...

ab_io 8y ago

Choose a documentation source-of-truth (Confluence, etc) and keep a changelog[1]. I've also seen a dedicated email group (i.e. change-mngt@company.com) used for tracking non-code changes. [1]: https://keepachangelog.com/en/1.0.0/ https://keepachangelog.com/en/1.0.0/

bmurphy1976 8y ago

We need more context. Server configuration should definitely be in code/source control these days. Database changes, well, what kind of changes are you talking about?

ryeguy_24 8y ago
These days, as we have multiple departments and services, it’s hard to track down any issues we identify without knowing what kind of changes have happened to potentially cause it. As we don’t have a single development platform (multiple acquisitionss) this is hard for us at least until we have unified systems (will take time). I’m referring to any infrastructure or operational change to the stack. For example, deployments/roll outs, scaling servers up or down, port changes, updates to software on key servers, changes to database table structure, manual changes to database data. Any changes at all to the production stack.

mattbillenstein 8y ago

It's all in code now... At least how I do it.

sandstrom 8y ago

Terraform checked into version control is pretty common

sciurus 8y ago

Whether something is "in code" or not is orthogonal to the problem of tracking infrastructure changes. E.G. You can commit a change to Terraform, but that doesn't tell someone when "terraform apply" occurred. What you should use depends on what tools you have available. We currently use Datadog, so we use their API for publishing events. We can then search for them in the event stream or overlay them on dashboards. You could look at the datastores Grafana supports for annotations and choose whichever of them you're comfortable operating. http://docs.grafana.org/reference/annotations/ http://docs.grafana.org/reference/annotations/

hunter_n 8y ago

You might want to take a look at Architectural Decision Records in addition to Change logs. A lightweight format for tracking changes to infra that can be checked into source control. One example - https://github.com/joelparkerhenderson/architecture_decision_record https://github.com/joelparkerhenderson/architecture_decision...