2 ms·

With proper tracking it is not difficult to see different versions of packages installed. Checking run books/OPs logs would also lead to a quick understanding t

With proper tracking it is not difficult to see different versions of packages installed. Checking run books/OPs logs would also lead to a quick understanding that Staging was updated and Production wasn't. Either of those two things would make it relatively easy to find the discrepancy in versions. Then, as they noted, a review of changelogs would have led to the patch they found. If you don't have processes and documentation to quickly point out those kinds of differences between environments, you're doing something wrong.

dap 11y ago
It's not that you can't find _some_ differences that way. As others have you pointed out, there should be little difference between the software deployed in staging and production. But there are can be differences in performance and hardware configuration (unless you can afford to mirror your production deployment exactly). Most unavoidably, there are problems you only hit after cumulative amounts of uptime or load[1], which is nearly impossible to simulate in a preproduction environment. [1] For an example, see our recent outage related having seen 200M PostgreSQL transaction: https://www.joyent.com/blog/manta-postmortem-7-27-2015 https://www.joyent.com/blog/manta-postmortem-7-27-2015
- edwhitesell 11y ago
  Fair enough. I guess my point should have been: With proper tracking of changes and work done between the environments, finding differing versions could be the lowest hanging fruit. That was my response to your original question of how they found got to haproxy as the root cause so quickly. Sure, mirroring hardware and performance can help, if you can afford it (which is rare). However, you'd also probably agree it's pretty rare to find an issue caused cumulative amounts of uptime or load. Mis-matched software versions or configurations are a far more likely culprit in my experience. Horses not zebras...

dap 11y ago

It's not that you can't find _some_ differences that way. As others have you pointed out, there should be little difference between the software deployed in staging and production. But there are can be differences in performance and hardware configuration (unless you can afford to mirror your production deployment exactly). Most unavoidably, there are problems you only hit after cumulative amounts of uptime or load[1], which is nearly impossible to simulate in a preproduction environment. [1] For an example, see our recent outage related having seen 200M PostgreSQL transaction: https://www.joyent.com/blog/manta-postmortem-7-27-2015 https://www.joyent.com/blog/manta-postmortem-7-27-2015

edwhitesell 11y ago
Fair enough. I guess my point should have been: With proper tracking of changes and work done between the environments, finding differing versions could be the lowest hanging fruit. That was my response to your original question of how they found got to haproxy as the root cause so quickly. Sure, mirroring hardware and performance can help, if you can afford it (which is rare). However, you'd also probably agree it's pretty rare to find an issue caused cumulative amounts of uptime or load. Mis-matched software versions or configurations are a far more likely culprit in my experience. Horses not zebras...