4 ms·

This is a great write-up of a tough technical problem, and I definitely agree with their conclusion that "it's worth taking the time to understand your stack."

by dap 11y ago

This is a great write-up of a tough technical problem, and I definitely agree with their conclusion that "it's worth taking the time to understand your stack." I especially liked that it wasn't preachy. It just explained their problem and how they solved it. (It seems a lot of write-ups end up offering very specific technical advice that grossly overgeneralizes from their experience.) My only question is how they got to haproxy as the root cause so quickly. In my experience, comparing what's different between production and staging is a long shot because there's so much that's different. Obviously workload can matter a lot, and so can uptime and time since upgrade. So I'm curious if haproxy was the first thing they saw that was different or if they just didn't write about the dead ends.

rocky1138 11y ago
It might not be all that different if they're using Docker.
edwhitesell 11y ago
With proper tracking it is not difficult to see different versions of packages installed. Checking run books/OPs logs would also lead to a quick understanding that Staging was updated and Production wasn't. Either of those two things would make it relatively easy to find the discrepancy in versions. Then, as they noted, a review of changelogs would have led to the patch they found. If you don't have processes and documentation to quickly point out those kinds of differences between environments, you're doing something wrong.
- dap 11y ago
  It's not that you can't find _some_ differences that way. As others have you pointed out, there should be little difference between the software deployed in staging and production. But there are can be differences in performance and hardware configuration (unless you can afford to mirror your production deployment exactly). Most unavoidably, there are problems you only hit after cumulative amounts of uptime or load[1], which is nearly impossible to simulate in a preproduction environment. [1] For an example, see our recent outage related having seen 200M PostgreSQL transaction: https://www.joyent.com/blog/manta-postmortem-7-27-2015 https://www.joyent.com/blog/manta-postmortem-7-27-2015
  - edwhitesell 11y ago
    Fair enough. I guess my point should have been: With proper tracking of changes and work done between the environments, finding differing versions could be the lowest hanging fruit. That was my response to your original question of how they found got to haproxy as the root cause so quickly. Sure, mirroring hardware and performance can help, if you can afford it (which is rare). However, you'd also probably agree it's pretty rare to find an issue caused cumulative amounts of uptime or load. Mis-matched software versions or configurations are a far more likely culprit in my experience. Horses not zebras...
Sinjo 11y ago
Thanks! Glad you enjoyed it. The main thing that narrowed it down was thinking about the request path from the app server to Elasticsearch. That pointed us at the load balancers (we tried a request directly from app -> Elasticsearch in production to verify they were the problem). You're right that HAProxy wasn't the first place we looked once we were there, though. If I remember right, we started by diffing `sysctl` output to see if Ubuntu had tweaked something between versions.
developer1 11y ago
To pitch in some preachy advice: with some time and effort, staging and production environments should be essentially identical. Sure, production is generally going to have more servers handling each service (more web nodes, more db/storage nodes, multiple load balancers, etc.), but the setup and deployments should not be any different. All too often, we think of staging as nothing more than a clone of the development environment on a remote server where QA can get at it. If you run your load balancer (haproxy), web server (apache, nginx), database (mysql, postgres, mondo, elasticsearch), caching (memcached, redis) all on a single server for staging, you're eventually going to stumble on this kind of hard-to-diagnose problem. One of the main disservices you are doing to yourself with a single-server staging is that all of your traffic is going to be travelling over localhost or, worse, over unix sockets. You're not even testing basic network latency or performance. Staging should really be considered a first-level production. Its configuration and maintenance should be handled with the same attention to detail as production.

by dap 11y ago

rocky1138 11y ago

It might not be all that different if they're using Docker.

edwhitesell 11y ago

With proper tracking it is not difficult to see different versions of packages installed. Checking run books/OPs logs would also lead to a quick understanding that Staging was updated and Production wasn't. Either of those two things would make it relatively easy to find the discrepancy in versions. Then, as they noted, a review of changelogs would have led to the patch they found. If you don't have processes and documentation to quickly point out those kinds of differences between environments, you're doing something wrong.

dap 11y ago
It's not that you can't find _some_ differences that way. As others have you pointed out, there should be little difference between the software deployed in staging and production. But there are can be differences in performance and hardware configuration (unless you can afford to mirror your production deployment exactly). Most unavoidably, there are problems you only hit after cumulative amounts of uptime or load[1], which is nearly impossible to simulate in a preproduction environment. [1] For an example, see our recent outage related having seen 200M PostgreSQL transaction: https://www.joyent.com/blog/manta-postmortem-7-27-2015 https://www.joyent.com/blog/manta-postmortem-7-27-2015
- edwhitesell 11y ago
  Fair enough. I guess my point should have been: With proper tracking of changes and work done between the environments, finding differing versions could be the lowest hanging fruit. That was my response to your original question of how they found got to haproxy as the root cause so quickly. Sure, mirroring hardware and performance can help, if you can afford it (which is rare). However, you'd also probably agree it's pretty rare to find an issue caused cumulative amounts of uptime or load. Mis-matched software versions or configurations are a far more likely culprit in my experience. Horses not zebras...

Sinjo 11y ago

Thanks! Glad you enjoyed it. The main thing that narrowed it down was thinking about the request path from the app server to Elasticsearch. That pointed us at the load balancers (we tried a request directly from app -> Elasticsearch in production to verify they were the problem). You're right that HAProxy wasn't the first place we looked once we were there, though. If I remember right, we started by diffing `sysctl` output to see if Ubuntu had tweaked something between versions.

developer1 11y ago

To pitch in some preachy advice: with some time and effort, staging and production environments should be essentially identical. Sure, production is generally going to have more servers handling each service (more web nodes, more db/storage nodes, multiple load balancers, etc.), but the setup and deployments should not be any different. All too often, we think of staging as nothing more than a clone of the development environment on a remote server where QA can get at it. If you run your load balancer (haproxy), web server (apache, nginx), database (mysql, postgres, mondo, elasticsearch), caching (memcached, redis) all on a single server for staging, you're eventually going to stumble on this kind of hard-to-diagnose problem. One of the main disservices you are doing to yourself with a single-server staging is that all of your traffic is going to be travelling over localhost or, worse, over unix sockets. You're not even testing basic network latency or performance. Staging should really be considered a first-level production. Its configuration and maintenance should be handled with the same attention to detail as production.