4 ms·

Few questions if someone either from ZenPayroll or having similar experience can answer - 1. How you handle maintenance like upgrades in applications hosted on

Few questions if someone either from ZenPayroll or having similar experience can answer - 1. How you handle maintenance like upgrades in applications hosted on multiple machines? Do you have a particular maintenance time period where you bring down the whole application? For some applications, it is not possible to shutdown the website completely. Even if you try to upgrade the application servers one by one there might be active users on that server. How do you handle this situation? 2. The database tier is the hardest to scale. How do you do it? There is an option to go master\slave route but it is hard. For performance purposes we can't always talk with database. How you sync data in case you plan to use cache. Thanks.

volkadav 12y ago
(I'm not a ZP employee, but I have worked in environments of significant scale a fair amount. The below is just my opinion...) #1: The usual approach is something like remove app servers from the load balancers' rotation, upgrade them, run some sort of tests to ensure the upgrade is ok, add them back into the LB rotation. Some environments might use user traffic as the test step (e.g. send 5% of your users at the newly upgraded machines, watch monitoring to see how response times and error counts behave). #2: The answer here really depends on what scale you're operating at. DB replication (e.g. read slaves if your traffic is more read-heavy that write heavy) and sufficiently high-spec machines combined with intelligent use of caching can take you a fairly long way. I hesitate to give hard numbers, but off the cuff probably 99% of sites will never need more than this. There's not really a recipe to follow for that, the data needs and design/ops parameters of an organization tend to be fairly specific to that org; more or less it's good schema design, thinking hard about how fresh data has to be at the edge, caching what you can with careful management of expiry times, and having an ops team comfortable with managing replication. Beyond that scale you start getting into globe-spanning multiple-datacenter voodoo rocket science. :) Ask amazon, google or facebook (some of their work has been mentioned publicly, things like cassandra from fb and various papers published by google/amazon).

volkadav 12y ago

(I'm not a ZP employee, but I have worked in environments of significant scale a fair amount. The below is just my opinion...) #1: The usual approach is something like remove app servers from the load balancers' rotation, upgrade them, run some sort of tests to ensure the upgrade is ok, add them back into the LB rotation. Some environments might use user traffic as the test step (e.g. send 5% of your users at the newly upgraded machines, watch monitoring to see how response times and error counts behave). #2: The answer here really depends on what scale you're operating at. DB replication (e.g. read slaves if your traffic is more read-heavy that write heavy) and sufficiently high-spec machines combined with intelligent use of caching can take you a fairly long way. I hesitate to give hard numbers, but off the cuff probably 99% of sites will never need more than this. There's not really a recipe to follow for that, the data needs and design/ops parameters of an organization tend to be fairly specific to that org; more or less it's good schema design, thinking hard about how fresh data has to be at the edge, caching what you can with careful management of expiry times, and having an ops team comfortable with managing replication. Beyond that scale you start getting into globe-spanning multiple-datacenter voodoo rocket science. :) Ask amazon, google or facebook (some of their work has been mentioned publicly, things like cassandra from fb and various papers published by google/amazon).