5 ms·

> Stupidest thing I've ever had to do. I don't really understand the "ranty" tone. The client had very specific requirements and the author came up with an eff

> Stupidest thing I've ever had to do. I don't really understand the "ranty" tone. The client had very specific requirements and the author came up with an effective solution and was fully paid to deliver it. Sounds like a win for everyone.

ianhawes 6y ago
I believe the expectations of having 0 downtime was not expressed until the day of the transfer.
- dna_polymerase 6y ago
  In addition to that the customer runs a single server but expects the guys to maintain a property not even feasible at Google scale: Zero downtime. Overall the whole thing was just ridiculous, but luckily the customer got a nice bill in the end.
  - pc86 6y ago
    To be fair, Google maintains zero downtime for small time scales like this a lot. Most of the time, actually.
    - sokoloff 6y ago
      I’m having a hard time following what “zero downtime most of the time, actually” really means. https://m.youtube.com/watch?v=IKiSPUc2Jck&t=81s https://m.youtube.com/watch?v=IKiSPUc2Jck&t=81s
      - im3w1l 6y ago
        It means that downtime is chunky.
      - buran77 6y ago
        > zero downtime for small time scales... Most of the time I read it as "if you take small enough discrete time intervals they won't overlap with any downtime". Or in other words "no downtime between downtimes". Yes, it's very in line with your video.
    - kevincox 6y ago
      Are you talking about Google Compute Engine? In that case yes, because by default VMs are live migrated between physical hosts. This can be done for schedule maintenance or upon signs that the machine is likely to fail. Furthermore there are no physical disks for a GCE VM which is one of the more common failure points. The result of this is that GCE VMs often survive for months or years without downtime. Note that the SLA allows more than 3 hours of downtime per month. https://cloud.google.com/compute/sla https://cloud.google.com/compute/sla For physical servers the uptime is typically quite small. Of course Google isn't optimizing for server uptime so it isn't fair to say "well even Google can't do it".
intpx 6y ago
probably because proper architecture (clustering, HA etc) and planning would have never made this an issue. This is still an extremely risky operation, hot swapping power and switching interfaces on the fly all while sitting on a cart in a corridor. In any disruptive work there is never a guarantee of no downtime for affected assets. I know the OP came in as a consultant, but If I was the MSP tech, I would have demanded a paper trail a mile long to cover my ass if this went sideways and If i was the account manager for the client, I would have refused the work. Its not good business to agree to do work where you know there is a better than good chance there will be an outage and your client is saying they wont pay if there is an outage. Even agreeing to it puts you in a bad spot for future work. I guess as an outside consultant, bewilderment is a better reaction than ranting, but this is the kind of shit that drives ops folks crazy
- addHocker 6y ago
  The whole setup is risky. And a customer who demands a 0 downtime while driving price down on the setup, sounds stupid from the start.
- dspillett 6y ago
  > planning would have never made this an issue Hard work is wonderful stuff. Days and weeks of it can save you whole hours of planning.
- jabroni_salad 6y ago
  My SOWs leave zero room for 'and it will go flawlessly or you wont get paid at all'. If you are occupying my time in a way that makes me unable to serve other clients you will definitely pay for it.
jedimastert 6y ago
> Me: You didn't notify them of scheduled maintenance like we discussed on Friday? It appears that the client didn't have the specific requirements on initial consult.
throwaway0a5e 6y ago
Reddit (for reasons related to user demographics and feedback loops) rewards certain types of writing and implied viewpoints. Following best practices and rules is one of those things. This server migration clearly runs counter to established wisdom so OP using a writing style of "look how terrible and asinine this was" will be rewarded and gain traction much more than a "look how interesting this was" writing style.
- user5994461 6y ago
  It's reddit /sysadmin, the channel is dedicated to rants and horrible experiences from sysadmin and helpdesk folks. It's quite sad IMO, don't recommend to go there unless you want to have a bad day reading about the most horrific work environments and bad practices in the world.
  - DoreenMichele 6y ago
    Perhaps somewhat similarly, r/TalesFromRetail is devoted to kvetching about your job in the retail sector, but it's really not a depressing place. There are a lot of rules and expectations about how you tell your story. You aren't supposed to outright dox anyone or veer into genuine trash talk. It's not supposed to be negative per se. It's supposed to be entertaining. It's an art form. It's not everyone's cup of tea, just like horror isn't everyone's cup of tea. But people often watch horror movies for catharsis, not because they want to be depressed and wallowing in self pity. Storytelling is often about educating people about things you can't speak about more directly. It's often a way of sharing wisdom in an inoffensive manner and one that will stick because people will actually pay attention, unlike when you are giving them some dry lecture about some problem they haven't yet had and don't yet care about. But if you entertain them, they will read it anyway and that story may stick with them. And then six months or a year later when they have the same problem, they will actually remember how someone else handled the same issue and it will turn a potentially nightmarish scenario into "Meh, I just did the same thing that guy on Reddit did to his shitty boss/customer/coworker. Worked like a charm. Moving on."
- me_me_me 6y ago
  <any group of 2 or more people> (for reasons related to user demographics and feedback loops) rewards certain types of writing and implied viewpoints. This is literally the basis of human interactions, thats how we humans work at every scale to form friendships/families/societies/nations.
swarnie_ 6y ago
I see reddit so i assume this is the sysadmin subreddit? They're famous for not being a cheery bunch. Because reddit's demographic does swing younger the sub used to be filled with endless posts about being socially incompetent or possessing 0 business craft. Does anyone know if it improved?
some_random 6y ago
The client was an asshole who demanded 100% uptime and stated that they wouldn't pay if there was any downtime at all. The rant is entirely justified.

ianhawes 6y ago

I believe the expectations of having 0 downtime was not expressed until the day of the transfer.

dna_polymerase 6y ago
In addition to that the customer runs a single server but expects the guys to maintain a property not even feasible at Google scale: Zero downtime. Overall the whole thing was just ridiculous, but luckily the customer got a nice bill in the end.
- pc86 6y ago
  To be fair, Google maintains zero downtime for small time scales like this a lot. Most of the time, actually.
  - sokoloff 6y ago
    I’m having a hard time following what “zero downtime most of the time, actually” really means. https://m.youtube.com/watch?v=IKiSPUc2Jck&t=81s https://m.youtube.com/watch?v=IKiSPUc2Jck&t=81s
    - im3w1l 6y ago
      It means that downtime is chunky.
    - buran77 6y ago
      > zero downtime for small time scales... Most of the time I read it as "if you take small enough discrete time intervals they won't overlap with any downtime". Or in other words "no downtime between downtimes". Yes, it's very in line with your video.
  - kevincox 6y ago
    Are you talking about Google Compute Engine? In that case yes, because by default VMs are live migrated between physical hosts. This can be done for schedule maintenance or upon signs that the machine is likely to fail. Furthermore there are no physical disks for a GCE VM which is one of the more common failure points. The result of this is that GCE VMs often survive for months or years without downtime. Note that the SLA allows more than 3 hours of downtime per month. https://cloud.google.com/compute/sla https://cloud.google.com/compute/sla For physical servers the uptime is typically quite small. Of course Google isn't optimizing for server uptime so it isn't fair to say "well even Google can't do it".

intpx 6y ago

probably because proper architecture (clustering, HA etc) and planning would have never made this an issue. This is still an extremely risky operation, hot swapping power and switching interfaces on the fly all while sitting on a cart in a corridor. In any disruptive work there is never a guarantee of no downtime for affected assets. I know the OP came in as a consultant, but If I was the MSP tech, I would have demanded a paper trail a mile long to cover my ass if this went sideways and If i was the account manager for the client, I would have refused the work. Its not good business to agree to do work where you know there is a better than good chance there will be an outage and your client is saying they wont pay if there is an outage. Even agreeing to it puts you in a bad spot for future work. I guess as an outside consultant, bewilderment is a better reaction than ranting, but this is the kind of shit that drives ops folks crazy

addHocker 6y ago
The whole setup is risky. And a customer who demands a 0 downtime while driving price down on the setup, sounds stupid from the start.
dspillett 6y ago
> planning would have never made this an issue Hard work is wonderful stuff. Days and weeks of it can save you whole hours of planning.
jabroni_salad 6y ago
My SOWs leave zero room for 'and it will go flawlessly or you wont get paid at all'. If you are occupying my time in a way that makes me unable to serve other clients you will definitely pay for it.

jedimastert 6y ago

> Me: You didn't notify them of scheduled maintenance like we discussed on Friday? It appears that the client didn't have the specific requirements on initial consult.

throwaway0a5e 6y ago

Reddit (for reasons related to user demographics and feedback loops) rewards certain types of writing and implied viewpoints. Following best practices and rules is one of those things. This server migration clearly runs counter to established wisdom so OP using a writing style of "look how terrible and asinine this was" will be rewarded and gain traction much more than a "look how interesting this was" writing style.

user5994461 6y ago
It's reddit /sysadmin, the channel is dedicated to rants and horrible experiences from sysadmin and helpdesk folks. It's quite sad IMO, don't recommend to go there unless you want to have a bad day reading about the most horrific work environments and bad practices in the world.
- DoreenMichele 6y ago
  Perhaps somewhat similarly, r/TalesFromRetail is devoted to kvetching about your job in the retail sector, but it's really not a depressing place. There are a lot of rules and expectations about how you tell your story. You aren't supposed to outright dox anyone or veer into genuine trash talk. It's not supposed to be negative per se. It's supposed to be entertaining. It's an art form. It's not everyone's cup of tea, just like horror isn't everyone's cup of tea. But people often watch horror movies for catharsis, not because they want to be depressed and wallowing in self pity. Storytelling is often about educating people about things you can't speak about more directly. It's often a way of sharing wisdom in an inoffensive manner and one that will stick because people will actually pay attention, unlike when you are giving them some dry lecture about some problem they haven't yet had and don't yet care about. But if you entertain them, they will read it anyway and that story may stick with them. And then six months or a year later when they have the same problem, they will actually remember how someone else handled the same issue and it will turn a potentially nightmarish scenario into "Meh, I just did the same thing that guy on Reddit did to his shitty boss/customer/coworker. Worked like a charm. Moving on."
me_me_me 6y ago
<any group of 2 or more people> (for reasons related to user demographics and feedback loops) rewards certain types of writing and implied viewpoints. This is literally the basis of human interactions, thats how we humans work at every scale to form friendships/families/societies/nations.

swarnie_ 6y ago

I see reddit so i assume this is the sysadmin subreddit? They're famous for not being a cheery bunch. Because reddit's demographic does swing younger the sub used to be filled with endless posts about being socially incompetent or possessing 0 business craft. Does anyone know if it improved?

some_random 6y ago

The client was an asshole who demanded 100% uptime and stated that they wouldn't pay if there was any downtime at all. The rant is entirely justified.