22 ms·

Moved a server from one building to another with zero downtime

1043 points by huhtenberg 6y ago

PinguTS 6y ago
It's been done 7 years ago even using public transport. https://www.reddit.com/r/uptimeporn/comments/1kf26r/moving_a_server_without_turning_it_off_to/ https://www.reddit.com/r/uptimeporn/comments/1kf26r/moving_a...
- jagermo 6y ago
  The most dangerous part is them expecting the 3G to be available during the subway ride.
  - deftnerd 6y ago
    I'm surprised they weren't stopped by police to investigate a very suspicious heavily loaded cart on the Subway. It easily could have been 300lbs of explosives on that cart.
  - mercora 6y ago
    in Germany mobile networks work just fine in the subway as ISPs have deployed hardware there. I actually have more issues with the network when using classical railroad transport...
- csours 6y ago
  I really thought this post on HN was going to be that story. Thanks for digging it up.
panpanna 6y ago
Really disappointed they didn't use a wireless network of some kind.
- lordnacho 6y ago
  My first thought as well. Set up WiFi along the path, basically turn the machine into a laptop. But I think there might be a disconnect when you change base stations? At least when I move my laptop between rooms in my house there's often a momentary problem while on video call. The other way I'd do it is more similar to described. Create redundant network paths to the server, then cut one.
- dbalatero 6y ago
  I wouldn't, the risk of disconnects is high.
  - panpanna 6y ago
    But the risk of sometime tripping over your looong cat6 and breaking the network is not negligible either...
D895n9o33436N42 6y ago
This reminds me of a famously obtuse and obdurate boss who asked for things that were utterly impossible. He had delusions of grandeur which left him convinced that he and only he was qualified to challenge the “cheap, fast, good - pick any two” triangle. Naturally, I did my best to explain the laws of physics to him, but he wouldn’t hear it. In a spectacular display of Stockholm syndrome I did my best to appease him for four years, but, as many of you can surely predict by this point in the story, I failed in every possible way and eventually gave up. Just wish I could have my four years back. I was glad to read that OP at least got paid well for his efforts.
- Tade0 6y ago
  I applaud you for being able to stand four years of this. I usually get fired from such positions in less than two.
  - sleepybrett 6y ago
    I usually walk out about six months in if not sooner. Maybe it's just because I spent so much time freelancing that I had enough experience to recognize a no-win situation.
jtbayly 6y ago
Seems very risky. Not something I’d want to do if minimum downtime was the goal. One wrong piece of gravel ends up with catastrophic failure instead of 5 minutes of downtime.
- dmurray 6y ago
  But the goal was zero downtime, not minimum downtime. The client made it clear that 5 minutes of downtime was equivalent to catastrophic failure. So they correctly found a solution that reduced the chance of "5 minutes of downtime", at the expense of an increased risk of catastrophic failure.
  - jtbayly 6y ago
    I understand that. I just doubt that the risk was worth it, if downtime is such a big deal.
fooblat 6y ago
> Stupidest thing I've ever had to do. I don't really understand the "ranty" tone. The client had very specific requirements and the author came up with an effective solution and was fully paid to deliver it. Sounds like a win for everyone.
- ianhawes 6y ago
  I believe the expectations of having 0 downtime was not expressed until the day of the transfer.
  - dna_polymerase 6y ago
    In addition to that the customer runs a single server but expects the guys to maintain a property not even feasible at Google scale: Zero downtime. Overall the whole thing was just ridiculous, but luckily the customer got a nice bill in the end.
    - pc86 6y ago
      To be fair, Google maintains zero downtime for small time scales like this a lot. Most of the time, actually.
      - sokoloff 6y ago
        I’m having a hard time following what “zero downtime most of the time, actually” really means. https://m.youtube.com/watch?v=IKiSPUc2Jck&t=81s https://m.youtube.com/watch?v=IKiSPUc2Jck&t=81s
        im3w1l 6y ago
        It means that downtime is chunky.
        buran77 6y ago
        > zero downtime for small time scales... Most of the time I read it as "if you take small enough discrete time intervals they won't overlap with any downtime". Or in other words "no downtime between downtimes". Yes, it's very in line with your video.
macintux 6y ago
At my first job we were starting up the company and didn’t really know what we were doing; one early server was sitting on a folding table and its power cord was wrapped around a leg, so just to replace the table with something more robust involved downtime.
- elliotpage 6y ago
  I've been there, solidarity for cheap furniture based maintainence windows.
- em-bee 6y ago
  the careful application of a saw or an angle grinder would have made it possible to remove the folding table without unplugging the power cord. :-)
- crumpled 6y ago
  I heard an anecdote about a company splicing some fiber cable in the middle of a utility van and having to cut the van apart at the end.
Humphrey 6y ago
I haven't read the article, but I'm reminded of that episode of Seinfeld and the frogger arcade game
- dfsegoat 6y ago
  I have pondered this exact scenario (server move w/0 downtime) - because of watching that episode - wouldn't have thought about it otherwise. ..It's interesting how pop-culture and your chosen profession intersect, at times.
g051051 6y ago
I guess servers have gotten a lot more robust in the last decade...there's no way any server I ever managed would survive something like that.
- maeln 6y ago
  A lot of server are SSD-only these days which make them less fragile. Still, I really wouldn't see myself pushing a running server in a cart.
  - tyingq 6y ago
    Yeah, there's certainly still things like riser cards and connectors that could come unseated due to vibration.
    - marcosdumay 6y ago
      That's probably a problem for the next guy that takes an ops job there. Loose pieces often don't disconnect right at the same instant, and even when they do, memory caches usually postpone the failures.
  - throwaway744678 6y ago
    On a parking lot, no less! Let's hope it will not rain on the way!
    - em-bee 6y ago
      umbrellas over the switches and the cart... (means extra billable hours for the extra manhours needed to hold the umbrellas)
anfractuosity 6y ago
Reminds me of this - https://www.youtube.com/watch?v=vQ5MA685ApE https://www.youtube.com/watch?v=vQ5MA685ApE 'Moving online webserver using public transport'
- tyingq 6y ago
  The Indiana Bell building move is pretty impressive. http://www.paul-f.com/ibmove.html http://www.paul-f.com/ibmove.html
  - sschueller 6y ago
    https://www.youtube.com/watch?v=CNqul9TfJwI https://www.youtube.com/watch?v=CNqul9TfJwI
  - joncrane 6y ago
    Wow this is literally the exact same thing as the OP but for an entire building. Insane.
- zomglings 6y ago
  In the rain!
- salzig 6y ago
  had the same thought :)
umarniz 6y ago
Interesting read, makes me wonder as a thought experiment if it counts as downtime if the latency of commands on the machines rises to 5 minutes? You could clone the VM to another instance and record commands going to VM1 and replay them to VM2 after 5 minutes. This whole brain fart of mine doesn't make much sense but if you play along with it, does it still count as a downtime or just very high latency?
- tyingq 6y ago
  That works where you have control over all of the timeouts and failure detection at every level and layer. TCP keepalives, for example, could thwart you. Or client side timeouts, or firewall connection state tables, etc. 5 minutes of unplanned downtime in a pub/sub setup could easily go unnoticed, since that setup is typically tuned for long timeouts and/or repeated retries.
- pc86 6y ago
  Wouldn't requests time out on the client side long before five minutes?
  - heavenlyblue 6y ago
    I don’t know whether it’s the software in general, but ever since I’ve started using Three 4G broadband in the UK; all of the software started behaving really weirdly (lots of lockups, hangs, etc). Apps often need to be restarted. If you do a ping during “bad weather”, you can see that they buffer up to 5 minutes of packets (i.e. there will be no communication for some time, then you’ll receive a bunch of them with a huge latency with sequences intact). So I would assume a lot of software could even work that way. I think a lot of software don’t set any (TCP) timeouts at all.
- NateEag 6y ago
  It depends on how downtime is defined in the contract. That sounds like I'm being snarky but I mean it - whether an actual legal contract or just the documentation given to users, any system where downtime matters should have some discussion of what impacts downtime can have and how it's measured and managed. That documentation is what defines "downtime". I'll add that what you've described is a sort of low-fi manual version of DB replication (https://en.m.wikipedia.org/wiki/Replication_(computing) https://en.m.wikipedia.org/wiki/Replication_(computing)).
user5994461 6y ago
I am so scared to imagine what would happen if there was any issue during the move (very likely when dragging live cables and powers over hundreds of meters). The client would immediately refuse to pay anything because he was very clear he wouldn't pay a thing if there is downtime. Then, the next contractor would be super quick to judge you and the situation, reinforcing that you were an incompetent idiot and the client was right to kick you away on the spot and not pay a dime. Glad it went well in the end. There is so much to lose for the person trying to help.
- willcipriano 6y ago
  This is a junior sysadmin I suspect. With a bit more experience you'd learn to say something along the lines of "no downtime, sure, that will be 30 grand" and the ability for downtime will suddenly materialize. Him and his friend did this big song and dance, took a huge risk and only got paid for ten hours worth of work in the end.
  - zrail 6y ago
    $30k, 80% up front, strict liability waiver that says I’m not responsible for loss of business or anything else if there is downtime.
    - pedrocr 6y ago
      Now you're over-charging massively. If you have no liability and are guaranteed pay, charging for just double hourly rate is more than enough as a "stupid and non-standard requirements" kind of thing.
    - imtringued 6y ago
      You can't get paid upfront and at the same time get a liability waiver. For a 100% guarantee with full liability $30k doesn't actually sound ridiculous because it would require obtaining 100% identical hardware and doing at least one test run on that hardware before actually doing it on the production hardware. What the contractor did is basically "wing it", explain a way to get zero downtime to the client and then not actually offer a guarantee by doing the operation straight on the production hardware. Really this was more about convincing (ie bullshitting your way through) the client to let you do the work than actually doing it properly and for a huge sum of money. It wouldn't surprise me if there was actual downtime for a few seconds and the client simply didn't notice it.
neilv 6y ago
Good thing the server had two power supplies. There was a YouTube video (which I can't immediately find) of people moving a server across town, on the train, without powering it off, and, IIRC, they had to splice the UPS into the power cable. When it's done for pay rather than for fun, and payment is conditioned on zero downtime, I hope they charged a premium to make up for the risk of no pay. Offhand, I don't know what's a good way to do that -- I've never had a consulting client demand terms like that for billed-by-the-hour work.
- kijin 6y ago
  Effective hourly rate = base hourly rate * risk. Risk = client risk * task risk. Client risk is based on your past experience with the same client. If they're prone to demand last-minute changes or stupid stuff, they get charged a higher rate on every project afterward. Jacking up the client risk factor is also a nice way to fire a client you don't want.
- discordance 6y ago
  https://www.youtube.com/watch?v=vQ5MA685ApE https://www.youtube.com/watch?v=vQ5MA685ApE
Milank 6y ago
No downtime is acceptable, but they have only one server? What if a technical failure happen? What if there's a fire in the server room? What if there is an earthquake and the building collapses? What if... many things can happen that can result in a long, long downtime with this tactics. If uptime is so crucial, the system should be setup in such way that moving one server should be a peace of cake, not a spec-ops mission.
- redwood 6y ago
  Remind me of how IBM positions mainframes: they are so highly available that you simply never let them shut down.
  - lasereyes136 6y ago
    IBM Mainframes are designed to be serviced while running so if you have multiple CPUs you can offline one at a time for upgrade it without the whole mainframe going down. Big Sun Solaris boxes where built like at as well. If your mainframe had only one CPU, you did have to turn it off in order to service it. But you could upgrade the OS without turning it off. While they aren't cool tech now, mainframes are a marvel of hardware engineering.
  - chasd00 6y ago
    plus, i would imagine turning them on and bringing them online isn't just a press of a button.
    - MrMorden 6y ago
      It's not. https://web.archive.org/web/20190324191654/https://www.ibm.com/support/knowledgecenter/zosbasics/com.ibm.zos.zsysprog/zsysprogc_initialization.htm https://web.archive.org/web/20190324191654/https://www.ibm.c... (archive.org link because ibm.com apparently isn't hosted on a mainframe.)
- walrus01 6y ago
  From an ISP perspective this seems like the sort of company that orders one $250 a month business DIA circuit (at a price point where there is no ISP ROI for building a true ring topology to feed a stub customer) and has no backup circuit. Then the inevitable happens like a dump truck 2km away with a raised dump driving through aerial fiber and causing an 18 hour outage. Some circuits might average 5 to 7 nines of uptime over a year, but the next year is dump truck time... You can never truly be certain.
walrus01 6y ago
Sort of on the subject, i've seen a brochure for a specialty product marketed to law enforcement. It's meant for use with the seizure of live, powered on desktop PCs and similar that have a high likelihood of full disk encryption. Essentially it's a medium sized double conversion ups, with a really high quality sine wave inverter, and some electronics that can match phase with a live 120vac 60Hz circuit. And a tool kit which consists of the insulated electrical hand tools needed to do a midspan removal of the cable jacket and splice into the wires in an ordinary PC power cable. The person using it is of course supposed to be trained in advance, and competent at the process of attaching the UPS to the live circuit.
- throwaway744678 6y ago
  Wouldn't it be safer to open the case and connect some kind of battery + adapter after the power supply?
  - walrus01 6y ago
    Splicing into the many wires that is an atx+12v power connector, between the output of the power supply and the motherboard is way more fiddly than just dealing with the hot and neutral on an ordinary $5 PC power cord. You could also never be certain what weird ziptie and cable management system (or lack thereof) might exist in a home built x86 PC case, or if there's any room for hands to work at all... I think the thing I saw is also meant to deal equally well with a commodity x86 PC built from parts, or an Intel NUC size thing, or a corporate desktop machine with proprietary internal wiring like a slimline Dell, Lenovo, HP, etc.
  - ajross 6y ago
    Safer for the operator? Sure. But certainly not for the device, if you're trying to keep it operating. An ATX power supply has 24 pins at 5 different voltage levels (plus any auxilliary power connectors for the GPU and drives, etc...), and motherboards are a lot less tolerant of spikes and transients than the PS on the other side. Dealing with AC power isn't really that dangerous if you're careful.
nobrains 6y ago
I don't know. If the "boss" was charged "4.5 hours of work, 2 hours of consultancy, and 4.5 hours of consultant", and assuming he would have been charged half of that with downtime, maybe the boss did get a good deal. We don't know the cost of downtime for him. I mean if he had access to technical resources who were willing and capable to do this for him, he chose to do it.
- notwhereyouare 6y ago
  I personally find it hard to believe that a rough estimate of $450 for the job (spitballing $45/hr for 10 hours) is less than 5 minutes of downtime and they only have 1 server. Then again, could easily be wrong
  - nobrains 6y ago
    You cannot compare it to zero. You have to compare it to the cost of doing it with the downtime. There would be cost to that as well. It will not be free.
- moduspol 6y ago
  It's also possible that "downtime" has different meanings to different people. The client may be seeing "downtime" as the net result of what happened the last few times the server was "down," which could have been for any number of reasons (potentially even unrelated to the server itself). When you get clients describing things like this, it's possible they've been promised things about this server before by other consultants that didn't pan out. They don't want to give you the full details because then you'll recommend a different route that they don't want to take (justifiably or not). It's easier for them to frame the problem to a consultant in a way that allows for only one potential solution, even if perhaps better ones exist, because the guy in charge of making the decision isn't technically skilled enough to assess whether others proposed by consultants are as viable. And, of course, one might read a little into why there exists a "boss" with such a highly-critical IT need that is hiring a consultant to do work like this, and thinks that threatening to not pay at all if there is any downtime is the best way to do it. I mean, what if they opened the door to this closet and it grazed a power cable on the floor and the machine just shut off? Why even bother staying around to bring things back up? It wasn't your fault and there's already downtime: you're not getting paid.
growt 6y ago
Setting up a new server at the new location and moving the VMs one by one to the new server as they become idle should be possible without downtime. But maybe there were other requirements (like no new/additional hardware) that weren't mentioned in the article.
xyst 6y ago
Why wouldn’t cloning the VMs to a second server, then split the traffic between the primary and secondary server work? Once traffic to the second server is confirmed, you could shut off the second server and haul it off to the new location. I would probably still charge a much higher rate since the owner was an arse, but at least you would get back your 7-8 hours.
- blcArmadillo 6y ago
  It sounds like there was no second server.
- viraptor 6y ago
  You're making assumptions about what's running on the servers. Let's say it's a VoIP conference server with a shared dedicated room - effectively you have an ongoing session shared between multiple connection and you cannot stop it. Or you have stateful local processing so you can't "split the traffic". Or a number of other limitations...
- AnIdiotOnTheNet 6y ago
  Not all services can be load balanced in this way Live migration of VMs would have been a better option, which was brought up in the reddit comments and dismissed because HyperV live migration is spotty. While I'd have to agree with that assessment, it isn't so spotty that what they actually did was less risky.
- morphogenesis 6y ago
  Database inconsistency for one thing. This works for frontend web services but how do you reconcile the writes between the two servers?
exabrial 6y ago
Reminds me of the "hot slide" technique used for old telephone switches
- ansible 6y ago
  They did some crazy stuff in the old days. Like when they moved a telephone exchange live... the whole building.
social_quotient 6y ago
Slight topic drift - Any thoughts on how the pandemic might materially change assumptions about an onsite/onprem being better than cloud or manage data center when the code people are now actually remote to the “Local” infrastructure. Something specific to the reality of the pandemic strikes me as something that would make the die hard local only folks have to start rethinking the position. (Not to suggest it’s bad, just different now that a primary assumption about people work in the office is less true)
- ocdtrekkie 6y ago
  I feel a lot safer knowing I'm controlling all the variables during a global crisis, actually. This article provides an example of how when you operate on prem, literally any crazy option remains on the table for you. If you asked your cloud provider to do this, it'd be a no.
- AnIdiotOnTheNet 6y ago
  As someone who works in a very anti-cloud company culture (which I happen to agree with), this incident has had no effect whatsoever on that mindset. We don't dislike cloud because it is accessed remotely, we dislike cloud because of the lack of control we have over everything running there. If something happens and our local systems have a problem, there are people here, like myself, who's highest priority will be fixing it and second highest priority will be communicating the status of that. Your problems are never a priority to a cloud vendor and communicating with you is even less of a priority. That's before we even get into the absurd expenses and reliance on big fat pipes.
chisleu 6y ago
I didn't want to lose my many months of uptime for a lan party back in 1999/2000 and we used the UPS to migrate my linux box across town for some Quake 3 Arena action. Things were so much simpler back then.
mercora 6y ago
i wonder if its really possible to do the initial setup of the ethernet failover without interruption. i have never done this, but i would expect the interfaces themselves will become unavailable for direct use and you get a completely fresh virtual ethernet interface which represents whatever physical interface is currently active... at least this is what happens when you add an ethernet interface to a bridge in linux...
webscalist 6y ago
Could've been cheaper to buy/rent another server, put it on the new location, set up redundancy/replication, power off the old server, move it to the new location, return the new server. Or just keep it for sanity.
imglorp 6y ago
The moving server on cart part made me nervous. If there was any rotating rust in there, bouncing across the parking lot would make things difficult for flying heads. I'd have hand carried it from stage to stage, setting it on a padded cart each stop, treating it like sweating TNT.
rafaelturk 6y ago
Fun reading. But my advice is never accept a job like this. This could easily become 2 weeks down time
codingdave 6y ago
I'm surprised that part of the story wasn't to drill down into the requirements. No downtime ever? Not even at 3 AM on a Saturday? I've found that when people are being unreasonable it is because they haven't split out their true needs from their first idea of how to meet those needs. In this case the true need is zero impact to users. The owner translated that to "zero downtime", and then didn't accept alternative solutions that still would have met his true business need.
robin_reala 6y ago
I always remember this post by the Amsterdam Police who managed to maintain their uptime on a VMS cluster despite moving data centres in the middle: http://web.archive.org/web/20120229042903/http://www.openvms.org/stories.php?story=03/11/28/7758863 http://web.archive.org/web/20120229042903/http://www.openvms...
digitalsushi 6y ago
I had to search the reddit commits for 'vmotion'. They have it covered. This anecdote is an amazingly good story for telling at the pub over a few beers. It's a terrible story for a strategy. If this is a mountain, my molehill is that one night in the late 90s, I got paged cause the SMTP outbound server was overheating. At midnight I drive across sleepy NH backroads, and stopped at a Wendys to get a chicken sandwich and iced tea, for the caffeine. When I got to the server room, I pulled the 2U Dell server out of the rack and discovered the CPU cooling fan had seized up. Mind you, this is a New Hampshire data center in 1999, and it has a filing cabinet with manilla folders, and carpeted floors. This thing was never prepared for any disasters. A half hour later, the SMTP server was up and running cool again. I greased the fan with the mayonnaise from my sandwich.
- dsr_ 6y ago
  The (probably soybean) oil is a fine lubricant, but the constant motion should cause the egg proteins to coagulate. How long did it operate before you replaced the fan properly?
  - emeraldd 6y ago
    I can't help smiling at this analysis. It feels like something you'd hear from a sci-fi story engineer working on an rundown ship that just keeps going no matter what ...
    - staticvoidmaine 6y ago
      That book is Expeditionary Force
      - novaleaf 6y ago
        books 1 to 3 are awesome. after that the author forgets how to advance the plot while still churning out a new book more than once/year. I gave up on book 7, so yeah, I tried sticking with it.
geocrasher 6y ago
I once shut down a PC, moved it to another desk, and it wouldn't power back on. Another time I moved a server to another rack. It had 2 years uptime. Had to power it down, and it wouldn't power back on. Both required PSU replacements. Had I moved them while powered on I can only imagine the fun times. Perhaps they should have just told the customer they couldn't find it: https://www.theregister.com/2001/04/12/missing_novell_server_discovered_after/ https://www.theregister.com/2001/04/12/missing_novell_server...
ericyan 6y ago
The consultants really should told the client that if all you have is a single server then there is no such thing as "zero downtime".
neya 6y ago
Dude has ONE server and talks about having 0 downtime for his clients? What the hell?! In a way, this is Darwinism for the IT industry and I'm happy the people involved got paid well. Due probably paid as much as it woulda costed him a new server. I bet he'll never forget this lesson.
Zenst 6y ago
Interesting story and one that has played out a few times, I'm aware of a couple verbatim to that. Another - used power extension leads to cover power. Key being systems with dual power units (most servers do) and networking so you can switch from one run to another. But have known some large companies who have in their history, done things like this and other creative solutions to impossible problems.
cpuguy83 6y ago
I'm picturing that Seinfeld episode where George tries to move the Frogger arcade from a restaurant that is shutting down but doesn't want to lose his high score.
- will_pseudonym 6y ago
  HOLES! I need HOLES! :)
kuon 6y ago
When I was younger (read 20 years ago), I did crazy things like that, not over that long distance, but moving live servers in different racks. Now that I am older, I don't think I would do it anymore, too much stress for a small reward. Also today, most of the time, I am able to "talk out" customers of crazy requirements, while I would just have said "OK let's do it" in my younger years.
devchix 6y ago
I recall sometime in the mid 2000s there was a fever for achieving five-9s (99.999% uptime, I think -- it became fodder for a few episodes of Mr. Robot). Not that the metric ever went away, but back then a lot of BigIron(TM) vendors advertised achieving five-9s by replacing hardware while the OS remained running and continuing service. Sun 15K and 25K series (Gilfoyle had a used one in the garage running his network) were behemoths whose mem/cpu boards you could swap out wholesale while the entire frame and backplane was powered on, and while the OS the board came out of remains functioning. There were many caveats around the procedure but it worked. Execs and sales guys loved those demos. These monsters were expensive and banks and energy conglomerates were buying them by the dozens. There was also a big todo about hot swappable drives. The idea that you could be doing hardware maintenance while the machine was still running was a novelty, something like brain surgery while the patient was not only awake, but awake and eating, driving his car, talking on the phone, etc. A decade later I look back with deep surprise that we didn't think to abstract out the service instead of the hardware. I don't know how many of those behemoths are still being bought, now I work almost exclusively with small server instances that can come and go on the fly. Micro services and AWS have taken five-9s in a different direction. I frequently think of Sun as a failed Hephaestus, in a Christopher Nolan film he would be brilliant but could only turn out clumsy tools because of his deformity, he hates the things he makes so he throws them away before completion. Men find these cast-offs and temper and refine them.
- chasd00 6y ago
  i remember those. A friend of mine was a network engineer at a local datacenter ( UUNET then MCI pre-scandal ) and said companies were buying Suns for everything no matter how trivial. He worked a night shift and i use to go hang out with him in the noc and download movies (residential bandwidth was not what it is today). Odd he nor i ever get in any trouble for that heh.
- tlack 6y ago
  Well as I recall there were a few reasons that people focused on reliability in hardware in the late 90s: 1. Shared state storage systems that supported replication were rare (I think Oracle and Informix maybe?) 2. Virtualization software was in its infancy (did SunOS have something before Solaris?) 3. RAM and hardware were waaaaay more expensive, meaning you often had to buy more pure metal just to answer questions fast enough At least that's my take on it based on my dim faded memories
tobyhinloopen 6y ago
10 hours investment for no downtime seems like a good deal for the owner
- topkai22 6y ago
  Depends on if he really has customers accessing the system "all the time." Besides, as pretty much everyone has noted, running a zero-downtime system on a single physical machine in what sounds like is just a normal cable room is kind of nuts. Those 10h would have been much better spent to move that puppy to someone else's data center and get some redundancy. Although reading between the lines, maybe the lease was up and they were waiting to the last minute to move it.
linsomniac 6y ago
Lower stakes, but ~15 years ago a friend had a Linux box in the corner that had huge uptime. I want to say the uptime started shortly after the kernel patch that fixed the 400-ish day overflow of the uptime counter. He moved to a new home and very carefully moved the running server using it's UPS. He didn't have to worry about keeping networking up though. I used to be all about long uptimes. I eventually started seeing long uptimes as a negative though. A long uptime probably means patches have not been applied.
- jccooper 6y ago
  I also did that once, about the same timeframe, specifically to preserve an uptime. I think the cult of runtime came about simply because it was impressive that a personal computer could stay running for more than a few days when most of the world ran Win95. And because development cycles were longer and there weren't a lot of network threats.
JeroenKnoops1 6y ago
Reminds me of the OpenVMS clusters.. Police in Amsterdam celebrated in 2007 an uptime of 10 years of their cluster. In this period, all hardware was replace, and half of it was moved to another location 7 km away. All data moved from DAS disks to SAN without one application needed to be stopped. Also VMS was upgraded from 6.2 to 7.3-2. The VMS cluster did not go down during all of these changes. I <3 OpenVMS
- JeroenKnoops1 6y ago
  During Y2K I've also had to shutdown various OpenVMS servers with uptime over 10 years... Only because of company policies, not because OpenVMS required the reboot.
- tandr 6y ago
  Would be interesting to know if it is still up and running?
tzs 6y ago
I needed to restart a server where I worked. My boss was complaining about the revenue loss during the down time. I knew the revenue loss (if there even was any, as opposed to a couple of minutes of revenue simply shifting to a few minutes later...) would be well under a dollar. So I listened to him whine for a couple minutes, then tossed a dollar on his desk, told him that would cover it so he could shut up now, and rebooted the server. Warning: you should probably only try this if you are good friends with your boss. That boss had been my best friend for years before I came to work for his company.
jfcorbett 6y ago
Reminds me of the time where IT at a previous employer told us that due to a "new IT strategy", our production cluster that had been sitting comfortably in the basement for years had to be moved to an "approved IT hub facility"... in another office 500 km away and across the North Sea. There was downtime. Promptly after our cluster settled into this wonderful new facility, a cooling pipe in the ceiling leaked on it, frying 1/3 of our nodes.
- yjftsjthsd-h 6y ago
  On a personal selfish level I was quite happy to see our workloads moving to datacenters that we couldn't (reasonably) physically access, because it replaced "can you go drive to the DC and replace a failing disk" with "we put in the request for smart hands to replace the failing disk". Of course, there's some notable tradeoffs, but it makes me feel better when the business decides to do such things...
arethuza 6y ago
This reminds me of a small company I joined many years ago that did deployments by RAID - find a working server (possibly at a customer site) swap in a blank HD, wait for it to rebuild then take it and put in a new server and repeat the process. Like finding people who argue against revision control systems, it's really quite a challenge convincing people why things like this are a bad idea - after all "it works!".
- yjftsjthsd-h 6y ago
  That's... actually fascinating, if in a slightly insane way. There's pets, there's cattle... and apparently there's a herd of cloned pets, which I'd somehow never considered before:)
rafaelturk 6y ago
Pictures please!
mercora 6y ago
when i was younger i was super proud that i could replace my disk while i kept working on the device. i would put the new disk into my LVM volume group moved all extends to the new disk and dropped the old disk out of the VG afterwards, when done i could just unplug it and be done without halting work except for kicking off the process.
Johnny555 6y ago
Decades ago an ISP I was colocated at did the same thing. I don't remember the exact details, but it was a DNS server and they either couldn't log in or were relying on the zone files cached in memory or something but for some reason they couldn't power it off. It was already plugged into a UPS, but they had to cut one of the posts off the rack to get the server out without unplugging it, then they plugged that UPS into a bigger UPS on a cart and wheeled it to the new data center they built out in the building next door. The world was much different at the time -- this coloc provider had a good reputation, yet.... they had a keg of beer in the corner of the server room and a stack of adult magazines in the men's room.
Reedx 6y ago
That reminds me of the Pixar incident where Toy Story 2 was accidentally deleted while in production and had no working backups. Luckily one employee was working from home (rare at the time!) and had a copy of the entire movie on her desktop computer. Which they very carefully moved back to the office and were able to restore from that. https://www.youtube.com/watch?v=7MAedEXri7c https://www.youtube.com/watch?v=7MAedEXri7c
tzury 6y ago
I once was called in to export data from a DOS program that had no export option. Single Author died of heart issues and the company needed the data for the migration. After several attempts to understand the binary format I gave up and ended up printing tabular reports to LPT1 which I connected my laptop to, extracting it and rebuilding CSV files. Lucky enough, printing those days were the most important feature of a business app.
fredley 6y ago
This is the kind of content I've only ever seen previously in TDWTF (which is entirely this sort of content...) https://thedailywtf.com/ https://thedailywtf.com/
merb 6y ago
meanwhile in germany, german telekom have their connect ip lines (leased lines..., company internet..) shutdown since tuesday morning. so a downtime of over 48 hours, besides a sla that no downtime will be longer than 8 hours and a availability of 99,9%. what a crazy world.
kyuudou 6y ago
It's called vMotion
akssri 6y ago
Was it George Costanza ? https://m.youtube.com/watch?v=a-FbktgqCqY https://m.youtube.com/watch?v=a-FbktgqCqY
johnklos 6y ago
I've done something like this - server running off of UPS moved from one building in Manhattan to another about 1/4 mile away, in snow... Not for someone with weak arms.
BrianB 6y ago
It's been done. https://i.cdn.turner.com/v5cache/TBS/Images/Dynamic/i439/seinfeld-s9e18-1600x900-800x450_071620141129.jpg https://i.cdn.turner.com/v5cache/TBS/Images/Dynamic/i439/sei...
neycoda 6y ago
I wonder if there was any legit reason to require no downtime. Otherwise the owner doesn't understand what downtime means for his business.
jcrawfordor 6y ago
I once basically spent a summer doing this, not over a parking lot but to consolidate the remaining equipment in a large number of racks into a few new ones - this was a former sales office of a megacorporation that had been built to have its 1970s-era computer room proudly displayed through windows into the main conference room, a very weird setup without the context that in said '70s that conference room was used to pitch prospective customers on business automation. Anyway, by the time I was there it was still a '70s-vintage large computer room but now massively overprovisioned on space, cooling, etc, particularly with most IT functions having moved to corporate. A decision was made to repurpose part of it as a test lab and move all the actual remaining equipment to three racks in the corner. I'd do about two servers a day in between other things, taking advantage of redundant power supplies to transfer the PSUs one at a time to extension cords, swap to a long network cable fast enough that TCP sessions probably didn't time out, and then unrack onto a hydraulic lift card and do the same procedure the other way. I presented this at the start as far from a guaranteed strategy - that it would minimize downtime but there would inevitably be some due to mistakes. None of this was really that critical. There were a few devices that were pretty old and poorly maintained, we agreed up front that if these lost power for some reason and then failed to boot, we would just say they'd lived long lives and purchase replacements. I guess the point is that this whole situation was kind of unusual and I would generally not recommend doing this, we were lucky that all the equipment left had stakeholders that acknowledged it was legacy stuff and they could tolerate losing it. The irony is, of course, that it went perfectly. So far as I know there was not a single problem experienced through the whole thing. I even managed to swap the phone lines to the (surprisingly busy!) legacy fax server when each was out of use.
- groby_b 6y ago
  > There were a few devices that were pretty old and poorly maintained, > if these lost power for some reason and then failed[... we'd] purchase replacements. And the sysadmin let that opportunity pass??
  - jcrawfordor 6y ago
    Heh, I know of a couple of people who would have been far happier if the fax server specifically was somehow accidentally tipped down a stairwell... by this time there were enough dirt cheap "cloud" fax services that it really didn't make sense to keep the on-prem system, particularly since the office had been almost entirely migrated to VoIP except for a few oddball devices like that fax setup. But that whole thing just becomes a story of the internal woes of that particular megacorporation, from the computer room to the front office there was a whole lot of stuff just being kept over from the '90s.
noobermin 6y ago
Sorry but this is ridiculous. It's a great story of a feat of sysadminery, but the client should have just accepted some down time, even a few hours. The level of entitlement from some clients people get is just infuriating. Even down to calling him back for not agreeing to help, what an infuriating person. That was my main take away from this. Endeavor to be the sort of person who can refuse clients, the entire idea that "the customer is always right" enables so much ridiculous behavior.
deleted 6y ago
[deleted]
pengaru 6y ago
Decades ago working in a sysadmin role at a hosting company I had a similar situation. The solution I came up with was to fashion a custom male<->male power cord, like a gender changer, from some broken ATX PSU scraps we had laying around. By rearranging the power sockets from multiple donors, two male power cords could be connected on a single enclosure. Internally the sockets were simply bridged, otherwise the PSU was basically gutted. With this goofy metal box having two male power cords dangling from it in hand, I just used a very long extension cord plugged into an outlet on the same AC phase as the existing server's power source. The extension cord powered one of the bridge cords. The other bridge cord plugged into the server's existing - and hot - power strip, forming a redundant power source. Now the power strip could be unplugged from the primary power source without losing power, and we just moved the server to the new location with the bridge box and power strip in tow. If memory serves the only tricky part was determining which outlet at the new home was on a compatible circuit. We didn't have much in the way of electronics tools, no oscilloscopes or anything. Even the soldering involved to make the bridge box was done using my personal soldering iron, which just happened to be in the office because some of us raced RC cars there after hours. I think I just used an incandescent desk lamp to verify a normal brightness on the bridged circuit before proceeding with the server, but it's been a while. I wonder how many people have fashioned AC power cord gender changers throughout history... :)
shireboy 6y ago
You realize the client is condescendingly mocking the guy for saying it can’t be done now, and will expect this next time they run updates on the server, which is to say never
partiallypro 6y ago
This reminds me of the Seinfeld episode with Frogger
aleem 6y ago
Sometimes it's better to seek forgiveness than to seek permission. Saturday 3AM shift with a 5 minute downtime would work just as wwll. Unless this server has had historically 100% uptime this would go unnoticed.
aeno 6y ago
A real classic comes to mind: https://www.youtube.com/watch?v=vQ5MA685ApE https://www.youtube.com/watch?v=vQ5MA685ApE Moving a running server about 7km through public transport without downtime.
snvzz 6y ago
Incredible. Five minutes of VM downtime were not acceptable. And yet, they had a single VM host. Should it catch fire (hardware FAILS!!!), what then?
lightlyused 6y ago
I seem to remember a similar story from another site. I'm thinking it was thedailywtf.com, but I can't find it.

1043 points by huhtenberg 6y ago

It's been done 7 years ago even using public transport. https://www.reddit.com/r/uptimeporn/comments/1kf26r/moving_a_server_without_turning_it_off_to/ https://www.reddit.com/r/uptimeporn/comments/1kf26r/moving_a...

jagermo 6y ago
The most dangerous part is them expecting the 3G to be available during the subway ride.
- deftnerd 6y ago
  I'm surprised they weren't stopped by police to investigate a very suspicious heavily loaded cart on the Subway. It easily could have been 300lbs of explosives on that cart.
- mercora 6y ago
  in Germany mobile networks work just fine in the subway as ISPs have deployed hardware there. I actually have more issues with the network when using classical railroad transport...
csours 6y ago
I really thought this post on HN was going to be that story. Thanks for digging it up.

panpanna 6y ago

Really disappointed they didn't use a wireless network of some kind.

lordnacho 6y ago
My first thought as well. Set up WiFi along the path, basically turn the machine into a laptop. But I think there might be a disconnect when you change base stations? At least when I move my laptop between rooms in my house there's often a momentary problem while on video call. The other way I'd do it is more similar to described. Create redundant network paths to the server, then cut one.
dbalatero 6y ago
I wouldn't, the risk of disconnects is high.
- panpanna 6y ago
  But the risk of sometime tripping over your looong cat6 and breaking the network is not negligible either...

D895n9o33436N42 6y ago

This reminds me of a famously obtuse and obdurate boss who asked for things that were utterly impossible. He had delusions of grandeur which left him convinced that he and only he was qualified to challenge the “cheap, fast, good - pick any two” triangle. Naturally, I did my best to explain the laws of physics to him, but he wouldn’t hear it. In a spectacular display of Stockholm syndrome I did my best to appease him for four years, but, as many of you can surely predict by this point in the story, I failed in every possible way and eventually gave up. Just wish I could have my four years back. I was glad to read that OP at least got paid well for his efforts.

Tade0 6y ago
I applaud you for being able to stand four years of this. I usually get fired from such positions in less than two.
- sleepybrett 6y ago
  I usually walk out about six months in if not sooner. Maybe it's just because I spent so much time freelancing that I had enough experience to recognize a no-win situation.

jtbayly 6y ago

Seems very risky. Not something I’d want to do if minimum downtime was the goal. One wrong piece of gravel ends up with catastrophic failure instead of 5 minutes of downtime.

dmurray 6y ago
But the goal was zero downtime, not minimum downtime. The client made it clear that 5 minutes of downtime was equivalent to catastrophic failure. So they correctly found a solution that reduced the chance of "5 minutes of downtime", at the expense of an increased risk of catastrophic failure.
- jtbayly 6y ago
  I understand that. I just doubt that the risk was worth it, if downtime is such a big deal.

fooblat 6y ago

> Stupidest thing I've ever had to do. I don't really understand the "ranty" tone. The client had very specific requirements and the author came up with an effective solution and was fully paid to deliver it. Sounds like a win for everyone.

ianhawes 6y ago
I believe the expectations of having 0 downtime was not expressed until the day of the transfer.
- dna_polymerase 6y ago
  In addition to that the customer runs a single server but expects the guys to maintain a property not even feasible at Google scale: Zero downtime. Overall the whole thing was just ridiculous, but luckily the customer got a nice bill in the end.
  - pc86 6y ago
    To be fair, Google maintains zero downtime for small time scales like this a lot. Most of the time, actually.
    - sokoloff 6y ago
      I’m having a hard time following what “zero downtime most of the time, actually” really means. https://m.youtube.com/watch?v=IKiSPUc2Jck&t=81s https://m.youtube.com/watch?v=IKiSPUc2Jck&t=81s
      - im3w1l 6y ago
        It means that downtime is chunky.
      - buran77 6y ago
        > zero downtime for small time scales... Most of the time I read it as "if you take small enough discrete time intervals they won't overlap with any downtime". Or in other words "no downtime between downtimes". Yes, it's very in line with your video.

macintux 6y ago

At my first job we were starting up the company and didn’t really know what we were doing; one early server was sitting on a folding table and its power cord was wrapped around a leg, so just to replace the table with something more robust involved downtime.

elliotpage 6y ago
I've been there, solidarity for cheap furniture based maintainence windows.
em-bee 6y ago
the careful application of a saw or an angle grinder would have made it possible to remove the folding table without unplugging the power cord. :-)
crumpled 6y ago
I heard an anecdote about a company splicing some fiber cable in the middle of a utility van and having to cut the van apart at the end.

Humphrey 6y ago

I haven't read the article, but I'm reminded of that episode of Seinfeld and the frogger arcade game

dfsegoat 6y ago
I have pondered this exact scenario (server move w/0 downtime) - because of watching that episode - wouldn't have thought about it otherwise. ..It's interesting how pop-culture and your chosen profession intersect, at times.

g051051 6y ago

I guess servers have gotten a lot more robust in the last decade...there's no way any server I ever managed would survive something like that.

maeln 6y ago
A lot of server are SSD-only these days which make them less fragile. Still, I really wouldn't see myself pushing a running server in a cart.
- tyingq 6y ago
  Yeah, there's certainly still things like riser cards and connectors that could come unseated due to vibration.
  - marcosdumay 6y ago
    That's probably a problem for the next guy that takes an ops job there. Loose pieces often don't disconnect right at the same instant, and even when they do, memory caches usually postpone the failures.
- throwaway744678 6y ago
  On a parking lot, no less! Let's hope it will not rain on the way!
  - em-bee 6y ago
    umbrellas over the switches and the cart... (means extra billable hours for the extra manhours needed to hold the umbrellas)

anfractuosity 6y ago

Reminds me of this - https://www.youtube.com/watch?v=vQ5MA685ApE https://www.youtube.com/watch?v=vQ5MA685ApE 'Moving online webserver using public transport'

tyingq 6y ago
The Indiana Bell building move is pretty impressive. http://www.paul-f.com/ibmove.html http://www.paul-f.com/ibmove.html
- sschueller 6y ago
  https://www.youtube.com/watch?v=CNqul9TfJwI https://www.youtube.com/watch?v=CNqul9TfJwI
- joncrane 6y ago
  Wow this is literally the exact same thing as the OP but for an entire building. Insane.
zomglings 6y ago
In the rain!
salzig 6y ago
had the same thought :)

umarniz 6y ago

Interesting read, makes me wonder as a thought experiment if it counts as downtime if the latency of commands on the machines rises to 5 minutes? You could clone the VM to another instance and record commands going to VM1 and replay them to VM2 after 5 minutes. This whole brain fart of mine doesn't make much sense but if you play along with it, does it still count as a downtime or just very high latency?

tyingq 6y ago
That works where you have control over all of the timeouts and failure detection at every level and layer. TCP keepalives, for example, could thwart you. Or client side timeouts, or firewall connection state tables, etc. 5 minutes of unplanned downtime in a pub/sub setup could easily go unnoticed, since that setup is typically tuned for long timeouts and/or repeated retries.
pc86 6y ago
Wouldn't requests time out on the client side long before five minutes?
- heavenlyblue 6y ago
  I don’t know whether it’s the software in general, but ever since I’ve started using Three 4G broadband in the UK; all of the software started behaving really weirdly (lots of lockups, hangs, etc). Apps often need to be restarted. If you do a ping during “bad weather”, you can see that they buffer up to 5 minutes of packets (i.e. there will be no communication for some time, then you’ll receive a bunch of them with a huge latency with sequences intact). So I would assume a lot of software could even work that way. I think a lot of software don’t set any (TCP) timeouts at all.
NateEag 6y ago
It depends on how downtime is defined in the contract. That sounds like I'm being snarky but I mean it - whether an actual legal contract or just the documentation given to users, any system where downtime matters should have some discussion of what impacts downtime can have and how it's measured and managed. That documentation is what defines "downtime". I'll add that what you've described is a sort of low-fi manual version of DB replication (https://en.m.wikipedia.org/wiki/Replication_(computing) https://en.m.wikipedia.org/wiki/Replication_(computing)).

user5994461 6y ago

I am so scared to imagine what would happen if there was any issue during the move (very likely when dragging live cables and powers over hundreds of meters). The client would immediately refuse to pay anything because he was very clear he wouldn't pay a thing if there is downtime. Then, the next contractor would be super quick to judge you and the situation, reinforcing that you were an incompetent idiot and the client was right to kick you away on the spot and not pay a dime. Glad it went well in the end. There is so much to lose for the person trying to help.

willcipriano 6y ago
This is a junior sysadmin I suspect. With a bit more experience you'd learn to say something along the lines of "no downtime, sure, that will be 30 grand" and the ability for downtime will suddenly materialize. Him and his friend did this big song and dance, took a huge risk and only got paid for ten hours worth of work in the end.
- zrail 6y ago
  $30k, 80% up front, strict liability waiver that says I’m not responsible for loss of business or anything else if there is downtime.
  - pedrocr 6y ago
    Now you're over-charging massively. If you have no liability and are guaranteed pay, charging for just double hourly rate is more than enough as a "stupid and non-standard requirements" kind of thing.
  - imtringued 6y ago
    You can't get paid upfront and at the same time get a liability waiver. For a 100% guarantee with full liability $30k doesn't actually sound ridiculous because it would require obtaining 100% identical hardware and doing at least one test run on that hardware before actually doing it on the production hardware. What the contractor did is basically "wing it", explain a way to get zero downtime to the client and then not actually offer a guarantee by doing the operation straight on the production hardware. Really this was more about convincing (ie bullshitting your way through) the client to let you do the work than actually doing it properly and for a huge sum of money. It wouldn't surprise me if there was actual downtime for a few seconds and the client simply didn't notice it.

neilv 6y ago

Good thing the server had two power supplies. There was a YouTube video (which I can't immediately find) of people moving a server across town, on the train, without powering it off, and, IIRC, they had to splice the UPS into the power cable. When it's done for pay rather than for fun, and payment is conditioned on zero downtime, I hope they charged a premium to make up for the risk of no pay. Offhand, I don't know what's a good way to do that -- I've never had a consulting client demand terms like that for billed-by-the-hour work.

kijin 6y ago
Effective hourly rate = base hourly rate * risk. Risk = client risk * task risk. Client risk is based on your past experience with the same client. If they're prone to demand last-minute changes or stupid stuff, they get charged a higher rate on every project afterward. Jacking up the client risk factor is also a nice way to fire a client you don't want.
discordance 6y ago
https://www.youtube.com/watch?v=vQ5MA685ApE https://www.youtube.com/watch?v=vQ5MA685ApE

Milank 6y ago

No downtime is acceptable, but they have only one server? What if a technical failure happen? What if there's a fire in the server room? What if there is an earthquake and the building collapses? What if... many things can happen that can result in a long, long downtime with this tactics. If uptime is so crucial, the system should be setup in such way that moving one server should be a peace of cake, not a spec-ops mission.

redwood 6y ago
Remind me of how IBM positions mainframes: they are so highly available that you simply never let them shut down.
- lasereyes136 6y ago
  IBM Mainframes are designed to be serviced while running so if you have multiple CPUs you can offline one at a time for upgrade it without the whole mainframe going down. Big Sun Solaris boxes where built like at as well. If your mainframe had only one CPU, you did have to turn it off in order to service it. But you could upgrade the OS without turning it off. While they aren't cool tech now, mainframes are a marvel of hardware engineering.
- chasd00 6y ago
  plus, i would imagine turning them on and bringing them online isn't just a press of a button.
  - MrMorden 6y ago
    It's not. https://web.archive.org/web/20190324191654/https://www.ibm.com/support/knowledgecenter/zosbasics/com.ibm.zos.zsysprog/zsysprogc_initialization.htm https://web.archive.org/web/20190324191654/https://www.ibm.c... (archive.org link because ibm.com apparently isn't hosted on a mainframe.)
walrus01 6y ago
From an ISP perspective this seems like the sort of company that orders one $250 a month business DIA circuit (at a price point where there is no ISP ROI for building a true ring topology to feed a stub customer) and has no backup circuit. Then the inevitable happens like a dump truck 2km away with a raised dump driving through aerial fiber and causing an 18 hour outage. Some circuits might average 5 to 7 nines of uptime over a year, but the next year is dump truck time... You can never truly be certain.

walrus01 6y ago

Sort of on the subject, i've seen a brochure for a specialty product marketed to law enforcement. It's meant for use with the seizure of live, powered on desktop PCs and similar that have a high likelihood of full disk encryption. Essentially it's a medium sized double conversion ups, with a really high quality sine wave inverter, and some electronics that can match phase with a live 120vac 60Hz circuit. And a tool kit which consists of the insulated electrical hand tools needed to do a midspan removal of the cable jacket and splice into the wires in an ordinary PC power cable. The person using it is of course supposed to be trained in advance, and competent at the process of attaching the UPS to the live circuit.

throwaway744678 6y ago
Wouldn't it be safer to open the case and connect some kind of battery + adapter after the power supply?
- walrus01 6y ago
  Splicing into the many wires that is an atx+12v power connector, between the output of the power supply and the motherboard is way more fiddly than just dealing with the hot and neutral on an ordinary $5 PC power cord. You could also never be certain what weird ziptie and cable management system (or lack thereof) might exist in a home built x86 PC case, or if there's any room for hands to work at all... I think the thing I saw is also meant to deal equally well with a commodity x86 PC built from parts, or an Intel NUC size thing, or a corporate desktop machine with proprietary internal wiring like a slimline Dell, Lenovo, HP, etc.
- ajross 6y ago
  Safer for the operator? Sure. But certainly not for the device, if you're trying to keep it operating. An ATX power supply has 24 pins at 5 different voltage levels (plus any auxilliary power connectors for the GPU and drives, etc...), and motherboards are a lot less tolerant of spikes and transients than the PS on the other side. Dealing with AC power isn't really that dangerous if you're careful.

nobrains 6y ago

I don't know. If the "boss" was charged "4.5 hours of work, 2 hours of consultancy, and 4.5 hours of consultant", and assuming he would have been charged half of that with downtime, maybe the boss did get a good deal. We don't know the cost of downtime for him. I mean if he had access to technical resources who were willing and capable to do this for him, he chose to do it.

notwhereyouare 6y ago
I personally find it hard to believe that a rough estimate of $450 for the job (spitballing $45/hr for 10 hours) is less than 5 minutes of downtime and they only have 1 server. Then again, could easily be wrong
- nobrains 6y ago
  You cannot compare it to zero. You have to compare it to the cost of doing it with the downtime. There would be cost to that as well. It will not be free.
moduspol 6y ago
It's also possible that "downtime" has different meanings to different people. The client may be seeing "downtime" as the net result of what happened the last few times the server was "down," which could have been for any number of reasons (potentially even unrelated to the server itself). When you get clients describing things like this, it's possible they've been promised things about this server before by other consultants that didn't pan out. They don't want to give you the full details because then you'll recommend a different route that they don't want to take (justifiably or not). It's easier for them to frame the problem to a consultant in a way that allows for only one potential solution, even if perhaps better ones exist, because the guy in charge of making the decision isn't technically skilled enough to assess whether others proposed by consultants are as viable. And, of course, one might read a little into why there exists a "boss" with such a highly-critical IT need that is hiring a consultant to do work like this, and thinks that threatening to not pay at all if there is any downtime is the best way to do it. I mean, what if they opened the door to this closet and it grazed a power cable on the floor and the machine just shut off? Why even bother staying around to bring things back up? It wasn't your fault and there's already downtime: you're not getting paid.

growt 6y ago

Setting up a new server at the new location and moving the VMs one by one to the new server as they become idle should be possible without downtime. But maybe there were other requirements (like no new/additional hardware) that weren't mentioned in the article.

xyst 6y ago

Why wouldn’t cloning the VMs to a second server, then split the traffic between the primary and secondary server work? Once traffic to the second server is confirmed, you could shut off the second server and haul it off to the new location. I would probably still charge a much higher rate since the owner was an arse, but at least you would get back your 7-8 hours.

blcArmadillo 6y ago
It sounds like there was no second server.
viraptor 6y ago
You're making assumptions about what's running on the servers. Let's say it's a VoIP conference server with a shared dedicated room - effectively you have an ongoing session shared between multiple connection and you cannot stop it. Or you have stateful local processing so you can't "split the traffic". Or a number of other limitations...
AnIdiotOnTheNet 6y ago
Not all services can be load balanced in this way Live migration of VMs would have been a better option, which was brought up in the reddit comments and dismissed because HyperV live migration is spotty. While I'd have to agree with that assessment, it isn't so spotty that what they actually did was less risky.
morphogenesis 6y ago
Database inconsistency for one thing. This works for frontend web services but how do you reconcile the writes between the two servers?

exabrial 6y ago

Reminds me of the "hot slide" technique used for old telephone switches

ansible 6y ago
They did some crazy stuff in the old days. Like when they moved a telephone exchange live... the whole building.

social_quotient 6y ago

Slight topic drift - Any thoughts on how the pandemic might materially change assumptions about an onsite/onprem being better than cloud or manage data center when the code people are now actually remote to the “Local” infrastructure. Something specific to the reality of the pandemic strikes me as something that would make the die hard local only folks have to start rethinking the position. (Not to suggest it’s bad, just different now that a primary assumption about people work in the office is less true)

ocdtrekkie 6y ago
I feel a lot safer knowing I'm controlling all the variables during a global crisis, actually. This article provides an example of how when you operate on prem, literally any crazy option remains on the table for you. If you asked your cloud provider to do this, it'd be a no.
AnIdiotOnTheNet 6y ago
As someone who works in a very anti-cloud company culture (which I happen to agree with), this incident has had no effect whatsoever on that mindset. We don't dislike cloud because it is accessed remotely, we dislike cloud because of the lack of control we have over everything running there. If something happens and our local systems have a problem, there are people here, like myself, who's highest priority will be fixing it and second highest priority will be communicating the status of that. Your problems are never a priority to a cloud vendor and communicating with you is even less of a priority. That's before we even get into the absurd expenses and reliance on big fat pipes.

chisleu 6y ago

I didn't want to lose my many months of uptime for a lan party back in 1999/2000 and we used the UPS to migrate my linux box across town for some Quake 3 Arena action. Things were so much simpler back then.

mercora 6y ago

i wonder if its really possible to do the initial setup of the ethernet failover without interruption. i have never done this, but i would expect the interfaces themselves will become unavailable for direct use and you get a completely fresh virtual ethernet interface which represents whatever physical interface is currently active... at least this is what happens when you add an ethernet interface to a bridge in linux...

webscalist 6y ago

Could've been cheaper to buy/rent another server, put it on the new location, set up redundancy/replication, power off the old server, move it to the new location, return the new server. Or just keep it for sanity.

imglorp 6y ago

The moving server on cart part made me nervous. If there was any rotating rust in there, bouncing across the parking lot would make things difficult for flying heads. I'd have hand carried it from stage to stage, setting it on a padded cart each stop, treating it like sweating TNT.

rafaelturk 6y ago

Fun reading. But my advice is never accept a job like this. This could easily become 2 weeks down time

codingdave 6y ago

I'm surprised that part of the story wasn't to drill down into the requirements. No downtime ever? Not even at 3 AM on a Saturday? I've found that when people are being unreasonable it is because they haven't split out their true needs from their first idea of how to meet those needs. In this case the true need is zero impact to users. The owner translated that to "zero downtime", and then didn't accept alternative solutions that still would have met his true business need.

robin_reala 6y ago

I always remember this post by the Amsterdam Police who managed to maintain their uptime on a VMS cluster despite moving data centres in the middle: http://web.archive.org/web/20120229042903/http://www.openvms.org/stories.php?story=03/11/28/7758863 http://web.archive.org/web/20120229042903/http://www.openvms...

digitalsushi 6y ago

I had to search the reddit commits for 'vmotion'. They have it covered. This anecdote is an amazingly good story for telling at the pub over a few beers. It's a terrible story for a strategy. If this is a mountain, my molehill is that one night in the late 90s, I got paged cause the SMTP outbound server was overheating. At midnight I drive across sleepy NH backroads, and stopped at a Wendys to get a chicken sandwich and iced tea, for the caffeine. When I got to the server room, I pulled the 2U Dell server out of the rack and discovered the CPU cooling fan had seized up. Mind you, this is a New Hampshire data center in 1999, and it has a filing cabinet with manilla folders, and carpeted floors. This thing was never prepared for any disasters. A half hour later, the SMTP server was up and running cool again. I greased the fan with the mayonnaise from my sandwich.

dsr_ 6y ago
The (probably soybean) oil is a fine lubricant, but the constant motion should cause the egg proteins to coagulate. How long did it operate before you replaced the fan properly?
- emeraldd 6y ago
  I can't help smiling at this analysis. It feels like something you'd hear from a sci-fi story engineer working on an rundown ship that just keeps going no matter what ...
  - staticvoidmaine 6y ago
    That book is Expeditionary Force
    - novaleaf 6y ago
      books 1 to 3 are awesome. after that the author forgets how to advance the plot while still churning out a new book more than once/year. I gave up on book 7, so yeah, I tried sticking with it.

geocrasher 6y ago

I once shut down a PC, moved it to another desk, and it wouldn't power back on. Another time I moved a server to another rack. It had 2 years uptime. Had to power it down, and it wouldn't power back on. Both required PSU replacements. Had I moved them while powered on I can only imagine the fun times. Perhaps they should have just told the customer they couldn't find it: https://www.theregister.com/2001/04/12/missing_novell_server_discovered_after/ https://www.theregister.com/2001/04/12/missing_novell_server...

ericyan 6y ago

The consultants really should told the client that if all you have is a single server then there is no such thing as "zero downtime".

neya 6y ago

Dude has ONE server and talks about having 0 downtime for his clients? What the hell?! In a way, this is Darwinism for the IT industry and I'm happy the people involved got paid well. Due probably paid as much as it woulda costed him a new server. I bet he'll never forget this lesson.

Zenst 6y ago

Interesting story and one that has played out a few times, I'm aware of a couple verbatim to that. Another - used power extension leads to cover power. Key being systems with dual power units (most servers do) and networking so you can switch from one run to another. But have known some large companies who have in their history, done things like this and other creative solutions to impossible problems.

cpuguy83 6y ago

I'm picturing that Seinfeld episode where George tries to move the Frogger arcade from a restaurant that is shutting down but doesn't want to lose his high score.

will_pseudonym 6y ago
HOLES! I need HOLES! :)

kuon 6y ago

When I was younger (read 20 years ago), I did crazy things like that, not over that long distance, but moving live servers in different racks. Now that I am older, I don't think I would do it anymore, too much stress for a small reward. Also today, most of the time, I am able to "talk out" customers of crazy requirements, while I would just have said "OK let's do it" in my younger years.

devchix 6y ago

I recall sometime in the mid 2000s there was a fever for achieving five-9s (99.999% uptime, I think -- it became fodder for a few episodes of Mr. Robot). Not that the metric ever went away, but back then a lot of BigIron(TM) vendors advertised achieving five-9s by replacing hardware while the OS remained running and continuing service. Sun 15K and 25K series (Gilfoyle had a used one in the garage running his network) were behemoths whose mem/cpu boards you could swap out wholesale while the entire frame and backplane was powered on, and while the OS the board came out of remains functioning. There were many caveats around the procedure but it worked. Execs and sales guys loved those demos. These monsters were expensive and banks and energy conglomerates were buying them by the dozens. There was also a big todo about hot swappable drives. The idea that you could be doing hardware maintenance while the machine was still running was a novelty, something like brain surgery while the patient was not only awake, but awake and eating, driving his car, talking on the phone, etc. A decade later I look back with deep surprise that we didn't think to abstract out the service instead of the hardware. I don't know how many of those behemoths are still being bought, now I work almost exclusively with small server instances that can come and go on the fly. Micro services and AWS have taken five-9s in a different direction. I frequently think of Sun as a failed Hephaestus, in a Christopher Nolan film he would be brilliant but could only turn out clumsy tools because of his deformity, he hates the things he makes so he throws them away before completion. Men find these cast-offs and temper and refine them.

chasd00 6y ago
i remember those. A friend of mine was a network engineer at a local datacenter ( UUNET then MCI pre-scandal ) and said companies were buying Suns for everything no matter how trivial. He worked a night shift and i use to go hang out with him in the noc and download movies (residential bandwidth was not what it is today). Odd he nor i ever get in any trouble for that heh.
tlack 6y ago
Well as I recall there were a few reasons that people focused on reliability in hardware in the late 90s: 1. Shared state storage systems that supported replication were rare (I think Oracle and Informix maybe?) 2. Virtualization software was in its infancy (did SunOS have something before Solaris?) 3. RAM and hardware were waaaaay more expensive, meaning you often had to buy more pure metal just to answer questions fast enough At least that's my take on it based on my dim faded memories

tobyhinloopen 6y ago

10 hours investment for no downtime seems like a good deal for the owner

topkai22 6y ago
Depends on if he really has customers accessing the system "all the time." Besides, as pretty much everyone has noted, running a zero-downtime system on a single physical machine in what sounds like is just a normal cable room is kind of nuts. Those 10h would have been much better spent to move that puppy to someone else's data center and get some redundancy. Although reading between the lines, maybe the lease was up and they were waiting to the last minute to move it.

linsomniac 6y ago

Lower stakes, but ~15 years ago a friend had a Linux box in the corner that had huge uptime. I want to say the uptime started shortly after the kernel patch that fixed the 400-ish day overflow of the uptime counter. He moved to a new home and very carefully moved the running server using it's UPS. He didn't have to worry about keeping networking up though. I used to be all about long uptimes. I eventually started seeing long uptimes as a negative though. A long uptime probably means patches have not been applied.

jccooper 6y ago
I also did that once, about the same timeframe, specifically to preserve an uptime. I think the cult of runtime came about simply because it was impressive that a personal computer could stay running for more than a few days when most of the world ran Win95. And because development cycles were longer and there weren't a lot of network threats.

JeroenKnoops1 6y ago

Reminds me of the OpenVMS clusters.. Police in Amsterdam celebrated in 2007 an uptime of 10 years of their cluster. In this period, all hardware was replace, and half of it was moved to another location 7 km away. All data moved from DAS disks to SAN without one application needed to be stopped. Also VMS was upgraded from 6.2 to 7.3-2. The VMS cluster did not go down during all of these changes. I <3 OpenVMS

JeroenKnoops1 6y ago
During Y2K I've also had to shutdown various OpenVMS servers with uptime over 10 years... Only because of company policies, not because OpenVMS required the reboot.
tandr 6y ago
Would be interesting to know if it is still up and running?

tzs 6y ago

I needed to restart a server where I worked. My boss was complaining about the revenue loss during the down time. I knew the revenue loss (if there even was any, as opposed to a couple of minutes of revenue simply shifting to a few minutes later...) would be well under a dollar. So I listened to him whine for a couple minutes, then tossed a dollar on his desk, told him that would cover it so he could shut up now, and rebooted the server. Warning: you should probably only try this if you are good friends with your boss. That boss had been my best friend for years before I came to work for his company.

jfcorbett 6y ago

Reminds me of the time where IT at a previous employer told us that due to a "new IT strategy", our production cluster that had been sitting comfortably in the basement for years had to be moved to an "approved IT hub facility"... in another office 500 km away and across the North Sea. There was downtime. Promptly after our cluster settled into this wonderful new facility, a cooling pipe in the ceiling leaked on it, frying 1/3 of our nodes.

yjftsjthsd-h 6y ago
On a personal selfish level I was quite happy to see our workloads moving to datacenters that we couldn't (reasonably) physically access, because it replaced "can you go drive to the DC and replace a failing disk" with "we put in the request for smart hands to replace the failing disk". Of course, there's some notable tradeoffs, but it makes me feel better when the business decides to do such things...

arethuza 6y ago

This reminds me of a small company I joined many years ago that did deployments by RAID - find a working server (possibly at a customer site) swap in a blank HD, wait for it to rebuild then take it and put in a new server and repeat the process. Like finding people who argue against revision control systems, it's really quite a challenge convincing people why things like this are a bad idea - after all "it works!".

yjftsjthsd-h 6y ago
That's... actually fascinating, if in a slightly insane way. There's pets, there's cattle... and apparently there's a herd of cloned pets, which I'd somehow never considered before:)

rafaelturk 6y ago

Pictures please!

mercora 6y ago

when i was younger i was super proud that i could replace my disk while i kept working on the device. i would put the new disk into my LVM volume group moved all extends to the new disk and dropped the old disk out of the VG afterwards, when done i could just unplug it and be done without halting work except for kicking off the process.

Johnny555 6y ago

Decades ago an ISP I was colocated at did the same thing. I don't remember the exact details, but it was a DNS server and they either couldn't log in or were relying on the zone files cached in memory or something but for some reason they couldn't power it off. It was already plugged into a UPS, but they had to cut one of the posts off the rack to get the server out without unplugging it, then they plugged that UPS into a bigger UPS on a cart and wheeled it to the new data center they built out in the building next door. The world was much different at the time -- this coloc provider had a good reputation, yet.... they had a keg of beer in the corner of the server room and a stack of adult magazines in the men's room.

Reedx 6y ago

That reminds me of the Pixar incident where Toy Story 2 was accidentally deleted while in production and had no working backups. Luckily one employee was working from home (rare at the time!) and had a copy of the entire movie on her desktop computer. Which they very carefully moved back to the office and were able to restore from that. https://www.youtube.com/watch?v=7MAedEXri7c https://www.youtube.com/watch?v=7MAedEXri7c

tzury 6y ago

I once was called in to export data from a DOS program that had no export option. Single Author died of heart issues and the company needed the data for the migration. After several attempts to understand the binary format I gave up and ended up printing tabular reports to LPT1 which I connected my laptop to, extracting it and rebuilding CSV files. Lucky enough, printing those days were the most important feature of a business app.

fredley 6y ago

This is the kind of content I've only ever seen previously in TDWTF (which is entirely this sort of content...) https://thedailywtf.com/ https://thedailywtf.com/

merb 6y ago

meanwhile in germany, german telekom have their connect ip lines (leased lines..., company internet..) shutdown since tuesday morning. so a downtime of over 48 hours, besides a sla that no downtime will be longer than 8 hours and a availability of 99,9%. what a crazy world.

kyuudou 6y ago

It's called vMotion

akssri 6y ago

Was it George Costanza ? https://m.youtube.com/watch?v=a-FbktgqCqY https://m.youtube.com/watch?v=a-FbktgqCqY

johnklos 6y ago

I've done something like this - server running off of UPS moved from one building in Manhattan to another about 1/4 mile away, in snow... Not for someone with weak arms.

BrianB 6y ago

It's been done. https://i.cdn.turner.com/v5cache/TBS/Images/Dynamic/i439/seinfeld-s9e18-1600x900-800x450_071620141129.jpg https://i.cdn.turner.com/v5cache/TBS/Images/Dynamic/i439/sei...

neycoda 6y ago

I wonder if there was any legit reason to require no downtime. Otherwise the owner doesn't understand what downtime means for his business.

jcrawfordor 6y ago

I once basically spent a summer doing this, not over a parking lot but to consolidate the remaining equipment in a large number of racks into a few new ones - this was a former sales office of a megacorporation that had been built to have its 1970s-era computer room proudly displayed through windows into the main conference room, a very weird setup without the context that in said '70s that conference room was used to pitch prospective customers on business automation. Anyway, by the time I was there it was still a '70s-vintage large computer room but now massively overprovisioned on space, cooling, etc, particularly with most IT functions having moved to corporate. A decision was made to repurpose part of it as a test lab and move all the actual remaining equipment to three racks in the corner. I'd do about two servers a day in between other things, taking advantage of redundant power supplies to transfer the PSUs one at a time to extension cords, swap to a long network cable fast enough that TCP sessions probably didn't time out, and then unrack onto a hydraulic lift card and do the same procedure the other way. I presented this at the start as far from a guaranteed strategy - that it would minimize downtime but there would inevitably be some due to mistakes. None of this was really that critical. There were a few devices that were pretty old and poorly maintained, we agreed up front that if these lost power for some reason and then failed to boot, we would just say they'd lived long lives and purchase replacements. I guess the point is that this whole situation was kind of unusual and I would generally not recommend doing this, we were lucky that all the equipment left had stakeholders that acknowledged it was legacy stuff and they could tolerate losing it. The irony is, of course, that it went perfectly. So far as I know there was not a single problem experienced through the whole thing. I even managed to swap the phone lines to the (surprisingly busy!) legacy fax server when each was out of use.

groby_b 6y ago
> There were a few devices that were pretty old and poorly maintained, > if these lost power for some reason and then failed[... we'd] purchase replacements. And the sysadmin let that opportunity pass??
- jcrawfordor 6y ago
  Heh, I know of a couple of people who would have been far happier if the fax server specifically was somehow accidentally tipped down a stairwell... by this time there were enough dirt cheap "cloud" fax services that it really didn't make sense to keep the on-prem system, particularly since the office had been almost entirely migrated to VoIP except for a few oddball devices like that fax setup. But that whole thing just becomes a story of the internal woes of that particular megacorporation, from the computer room to the front office there was a whole lot of stuff just being kept over from the '90s.

noobermin 6y ago

Sorry but this is ridiculous. It's a great story of a feat of sysadminery, but the client should have just accepted some down time, even a few hours. The level of entitlement from some clients people get is just infuriating. Even down to calling him back for not agreeing to help, what an infuriating person. That was my main take away from this. Endeavor to be the sort of person who can refuse clients, the entire idea that "the customer is always right" enables so much ridiculous behavior.

deleted 6y ago

[deleted]

pengaru 6y ago

Decades ago working in a sysadmin role at a hosting company I had a similar situation. The solution I came up with was to fashion a custom male<->male power cord, like a gender changer, from some broken ATX PSU scraps we had laying around. By rearranging the power sockets from multiple donors, two male power cords could be connected on a single enclosure. Internally the sockets were simply bridged, otherwise the PSU was basically gutted. With this goofy metal box having two male power cords dangling from it in hand, I just used a very long extension cord plugged into an outlet on the same AC phase as the existing server's power source. The extension cord powered one of the bridge cords. The other bridge cord plugged into the server's existing - and hot - power strip, forming a redundant power source. Now the power strip could be unplugged from the primary power source without losing power, and we just moved the server to the new location with the bridge box and power strip in tow. If memory serves the only tricky part was determining which outlet at the new home was on a compatible circuit. We didn't have much in the way of electronics tools, no oscilloscopes or anything. Even the soldering involved to make the bridge box was done using my personal soldering iron, which just happened to be in the office because some of us raced RC cars there after hours. I think I just used an incandescent desk lamp to verify a normal brightness on the bridged circuit before proceeding with the server, but it's been a while. I wonder how many people have fashioned AC power cord gender changers throughout history... :)

shireboy 6y ago

You realize the client is condescendingly mocking the guy for saying it can’t be done now, and will expect this next time they run updates on the server, which is to say never

partiallypro 6y ago

This reminds me of the Seinfeld episode with Frogger

aleem 6y ago

Sometimes it's better to seek forgiveness than to seek permission. Saturday 3AM shift with a 5 minute downtime would work just as wwll. Unless this server has had historically 100% uptime this would go unnoticed.

aeno 6y ago

A real classic comes to mind: https://www.youtube.com/watch?v=vQ5MA685ApE https://www.youtube.com/watch?v=vQ5MA685ApE Moving a running server about 7km through public transport without downtime.

snvzz 6y ago

Incredible. Five minutes of VM downtime were not acceptable. And yet, they had a single VM host. Should it catch fire (hardware FAILS!!!), what then?

lightlyused 6y ago

I seem to remember a similar story from another site. I'm thinking it was thedailywtf.com, but I can't find it.