4 ms·
That really sounds like they haven't been backing up their server, and that they think of source control as the backup. Both of these things are pretty bad.
by onion2k 7y ago
That really sounds like they haven't been backing up their server, and that they think of source control as the backup. Both of these things are pretty bad.
- pmikesell 7y agoThis is common. Most places use their revision control system as backup for their source code and take backups of their database. This makes sense because the source code holds no state. Why would you back up the server? What would you do when you had 100 servers? You’d take backups of the running disk state of all of those? To what end?
- ajross 7y agoRight, it's not a problem with technique. There are lots of storage management paradigms whose backup and restore strategies don't involve filesystem copies of the running servers. The problem here was that there seemed to be no backup and restore strategy at all. That said, though: if you're a tiny group like this and don't have the time to invest in a gold plated storage design, "Just Back Up All The Servers" is a pretty reasonable choice to make.
- undoware 7y agoPrecisely to this end :) in that role I would typically back up the deploy directory (/opt/foo) and the /etc/ and parts of /var/ every day to Glacier. Of course this is all pre-docker and pre-k8s but this is 2013 we are talking about
- cwyers 7y agoWell, I think we have an answer to "what end" now. Yes, if you have a deployment strategy where you can reproducibly deploy from your source code repository, then you don't need to back up the servers. But until you've tested that that works, and I mean really tested it, you need server backups.
- onion2k 7y agoWhy would you back up the server? In case of things like this. Also it's usually quicker to recover a server using an image than it would be to provision from source in the case of, say, a hardware failure. What would you do when you had 100 servers? Something more scalable. There's no reason why you should pick an approach to solving a problem and stick with it forever. If the situation changes you do something more appropriate. That doesn't mean the small scale approach is wrong when the scale is small.
- ianmobbs 7y agoI mean, the better way of doing this is just keep your application code stateless and make frequent backups of your data stores. If the developers knew they had server code that wasn't checked into SVN, your solution makes more sense, but they just had no idea. Even in corporate, if I need to set up a new box for a service, I don't backup an image from another box and deploy it to the new one - I just pull my Docker image and run that.
- cwyers 7y agoYes, but part of keeping your application code stateless is that you need to actually do things to keep it that way. If you haven't done those things, it's easier to set up Tarsnap in a cron job than it is to make sure your deployment process is stateless.
- AmericanChopper 7y agoIf you have a service you haven’t built in 5 years, then I’d say you have a lot more important problems than server backups.
- gustavorg 7y agoWhy would you back up the server? Because the rule number one of the programmers club, is backup everything, everywhere, all the time, if you don't remember do it again, and if you're sure you have enough backups do it again anyway. I even do backup of my backups everyday (in pendrives, in cds, in ancient scrolls, etc)
- wetpaste 7y agoIn the age of treating servers (or containers) like cattle instead of pets, the "Back up everything" mantra has fallen by the wayside. In order to get away with selective backups you have to know exactly where long-term state is stored and you need to have the infrastructure in place to manage re-provisioning everything and restoring snapshots. It's not something you can tack on later. Iterate, test, integrate, document, audit, review. It ends up being much more complicated than periodic wholesale snapshots on a server. There's a certain elegance and assurance you get from this that has been lost with the times, akin to how monolithic server software with all functionality natively available in the code has gone away in favor of microservices. Now you have message queues, k/v stores, caches, search engines as a microservices that are tacked on to the core services and rarely fully understood by the engineering team and containing more functionality than the codebase ever really utilizes. Ends up being more complicated in manage in a lot of ways. I think the emergence of microservices is one of the driving forces behind selective state backups, because you can never back up the entire state at once, everything is too spread out. You're not going to back up the running state of the k8s node, or whatever
- LanceH 7y agoIt's more "make everything restorable" rather than "back up everything."
- matwood 7y agoIf the servers are still treated as pets, then they need to be backed up. Outside of code there would be configuration. Keep in mind that once the servers are treated as cattle then individual server backups are typically no longer needed. The developer in this case would likely not be in this situation then.
- tomc1985 7y agoShit like this happens more often than one might think. My last job we had one magic disk image that worked and anything else we tried to handroll didn't. (The app itself was written in an ancient version of Django.) While working on other things I discovered modifications written directly to cached versions of the Django instance itself, which were not committed to any source control. He simply edited the instance of Django in the disk image, which was then replicated to our various deployments. The worst part was the mods this guy made were perfectly capable of running within our project, from code in source control. Why he took the approach he did was a complete mystery to us.
- Shish2k 7y agoMy last job had similar -- our internal tools were a buggy mess, to the point where customer support spent upwards of 75% of their time fighting the software instead of supporting customers - but the CEO decided that the dev team would not fix any bugs that aren't DIRECTLY affecting end-users, nor would the dev team "waste time" reviewing patches from frustrated tech-literate support staff fixing internal bugs for themselves. However, the support staff still had root logins to all our servers, so they followed the letter of the law by not having their patches go through the code review / source control / gradual rollout process...
- deleted 7y ago[deleted]
- jniedrauer 7y agoThis is actually a pretty reasonable way to run things In The Cloud™. If your servers are transient and provisioned on demand, then backing them up is largely meaningless. Source control is the backup.
- hinkley 7y agoBut you're supposed to prove it's the backup. It's part of the deal. As one writer said early in the unraveling of Agile, it's like you were told that if you eat your vegetables you can have desert, and now we just want desert but don't want to eat our vegetables.
- OJFord 7y ago> That really sounds like they haven't been backing up their server Nor rolling it and both testing this procedure and putting people off mutating the server.
- hinkley 7y agoMy next question is "why didn't the QA team find this?" And, "Do you have a QA team," he said, expecting the answer "no".
- adamhepner 7y agoMy next comment as a QA: "I told you so" God, the many times I had told them so months before difficult situations arrived, just to be swiftly ignored by business because some devs didn't want to bother, and wanted to get to more shiny and fancy and interesting problems.