5 ms·

I think there is a big difference here. I am responding to a reply that wanted remove any responsibility from running such non-since as npm on a productions ser

I think there is a big difference here. I am responding to a reply that wanted remove any responsibility from running such non-since as npm on a productions server -- note I did not reply to "crap I hosed my stuff". But lets take a closer look at your comments. > Not everyone is blessed with working in a mature, well-funded environment full of experts. Maybe we're talking about a new or small organisation that simply doesn't have the resources and/or knowledge to isolate things with containers or VMs and related admin tools. These are not excuses to knowing your trade. And the size and funding of your environment should not stop you from practicing your trade well. > Maybe even taking out a staging server is still going to waste significant time resetting everything, blocking other development/deployment jobs in the meantime. I fundamentally disagree. Having a stating environment will always cut cost and can't EVER be considered to "waste significant time". It can only save time and improve your product. Its these types of attitudes that result in your service going down and lost of real revenue and ultimately the failure of the project. Taking time to setup proper staging environments always pays back in spades. > Maybe we're not talking about a server at all, but a developer's personal development workstation where they just use NPM to install a few Node-based tools. Yeah, maybe we are talking about a developers personal station, nope, we are talking about production servers. Nuking a developers station is not even on the same scale as nuking a production system. And had somebody complained about nuking their dev environment my reply would have been about not running tools as root. > Maybe we're not talking about a server at all, but a developer's personal development workstation where they just use NPM to install a few Node-based tools. Again, I am okay with a developers system being nuked, at least it was not production! > It's all very well saying npm shouldn't be run on production servers, but that doesn't really address the fundamental problem. Do we also ban system package managers, and say the only way to deploy anything is via some sort of imaging tool? What if there's an equivalent screw-up in that orchestration tool and it bricks all 100 servers at once? I would not advise running system package managers on production servers either -- not unless your staging environment had passed such test first. But that being said I am a big fan of fresh install and migrate -- where the migration code is something I own and can test to ensure it works before using it. If you have 100 servers then you should have the resources to handle setting up testing environments to ensure your production rolls out. You should also not update 100 servers at the same time. Production is production is production is production is production! You don't run things the first time ever in production. If you want your product, company whatever to succeeded then there really is NO excuse for you not to have good practices when building and deploying software. You can come up with 2^64 what if's but if you are running something that for the first time and it nukes your system you are at fault. Things like testing environments or staging environments were created to just dream about, and talk about when things go bad with deploying directly to production. These things came about because they bring real value to a project. The notion that these things are a waste, or cost too much just non-since. Anybody in this industry of deploying software to servers needs to stand-up to these ideas that these good practices are too costly. These are the ideas and notions I expect from executive teams who have never coded a line of code, the accountants trying to save money, and managers who only care about the next quarter. I don't expect to find these ideas on sites like HN or from peers in the industry, but when I do I think it is important to take a hard line and not allow the notion of bad programming, and deployment practices to be unmet with rebuke for fear of hurting somebodies feelings. So while I clearly towed a hard line in this reply Silhouette, please do not take this as a personal rebuke or attack at you. I am upset with the ideas, and the notion hat we have to settle for less, and have results like production servers falling on their face when we as a industry already know the answers to the problem, and have the solutions to minimize downtime and provide truly awesome software to others.

Silhouette 9y ago
These are not excuses to knowing your trade. And the size and funding of your environment should not stop you from practicing your trade well. It's all very well saying "know your trade", but the reality is that most organisations aren't running state-of-the-art orchestration tools. Heck, not so many years ago, many of these modern tools didn't even exist yet, and they've had plenty of problems of their own that make keeping up with the bleeding edge dangerous in itself. So, while it might not be ideal compared with modern management tools, I think it's neither unusual nor unreasonable in many real world environments for someone to expect to deploy a standard set of packages on a production server using the normal deployment tools and a controlled configuration file, and expect it to work without destroying that world. Speaking of funding, that affects everything in an environment like a bootstrapped startup or a small non-profit, even things like whether you can afford physically separate machines to run each level of testing/staging/whatever, or whether you can afford to hire someone who understands the recent generation of tools that deploy a snapshot in one form or another instead. It's totally unrealistic to expect this sort of organisation to have mature, state-of-the-art configuration management and deployment systems in place from day one. Hopefully even in the early stages you would still have some sort of staging set up, and I think you misread my comment there; I was in no way advocating not having staging servers. I was only observing that even if you take out staging catastrophically rather than production, it can still be a pain to set everything back up, just less of a pain than losing production while you're doing it. Again, I am okay with a developers system being nuked, at least it was not production! You're OK with a developer's entire workstation being taken out, at best losing everything they've done since last night's backup and then probably taking another half-day to restore from backups if everything goes smoothly? I'm not OK with that, and somehow I doubt most developers would be either. If you have 100 servers then you should have the resources to handle setting up testing environments to ensure your production rolls out. You should also not update 100 servers at the same time. Right, but how many organisations have 100 production servers? If you've reached that scale, you're already probably in some sort of 1% group, and obviously you might have far more resources available to deploy management infrastructure around those servers. Anybody in this industry of deploying software to servers needs to stand-up to these ideas that these good practices are too costly. That philosophy might be something you can afford once you're no longer operating in small/early mode, if you get that far. But while you're still worrying about say getting from MVP to ramen profitability in your startup, everything is too costly, and you never have the luxury of doing the ideal thing everywhere right now. Hoping for basic staging isn't out of the question. Hoping for a full-time ops person to deploy the best-in-class orchestration tools that came out last week because you can't trust running apt to install security updates on your production Debian servers without destroying them is probably beyond your wildest dreams. It's not that I disagree with you on the ideal situation. I just see that an ideal is what it is. Many, many organisations will not have the luxury of doing everything ideally, because they lack the time, people, budget, knowledge or omnipotence to do it all at once. That's the nature of running businesses. It's not unreasonable to expect that when you have to prioritise, the risk of your basic package management tools nuking your entire system should be negligible, and I still think it's unfair to criticise the victims of such a spectacular screw-up until you've walked a mile in their shoes and seen what they would have had to give up somewhere else to get that extra level of protection against something that obviously should never have happened.

Silhouette 9y ago

These are not excuses to knowing your trade. And the size and funding of your environment should not stop you from practicing your trade well. It's all very well saying "know your trade", but the reality is that most organisations aren't running state-of-the-art orchestration tools. Heck, not so many years ago, many of these modern tools didn't even exist yet, and they've had plenty of problems of their own that make keeping up with the bleeding edge dangerous in itself. So, while it might not be ideal compared with modern management tools, I think it's neither unusual nor unreasonable in many real world environments for someone to expect to deploy a standard set of packages on a production server using the normal deployment tools and a controlled configuration file, and expect it to work without destroying that world. Speaking of funding, that affects everything in an environment like a bootstrapped startup or a small non-profit, even things like whether you can afford physically separate machines to run each level of testing/staging/whatever, or whether you can afford to hire someone who understands the recent generation of tools that deploy a snapshot in one form or another instead. It's totally unrealistic to expect this sort of organisation to have mature, state-of-the-art configuration management and deployment systems in place from day one. Hopefully even in the early stages you would still have some sort of staging set up, and I think you misread my comment there; I was in no way advocating not having staging servers. I was only observing that even if you take out staging catastrophically rather than production, it can still be a pain to set everything back up, just less of a pain than losing production while you're doing it. Again, I am okay with a developers system being nuked, at least it was not production! You're OK with a developer's entire workstation being taken out, at best losing everything they've done since last night's backup and then probably taking another half-day to restore from backups if everything goes smoothly? I'm not OK with that, and somehow I doubt most developers would be either. If you have 100 servers then you should have the resources to handle setting up testing environments to ensure your production rolls out. You should also not update 100 servers at the same time. Right, but how many organisations have 100 production servers? If you've reached that scale, you're already probably in some sort of 1% group, and obviously you might have far more resources available to deploy management infrastructure around those servers. Anybody in this industry of deploying software to servers needs to stand-up to these ideas that these good practices are too costly. That philosophy might be something you can afford once you're no longer operating in small/early mode, if you get that far. But while you're still worrying about say getting from MVP to ramen profitability in your startup, everything is too costly, and you never have the luxury of doing the ideal thing everywhere right now. Hoping for basic staging isn't out of the question. Hoping for a full-time ops person to deploy the best-in-class orchestration tools that came out last week because you can't trust running apt to install security updates on your production Debian servers without destroying them is probably beyond your wildest dreams. It's not that I disagree with you on the ideal situation. I just see that an ideal is what it is. Many, many organisations will not have the luxury of doing everything ideally, because they lack the time, people, budget, knowledge or omnipotence to do it all at once. That's the nature of running businesses. It's not unreasonable to expect that when you have to prioritise, the risk of your basic package management tools nuking your entire system should be negligible, and I still think it's unfair to criticise the victims of such a spectacular screw-up until you've walked a mile in their shoes and seen what they would have had to give up somewhere else to get that extra level of protection against something that obviously should never have happened.