4 ms·
> A lazy programmer do not deploy in production, they instruct Jenkins to do that. Therefore a lazy programmer is not afraid of deploying on Friday afternoon.
by Sebb767 6y ago
> A lazy programmer do not deploy in production, they instruct Jenkins to do that. Therefore a lazy programmer is not afraid of deploying on Friday afternoon.
Just because Jenkins does the final "cp && restart" does not mean that critical bug will not ruin your weekend. The tests mentioned before help, but the unexpected is, per definition, unexpected.
In fact, when the house is burning, I found CI deploys to be quite a slowdown. But this might just be me.
- tigroferoce 6y agoOP here: I agree on some level. I have perfected my vim skills by actually modifying the production php source code in my youth. However as things get more and more complex, it helps to script them down so that you don't have to remember a set of commands in a given order. I see the deploy script in this case as a "documentation for lazy" :-).
- Sebb767 6y agoFair enough. Our CI does build+deploy and we're on a bit of a budget, so the process tends to be `git push`, wait 5-10 min for the pipeline to complete, check if it worked, repeat otherwise. But that's really something to optimize in our infrastructure; in an ideal case, I'm on your side :)
- gomseb 6y agoWell, I guess the point here is to rollback to the previous version and find and fix the bug on Monday.
- denimnerd42 6y agoI'm curious how many companies have a working rollback. I think the majority probably have to keep patching until they get it right or get it stable.
- lunfard00 6y agomaybe because by definition you lose data on rollbacks? You can re-run all events/messages (if your company architecture supports it) but that gets more complicated, usually patching is easier and faster. Also, downtime.
- Sebb767 6y agoSometimes, yes. But the bug might not immediately be obvious (for example a memory leak) or cause data corruption, in which case you need to get a backup - and you probably don't use a backup without manual approval. A good infrastructure will help you at every point, but it will still take time.
- hyko 6y agoYeah, not deploying on a Friday has nothing to do with a lack of CI; it’s related to the human culture of having a weekend, which reduces an organisation’s ability to handle the unforeseen consequences of pushing to production. “So just roll back” I hear you say, to which I respond: “it ain’t always that easy”. You can’t always just turn the reactor off once the meltdown is in progress.
- WrtCdEvrydy 6y agoWe've had this issue many times with dependency management. CI will rebuild after you revert everything and still fail to deploy if you pushed a versioned library from your broken code. Go into your environment and manually roll back to a working image... and learn a powerful lesson.
- vergessenmir 6y agoI'm not sure what you're saying here. I can see this being a problem when you have cross-team dependencies and your roll-back requires coordination. Even then it's manageable if your production deployments are version pinned, it only becomes a problem if schemas in your database or messaging have had non compatible changes and data has been written (i.e you have binary and data dependencies in a release) All this being said with the assumption that the CI/CD pipeline has been designed to best practices.
- yoz-y 6y agoDeploying often means a lot more than just “changing some binary somewhere in prod” too.
- runarberg 6y agoWhy is a developer deploying to production in the first place? To minimize the work for everyone the code should have gone through QA before it was passed to ops. Ops should be confident that QA did their job sufficiently (“A lazy programmer knows how to delegate”) and shouldn’t be afraid to start the deploy job whenever.
- mxz3000 6y agoNot all companies/teams have dedicated QA and Ops teams. In the (very large) company I work at, the engineers are responsible for writing the code, testing it, and supporting it once it's live. That includes being Oncall and being paged if something goes wrong.
- runarberg 6y agoFor a small company with a fairly stable feature release schedule, that is fine. But in a large setting where expectations for new features is high, that seems like a scam. That is it looks like they are trying to get the developer to work double (or triple) shifts but only pay for one.
- temporama1 6y agoDude, I'm so jealous that you haven't even heard of DevOps. I don't know where you must work but please give me a job.
- Silhouette 6y agoThe trouble seems to be that "DevOps" is the new "Agile". How often is it just an excuse to dump non-development responsibilities on the dev team, and to ignore boring but time-proven processes and tools that specialists might have preferred if you'd hired them for those responsibilities instead?
- jghn 6y agoWhy are there separate QA and Ops teams?
- _the_inflator 6y agoApple won’t help you during weekends. Our experience. There are dependencies, you at first don’t think of.
- void_mint 6y ago> I found CI deploys to be quite a slowdown. But this might just be me. It depends how you think about it I think. Meaning, it'll feel like quite a slowdown but exists to help mitigate errors made by humans experiencing varying degrees of panic while their house is burning. In my opinion CI deploys are meant to provide consistency, not speed (although people generally experience speed ups in their deployment process when there are no longer "works on my machine" type issues that pop up each deploy causing slowdowns in new and exciting ways). But I also think that programmers flaunt "Friday afternoon deploys" too much in either direction. It shouldn't be the end of the world but it also is just an unnecessary risk if you can help it.
- milesvp 6y agoYes. I say deploy in friday, but you better call it out explicitely. We’d call it cowboy friday anytime someone suggested deploying on friday. That person immediately volunteered to be on call. And if anything went south, they lost social capital and had to eat crow (mostly playfully, more seriously if it became a pattern). If enough people agreed the risk was worth the reward, and at least one QA was willing to vouch for the push, the yeah, push as long as there’s enough time to rollback before people leave for the day. Mostly, it was easier to not push on friday.