10 ms·

We ship the most code on Friday

69 points by dban 4y ago

litoE 4y ago
You never ship on Friday because, to a programmer, a weekend is an infinite amount of time.
CobrastanJorji 4y ago
> But following that reasoning, why not also avoid shipping after 4PM? Yes, exactly! Don't ship after 4 PM either!
- fooster 4y ago
  Yes, only ship code to production every other Tuesday from 10:30-10:30am.
  - lmm 4y ago
    I worked somewhere that did that and it was great. Everyone knew when the release was. If your code didn't make it into this one, well, no worries, it can go out in two weeks' time. And if something broke we had all day / all week to fix it.
    - throwaway98797 4y ago
      sounds awful for feature velocity
      - outworlder 4y ago
        Not all industries care about that, versus having fewer outages.
  - forgetfreeman 4y ago
    Some folks just cannot be satisfied with a warning and have to stick the fork in the outlet for themselves.
- 0x457 4y ago
  I think if you're afraid to ship after 4pm, then you should write better code and/or test it better. There are things I'm afraid to do late (say upgrading 3rd party software), but shipping a new version of my software? Don't see why not.
  - alar44 4y ago
    Right, just write perfect code! Things can and do break. It's impossible to test everything. That's why.
  - outworlder 4y ago
    > I think if you're afraid to ship after 4pm, then you should write better code and/or test it better. There are things I'm afraid to do late (say upgrading 3rd party software), but shipping a new version of my software? Don't see why not. Your wording seems to indicate that you work by yourself. That's fine, but things change when large teams and systems are involved. You can't vouch for every single change or the inter-dependencies between all systems. It's pretty elitist too. "Your code only breaks because you suck" doesn't make for good software engineering practices. Everyone makes mistakes.
    - 0x457 4y ago
      No, I don't work by myself. We are just good at writing tests and have a good QA team that writes good e2e tests.
    - lnxg33k1 4y ago
      That's why he wrote "test it better", it's intrinsic in that statement that he's well aware of the fallibility of humans
  - opportune 4y ago
    Really depends on what you’re working on. Less complex things that are easily tested, sure. For more complex things that are hard to fully test, bad idea. If you have a micro service that takes requests over / foo and /bar and returns JSON that needs to fit certain properties, and you’re submitting a change to fix a corner case bug or add a new property that isn’t read, sure. If you are working on low level networking infrastructure that is used in tandem with thousands of different workloads you need to support, won’t work.
- opportune 4y ago
  I think what’s missing the most is context. I work on infrastructure and we never ship after 4 or on weekends because we have to support so many different kinds of behavior, and if we break any of those behaviors (which may very well be relying on undefined behavior/non-compliant protocol implementations) we are breaking our customers. For people who work on applications they are often only hurting themselves if things break, so the risk profile is different. Maybe you don’t even have any customers, maybe your customers are consumers who are just not able to place orders. IMO these groups often talk past each other (although as someone who works on infrastructure I think we often consider application developers’ more than they consider us, because they are our customers, and to them we are doing best when they don’t need to think about us) which leads to all these arguments from “Ship fast ship often - nobody uses our mostly static website which is easy to test, so who gives a fuck if it breaks” to “canary every change and carefully A/B test it with long release times to ensure it is safe”.
  - CobrastanJorji 4y ago
    You know, that's a very good point. These arguments never bring up what's being built, what's the cost of things going wrong, etc. If you're running a Very Serious service with well-defined SLAs with a bunch of formal definitions and a suspicious number of 9s, your risk tolerance is entirely different from a startup with six highly caffeinated people who have naught but a dream and a three month financial runway before they're all out of a job. If devs from those two teams describe their release process to each other and don't mention what they do, they'll each probably conclude that the other is insane. One will vomit upon hearing the other describe a multi-week deployment scheme, and the other will vomit upon hearing the phrase "test in prod" said non-ironically.
mikl 4y ago
Is it just me, or wouldn’t their `git log | grep…` one-liner count commits to all branches that were eventually merged to master, not just direct commits to master? If so, unless they always squash when merging, this just shows that they have the most commits on Fridays, not deployments.
- gschier 4y ago
  We do squash commits to master, so there's only one commit per pull request. edit: I updated the image caption to specify. Thanks for pointing that out!
bradhe 4y ago
Not shipping on Fridays in 2023 is an infrastructure smell.
- noman-land 4y ago
  To me it seems like just good, clean living. If you have to deploy on Friday then you can, but if you don't have to, then it just seems like a good idea to do it while many people are paying attention instead of over a weekend when most people are off.
- pcdevils 4y ago
  Infra can make it easier to do it safer; it's not a panacea. Sure you can roll back; alert based on metrics / logs / error rates, but the knowledge required to automate around these data points tends to require the data to already exist, which won't be true with new features.
  - bradhe 4y ago
    Okay so don't always ship on a Friday but that doesn't seem like rationale to put a blanket rule in place.
- Jensson 4y ago
  No matter how you structure your infrastructure shipping an update can bring down your system. You can make it less likely, but I don't know any company that never bring down their systems with bad updates. Google does it, Amazon does it, Microsoft does it etc.
  - bradhe 4y ago
    Your infrastructure should heal itself if it comes “down.” A decent release pipeline can fix this. Those companies don’t practice release moratoriums on Fridays.
    - erik_seaberg 4y ago
      Infra won’t amend or refund bad transactions that were committed, I’ve seen a few people have to burn their weekends recovering from that.
marcosdumay 4y ago
Software that is mostly used Monday to Friday doesn't get incident reports at the weekend! News at 11! Deploying when you have low usage has a lot of benefits. But the article is convinced that the difference is really because they are much better than anybody else.
boredumb 4y ago
Shipping code on Friday is bad. Shipping things after 4PM is also bad. Why hire any QA? Why code review at all? All of these reduce the risk of bad or unintended software being in the wild and not releasing on friday or after 4pm reduce the amount of time it can be in the wild without harassing people who are otherwise on their off time. You can have a half dozen full time QA people and a horde of reviewers merging in your request into a staging instance that triggers E2E tests that complete and require a manager to merge into master to be automatically deployed via the same docker image into a cluster and the only guarantees you have is that it may be absolutely correct and that it may have an issue.
- Hermitian909 4y ago
  Hear hear. Shipping at 10am Monday-Thursday is heaven, almost every single incident requiring immediate attention occurs during business hours.
  - indigodaddy 4y ago
    Absolutely, the benefits of this approach far outweigh the downsides— and/or benefits of other approaches.
- truncate 4y ago
  Also, I presume lot of people like Fridays to be bit more relaxed, over trying to do something stressful like a release. From a more pessimistic perspective, as times goes -- code will likely get more messy and you'll likely have more customers (assumings things are going good), in which case the fact incidents are not happening now, wont mean it wont start happening eventually.
  - warbeforepeace 4y ago
    I ship on friday if im not on call for the weekend.
- outworlder 4y ago
  It really doesn't matter how robust your internal processes are or how good your QA team is. Some issues will slip through, and will be seen at very inopportune times, after the system has had time to soak for a bit. By all means deploy to staging on Friday and have some automated and load testing running on weekends if that's what you want to do. But there's really no reason that promotion to prod should happen in a Friday or just before any significant time off. It really doesn't have much of a justification. You want to "feel good" about your work week and ship on a Friday? Ok great. How is that benefiting your customers? It's not urgent, it can wait until Monday or what have you.
pcdevils 4y ago
Wouldn't it make more sense to count the weekend as a whole? So 22 on the weekend, negating the point they're making. All an incident on a Sunday says in that data, presumably is, that as no one was working no one noticed the issue on the Saturday.
kerblang 4y ago
I tend to agree that ideally you want to have a the right processes in place so that you can ship anytime. I am annoyed every single Friday that I have a stack of things ready but... nope, not on Friday. Many of us are a long ways away from that ideal. I recommend that you start with "anytime as in normal working hours", which means having load-balancing/failover in production so that you are mildly degraded but no real outage. A lot of shops never get to this point and you'll find that it's awesome compared to the old "just stay up late and do it off-hours" routine that creates even more problems. The thing I like most is the ease of doing incremental daily rollouts instead of big-banging the shit out of everything at once because staying up late every night for a week is unrealistic. Maybe someday you'll get to the ideal peak of everything-automatic-regression-testing that is unrealistic for most of us, but normal working hours is a pretty good second best.
night-rider 4y ago
> there’s no greater motivator than getting paged at 3AM because your past self forgot to migrate prod Over time, working remotely, many of my team are in different time-zones. Some in London, New York, and L.A. which all have wildly different time-zones. I have to be sometimes like a cat (who are known to sleep a lot since they need to build up enough energy to pounce on wild animals without notice and do that straight away without hesitation). So I'm used to being paged in real-time to 'pounce' on critical issues. I'm on-call like a fireman. If anyone's working remotely here reading this, this is a skill that is learned.
sisve 4y ago
I really in favor of being able to deploy on Friday's. I do think that they make a good point in mastering it. But there is one important point I'm think they have learned but they do not explicit say in this blog post. It's what you deploy. Context matters. Deploy most things as normal on Friday's. But you do hold off the release that is bigger than normal and involves some new infra on the backend side or is known to more memory heavy etc. But for bug fixes etc. Deploy. Works great here at least. And I am, or at least for most of my career, was the person that fix the problem. You ship it you fix it. Then it's up to the person that has the PR. A judgment call
paxys 4y ago
Considering barely anyone is using their application on the weekends, ~22% of the total incident reports coming on Saturday and Sunday proves the exact opposite of what they are trying to say in this post.
- outworlder 4y ago
  Nice catch.
- antoinealb 4y ago
  A lot (most?) incidents in my experience don't come from load but rather through change, be it config change, new releases, flag flip, etc.
- andreygrehov 4y ago
  Not necessarily. If their software is barely used on the weekends, then shipping on Friday has a smaller blast radius in the event of a downtime / bad deployment, eg fewer customers are impacted. However, there is a chance their work-life-balance metrics are not great, since the engineers have to deal with the incidents during the weekend. Btw, 102 incidents within a week. Whoa! That's a lot. Wouldn't want to work there. To put it into perspective, here at AWS my team has about 3 incidents within a week.
  - infinityio 4y ago
    > Btw, 102 incidents within a week. Whoa! That's a lot. Wouldn't want to work there. I might be wrong but I'm pretty sure that's every incident they've ever documented, grouped by day, as opposed to the data from last week Their status page seems to suggest ~99.97% uptime (about 3h/year down) recently
    - andreygrehov 4y ago
      You are right. The title of the image says "Total incidence counts across all time, grouped by day".
  - 4y ago
jupp0r 4y ago
So the reason is to create a weekly artificial deadline to increase productivity and motivation? Additionally the increased pain of any problems that might arise will incentivize creating procedures that avoid problems. Not sure if creating this kind of artificial stress is actually beneficial long term for the team.
xwowsersx 4y ago
Off-topic, but one of the projects listed that uses Railway is https://www.slip.so/ https://www.slip.so/ which I had not seen before. I've been looking for exactly this!
- xwowsersx 4y ago
  oh hmm, maybe abandonware at this point? The Discord invite has expired and https://www.slip.so/courses https://www.slip.so/courses won't load (did before, but now not..)
  - lemoneyeo 4y ago
    I know the founder, he pivoted to something else. https://twitter.com/KennethCassel https://twitter.com/KennethCassel
    - xwowsersx 4y ago
      Oh cool, thanks for the reply. I emailed him as well. Maybe I'll try to build this myself, even if only to use for my own course! Thanks for the reply.
      - ndneighbor 4y ago
        Angelo here (Support Engineer at Railway and not the OP) Kenneth is dope, he used it to launch https://www.vim.so/ https://www.vim.so/ - with that said, our landing page is overdue for a refresh and along with it, new customer references.
deleted 4y ago
[deleted]
kc10 4y ago
> automate the necessary checks to minimize human error minimize is the keyword here. I assume you have seen regressions with Friday pushes and team had to spend their weekends resolving that at least once. People will do mistakes, bugs will go into code, regressions will happen. It's just inevitable and no amount of automation can cover all your user paths. People will hate this unpredictability to plan their weekends and spend a peaceful weekend. IMO, it's not worth it.
zabzonk 4y ago
what i used to do - wednesday, 7 am (or whenever trading starts). go home a bit early if all has gone well,
deleted 4y ago
[deleted]
erik_seaberg 4y ago
You earn the privilege of deploying late friday by demonstrating you can go many months in a row without ever blowing up somebody’s weekend (urgently amending bad transactions or whatever). This team doesn’t sound like they’re there.
wonderwonder 4y ago
This is not the argument they think it is. Dont ship on Friday and that 9 incidents on Saturday goes much lower, maybe to 0. The Sunday count probably does the same. The true cost of shipping on Friday is likely the sum of Saturday and Sunday. This sounds like a directive from management telling them to never stop shipping and they tried to wrap a nice story and some questionable stats around it to make the team think they have it good. Doubt they pay OT for working on the weekend to fix what was broken on Friday either.
- taeric 4y ago
  But your counter isn't necessarily saying what you think, either. A lot depends on what the response was. For example, if the response was an aggressive rollback, having lower load reduced the blast radius. And fair on OT for weekend work, but again, a lot depends on what was done to fix. If it was a rollback and retry next week after fixing any findings, this is a lot less bad than the high energy frustration of trying to rollback during peak hours.
  - wonderwonder 4y ago
    Is it? I hear your point but in this case you are just making up the "rollback and retry next week" response [which is fine]. You may be right though. Either way I would prefer to have to fix it during work hours when I am getting paid than having to tell my kid I have to step away from his Saturday birthday party for a few hours to fix something that my work made me deploy the day before. It would get to the point where you have to request weekends off for big events. Total lack of separation between work and life. Work would constantly bleed into everything as you expect a call at any time on the weekend robbing you of your ability to just relax. Cant drink too many beers or smoke a joint on Saturday [or whatever your preferred vice is], work might call. Sounds terrible.
    - taeric 4y ago
      Agreed on that. My point here was only that we would need more details. I could see this being better than you portrayed. I could also see it being worse. :)
semireg 4y ago
It also helps to publish to the Mac App Store on Friday because you can (should be the default) elect to rollout over a 7 day period (1,2,5,10,25,50,100%). When you do this you can catch an error over the weekend with a small percentage of users and reaching 100% means you got through the whole week. If you do a 7 day rollout on Monday then you’ll be hitting those huge chunks of users over the weekend which may run against rollout strategy. This is to say nothing of review times… which have been decently fast the last few years. But really, as a solo developer I ship on Fridays because it coincides with a weekend where I’ll have time to deal with issues in a low stress way (not as busy). My situation is somewhat hybrid because I publish to MAS and direct to my app’s website.
lemoneyeo 4y ago
>> maintain the lowest incident rate on Saturday I make my team stay off Slack on weekends. Unless there is major incident. So would not ship on Friday to risk having them fix the lowest incidents.
manv1 4y ago
Many of the larger IT orgs I know do releases on Tuesday - because nobody wants to deal with crap on the weekends. Everyone is around, and the system is actually being used. This kind of deployment also brings more discipline to the process, because you have to do the switchover when users are actually using the system. It means people are a lot more conscientious about making sure their plans actually work. We try to push on Monday afternoons, but it generally turns into Monday night. At some point your app doesn't/can't really have downtime, which is also a factor.
Shorel 4y ago
Where I work, releases are forbidden on a Friday.
xs83 4y ago
"I'll take 'reasons I would never want to work at Railway for $1000' please Alex"
pruthvishetty 4y ago
Could it be because most sprints end on a Friday?
alentred 4y ago
I ride my bike almost every (working) day for several decades and I can truly say I master it. And yet I wear a helmet every time - that's basic risk management. Yes, the shipping process must be automated, tested, have safeguards (acceptance tests and an automated rollback process). But it doesn't justify taking a risk of ruining a weekend for your entire team.

69 points by dban 4y ago

litoE 4y ago

You never ship on Friday because, to a programmer, a weekend is an infinite amount of time.

CobrastanJorji 4y ago

> But following that reasoning, why not also avoid shipping after 4PM? Yes, exactly! Don't ship after 4 PM either!

fooster 4y ago
Yes, only ship code to production every other Tuesday from 10:30-10:30am.
- lmm 4y ago
  I worked somewhere that did that and it was great. Everyone knew when the release was. If your code didn't make it into this one, well, no worries, it can go out in two weeks' time. And if something broke we had all day / all week to fix it.
  - throwaway98797 4y ago
    sounds awful for feature velocity
    - outworlder 4y ago
      Not all industries care about that, versus having fewer outages.
- forgetfreeman 4y ago
  Some folks just cannot be satisfied with a warning and have to stick the fork in the outlet for themselves.
0x457 4y ago
I think if you're afraid to ship after 4pm, then you should write better code and/or test it better. There are things I'm afraid to do late (say upgrading 3rd party software), but shipping a new version of my software? Don't see why not.
- alar44 4y ago
  Right, just write perfect code! Things can and do break. It's impossible to test everything. That's why.
- outworlder 4y ago
  > I think if you're afraid to ship after 4pm, then you should write better code and/or test it better. There are things I'm afraid to do late (say upgrading 3rd party software), but shipping a new version of my software? Don't see why not. Your wording seems to indicate that you work by yourself. That's fine, but things change when large teams and systems are involved. You can't vouch for every single change or the inter-dependencies between all systems. It's pretty elitist too. "Your code only breaks because you suck" doesn't make for good software engineering practices. Everyone makes mistakes.
  - 0x457 4y ago
    No, I don't work by myself. We are just good at writing tests and have a good QA team that writes good e2e tests.
  - lnxg33k1 4y ago
    That's why he wrote "test it better", it's intrinsic in that statement that he's well aware of the fallibility of humans
- opportune 4y ago
  Really depends on what you’re working on. Less complex things that are easily tested, sure. For more complex things that are hard to fully test, bad idea. If you have a micro service that takes requests over / foo and /bar and returns JSON that needs to fit certain properties, and you’re submitting a change to fix a corner case bug or add a new property that isn’t read, sure. If you are working on low level networking infrastructure that is used in tandem with thousands of different workloads you need to support, won’t work.
opportune 4y ago
I think what’s missing the most is context. I work on infrastructure and we never ship after 4 or on weekends because we have to support so many different kinds of behavior, and if we break any of those behaviors (which may very well be relying on undefined behavior/non-compliant protocol implementations) we are breaking our customers. For people who work on applications they are often only hurting themselves if things break, so the risk profile is different. Maybe you don’t even have any customers, maybe your customers are consumers who are just not able to place orders. IMO these groups often talk past each other (although as someone who works on infrastructure I think we often consider application developers’ more than they consider us, because they are our customers, and to them we are doing best when they don’t need to think about us) which leads to all these arguments from “Ship fast ship often - nobody uses our mostly static website which is easy to test, so who gives a fuck if it breaks” to “canary every change and carefully A/B test it with long release times to ensure it is safe”.
- CobrastanJorji 4y ago
  You know, that's a very good point. These arguments never bring up what's being built, what's the cost of things going wrong, etc. If you're running a Very Serious service with well-defined SLAs with a bunch of formal definitions and a suspicious number of 9s, your risk tolerance is entirely different from a startup with six highly caffeinated people who have naught but a dream and a three month financial runway before they're all out of a job. If devs from those two teams describe their release process to each other and don't mention what they do, they'll each probably conclude that the other is insane. One will vomit upon hearing the other describe a multi-week deployment scheme, and the other will vomit upon hearing the phrase "test in prod" said non-ironically.

mikl 4y ago

Is it just me, or wouldn’t their `git log | grep…` one-liner count commits to all branches that were eventually merged to master, not just direct commits to master? If so, unless they always squash when merging, this just shows that they have the most commits on Fridays, not deployments.

gschier 4y ago
We do squash commits to master, so there's only one commit per pull request. edit: I updated the image caption to specify. Thanks for pointing that out!

bradhe 4y ago

Not shipping on Fridays in 2023 is an infrastructure smell.

noman-land 4y ago
To me it seems like just good, clean living. If you have to deploy on Friday then you can, but if you don't have to, then it just seems like a good idea to do it while many people are paying attention instead of over a weekend when most people are off.
pcdevils 4y ago
Infra can make it easier to do it safer; it's not a panacea. Sure you can roll back; alert based on metrics / logs / error rates, but the knowledge required to automate around these data points tends to require the data to already exist, which won't be true with new features.
- bradhe 4y ago
  Okay so don't always ship on a Friday but that doesn't seem like rationale to put a blanket rule in place.
Jensson 4y ago
No matter how you structure your infrastructure shipping an update can bring down your system. You can make it less likely, but I don't know any company that never bring down their systems with bad updates. Google does it, Amazon does it, Microsoft does it etc.
- bradhe 4y ago
  Your infrastructure should heal itself if it comes “down.” A decent release pipeline can fix this. Those companies don’t practice release moratoriums on Fridays.
  - erik_seaberg 4y ago
    Infra won’t amend or refund bad transactions that were committed, I’ve seen a few people have to burn their weekends recovering from that.

marcosdumay 4y ago

Software that is mostly used Monday to Friday doesn't get incident reports at the weekend! News at 11! Deploying when you have low usage has a lot of benefits. But the article is convinced that the difference is really because they are much better than anybody else.

boredumb 4y ago

Shipping code on Friday is bad. Shipping things after 4PM is also bad. Why hire any QA? Why code review at all? All of these reduce the risk of bad or unintended software being in the wild and not releasing on friday or after 4pm reduce the amount of time it can be in the wild without harassing people who are otherwise on their off time. You can have a half dozen full time QA people and a horde of reviewers merging in your request into a staging instance that triggers E2E tests that complete and require a manager to merge into master to be automatically deployed via the same docker image into a cluster and the only guarantees you have is that it may be absolutely correct and that it may have an issue.

Hermitian909 4y ago
Hear hear. Shipping at 10am Monday-Thursday is heaven, almost every single incident requiring immediate attention occurs during business hours.
- indigodaddy 4y ago
  Absolutely, the benefits of this approach far outweigh the downsides— and/or benefits of other approaches.
truncate 4y ago
Also, I presume lot of people like Fridays to be bit more relaxed, over trying to do something stressful like a release. From a more pessimistic perspective, as times goes -- code will likely get more messy and you'll likely have more customers (assumings things are going good), in which case the fact incidents are not happening now, wont mean it wont start happening eventually.
- warbeforepeace 4y ago
  I ship on friday if im not on call for the weekend.
outworlder 4y ago
It really doesn't matter how robust your internal processes are or how good your QA team is. Some issues will slip through, and will be seen at very inopportune times, after the system has had time to soak for a bit. By all means deploy to staging on Friday and have some automated and load testing running on weekends if that's what you want to do. But there's really no reason that promotion to prod should happen in a Friday or just before any significant time off. It really doesn't have much of a justification. You want to "feel good" about your work week and ship on a Friday? Ok great. How is that benefiting your customers? It's not urgent, it can wait until Monday or what have you.

pcdevils 4y ago

Wouldn't it make more sense to count the weekend as a whole? So 22 on the weekend, negating the point they're making. All an incident on a Sunday says in that data, presumably is, that as no one was working no one noticed the issue on the Saturday.

kerblang 4y ago

I tend to agree that ideally you want to have a the right processes in place so that you can ship anytime. I am annoyed every single Friday that I have a stack of things ready but... nope, not on Friday. Many of us are a long ways away from that ideal. I recommend that you start with "anytime as in normal working hours", which means having load-balancing/failover in production so that you are mildly degraded but no real outage. A lot of shops never get to this point and you'll find that it's awesome compared to the old "just stay up late and do it off-hours" routine that creates even more problems. The thing I like most is the ease of doing incremental daily rollouts instead of big-banging the shit out of everything at once because staying up late every night for a week is unrealistic. Maybe someday you'll get to the ideal peak of everything-automatic-regression-testing that is unrealistic for most of us, but normal working hours is a pretty good second best.

night-rider 4y ago

> there’s no greater motivator than getting paged at 3AM because your past self forgot to migrate prod Over time, working remotely, many of my team are in different time-zones. Some in London, New York, and L.A. which all have wildly different time-zones. I have to be sometimes like a cat (who are known to sleep a lot since they need to build up enough energy to pounce on wild animals without notice and do that straight away without hesitation). So I'm used to being paged in real-time to 'pounce' on critical issues. I'm on-call like a fireman. If anyone's working remotely here reading this, this is a skill that is learned.

sisve 4y ago

I really in favor of being able to deploy on Friday's. I do think that they make a good point in mastering it. But there is one important point I'm think they have learned but they do not explicit say in this blog post. It's what you deploy. Context matters. Deploy most things as normal on Friday's. But you do hold off the release that is bigger than normal and involves some new infra on the backend side or is known to more memory heavy etc. But for bug fixes etc. Deploy. Works great here at least. And I am, or at least for most of my career, was the person that fix the problem. You ship it you fix it. Then it's up to the person that has the PR. A judgment call

paxys 4y ago

Considering barely anyone is using their application on the weekends, ~22% of the total incident reports coming on Saturday and Sunday proves the exact opposite of what they are trying to say in this post.

outworlder 4y ago
Nice catch.
antoinealb 4y ago
A lot (most?) incidents in my experience don't come from load but rather through change, be it config change, new releases, flag flip, etc.
andreygrehov 4y ago
Not necessarily. If their software is barely used on the weekends, then shipping on Friday has a smaller blast radius in the event of a downtime / bad deployment, eg fewer customers are impacted. However, there is a chance their work-life-balance metrics are not great, since the engineers have to deal with the incidents during the weekend. Btw, 102 incidents within a week. Whoa! That's a lot. Wouldn't want to work there. To put it into perspective, here at AWS my team has about 3 incidents within a week.
- infinityio 4y ago
  > Btw, 102 incidents within a week. Whoa! That's a lot. Wouldn't want to work there. I might be wrong but I'm pretty sure that's every incident they've ever documented, grouped by day, as opposed to the data from last week Their status page seems to suggest ~99.97% uptime (about 3h/year down) recently
  - andreygrehov 4y ago
    You are right. The title of the image says "Total incidence counts across all time, grouped by day".
- 4y ago

jupp0r 4y ago

So the reason is to create a weekly artificial deadline to increase productivity and motivation? Additionally the increased pain of any problems that might arise will incentivize creating procedures that avoid problems. Not sure if creating this kind of artificial stress is actually beneficial long term for the team.

xwowsersx 4y ago

Off-topic, but one of the projects listed that uses Railway is https://www.slip.so/ https://www.slip.so/ which I had not seen before. I've been looking for exactly this!

xwowsersx 4y ago
oh hmm, maybe abandonware at this point? The Discord invite has expired and https://www.slip.so/courses https://www.slip.so/courses won't load (did before, but now not..)
- lemoneyeo 4y ago
  I know the founder, he pivoted to something else. https://twitter.com/KennethCassel https://twitter.com/KennethCassel
  - xwowsersx 4y ago
    Oh cool, thanks for the reply. I emailed him as well. Maybe I'll try to build this myself, even if only to use for my own course! Thanks for the reply.
    - ndneighbor 4y ago
      Angelo here (Support Engineer at Railway and not the OP) Kenneth is dope, he used it to launch https://www.vim.so/ https://www.vim.so/ - with that said, our landing page is overdue for a refresh and along with it, new customer references.

deleted 4y ago

[deleted]

kc10 4y ago

> automate the necessary checks to minimize human error minimize is the keyword here. I assume you have seen regressions with Friday pushes and team had to spend their weekends resolving that at least once. People will do mistakes, bugs will go into code, regressions will happen. It's just inevitable and no amount of automation can cover all your user paths. People will hate this unpredictability to plan their weekends and spend a peaceful weekend. IMO, it's not worth it.

zabzonk 4y ago

what i used to do - wednesday, 7 am (or whenever trading starts). go home a bit early if all has gone well,

deleted 4y ago

[deleted]

erik_seaberg 4y ago

You earn the privilege of deploying late friday by demonstrating you can go many months in a row without ever blowing up somebody’s weekend (urgently amending bad transactions or whatever). This team doesn’t sound like they’re there.

wonderwonder 4y ago

This is not the argument they think it is. Dont ship on Friday and that 9 incidents on Saturday goes much lower, maybe to 0. The Sunday count probably does the same. The true cost of shipping on Friday is likely the sum of Saturday and Sunday. This sounds like a directive from management telling them to never stop shipping and they tried to wrap a nice story and some questionable stats around it to make the team think they have it good. Doubt they pay OT for working on the weekend to fix what was broken on Friday either.

taeric 4y ago
But your counter isn't necessarily saying what you think, either. A lot depends on what the response was. For example, if the response was an aggressive rollback, having lower load reduced the blast radius. And fair on OT for weekend work, but again, a lot depends on what was done to fix. If it was a rollback and retry next week after fixing any findings, this is a lot less bad than the high energy frustration of trying to rollback during peak hours.
- wonderwonder 4y ago
  Is it? I hear your point but in this case you are just making up the "rollback and retry next week" response [which is fine]. You may be right though. Either way I would prefer to have to fix it during work hours when I am getting paid than having to tell my kid I have to step away from his Saturday birthday party for a few hours to fix something that my work made me deploy the day before. It would get to the point where you have to request weekends off for big events. Total lack of separation between work and life. Work would constantly bleed into everything as you expect a call at any time on the weekend robbing you of your ability to just relax. Cant drink too many beers or smoke a joint on Saturday [or whatever your preferred vice is], work might call. Sounds terrible.
  - taeric 4y ago
    Agreed on that. My point here was only that we would need more details. I could see this being better than you portrayed. I could also see it being worse. :)

semireg 4y ago

It also helps to publish to the Mac App Store on Friday because you can (should be the default) elect to rollout over a 7 day period (1,2,5,10,25,50,100%). When you do this you can catch an error over the weekend with a small percentage of users and reaching 100% means you got through the whole week. If you do a 7 day rollout on Monday then you’ll be hitting those huge chunks of users over the weekend which may run against rollout strategy. This is to say nothing of review times… which have been decently fast the last few years. But really, as a solo developer I ship on Fridays because it coincides with a weekend where I’ll have time to deal with issues in a low stress way (not as busy). My situation is somewhat hybrid because I publish to MAS and direct to my app’s website.

lemoneyeo 4y ago

>> maintain the lowest incident rate on Saturday I make my team stay off Slack on weekends. Unless there is major incident. So would not ship on Friday to risk having them fix the lowest incidents.

manv1 4y ago

Many of the larger IT orgs I know do releases on Tuesday - because nobody wants to deal with crap on the weekends. Everyone is around, and the system is actually being used. This kind of deployment also brings more discipline to the process, because you have to do the switchover when users are actually using the system. It means people are a lot more conscientious about making sure their plans actually work. We try to push on Monday afternoons, but it generally turns into Monday night. At some point your app doesn't/can't really have downtime, which is also a factor.

Shorel 4y ago

Where I work, releases are forbidden on a Friday.

xs83 4y ago

"I'll take 'reasons I would never want to work at Railway for $1000' please Alex"

pruthvishetty 4y ago

Could it be because most sprints end on a Friday?

alentred 4y ago

I ride my bike almost every (working) day for several decades and I can truly say I master it. And yet I wear a helmet every time - that's basic risk management. Yes, the shipping process must be automated, tested, have safeguards (acceptance tests and an automated rollback process). But it doesn't justify taking a risk of ruining a weekend for your entire team.