5 ms·

Maybe I am just an old fuddy duddy conservative, but this struck me from the post: “In the grand scheme of things, one week isn’t that long. But to us, it felt

Maybe I am just an old fuddy duddy conservative, but this struck me from the post: “In the grand scheme of things, one week isn’t that long. But to us, it felt like forever. We are constantly iterating and release multiple changes every day”. I assume they mean multiple production releases? Is this because the product lacks maturity or stability, or is it just your culture? I am asking because I am trying to imagine the impact of this on existing customers. It sounds like an awful lot of churn. This obviously happens a lot in the “you are the product” space like Facebook, Google, etc. But this looks to be a data analytics product with paid tiers. Curious what tooling and processes you have to support this, and how you keep customers happy with this model.

mariosisters 4y ago
I think it’s that you are an old fuddy duddy :P Actually, if you work with SMBs/enterprises, I agree with you on customer facing changes. In my past life we would ship very frequently (often more than once a day) but always had to feature flag changes that large clients might see or be affected by. Even something as simple as tweaking the layout of a core flow could cause support headaches and angry customers — customers worth 10s of thousands of dollars per month. Is it worth losing a customer to CD a new button placement?
- chrisandchris 4y ago
  I can only image how clean code looks & works that is full of feature flags. Glad that I don't need to do that to often :)
  - tempest_ 4y ago
    They keep adding them until they need an internal library to manage their collection of feature flags. Then after a while they graduate to feature flags as a service (of which there are a bunch of cloud services trying to have a go at)
  - mariosisters 4y ago
    The right approach is to immediately remove the flags after rollout… The actual approach is to maintain a million fucking feature flags, ensuring that almost all possible combinations are essentially untested… better hope you did a good job separating concerns!
alrocar 4y ago
Yep, it's part of our culture, we do many releases per day to constantly iterate things. Also as in other projects there are maintenance and bug fixing we want to bring to production as soon as possible. Our context is the one of a startup that is constantly validating things, also in our context a release does not necessarily mean releasing to the users, sometimes stuff is behind feature flags or for beta testing.
drewcoo 4y ago
Hey there! I'm an old fuddy duddy, too. Continuous deployment has been around long enough that even IBM (remember never getting fired for buying IBM?) talks about it. https://www.ibm.com/topics/continuous-deployment https://www.ibm.com/topics/continuous-deployment "Dark deploys" and "feature flags" are often used to keep customers safe from incomplete features while still giving all of the advantages of CD plus allowing testing in production. I'd never heard of Flagship, but this is a nice writeup on that (kudos, Flagship.io): https://www.flagship.io/glossary/dark-launch/ https://www.flagship.io/glossary/dark-launch/
tetha 4y ago
Mh, I'm interacting with teams with wildly different release strategies and stability requirements at work. One of the more fundamental things actually pushing towards faster releases is what I call the relativistic deployment speed. We have products that will need at least 2 months to get a remotely deployable version ready. The average fast hotfix usually takes more like 4 months until an installation on a prod system actually can start. Our fastest products can go from code to prod in like 15 minutes with the automated tests being the bottleneck. This in turn shapes choices for the product managers, but also for security. If something like Log4shell hit these slow products, I'd have to plan to be vulnerable for two months at least, and usually more like 4 - 8 months depending on the customers. I have no choice, because that's their light speed of deployment. No code goes to prod faster than two months latency. That, quite frankly, fucking sucks. Other products were much better in that situation. We were lucky to have the right devs around, but we went from the decision to emergency log4shell at an utmost risky speed to the first log4shell patches in prod of many within 30 minutes. However, that's not the normal speed, and that's when you get into the second decision area. Given a lightspeed of deployment, how fast do you want to go? Some of our possibly faster moving products are B2B products, with a lot of internal training for support and consulting going into a release, and also training at customers happening for larger customers. This means, product chooses to only release bigger changes and heavily customer-visible changes every 6 weeks. They could do this a lot faster, but they choose to slow down because it fits their customers well. And for example, december is usually frozen entirely because customers want to. But then there is the third decision area. What happens if there is an entirely customer invisible change, such as an optimization in database handling, some internal metric generation for an optimization, or an internal change to prepare a new feature for the next scheduled rollout? And we have the tested, vetted and working option to just push that into prod without downtime, with also gives us opportunity to build experience with, and confidence into our no-downtime deployment system? I don't see a reason why I wouldn't exercise this daily at least once.
igetspam 4y ago
Read the State of DevOps reports over the years and you'll see why this is the direction we're all heading now. It turns out all that safety we thought we were building by making complex commit flows to multiple branches and environments was not only more complex than it needed to be but has also slowed us down and not made things better. Truly based development is back again and this time with data. Push early, push often, push small changes and iterate quickly. It's not just easier but it also seems to increase quality. (There are a lot of reasons why this turns out to be true. Read Accelerate. I won't do a better job explaining in a comment.)
scott_w 4y ago
This is considered the norm for high performing product teams in the modern day. We keep customers happy because we push changes live incrementally, reduce our chances of major outages and improve our response time when they do occur.
- hedora 4y ago
  As a customer, if I find a competitor that does not do this, then I will switch to it. For example, I cancelled my netflix subscription because they are unable to reliably operate microservices, and the UI was always in some semi-broken state. As a software engineer, this stressed me out during my relaxing TV time. Even if continuous delivery is somehow reliably delivered, if the changes are customer visible, then they break my muscle memory, and increase my cognitive load -- I have to re-learn the damned UI every fucking time I log in. If the changes are not customer visible, then what business value to they deliver?
  - scott_w 4y ago
    And yet the numbers show that companies that continuously delivery updates massively outperform those that don’t. You claim to make decisions based on a company’s engineering practices but I can guarantee that you have no idea about the engineering practices of any companies that you do support actually are.
    - deely3 4y ago
      Wait, can I see these numbers grouped by industries if possible? Please?
      - scott_w 4y ago
        Read any book on the subject, they’ve done the research for you. The Phoenix Project, The State of DevOps Report. Spend some time in companies that move slow vs fast and you’ll see the difference in their success first hand. You’ll see the metrics on their incidents and severity and customer satisfaction with them. Oh, and the fact two companies mentioned (Google and Facebook) are two of the most successful companies on earth.
recfab 4y ago
This just sounds like Continuous Delivery. We never achieved it in my last job, so I can't speak from experience, but my understanding is that typically "deploy" is separated from "release" using feature flags of some kind.
- drewcoo 4y ago
  It sounds like continuous deployment, not continuous delivery. Continuous deployment deploys code to production frequently, as soon as it's ready. Continuous delivery has some ready-to deliver branch that's constantly being updated as above, but they're not deployed to production until someone (Product Owner?) or something (Yay - end of sprint!) triggers it. Different people may use the word release for at least this many things: 1) a deployment, 2) an unveiling via feature flags, 3) a public announcement despite the code already having been live.
- Scubabear68 4y ago
  The article starts with “Last year, we made the difficult decision to stop deploying any changes to production for one week” and goes on to talk about releases. In that context I assume this means they make multiple production releases per day (which makes me shudder). I am curious how they do this while maintaining high quality and not driving customers insane.
  - fragmede 4y ago
    > multiple production releases per day Most (but not all) SaaS businesses are expected to these days, so I'm curious what your business/industry is that not only that you don't, but that it gives you the shudders.
    - Scubabear68 4y ago
      I consult across a few different industries, but it includes SaaS offerings. Many places could do multiple deploys to prod a day, but choose weekly releases or other longer cadences. This is to allow for documentation, client notifications, etc. It also is more efficient, constantly releasing requires a lot of resources, as this blog implies they are churning dozens of K8 pods several times a day. I would think constant releasing would also make debugging prod issues pure hell.
lelandfe 4y ago
That’s continuous delivery, right? You make great tests and you should feel comfortable releasing after review.
- mariosisters 4y ago
  In my experience and to parent’s point, it’s not about your comfort it’s about documenting, notifying clients, updating support, etc. All the non-code parts of selling software. As you suggest if the code has been reviewed, tested and merged, it “should” be ready to go. Right?

mariosisters 4y ago

I think it’s that you are an old fuddy duddy :P Actually, if you work with SMBs/enterprises, I agree with you on customer facing changes. In my past life we would ship very frequently (often more than once a day) but always had to feature flag changes that large clients might see or be affected by. Even something as simple as tweaking the layout of a core flow could cause support headaches and angry customers — customers worth 10s of thousands of dollars per month. Is it worth losing a customer to CD a new button placement?

chrisandchris 4y ago
I can only image how clean code looks & works that is full of feature flags. Glad that I don't need to do that to often :)
- tempest_ 4y ago
  They keep adding them until they need an internal library to manage their collection of feature flags. Then after a while they graduate to feature flags as a service (of which there are a bunch of cloud services trying to have a go at)
- mariosisters 4y ago
  The right approach is to immediately remove the flags after rollout… The actual approach is to maintain a million fucking feature flags, ensuring that almost all possible combinations are essentially untested… better hope you did a good job separating concerns!

alrocar 4y ago

Yep, it's part of our culture, we do many releases per day to constantly iterate things. Also as in other projects there are maintenance and bug fixing we want to bring to production as soon as possible. Our context is the one of a startup that is constantly validating things, also in our context a release does not necessarily mean releasing to the users, sometimes stuff is behind feature flags or for beta testing.

drewcoo 4y ago

Hey there! I'm an old fuddy duddy, too. Continuous deployment has been around long enough that even IBM (remember never getting fired for buying IBM?) talks about it. https://www.ibm.com/topics/continuous-deployment https://www.ibm.com/topics/continuous-deployment "Dark deploys" and "feature flags" are often used to keep customers safe from incomplete features while still giving all of the advantages of CD plus allowing testing in production. I'd never heard of Flagship, but this is a nice writeup on that (kudos, Flagship.io): https://www.flagship.io/glossary/dark-launch/ https://www.flagship.io/glossary/dark-launch/

tetha 4y ago

Mh, I'm interacting with teams with wildly different release strategies and stability requirements at work. One of the more fundamental things actually pushing towards faster releases is what I call the relativistic deployment speed. We have products that will need at least 2 months to get a remotely deployable version ready. The average fast hotfix usually takes more like 4 months until an installation on a prod system actually can start. Our fastest products can go from code to prod in like 15 minutes with the automated tests being the bottleneck. This in turn shapes choices for the product managers, but also for security. If something like Log4shell hit these slow products, I'd have to plan to be vulnerable for two months at least, and usually more like 4 - 8 months depending on the customers. I have no choice, because that's their light speed of deployment. No code goes to prod faster than two months latency. That, quite frankly, fucking sucks. Other products were much better in that situation. We were lucky to have the right devs around, but we went from the decision to emergency log4shell at an utmost risky speed to the first log4shell patches in prod of many within 30 minutes. However, that's not the normal speed, and that's when you get into the second decision area. Given a lightspeed of deployment, how fast do you want to go? Some of our possibly faster moving products are B2B products, with a lot of internal training for support and consulting going into a release, and also training at customers happening for larger customers. This means, product chooses to only release bigger changes and heavily customer-visible changes every 6 weeks. They could do this a lot faster, but they choose to slow down because it fits their customers well. And for example, december is usually frozen entirely because customers want to. But then there is the third decision area. What happens if there is an entirely customer invisible change, such as an optimization in database handling, some internal metric generation for an optimization, or an internal change to prepare a new feature for the next scheduled rollout? And we have the tested, vetted and working option to just push that into prod without downtime, with also gives us opportunity to build experience with, and confidence into our no-downtime deployment system? I don't see a reason why I wouldn't exercise this daily at least once.

igetspam 4y ago

Read the State of DevOps reports over the years and you'll see why this is the direction we're all heading now. It turns out all that safety we thought we were building by making complex commit flows to multiple branches and environments was not only more complex than it needed to be but has also slowed us down and not made things better. Truly based development is back again and this time with data. Push early, push often, push small changes and iterate quickly. It's not just easier but it also seems to increase quality. (There are a lot of reasons why this turns out to be true. Read Accelerate. I won't do a better job explaining in a comment.)

scott_w 4y ago

This is considered the norm for high performing product teams in the modern day. We keep customers happy because we push changes live incrementally, reduce our chances of major outages and improve our response time when they do occur.

hedora 4y ago
As a customer, if I find a competitor that does not do this, then I will switch to it. For example, I cancelled my netflix subscription because they are unable to reliably operate microservices, and the UI was always in some semi-broken state. As a software engineer, this stressed me out during my relaxing TV time. Even if continuous delivery is somehow reliably delivered, if the changes are customer visible, then they break my muscle memory, and increase my cognitive load -- I have to re-learn the damned UI every fucking time I log in. If the changes are not customer visible, then what business value to they deliver?
- scott_w 4y ago
  And yet the numbers show that companies that continuously delivery updates massively outperform those that don’t. You claim to make decisions based on a company’s engineering practices but I can guarantee that you have no idea about the engineering practices of any companies that you do support actually are.
  - deely3 4y ago
    Wait, can I see these numbers grouped by industries if possible? Please?
    - scott_w 4y ago
      Read any book on the subject, they’ve done the research for you. The Phoenix Project, The State of DevOps Report. Spend some time in companies that move slow vs fast and you’ll see the difference in their success first hand. You’ll see the metrics on their incidents and severity and customer satisfaction with them. Oh, and the fact two companies mentioned (Google and Facebook) are two of the most successful companies on earth.

recfab 4y ago

This just sounds like Continuous Delivery. We never achieved it in my last job, so I can't speak from experience, but my understanding is that typically "deploy" is separated from "release" using feature flags of some kind.

drewcoo 4y ago
It sounds like continuous deployment, not continuous delivery. Continuous deployment deploys code to production frequently, as soon as it's ready. Continuous delivery has some ready-to deliver branch that's constantly being updated as above, but they're not deployed to production until someone (Product Owner?) or something (Yay - end of sprint!) triggers it. Different people may use the word release for at least this many things: 1) a deployment, 2) an unveiling via feature flags, 3) a public announcement despite the code already having been live.
Scubabear68 4y ago
The article starts with “Last year, we made the difficult decision to stop deploying any changes to production for one week” and goes on to talk about releases. In that context I assume this means they make multiple production releases per day (which makes me shudder). I am curious how they do this while maintaining high quality and not driving customers insane.
- fragmede 4y ago
  > multiple production releases per day Most (but not all) SaaS businesses are expected to these days, so I'm curious what your business/industry is that not only that you don't, but that it gives you the shudders.
  - Scubabear68 4y ago
    I consult across a few different industries, but it includes SaaS offerings. Many places could do multiple deploys to prod a day, but choose weekly releases or other longer cadences. This is to allow for documentation, client notifications, etc. It also is more efficient, constantly releasing requires a lot of resources, as this blog implies they are churning dozens of K8 pods several times a day. I would think constant releasing would also make debugging prod issues pure hell.

lelandfe 4y ago

That’s continuous delivery, right? You make great tests and you should feel comfortable releasing after review.

mariosisters 4y ago
In my experience and to parent’s point, it’s not about your comfort it’s about documenting, notifying clients, updating support, etc. All the non-code parts of selling software. As you suggest if the code has been reviewed, tested and merged, it “should” be ready to go. Right?