3 ms·

The title is a bit click-baity and misleading. DevOps isn't dead it's just becoming centralized. The rub seems to be that quality managed services are widely

The title is a bit click-baity and misleading. DevOps isn't dead it's just becoming centralized. The rub seems to be that quality managed services are widely available and affordable enough to obviate the need for in-house DevOps.

toomuchtodo 10y ago
> The rub seems to be that quality managed services are widely available and affordable enough to obviate the need for in-house DevOps. I have yet to find developers who are throwing themselves at carrying a pager. DevOps isn't going away. Its just less relevant in smaller teams. Larger teams are always going to have an ops team.
- lisivka 10y ago
  DevOps does not need to carry a pager. DevOps will automate disaster recovery.
  - toomuchtodo 10y ago
    I'm a devops engineer, and I assure you, I have been on call for the last two years in my current position, attending to application and infrastructure alarms. All disaster recovery cannot be automated. A sanity check is required before your Elasticsearch or Redis clusters decide to f___ their datastore. EDIT: Or perhaps RDS is to your liking? We've had our replicas (off the same master) in AWS fail, at which point they would not further replicate. We needed to fail our app over to the master entirely, reduce its workload to not kill the master, terminate all of the replicas, relaunch all of the replicas, and then re-apply the workload back to the new replicas.
    - creshal 10y ago
      > All disaster recovery cannot be automated. See also: https://github.com/blog/1261-github-availability-this-week https://github.com/blog/1261-github-availability-this-week > The automated failover of our main production database could be described as the root cause of both of these downtime events. In each situation in which that occurred, if any member of our operations team had been asked if the failover should have been performed, the answer would have been a resounding no. There are many situations in which automated failover is an excellent strategy for ensuring the availability of a service. After careful consideration, we've determined that ensuring the availability of our primary production database is not one of these situations. To this end, we've made changes to our Pacemaker configuration to ensure failover of the 'active' database role will only occur when initiated by a member of our operations team.
      - toomuchtodo 10y ago
        Pinterest as well uses manual intervention: https://engineering.pinterest.com/blog/open-sourcing-pinterest-mysql-management-tools https://engineering.pinterest.com/blog/open-sourcing-pintere...
    - lisivka 10y ago
      All disaster recovery in general case cannot be automated, you are right. But it definitely can be automated to the point, when problem can wait till next Monday. To recover any disaster automagically, you need five things: your actual data, your working software, a working hardware, a working network connection, money. All that, except actual data, can be prepared ahead of time.
      - toomuchtodo 10y ago
        If your datastore can run in a degraded state through a weekend, its not a critical datastore (or you're shedding writes or making other consistency compromises).
        lisivka 10y ago
        If you have a single point of failure, then it is your fault.
tyingq 10y ago
>>DevOps isn't dead it's just becoming centralized. Should be fun for the Fortune 500's. Part of the justification of moving to the cloud was laying off the sysadmins and other people that did "DevOps" before it was called that. Once it's centralized, the human instinct to "ensure survival of the group" means the DevOps team will get bloated in headcount, budget, tools, etc. I assume that will be followed by some new wave of features called the "AI Cloud" or something that manages itself, allowing you to now lay off the new centralized DevOps team :)

toomuchtodo 10y ago

> The rub seems to be that quality managed services are widely available and affordable enough to obviate the need for in-house DevOps. I have yet to find developers who are throwing themselves at carrying a pager. DevOps isn't going away. Its just less relevant in smaller teams. Larger teams are always going to have an ops team.

lisivka 10y ago
DevOps does not need to carry a pager. DevOps will automate disaster recovery.
- toomuchtodo 10y ago
  I'm a devops engineer, and I assure you, I have been on call for the last two years in my current position, attending to application and infrastructure alarms. All disaster recovery cannot be automated. A sanity check is required before your Elasticsearch or Redis clusters decide to f___ their datastore. EDIT: Or perhaps RDS is to your liking? We've had our replicas (off the same master) in AWS fail, at which point they would not further replicate. We needed to fail our app over to the master entirely, reduce its workload to not kill the master, terminate all of the replicas, relaunch all of the replicas, and then re-apply the workload back to the new replicas.
  - creshal 10y ago
    > All disaster recovery cannot be automated. See also: https://github.com/blog/1261-github-availability-this-week https://github.com/blog/1261-github-availability-this-week > The automated failover of our main production database could be described as the root cause of both of these downtime events. In each situation in which that occurred, if any member of our operations team had been asked if the failover should have been performed, the answer would have been a resounding no. There are many situations in which automated failover is an excellent strategy for ensuring the availability of a service. After careful consideration, we've determined that ensuring the availability of our primary production database is not one of these situations. To this end, we've made changes to our Pacemaker configuration to ensure failover of the 'active' database role will only occur when initiated by a member of our operations team.
    - toomuchtodo 10y ago
      Pinterest as well uses manual intervention: https://engineering.pinterest.com/blog/open-sourcing-pinterest-mysql-management-tools https://engineering.pinterest.com/blog/open-sourcing-pintere...
  - lisivka 10y ago
    All disaster recovery in general case cannot be automated, you are right. But it definitely can be automated to the point, when problem can wait till next Monday. To recover any disaster automagically, you need five things: your actual data, your working software, a working hardware, a working network connection, money. All that, except actual data, can be prepared ahead of time.
    - toomuchtodo 10y ago
      If your datastore can run in a degraded state through a weekend, its not a critical datastore (or you're shedding writes or making other consistency compromises).
      - lisivka 10y ago
        If you have a single point of failure, then it is your fault.

tyingq 10y ago

>>DevOps isn't dead it's just becoming centralized. Should be fun for the Fortune 500's. Part of the justification of moving to the cloud was laying off the sysadmins and other people that did "DevOps" before it was called that. Once it's centralized, the human instinct to "ensure survival of the group" means the DevOps team will get bloated in headcount, budget, tools, etc. I assume that will be followed by some new wave of features called the "AI Cloud" or something that manages itself, allowing you to now lay off the new centralized DevOps team :)