10 ms·

> If you’re a Software Engineer/Developer, then consider that a service (at least, for me), is a piece of code running in a live production system, that YOU wro

by chrsig 4y ago

> If you’re a Software Engineer/Developer, then consider that a service (at least, for me), is a piece of code running in a live production system, that YOU wrote, only YOU know how it works, thus YOU own. I've grown unfond of this attitude. I most certainly don't own it. I have no IP rights to it at all. We're both being paid to solve different facets of the same problem. Coming at me with "this is your problem" isn't going to foster a collaborative environment with me. Which is much more pleasant than an adversarial environment. Also: I'm not the only one that knows how it works, it's been peer reviewed in no small part to reduce my bus factor. All documentation requested is perfectly reasonable, and should be part of the organizations standard operating procedure. If it's not part of the SOP, then no, you wont have those things. You need to work at a cultural level to change that, and for that you're much better off making allies than anything else. Make it clear how those things help you, and what you'll do to make the developers life easier when you don't need to worry about the basics. If altruism fails you, you can usually count on people to act in their own best interests.

chidog12 4y ago
"I've grown unfond of this attitude. I most certainly don't own it. I have no IP rights to it at all. We're both being paid to solve different facets of the same problem." If you are a dev on the team that owns that service then it's you and your team's responsibility to answer all of these questions... Even Org's SOP would end up reaching back to the team who owns the service if problem's arises...
- hayst4ck 4y ago
  When you give responsibility to teams themselves the result is O(1) size problems becoming O(teams) sized problems. > How can I check the health of the service? In the definition of service, you define a field for health check script. > How can I safely and gracefully restart the service? This will exist within the script used to push new code. > Does it has any external dependencies? This could be defined in the service configuration and used for setting up integration tests and automatically generating a dependency dashboard. > Do you have a playbook, or sequence of steps, to bring the service back up? You could generate a field in the service defintion to automatically generate a dashboard and include the playbook link at the top of the page. > Do you use appropriate logging levels depending on the environments? Production could be extremely opinionated about what acceptable logging looks like, forced via code review. Log level could be defined in service config. > Are you logging to stdout? Why would any production service get to choose? Service owners shouldn't be able to log into machines. > Are you measuring the RED signals? Required fields in service config that could be used to generate a service dashboard. > Is there any documentation/design specification for the service? Required config field. > Are you using gRPC or REST? Trivial grep. > How does the data flow through the service? This is complicated, but can probably be easily replaced by asking what state your service keeps and how it's stored. This is the only question I think the author should/needs to ask. > Do you have any PII/Sensitive data flowing through the service? While this question is important, this is one of the problems that has to be a particular person's responsibility. Any dev that answers anything but "probably not, but I don't know" shouldn't be trusted. > What is the testing coverage for this service? Some form of this would exist in a service config. I don't think the question of responsibility is as simple as "it's the team's problem."
  - dopidopHN 4y ago
    Hey I see “service config” referenced a lot in that thread, but your answers has the more occurrences. I’m not sure I follow what it is. A technical construct, like a code template or a API that services implements ? Or a process constructs, like a SOP to follow with checkboxes? Thanks
    - hayst4ck 4y ago
      succinctly: A service config is the authoritative source of truth for what a service is in a format that can be (is) consumed by tooling. A lot of software development is about generating abstractions. "Service" is a possible abstraction someone might want to generate and develop. I think a service abstraction can be defined by: A blob of code A set of machines to run it on A way to stop and start it A method to load balance to it So it would make sense to create a yaml config file committed to a repo containing something like: services: [ { name: "CoolAppServerName.prod", build_script: "./bin/buildCoolAppServerName.py", start_script: "./bin/startCoolAppServerName.py", stop_script: "./bin/stopCoolAppServerName.py", hosts:[ "host_1", "host_2", ], slb_name: "CoolAppServerName.prod", }, {...}, ] Once you have a definition, it can be extended to meet growing needs. You might choose to do something like: { name: "CoolAppServerName.prod", key_metrics: [ "CoolAppServerName.prod.5xx", "CoolAppServerName.prod.latency_percentiles", ], owner: "CoolTeam", ..., } And then you could generate a webpage with a dropdown where "CoolAppServerName.prod" is an option and the dashboard including graphs for the time series metrics "CoolAppServerName.prod.5xx" and "CoolAppServerName.prod.latency_percentiles" automatically show up. Maybe instead of having service names in the dropdown you have owner names in the dropdown. You could potentially write some code that attempts to validate no significant changes in those metrics and use it to automatically verify that newly pushed code didn't take down the website. Service config means creating an authoritative service identifier (authoritative because it's the only identifier used in tooling) and then attaching a configuration to it. Facebook and google have (or at least at some point had) tupperware and borg respectively, that are basically custom verisons of the above extended for their infrastructures.
- jamesfinlayson 4y ago
  Agreed - in a previous job people shipped garbage code frequently and when there was a problem they didn't want to hear about it, because "everyone owned the code".
  - waynesonfire 4y ago
    How else would you get promoted? Get with the program.
- Jochim 4y ago
  I hate separate infrastructure teams with a passion for this reason. Far too frequently you end up in a situation where someone makes an environment change and blows everything up because they have no understanding of the services they're stewarding. If you want me to take responsibility, my team should be managing the service end to end. I feel really strongly against this division of responsibility in software teams. It too often leads to holding up progress and hostile interactions due to each team pursuing their own priorities.
  - namaria 4y ago
    All counsel these days, from 'devops' or 'sre' bodies of knowledge is: development and operations are two sides of the same system, they should be integrated better. Companies: got it, create new title/team, in charge of this integration. Seriously?
  - Aeolun 4y ago
    > If you want me to take responsibility, my team should be managing the service end to end. This. I really do not enjoy being called up in the middle of the night to walk a group of people that know absolutely nothing about the system through the steps they need to resolve the issue, because nobody wants to give the “dev” team access to the production environment. I think the solution that best aligns incentives is the one where the people introducing issues are also the ones called up (and able) to fix them.
    - freeone3000 4y ago
      Ah, developers empowered to do operations. We should have a catchy name for it... "opsdevs"? :P Seriously, this is the original idea of the DevOps principals. But they run straight into CIS requirement that "developers do not have access to production code" and the ISO 27001 v2013 requirement of separation of responsibilities. So it'd be great if it happens, it just can't happen in the big B2B spaces.
      - sethammons 4y ago
        We allow devs to do things in prod. We are a public company. Sox, Hipaa, ISO27001, GDPR, and all that. Every dev on my team has access to their prod servers and databases (but no access to other team's stuff usually). We deploy multiple times a day. We handle our own oncall. We process billions of individual requests daily for millions of users. We have several thousand employees. Our compliance requires that all code be reviewed and pass quality assurance before merging and that all prod changes be documented. That means Dev1 writes the code, the unit and integration tests, sets the right configs in each environment, updates the dashboards for any updated metrics, sets up alerts, and updates runbooks. Dev2 reviews the work, pushes back when any of the above needs more work, and then documents on the jira ticket how they verified stuff. Dev1 or Dev2 merge the code, observe the build, and ensure the code rolls out to prod. When something goes wrong, the oncall dev on the team is paged and can access all prod systems, and can log in, start and kill things, move files, etc.
hayst4ck 4y ago
Ownership is important to be explicit about, less as a means of assigning blame, but more as a means of coordination and resource allocation. The author is using ownership as a tool to avoid responsibility, and is thus creating an `us vs them` mindset rather than an `us vs the problem` mindset. Having a strong definition of ownership (like committing your organizational structure to your monorepo as config file) is invaluable for building tooling. If you have a strong definition of code ownership it allows for things like people less familiar with a particular piece of code being able to make changes with the approval of the owners, while simultaneously notifying them of the change. Likewise, if you are working on a platform that multiple teams use, you can write tooling that automatically assigns bugs or tickets. Ownership problems and "us vs them" is a clear sign of poor leadership. Most devs that experience it become cyncial or hostile without being able to understand that it is leadership that failed them.
- atmosx 4y ago
  > Having a strong definition of ownership (like committing your organizational structure to your monorepo as config file) is invaluable for building tooling. Having a strong ownership can become toxic very quickly.
  - hayst4ck 4y ago
    Ownership is not toxic. Leadership is toxic. It is really important to understand that toxicity and hostility is a function of leadership (or lack of it). I highly recommend the book Extreme Ownership. That book explains the mechanics of toxic environments.
scarface74 4y ago
> We're both being paid to solve different facets of the same problem Code that is not running correctly in production is worthless. If you write code and haven’t thought about all of the implications that it takes to make it run correctly you haven’t produce business value. Yes, I have been a developer for 25+ years professionally. But for the last 10, I’ve also thought through all of the topics that the author has delineated in his article. Yes I consider myself to be an competent “DevOps” engineer as long as it is on AWS and can go from empty AWS account to a fully functional infrastructure using IAC, a CI/CD pipeline, monitoring, alerting, centralizing logging etc. Knowing that I will either be the person doing the “DevOps” or working with the person who is informs the design of my development.
- lmm 4y ago
  > Code that is not running correctly in production is worthless. If you write code and haven’t thought about all of the implications that it takes to make it run correctly you haven’t produce business value. I wish this were true, but the number of billion dollar companies running code that doesn't run correctly shows it isn't.
  - patmcc 4y ago
    If it brings in revenue it's running correctly, as far as the business is concerned.
    - lmm 4y ago
      Well if you want to use that definition then my point is that "correct" in that sense doesn't (necessarily) require any thinking through.
Galanwe 4y ago
Agree 100% Although it's sane that developers keep contact with the reality of maitaining the live applications they wrote, it just doesn't scale to ask them to fully support them. There is an infinite amount of maintenance for any live system. No service of any magnitude just "works" in production indefinitely, at the very least because this service interacts with others that will fail. If developers are responsible for every live system they publish, they will get locked on after a finite amount of service they published, and leave because of maintenance boredom. There needs to be a reasonable amount of documentation written, explanations given, level 2 support taken, but that's it, the maintenance is for ops.
- Spivak 4y ago
  I don’t know what you think is magically different about an infra person who can apparently maintain arbitrary numbers of services and is somehow immune to the same boredom you fall under. And look, infra people are harder to come by and more expensive than devs so no company wants to waste our time bumping dependencies and fixing non-infra related bugs. If you’re big enough to have a team or teams of maintainers then they would go there. If not then it’s ship it maintain it.
- quickthrower2 4y ago
  This is only true for badly written or architected systems. Usually it is a team. If you move around different teams you are probably only maintaining $currentteams apps, not everything you touched. As such any team should ensure integrity of it’s code regardless of people coming in and out. Hiring, levels, code reviews etc. help this. Bunch of grads with no source control and write code directly in production without review would be the opposite. SRE is a hat not a title or a team at small companies so this is how it has to be done. Also AWS probably works this way from what I have read. The team like a mini business.
klodolph 4y ago
> We're both being paid to solve different facets of the same problem. Coming at me with "this is your problem" isn't going to foster a collaborative environment with me. SRE here. My takeaway from this is: If you want SRE support running this service, then you need to provide SREs with knowledge of how the system works. As long as only the devs have this knowledge, it's a bit unfair to put the SREs on the hook for supporting it. Maybe I'm reading between the lines too much--the wording in the article is sloppy at best, and at worst, it doesn't actually say what I'm saying. It's nice that your code has been through peer review and other people on your team know how it works too. That's less helpful for the SREs running it. SREs bear the burden of the pager--sometimes getting woken up at odd hours of the night to fix problems that were, in a sense, created by developers. The SOP for getting SRE support for new services should include things like runbooks and design reviews. SREs should be in the loop when you figure out what metrics to expose from your service, because SREs will be the ones using those metrics to figure out the alerting systems. Very few companies have decent "SOP" for SRE support--there are a few companies which are really good at it, like Google, and then a long tail of companies which dump services on SREs without including SREs in the process. IMO--the right thing to do is to give SRE teams the power to say "no" and refuse to take the pager for any particular service, barring exceptional circumstances. There's a deeper discussion to be had about why this should be the case--basically, devs and SREs have different incentives, and neither team should be put in a subordinate position to the other, because both teams have goals that support the business.
- minaguib 4y ago
  It's interesting that this: > the right thing to do is to give SRE teams the power to say "no" and refuse to take the pager for any particular service Was a tenet in the original Google SRE material. SREs help operate well-behaving services using engineering best-practices. Services that fail to behave well and bust their error and support budgets repeatedly go back to the teams that wrote them.
  - GauntletWizard 4y ago
    This is important to stress. SRE often takes first-line oncall, but "We" support the service. On a well behaved service, having SRE support is an extra debugging-focused engineer when things go wrong. But that engineer is rarely called up, and can support the common parts of failure (network issues, hardware failures, the things that are out of any one team's service team's control) for multiple teams.
_8j50 4y ago
"Own" in this context means be responsible for. Peer review does not transfer ownership to peers but it does reduce risk of your code breaking. Other people should not understand your own code (and intent) more than you, so long as that is correct, while everyone in dev/ops shares responsibility of delivering service, you own and are responsible for the part of that service delivery that you authored simply because you are the person most qualified to resolve any issue that arise from that code breaking. If an SRE is keeping your code running then they own the code's uptime in as much as you have been able to communicate the parameters and configuration of the app, but at the end of the day, when there is a bug in the code you are the best person to fix it, so you own it.
- chrsig 4y ago
  > "Own" in this context means be responsible for. I understand this, this is what I was trying to direct my distaste at. Unfortunately, it seems to have been misconstrued by responders. > you own and are responsible for the part of that service delivery No, I'm just responsible for that part of the service delivery. I own none of it. I object to it in the same way that I object to hearing "we're a family" from leadership.
  - _8j50 4y ago
    So long as your bosses are ok with that, who cares. Power to you. I was just stating the obvious typical expectation I guess.
runamok 4y ago
It's a question of aligned incentives. If your team (not you necessarily) only have a priority to ship features than your team needs to be alerted for performance and reliability issues because you are thus incentivized to fix said issues. If these issues are thrown over the fence to some other team who doesn't know your codebase and have the domain knowledge to improve said code then things (usually) never get fixed. Likewise error budgets that say "your service has not been reliable enough this period so the next sprint will be dedicated to improving that and no new features will ship" is another way to make sure quality is not an afterthought.
disruptiveink 4y ago
Isn't this (you devs own your app, I don't give a shit about it, I just want it to not page me) the classic old school "Sysadmin" approach that supposedly DevOps was supposed to counter? How can anyone say this and claim to be doing anything remotely near DevOps?
- jkukul 4y ago
  I second this. DevOps was supposed to result in developers also being responsible for deploying and monitoring their applications. It's either a DevOps culture or a classical Dev + SysAdmin setup with clear boundaries and where a developed application is being thrown over the fence to SysAdmins. The latter will always result in mutual animosities between the two parties.
  - fragmede 4y ago
    It's part of the deal for modern SaaS software deployment. If the devs want to deploy more often than once a quarter, then they need to be mindful of the operational burden they impose on the ops team. The way to drive that home is for them to carry the pager part of the time.
  - pards 4y ago
    > DevOps was supposed to result in developers also being responsible for deploying and monitoring their applications No, it was supposed to be dev and ops collaborating with mutual respect and empathy.
bambax 4y ago
> If it's not part of the SOP, then no, you wont have those things Isn't this "adversarial" as well? Why would you withhold that information just because the SOP don't make you provide it? What will happen then is that eventually the service will break, nobody will know how to fix it, and they will come and ask you. If you're no longer there, the service will be decommissioned and all your work will have been in vain. I don't see a net benefit for any of this.
- chrsig 4y ago
  I didn't mean it adversarially, but I can see how you came to that conclusion. Communication is hard. I was trying to convey that they wouldn't exist because it hadn't been asked for. People aren't mind readers, ops has to vocalize their needs. Which is what the post was doing. And I don't think those needs are unreasonable, either. My point was more that the work to make sure those things are available needs to happen higher up than you'll be able to reach by talking to any individual developer.
  - bambax 4y ago
    Aaah, ok! Sorry for the misunderstanding, and thanks for the explanation. (I was mostly in agreement with the OP and was surprised by the level of opposition to it in the comments.) Yes, it's better if the information asked for in the original post is in the SOP, but if it's not, then asking for it each time somebody sends something to production, while suboptimal, is better than nothing IMHO.
treeman79 4y ago
Dealt with this at a company. Eventually the developers just gave up on all development work and became operations. The actual operations team kept k8 cluster alive. Developers had to do everything else. Eventually people would get the hang of the operations side but by that point they were burnt out and quit.

by chrsig 4y ago

chidog12 4y ago

"I've grown unfond of this attitude. I most certainly don't own it. I have no IP rights to it at all. We're both being paid to solve different facets of the same problem." If you are a dev on the team that owns that service then it's you and your team's responsibility to answer all of these questions... Even Org's SOP would end up reaching back to the team who owns the service if problem's arises...

hayst4ck 4y ago
When you give responsibility to teams themselves the result is O(1) size problems becoming O(teams) sized problems. > How can I check the health of the service? In the definition of service, you define a field for health check script. > How can I safely and gracefully restart the service? This will exist within the script used to push new code. > Does it has any external dependencies? This could be defined in the service configuration and used for setting up integration tests and automatically generating a dependency dashboard. > Do you have a playbook, or sequence of steps, to bring the service back up? You could generate a field in the service defintion to automatically generate a dashboard and include the playbook link at the top of the page. > Do you use appropriate logging levels depending on the environments? Production could be extremely opinionated about what acceptable logging looks like, forced via code review. Log level could be defined in service config. > Are you logging to stdout? Why would any production service get to choose? Service owners shouldn't be able to log into machines. > Are you measuring the RED signals? Required fields in service config that could be used to generate a service dashboard. > Is there any documentation/design specification for the service? Required config field. > Are you using gRPC or REST? Trivial grep. > How does the data flow through the service? This is complicated, but can probably be easily replaced by asking what state your service keeps and how it's stored. This is the only question I think the author should/needs to ask. > Do you have any PII/Sensitive data flowing through the service? While this question is important, this is one of the problems that has to be a particular person's responsibility. Any dev that answers anything but "probably not, but I don't know" shouldn't be trusted. > What is the testing coverage for this service? Some form of this would exist in a service config. I don't think the question of responsibility is as simple as "it's the team's problem."
- dopidopHN 4y ago
  Hey I see “service config” referenced a lot in that thread, but your answers has the more occurrences. I’m not sure I follow what it is. A technical construct, like a code template or a API that services implements ? Or a process constructs, like a SOP to follow with checkboxes? Thanks
  - hayst4ck 4y ago
    succinctly: A service config is the authoritative source of truth for what a service is in a format that can be (is) consumed by tooling. A lot of software development is about generating abstractions. "Service" is a possible abstraction someone might want to generate and develop. I think a service abstraction can be defined by: A blob of code A set of machines to run it on A way to stop and start it A method to load balance to it So it would make sense to create a yaml config file committed to a repo containing something like: services: [ { name: "CoolAppServerName.prod", build_script: "./bin/buildCoolAppServerName.py", start_script: "./bin/startCoolAppServerName.py", stop_script: "./bin/stopCoolAppServerName.py", hosts:[ "host_1", "host_2", ], slb_name: "CoolAppServerName.prod", }, {...}, ] Once you have a definition, it can be extended to meet growing needs. You might choose to do something like: { name: "CoolAppServerName.prod", key_metrics: [ "CoolAppServerName.prod.5xx", "CoolAppServerName.prod.latency_percentiles", ], owner: "CoolTeam", ..., } And then you could generate a webpage with a dropdown where "CoolAppServerName.prod" is an option and the dashboard including graphs for the time series metrics "CoolAppServerName.prod.5xx" and "CoolAppServerName.prod.latency_percentiles" automatically show up. Maybe instead of having service names in the dropdown you have owner names in the dropdown. You could potentially write some code that attempts to validate no significant changes in those metrics and use it to automatically verify that newly pushed code didn't take down the website. Service config means creating an authoritative service identifier (authoritative because it's the only identifier used in tooling) and then attaching a configuration to it. Facebook and google have (or at least at some point had) tupperware and borg respectively, that are basically custom verisons of the above extended for their infrastructures.
jamesfinlayson 4y ago
Agreed - in a previous job people shipped garbage code frequently and when there was a problem they didn't want to hear about it, because "everyone owned the code".
- waynesonfire 4y ago
  How else would you get promoted? Get with the program.
Jochim 4y ago
I hate separate infrastructure teams with a passion for this reason. Far too frequently you end up in a situation where someone makes an environment change and blows everything up because they have no understanding of the services they're stewarding. If you want me to take responsibility, my team should be managing the service end to end. I feel really strongly against this division of responsibility in software teams. It too often leads to holding up progress and hostile interactions due to each team pursuing their own priorities.
- namaria 4y ago
  All counsel these days, from 'devops' or 'sre' bodies of knowledge is: development and operations are two sides of the same system, they should be integrated better. Companies: got it, create new title/team, in charge of this integration. Seriously?
- Aeolun 4y ago
  > If you want me to take responsibility, my team should be managing the service end to end. This. I really do not enjoy being called up in the middle of the night to walk a group of people that know absolutely nothing about the system through the steps they need to resolve the issue, because nobody wants to give the “dev” team access to the production environment. I think the solution that best aligns incentives is the one where the people introducing issues are also the ones called up (and able) to fix them.
  - freeone3000 4y ago
    Ah, developers empowered to do operations. We should have a catchy name for it... "opsdevs"? :P Seriously, this is the original idea of the DevOps principals. But they run straight into CIS requirement that "developers do not have access to production code" and the ISO 27001 v2013 requirement of separation of responsibilities. So it'd be great if it happens, it just can't happen in the big B2B spaces.
    - sethammons 4y ago
      We allow devs to do things in prod. We are a public company. Sox, Hipaa, ISO27001, GDPR, and all that. Every dev on my team has access to their prod servers and databases (but no access to other team's stuff usually). We deploy multiple times a day. We handle our own oncall. We process billions of individual requests daily for millions of users. We have several thousand employees. Our compliance requires that all code be reviewed and pass quality assurance before merging and that all prod changes be documented. That means Dev1 writes the code, the unit and integration tests, sets the right configs in each environment, updates the dashboards for any updated metrics, sets up alerts, and updates runbooks. Dev2 reviews the work, pushes back when any of the above needs more work, and then documents on the jira ticket how they verified stuff. Dev1 or Dev2 merge the code, observe the build, and ensure the code rolls out to prod. When something goes wrong, the oncall dev on the team is paged and can access all prod systems, and can log in, start and kill things, move files, etc.

hayst4ck 4y ago

Ownership is important to be explicit about, less as a means of assigning blame, but more as a means of coordination and resource allocation. The author is using ownership as a tool to avoid responsibility, and is thus creating an `us vs them` mindset rather than an `us vs the problem` mindset. Having a strong definition of ownership (like committing your organizational structure to your monorepo as config file) is invaluable for building tooling. If you have a strong definition of code ownership it allows for things like people less familiar with a particular piece of code being able to make changes with the approval of the owners, while simultaneously notifying them of the change. Likewise, if you are working on a platform that multiple teams use, you can write tooling that automatically assigns bugs or tickets. Ownership problems and "us vs them" is a clear sign of poor leadership. Most devs that experience it become cyncial or hostile without being able to understand that it is leadership that failed them.

atmosx 4y ago
> Having a strong definition of ownership (like committing your organizational structure to your monorepo as config file) is invaluable for building tooling. Having a strong ownership can become toxic very quickly.
- hayst4ck 4y ago
  Ownership is not toxic. Leadership is toxic. It is really important to understand that toxicity and hostility is a function of leadership (or lack of it). I highly recommend the book Extreme Ownership. That book explains the mechanics of toxic environments.

scarface74 4y ago

> We're both being paid to solve different facets of the same problem Code that is not running correctly in production is worthless. If you write code and haven’t thought about all of the implications that it takes to make it run correctly you haven’t produce business value. Yes, I have been a developer for 25+ years professionally. But for the last 10, I’ve also thought through all of the topics that the author has delineated in his article. Yes I consider myself to be an competent “DevOps” engineer as long as it is on AWS and can go from empty AWS account to a fully functional infrastructure using IAC, a CI/CD pipeline, monitoring, alerting, centralizing logging etc. Knowing that I will either be the person doing the “DevOps” or working with the person who is informs the design of my development.

lmm 4y ago
> Code that is not running correctly in production is worthless. If you write code and haven’t thought about all of the implications that it takes to make it run correctly you haven’t produce business value. I wish this were true, but the number of billion dollar companies running code that doesn't run correctly shows it isn't.
- patmcc 4y ago
  If it brings in revenue it's running correctly, as far as the business is concerned.
  - lmm 4y ago
    Well if you want to use that definition then my point is that "correct" in that sense doesn't (necessarily) require any thinking through.

Galanwe 4y ago

Agree 100% Although it's sane that developers keep contact with the reality of maitaining the live applications they wrote, it just doesn't scale to ask them to fully support them. There is an infinite amount of maintenance for any live system. No service of any magnitude just "works" in production indefinitely, at the very least because this service interacts with others that will fail. If developers are responsible for every live system they publish, they will get locked on after a finite amount of service they published, and leave because of maintenance boredom. There needs to be a reasonable amount of documentation written, explanations given, level 2 support taken, but that's it, the maintenance is for ops.

Spivak 4y ago
I don’t know what you think is magically different about an infra person who can apparently maintain arbitrary numbers of services and is somehow immune to the same boredom you fall under. And look, infra people are harder to come by and more expensive than devs so no company wants to waste our time bumping dependencies and fixing non-infra related bugs. If you’re big enough to have a team or teams of maintainers then they would go there. If not then it’s ship it maintain it.
quickthrower2 4y ago
This is only true for badly written or architected systems. Usually it is a team. If you move around different teams you are probably only maintaining $currentteams apps, not everything you touched. As such any team should ensure integrity of it’s code regardless of people coming in and out. Hiring, levels, code reviews etc. help this. Bunch of grads with no source control and write code directly in production without review would be the opposite. SRE is a hat not a title or a team at small companies so this is how it has to be done. Also AWS probably works this way from what I have read. The team like a mini business.

klodolph 4y ago

> We're both being paid to solve different facets of the same problem. Coming at me with "this is your problem" isn't going to foster a collaborative environment with me. SRE here. My takeaway from this is: If you want SRE support running this service, then you need to provide SREs with knowledge of how the system works. As long as only the devs have this knowledge, it's a bit unfair to put the SREs on the hook for supporting it. Maybe I'm reading between the lines too much--the wording in the article is sloppy at best, and at worst, it doesn't actually say what I'm saying. It's nice that your code has been through peer review and other people on your team know how it works too. That's less helpful for the SREs running it. SREs bear the burden of the pager--sometimes getting woken up at odd hours of the night to fix problems that were, in a sense, created by developers. The SOP for getting SRE support for new services should include things like runbooks and design reviews. SREs should be in the loop when you figure out what metrics to expose from your service, because SREs will be the ones using those metrics to figure out the alerting systems. Very few companies have decent "SOP" for SRE support--there are a few companies which are really good at it, like Google, and then a long tail of companies which dump services on SREs without including SREs in the process. IMO--the right thing to do is to give SRE teams the power to say "no" and refuse to take the pager for any particular service, barring exceptional circumstances. There's a deeper discussion to be had about why this should be the case--basically, devs and SREs have different incentives, and neither team should be put in a subordinate position to the other, because both teams have goals that support the business.

minaguib 4y ago
It's interesting that this: > the right thing to do is to give SRE teams the power to say "no" and refuse to take the pager for any particular service Was a tenet in the original Google SRE material. SREs help operate well-behaving services using engineering best-practices. Services that fail to behave well and bust their error and support budgets repeatedly go back to the teams that wrote them.
- GauntletWizard 4y ago
  This is important to stress. SRE often takes first-line oncall, but "We" support the service. On a well behaved service, having SRE support is an extra debugging-focused engineer when things go wrong. But that engineer is rarely called up, and can support the common parts of failure (network issues, hardware failures, the things that are out of any one team's service team's control) for multiple teams.

_8j50 4y ago

"Own" in this context means be responsible for. Peer review does not transfer ownership to peers but it does reduce risk of your code breaking. Other people should not understand your own code (and intent) more than you, so long as that is correct, while everyone in dev/ops shares responsibility of delivering service, you own and are responsible for the part of that service delivery that you authored simply because you are the person most qualified to resolve any issue that arise from that code breaking. If an SRE is keeping your code running then they own the code's uptime in as much as you have been able to communicate the parameters and configuration of the app, but at the end of the day, when there is a bug in the code you are the best person to fix it, so you own it.

chrsig 4y ago
> "Own" in this context means be responsible for. I understand this, this is what I was trying to direct my distaste at. Unfortunately, it seems to have been misconstrued by responders. > you own and are responsible for the part of that service delivery No, I'm just responsible for that part of the service delivery. I own none of it. I object to it in the same way that I object to hearing "we're a family" from leadership.
- _8j50 4y ago
  So long as your bosses are ok with that, who cares. Power to you. I was just stating the obvious typical expectation I guess.

runamok 4y ago

It's a question of aligned incentives. If your team (not you necessarily) only have a priority to ship features than your team needs to be alerted for performance and reliability issues because you are thus incentivized to fix said issues. If these issues are thrown over the fence to some other team who doesn't know your codebase and have the domain knowledge to improve said code then things (usually) never get fixed. Likewise error budgets that say "your service has not been reliable enough this period so the next sprint will be dedicated to improving that and no new features will ship" is another way to make sure quality is not an afterthought.

disruptiveink 4y ago

Isn't this (you devs own your app, I don't give a shit about it, I just want it to not page me) the classic old school "Sysadmin" approach that supposedly DevOps was supposed to counter? How can anyone say this and claim to be doing anything remotely near DevOps?

jkukul 4y ago
I second this. DevOps was supposed to result in developers also being responsible for deploying and monitoring their applications. It's either a DevOps culture or a classical Dev + SysAdmin setup with clear boundaries and where a developed application is being thrown over the fence to SysAdmins. The latter will always result in mutual animosities between the two parties.
- fragmede 4y ago
  It's part of the deal for modern SaaS software deployment. If the devs want to deploy more often than once a quarter, then they need to be mindful of the operational burden they impose on the ops team. The way to drive that home is for them to carry the pager part of the time.
- pards 4y ago
  > DevOps was supposed to result in developers also being responsible for deploying and monitoring their applications No, it was supposed to be dev and ops collaborating with mutual respect and empathy.

bambax 4y ago

> If it's not part of the SOP, then no, you wont have those things Isn't this "adversarial" as well? Why would you withhold that information just because the SOP don't make you provide it? What will happen then is that eventually the service will break, nobody will know how to fix it, and they will come and ask you. If you're no longer there, the service will be decommissioned and all your work will have been in vain. I don't see a net benefit for any of this.

chrsig 4y ago
I didn't mean it adversarially, but I can see how you came to that conclusion. Communication is hard. I was trying to convey that they wouldn't exist because it hadn't been asked for. People aren't mind readers, ops has to vocalize their needs. Which is what the post was doing. And I don't think those needs are unreasonable, either. My point was more that the work to make sure those things are available needs to happen higher up than you'll be able to reach by talking to any individual developer.
- bambax 4y ago
  Aaah, ok! Sorry for the misunderstanding, and thanks for the explanation. (I was mostly in agreement with the OP and was surprised by the level of opposition to it in the comments.) Yes, it's better if the information asked for in the original post is in the SOP, but if it's not, then asking for it each time somebody sends something to production, while suboptimal, is better than nothing IMHO.

treeman79 4y ago

Dealt with this at a company. Eventually the developers just gave up on all development work and became operations. The actual operations team kept k8 cluster alive. Developers had to do everything else. Eventually people would get the hang of the operations side but by that point they were burnt out and quit.