7 ms·
Team Structure for Software Reliability Within Organizations
- DrScientist 7y agoThe article misses out the people element. Isn't it best to design for reliability in the first place? Don't you incentivize that best by having the same people supporting and app as well as developing? Nothing makes you write reliable software like being the person being called out, or dealing with the customer issue.
- karatestomp 7y agoI’m becoming more sympathetic over time to the notion that this everyone-must-do-everything movement is basically a failure. There’s too much. Be good with: two or three PM and issue tracker tools, a graphics or UI mockup program or three, various analytics tools and libraries, a payment system or three, several build and packaging systems, a couple reverse proxies, SSL provisioning, several ways to schedule jobs, a few message or event buses, a couple cloud management UIs (command line and browser!) and also WTF all the brand-names in there do, several major database systems with a few different query languages and paradigms, three or more programming languages for weekly use, various communication tools, your own command line environment, the typical environment on your deployment targets and there may be more than one of those, infra as code languages/tools, at least one CI system at any given time, git, the quirks and bullshit of dozens of major libraries across multiple languages, testing frameworks and practices, networking, and on and on. It’s plainly too much and functionally no-one’s doing all that well, which explains some of what we see from software in the wild. We could probably use about three specialist developers and a secretary for every do-everything developer like that.
- sethammons 7y agoIt is easy to swing too much towards self serve teams. The separation that works well for us is the team owns the code, the emissions of artifacts like logs and metrics. Alerts and such go to the team. However, there is a team that maintains the build system. A team that deals with maintaining the monitoring systems, a team that deals with network, hardware, proxies, etc. For us, the one we are learning to balance is the data layer. We've traditionally had a DBA team doing most of that. But the team building the service knows their data better. We are shifting to a DBA as a consultant resource model and seeing how that goes.
- rednerrus 7y agoYou're describing what Devops and SREs do.
- karatestomp 7y agoYes, at places that have them and where those roles are well-integrated and their teams reasonably well run, which is... not everywhere, by a long shot.
- pm90 7y agoDo you think the complexity may be a result of the diverse ecosystem of technologies we find ourselves in today? Never before have we seen so many ways of doing so many different things. Every problem has an OSS solution, companies deploy variations of those tools internally. The problem will only get more pronounced in the future... there seems no escaping unless you restrict the adoption and usage of new technologies.
- DrScientist 7y agoOf course you need some tech stack discipline and the dev team needs to operate as a team not a bunch of individuals - the team has to be able to support the system if a single developer wants to go on holiday. A mess of a tech stack is more of a problem for developers making future changes than it is for production ( if you have fully automated the deployment, monitoring and service recovery ). ie the problem of a mess of a tech stack isn't due to being developer led per se, it's a problem of a lack of team working and organisation - which can happen in any team or situation.
- hibbelig 7y agoThe article talks about adding SREs. As if the developers wrote buggy software and then the SREs fix the bugs. I’m sure I’m misunderstanding. What is meant?
- treerock 7y agoSounds like SRE are fixing bugs and introducing tools and processes that improve reliability. > There are two main types of reliability work. The first is mitigation, which is a linear fix that’s often referred to as firefighting...The second is change management, which is a non-linear fix that proactively reduces the defect rates through projects like migrating to better tools and refactoring spaghetti code. While SREs support both these types of work, they should spend more time on the latter.
- deleted 7y ago[deleted]
- madhadron 7y agoShucks, I was hoping for an analysis of team structure based on Microsoft's paper[1] [1]: https://www.microsoft.com/en-us/research/publication/the-influence-of-organizational-structure-on-software-quality-an-empirical-case-study/ https://www.microsoft.com/en-us/research/publication/the-inf...
- sethammons 7y agoRe SREs: we've taken a different approach. Devs own their systems. If the issue is network or hardware, there are more ops oriented folk that make that work. But when it comes to services, teams own their stuff. Bug a 2am causes a service to go down? The team is expected to start debugging it and pull people in as needed. The result? Shifting to this model from an SRE firefighting model has lowered total alerts, incidents, and outages. When we first shifted, people dreaded going on their on call rotation for their team; you expected to be paged multiple times a night. Fast forward and now I can count on one hand the amount of pages I've had on my last multiple on call rotations. Work is prioritized for system reliability because we feel the pain.
- fatnoah 7y ago>The result? Shifting to this model from an SRE firefighting model has lowered total alerts, incidents, and outages. When we first shifted, people dreaded going on their on call rotation for their team; you expected to be paged multiple times a night. I implemented a similar change at a previous company. Prior to the change, the operations team fronted all on-call issues. After, anything related to the software/services was handled by software teams. We went from nightly alerts to less than monthly. As a bonus, time to diagnose and repair went waayyyyy down due to much better logging. Note, better logging <> more logging in this case. It meant less spurious errors/warnings, more useful INFO, and better observability of the system.
- test1235 7y agoDid everyone get paid more to take on these responsibilities? I feel we're about to go through exactly the same thing at my company.
- sethammons 7y agoThe logic was that we are highly paid developers and are expected to build robust, reliable software with high availability. Typically a brutal on call night yields untracked time off.
- sokoloff 7y ago
- ignoramous 7y agoAWS, incidentally, spoke a tonne at re:Invent 2019 on how they run their operations. It is interesting contrast against the article because AWS takes a different approach (no dedicated SREs)-- The engineers that build the system run ops, too. Some references: [0] Colm MacCarthaigh, How to take control of systems, big & small, https://www.youtube-nocookie.com/embed/O8xLxNje30M https://www.youtube-nocookie.com/embed/O8xLxNje30M (2018) [1] Eric Brandwine, Aspirational security, https://www.youtube-nocookie.com/embed/ad9180b4Xew https://www.youtube-nocookie.com/embed/ad9180b4Xew [2] Marc Brooker, Amazon's approach to building resilient services, https://www.youtube-nocookie.com/embed/KLxwhsJuZ44 https://www.youtube-nocookie.com/embed/KLxwhsJuZ44 [3] Peter Ramensky, Amazon's approach to high-availability deployment, https://www.youtube-nocookie.com/embed/bCgD2bX1LI4 https://www.youtube-nocookie.com/embed/bCgD2bX1LI4 [4] Andy Troutman, Amazon's approach to running service-oriented teams, https://www.youtube-nocookie.com/embed/n1d20Yok000 https://www.youtube-nocookie.com/embed/n1d20Yok000 [5] Colm MacCarthaigh, Amazon's approach to security during development, https://www.youtube-nocookie.com/embed/NeR7FhHqDGQ https://www.youtube-nocookie.com/embed/NeR7FhHqDGQ [6] Becky Weiss, Amazon's approach to failing successfully, https://www.youtube-nocookie.com/embed/yQiRli2ZPxU https://www.youtube-nocookie.com/embed/yQiRli2ZPxU [7] Thomas Blood, Amazon's culture of innovation, https://www.youtube-nocookie.com/embed/2ZQKPUD7vKE https://www.youtube-nocookie.com/embed/2ZQKPUD7vKE [8] Andy Warfield and Seth Markle, Lessons from Amazon S3's culture of durability, https://www.youtube-nocookie.com/embed/DzRyrvUF-C0 https://www.youtube-nocookie.com/embed/DzRyrvUF-C0 [9] Colm MacCarthaigh, Lessons from Amazon's highest available data-planes, https://www.youtube-nocookie.com/embed/2L1S0zfnIzo https://www.youtube-nocookie.com/embed/2L1S0zfnIzo [10] Eric Brandwine, The tension between absolutes & ambiguity in security, https://www.youtube-nocookie.com/embed/GXTvlQXVCOs https://www.youtube-nocookie.com/embed/GXTvlQXVCOs (2018)
- coldcode 7y agoArticles like this always assume an organization is just one thing, in this case server and/or web. Our organization is dozens of server teams, internal apps, mobile apps and web clients, plus interactions with real world systems (and people). It's impossible to actually model reliability in such a complex environment.
- JMTQp8lwXL 7y agoI work in a publicly traded company. I do everything from requirements elicitation, implementation, unit testing and end-to-end testing, own the deployment process, as well configure logging and monitoring to ensure deployments continue to stay healthy. Our software's reliability is as strong or as brittle as each individual engineer deciding to implement or skip these processes, since most micro-services are made by groups of 2-5 people (and I'm currently a team of 2, and for a few months was a team of 1).
- jungturk 7y agoWould defining a (required) key set of observables not help to reduce the variability across teams? Or perhaps the issue you're describing is that teams aren't required to demonstrate any? In my experience (as a chief architect and engineering VP), laying out some baseline metrics closes much of the reliability variance across teams. As the teams get more competent at baking in the basics (overall load, latency, resource utilization, error rates, error events posted to chat) you can ratchet up the competency to include higher order observables (scaling events, business transactions, circuit statuses, traces, anomalies). Which is to say its been straightforward (in my experience) for most teams to raise the bar once they know there _is_ a bar and once they can see the bar.
- JMTQp8lwXL 7y agoTeams aren't required to demonstrate any. Engineers who understand the importance of quality bake in time, and management is supportive of self-imposed controls. Our NPS numbers are trash, though. But it's a different kind of quality issue: we aren't fighting fires every week. The problem is our API response codes are often wrong or unpredictable. The original engineers clearly did not understand the HTTP protocol.
- nradov 7y agoTeam culture and incentives matter more than structure.
- JoshMcguigan 7y agoHaving the right team and organizational structure can go a long way toward aligning incentives.
- halbritt 7y agoIn my experience, the best team structure depends on the organization. Certainly for smaller orgs and startups it is possible to hire a small set of engineers with the necessary abilities to handle the reliability work. In larger, more-established organizations, scrum teams tend to gravitate toward feature development in which it becomes difficult to prioritize reliability work even if the skills are present. With very many feature development teams, it becomes difficult to hire for a programmer that has the primary skill desired and the ancillary skill of reliability and systems operations. I'm currently of the belief that in most orgs, a team should be dedicated to this. There's the ongoing operations of a service, which I think is a shared duty, and the infrastructure engineering components necessary to abstract the infrastructure sufficiently well that it is uniform across feature-development teams and also reliable, scalable, and secure by default. As with anything, the devil is in the details.