9 ms·

Full cycle developers – Operate what you build (2018)

124 points by samdung 5y ago

adrianN 5y ago
>The primary upside of having a separate ops team was less developer interrupts when things were going well. And, you know, developers having time to get better at developing instead of spending their time learning all the tooling you need to understand and the experience you need to gain in addition to what you already have to know to be an effective developer. Specialization is not always a bad thing. By having everybody do everything you now just have a bunch of people that do most workflows only a couple of times a year and lack the routine to handle situations where the extra automation you might have built to help them is insufficient. This just adds a lot of stress for your employees. The primary upside is that you save headcount while things are going well.
- CraigJPerry 5y ago
  I think you’ve gone off to an extreme view. There’s a huge middle ground between “oh, it uses electricity? Yeah you want to speak to the developers if that’s not working” and “do not disturb, coding.” Having a platform engineering team, if Your company can initially afford to bootstrap such a thing, is a huge productivity accelerator for your developers but even with that in place, I’d say it’s still healthy for the dev to be first port of call for an app outage.
  - adrianN 5y ago
    It of course depends on the complexity of your product. For sufficiently simple things, a single developer can handle everything themselves. But it doesn't take much to reach a point where a single developer doesn't even understand all the software, let alone all the infrastructure around it. Sysadmining is hard, it's a tall order to expect someone you hired because they where good at inverting binary trees to be a competent sysadmin.
    - Aissen 5y ago
      It seems you already know how hard ops can be, so you might already know that a little development effort can often go a long way into alleviating the ops tasks, which is a win for everyone. I'm not sure every developer knows, or understands. Yes, you could follow a rule book, or a deployment framework, but you'll be better at what you do if know why you do it.
      - adrianN 5y ago
        Imo, a good way to teach developers how they can help other roles is by having a shadow program where developers work together with a different specialized role for a couple of days but don't have to carry the responsibility. That improves cross team communication a lot, while being much less stressful for your employees.
      - lmm 5y ago
        I think it's a big win if developers and operators are the same team - sit next to each other, do the same teambuilding exercises, have responsibility for the same set of products, pair-program together where appropriate. Slice your teams vertically rather than horizontally. I'm not convinced that it's worth going all the way to having every developer be an operator and vice versa.
        pjmlp 5y ago
        Having weared both hats multiple time, Dev and Ops, I think this is the right approach.
    - andrewstuart2 5y ago
      As a sysadmin who became a developer, I don't think I agree that it needs to be hard to operate software these days. We have tools that simplify, standardize, and aggregate a lot of the things that used to make systems administration much more complicated, such that operating the software you wrote very rarely actually requires a full set of systems administration knowledge. With good platforms, it's absolutely feasible to operate your software without needing the full knowledge of a sysadmin.
  - quickthrower2 5y ago
    The joke at one place I worked was developers were good at dealing with all devices. Except the phones!
- Cthulhu_ 5y ago
  I think the most important part of what a developer should learn from ops is the daily struggles. Observability, performance characteristics under load, debuggability, stability, etc. If you just build shit and throw it over a hedge for an ops team to manage, you'll never learn.
  - m_mueller 5y ago
    What we do is shared ops duty between the DEVs, including the lead. E.g. in a team of 5, each takes a working day a week. On-call for off hours issues is organized separately. I think this system works reasonably well to keep everyone aware of production issues.
  - nicoburns 5y ago
    As someone currently leading a small team that is responsible for absolutely everything technical at our company (from development to ops to IT and even UX), I do agree that it's a good learning experience. On the other hand it sure does take a lot of time out of the day. And I think all the context switching definitely leads me to be less productive than I would otherwise be. Of note for me: while development work often requires uninterrupted time and deep concentration, ops and customer support requests are often urgent and have to be dealt with right away. I think the ideal might be to get broad experience and then specialise a little later.
  - taneq 5y ago
    I think what's happening here is we're conflating "developer" with "product designer." And often, yeah, they're the same - but if you're in a big enough team that you have one person doing the designing and another person doing the software architecture and another doing coding, it's the designer hat which needs to be at the coal face watching the users use the product and understanding their pain points.
- xwolfi 5y ago
  Yeah I liked a lot the idea when working in a small startup as the ops team we removed was way slower than us, but now that I work in a company making actual money relying on actual regulatory deadline, I find myself wishing they'd reinstate the ops team they also removed to stop us from having to learn all the details on the gigantic infrastructure that can fail, from routers in another continent to red hat bugs to hedge fund client complaints... This move to make the dev team all-doing isn't really working as efficiently as the cost reduction made it appear.
  - qaq 5y ago
    it never does. Suits want a "generic employee" that can work on any problem. Makes planing way easier.
    - pjmlp 5y ago
      Aka consultant. As I was once told, you just need to be one chapter ahead of the customer.
  - draw_down 5y ago
    Isn’t it great that we’ve figured out a cool sounding phrase for what is basically cost savings? We’re full cycle developers bro, our company is definitely not just cheap!
- moksly 5y ago
  I think the mixture is more important. I don’t necessarily need my developers to know a whole lot about what our networking personal does. But I do need them to know how to configure a trusted partner in our ADFS, how to create and manage databases and how to operate things in Azure. We used to think differently, we used to do much more specialisation, but because operations needed to also code, they eventually became developers who could also do operations, which meant that some of them simply ended up replacing some of the developers who simply took up too much man hours to get things into production by comparison. It’s sort of a delicate line of course, because you’re right on some level. People can’t know everything. So now our split is that developers need to be capable of working with things like Azure AD, but they don’t need to know how to manage it. They need to be capable of creating databases and how to handle the security on them, but they don’t need to know how we operate our actual database clusters. And so on. On the flip side some of our operations staff need to know how to program in python to be able to handle disasters if something happens to our Azure Services or many back-ends. They also need to know how to develop our software robots. But I think the biggest challenge most people face isn’t really on the operations vs development front. It’s on the new development vs old development improvement front. This is likely not as big an issue at a place like Netflix where they don’t service a gazillion different business needs, but in more broader enterprise businesses it quickly becomes a resource issue, because nobody is really going to have updated knowledge on 400 different systems at all times. And you can’t have people sitting around waiting for update requests on systems that may not produce one in a year.
- ignoramous 5y ago
  > This just adds a lot of stress for your employees. The worst is when full-cycle developers (and less often, the managers) jump ship to another team, org, or company after it is clear what they've built is a big ball of unmanageable, painful mud.
  - Aeolun 5y ago
    Building that is certainly not limited to full-cycle developers. At the very least, there is 1 person that understands the entire thing.
    - ignoramous 5y ago
      I meant to highlight that full-cycle developers can and do turn away from owning the thing. When you've got an SRE org, there's stricter guard rails in-place to ensure you don't end up building a frankenstein monster. For the same reason the Cloud makes sense, Serverless makes sense (offload what you can on to world-class experts who do know what they're doing)... SRE orgs would provide for better RoI than own-what-you-build teams, over the long run. That is, anyway, my armchair take.
- hyperman1 5y ago
  As a developer , I have less interruptions now I can partially do my own operations. The old world was one where operations did not allow any dev access on production. So you get a ticket from a user, but you have no access to logs or databases or whatever, just a screenshot from the user. Performance testing is only allowed on dev, with a dataset of 2 heavily anonymized records. Deploying to prod required devs to specify paths and server names, but devs were not allowed to know what servers ran the application or what paths were available. Detailed deployment plans are 'optimized' by ops to remove 'unnecesary' steps, then for some strange reason the new version crashes 5 seconds after release, which is promptly declared a 'programmer error'. Preproduction and production were generally configured by different people, so plenty of differences between the environments exist. Neverending migration projects mean your application might accidentally partially disappear, with only user tickets as a signal that something happened. Monitoring is ignored by ops, as it produced more alerts than they can handle. I wonder why, I never got one. So now I as a dev also do ops. I added some monitoring, and every few days I manually check basic health parameters. Problems generally get solved without users even noticing, and I can schedule time for this and work uninterrupted after that. I know what I prefer.
kennethh 5y ago
By departmentalize you get silos and reduced efficiency across teams, by going the other way and having full cycle developers you reduce the silo effect and get alignment of incentives. Again the problem is that you need people who know a lot, and the mental load is very hard and not everybody is interested or able to handle that. As an old developer, this is very much how we operated before CI/CD was a thing in the early 2000. I remember we were responsible for developing, monitoring and support. In our team we had one week each where we did support and monitoring as the primary task.
- piokoch 5y ago
  I think this all depends on the complexity of operational tasks. If you have hosted MySql and a couple of servers, fine, developer can handle monitoring this. What if you have, say, DB2 cluster with HADR running, do you think you can monitor it and fix something if needed? Do you know ins and outs of share drives configuration using GFS. Kubernetes configuration details, if something goes wrong should every developer have sufficient knowledge to troubleshoot this? How about A10 load balancer, what average developer knows about this, can average developer fix failed virtual IP configuration or firewall rules on A10? Solving Red Hat Satellite cache issues? I don't even mention handling some complicated network setup with third parties connections through proprietary VPN solution, AWS cloud deployments using multitude of services that AWS offers or obscure stuff like zSeries IBM or Unisys mainframes, etc. Developer can be trained in all kind technologies, but there is truly a lot of them, this is huge body of knowledge, learning this will take a lot of time and money.
  - nijave 5y ago
    I think you're conflating platform engineering and operations. A DB2 cluster with HADR is another application that some engineering team has stood up. A consumer of the database isn't responsible for the implementation; they're responsible for how their code interacts with and uses the system another (engineering) team built. In that regard, if a developer sees their application degrading due to an issue with the underlying platform, they'd engage the developers that built that platform (and ideally those developers have built a system to detect problems before the "end user" system starts degrading) In the same vein, it's not expected every developer understands kernel and hardware development--there are abstractions and boundaries in place that separate responsibility. The recurring theme is the person that configured/coded/built is responsible for operating it
draw_down 5y ago
Always be thinking about what you can get out of your employer. They’re always thinking about what else they can get out of you.
5tefan 5y ago
"Eat your own dog food" is one of my guiding principles and it's very powerful.
- ghostwriter 5y ago
  "sure, but I'll eat it from my own plate" aka developers get to choose the tooling they will use and ops have no objections to it.
Stranger43 5y ago
The problem here as with most of the devops literature is that it's only really applicable for the rare usecase of where an entire company's worth of resources is put behind a single website/app deployment, where as the far more normal practice of having small implementation and operations teams manage a large back catalog of projects/deployment that's supposed to run stable with little need for day to day improvements/interventions, and which are often based on products that were oversold by some consultant/salesdrone 5 years ago as virtually maintenance free. And when it comes to big systemic problems with security/stability that plagues the IT industry it's usually this back catalog that's the real cause and the problem that gets the least attention.
- jiggawatts 5y ago
  Oh god, this. This is the real problem! Everyone wants to sell you the solution to problems where there are 10,000 servers for a single set of related apps managed by 1,000 developers. Think companies like NetFlix, where there are teams of people dedicated to specific parts of DevOps. Nobody has good solutions for legacy but still functional apps developed by some guy named Bob that was a contractor for 6 months and he's gone to some other gig. For example: Take Splunk, or Azure Log Analytics, or any similar tool. They work great if you get 100K hits per day. The graphs are beautifully smooth and the wealth of data makes it trivial to extract all sorts of insights. Approaches like A/B testing can accurately detect changes as small as 1% in many cases. But what do you do with the site that gets 5 hits per week, but is still important because what it does for each one of those hits is worth on the order of a thousands dollars? The tooling is a total letdown, basically nothing out there can help with apps like this. Trying to retrofit some legacy app into Kubernetes or whatever would cost more than app makes in revenue. Leaving it alone is just as bad, because eventually you have to upgrade the server it lives on and the load balancer in front of it. At which point it will break. Or it might not. How would you know? You only get 5 hits a week...
  - flukus 5y ago
    > But what do you do with the site that gets 5 hits per week, but is still important because what it does for each one of those hits is worth on the order of a thousands dollars? Log files and few shell scripts can go an awful long way, from simple single user apps to complex corporate environments with dozens of servers. They require almost zero upfront investment and you can build on them as needed. You won't get fancy A/B testing (without a lot of work) or give you a pretty UI with smooth graphs, but grep can happily tear through 100GB worth of log files and get you some of that good stuff you want from azure log analytics.
TobySKT 5y ago
How to Build a Food Delivery app for Restaurant? Starting a food delivery service today will turn into a prosperous business in the future. This guide will help you learn how to create a food delivery app from both business and technical perspectives https://steelkiwi.com/blog/how-build-food-delivery-app-for-restaurant/ https://steelkiwi.com/blog/how-build-food-delivery-app-for-r...
musiccog 5y ago
As an older full-stack/full-cycle developer for many years, this article made sense - and gave quite a bit of hope. If you can't dog-food your product, IMHO developers need to be as close to the coal face of their 'product' as possible. Per the article, assigning a team to a feature has allowed Netflix to get better coding outcomes. With that said, I can also see a potential downside to this development model. Once a new feature is stable enough, the number of people required to support it has to reduce. Wondering how Netflix solved this?
sdevonoes 5y ago
> We mitigate this by having an on-call rotation where developers take turns handling the deployment + operations + support responsibilities Sorry, but no. All the reasons that exist for developers doing on-call rotation are reasonable, yes, I admit that, but my reason to not do on-call rotation is also reasonable: I do not want to give my employer more than 40h/week. I just do not want more money in exchange for my free time. If your company doesn't provide that, then "alright, thank you" and I will keep looking. Now, if all companies start to require (paid) on-call rotation for developers: that would be a very sad tech scenario (at least for people like me).
- ghostwriter 5y ago
  Charge them with a significantly higher base pay if they are willing to include on-call into your list of responsibilites, or make sure your contract mentions exemption from on-call activities when you change an employer next time. In other words, make them see on-calls in salary budgets.
ofrzeta 5y ago
If you are in the business of developing and running web applications there's a lot overlap between dev and ops in terms of knowledge. If you have to troubleshoot something you need to know HTTP Headers, CORS and related topics. Who is in charge when the latest update of Chrome breaks your clients' websites? Wearing both hats certainly helped us fixing stuff easier by inserting headers through HAProxy although our regular job is programming. What about performance optimization? Can that slow SQL query be fixed through tuning the MySQL server or maybe just rewrite the code?
deleted 5y ago
[deleted]
jacksonkmarley 5y ago
I don't know how prevalent my opinion is amongst developers, but the reason I will never take another job with a support aspect is not because of some detailed assessment of the software lifecycle. It's because I don't like it. And I can't think of anything more annoying than being on-call for work outside regular work hours. So I think it's awesome that some people want to do this work. I hope they do it full-time, and that lets me do my thing full-time. Obviously many developers like the mixed model or accept it. I'm curious about whether I'm an outlier or lots of others feel the same?
- potamic 5y ago
  Does having a separate on-call really shield the devs from support? I assume in most cases on-call would need to escalate or reach out to the devs for issues and for that someone needs to be reachable on phone off hours?
  - jacksonkmarley 5y ago
    I mean I guess it's pretty dependent on the company and product, I just know if I see "on-call" or "support" in a job ad, I pass. Perhaps I'm just screening myself out of certain business areas. My experience was support during business hours as part of my job as a software engineer. I used to hate hearing the phone ring, and it was rarely if ever a problem directly related to what I was developing. I felt like it was just a huge disruption to my work.
- leokennis 5y ago
  I understand your attitude, and sort of agree with it. If I take a job with on call duty, it also means I will be woken up at night because of "your" mistakes (in code, testing etc.) and that is fine. But a very important aspect for me is that, if that happens, the developers give the correct priority to fixing the bug that woke me up at night. So, it's fine you only want to work during the day. But if I get woken up at night or in the weekend, you better start fixing the bug that caused it as soon as it's day again.
  - 5y ago
deleted 5y ago
[deleted]
trestenhortz 5y ago
This is not a good idea. Developers already have an incredible amount to learn without making them devops too. This is the sort of thing than non tech management would love though. It’s a great slogan. Essentially the sign of mismanagement dressed up as innovation.
aoetalks 5y ago
100% agree with this. I work on a service deployed in Azure processing PBs of data each day. We’re a team of about 8 engineers, and we have an on-call rotation. As a result, we’re incentivized to reduce on call noise and have the system automate remediation as much as possible. Granted, our company does have centralized teams that build tooling. So that does seem to be a prerequisite to having full cycle developers. If you’re building in the cloud, I think there’s less need to be a SysAdmin these days (I used to be be one), and Azure makes it pretty easy to have automatic updates, firewall rules, etc. We rarely touch our hosting environment (Service Fabric on VMSS). The on-call sucks at times, but I’d never trade it for how fast we’re able to ship.
deleted 5y ago
[deleted]
pmlnr 5y ago
This used to be called jack of all trades, master of none.
dijit 5y ago
So, I had this argument/conversation with Jedberg (who incidentally also works at Netflix); I said that: DevOps means different things to different people; To Some: it's the evolution of "Operations" to integrate them into the team so that operational issues are automated out and that developers can get greater velocity. To Others: it's a sysadmin who can use CI/CD and fumble through scripting something. To the majority: it's a developer who can fumble through installing a server or package, and this is jedbergs definition too, and one he argues is canonical. My opinion is that specialising is important. --- To this effect: it's _far_ too much cognitive load to understand how your OS works, your application, its framework, the cloud provider, the logging tools, SLI/SLOs and error budgets, on-call policies and alerting mechanisms, application level security, authentication of systems, authorization of systems, SSO, etc;etc;etc;etc; Because to learn all of those various things and weigh them in your head every time you make a decision is fatiguing, it took me 10 years to get "great" at Ops and a decent scripter; it's possible that Ops is easier than programming but from what I understand it's not easier, it's just different. With Development (exception: Javascript) you learn primitives and they rarely change. With Ops, the tools and landscape change _frequently_. 10 years ago it was Nagios, then Zabbix, then some combination of graphite and co, now it's prometheus, maybe Thanos or is it influxDB+Kapacitor? That's just alerting, there's new and changing solutions for logging, message queues, databases, IdP tooling (keycloak, vs FreeIPA vs AD), distributed tracing systems, heck even load balancers (GCE Ingress vs Nginx vs traefik vs etc;etc;etc). Hell, understanding how Kubernetes works is basically a full time job otherwise "black magic is happening" and debugging becomes a nightmare. Frameworks change or get replaced too, but it doesn't feel like it's close to the same rate as infrastructure software. I remember at college one of the professors said to me: "It's impossible to know everything about computers, because as you're learning: new things come out, and after you've learned something: it will change. The more you learn, the more you have to keep up to date until you can no longer keep up." I think maybe people don't think like that. I think we need to have people of different disciplines.
lamontcg 5y ago
I've always been a bit confused that this wasn't what DevOps was supposed to mean from the start. Devs should be producing easy-to-operate code and their oncall should therefore not be burdensome and there should be tight cycles to drive it to near zero.
ChrisArchitect 5y ago
so many different URLs, damn Medium previous discussion, 2 years ago, 20 upvotes, https://news.ycombinator.com/item?id=19481912 https://news.ycombinator.com/item?id=19481912
sunstone 5y ago
Development and operations are two different mindsets not typically found within the same person.
shreevari 5y ago
Hope this term doesn't hit recruiters dictionaries.

124 points by samdung 5y ago

>The primary upside of having a separate ops team was less developer interrupts when things were going well. And, you know, developers having time to get better at developing instead of spending their time learning all the tooling you need to understand and the experience you need to gain in addition to what you already have to know to be an effective developer. Specialization is not always a bad thing. By having everybody do everything you now just have a bunch of people that do most workflows only a couple of times a year and lack the routine to handle situations where the extra automation you might have built to help them is insufficient. This just adds a lot of stress for your employees. The primary upside is that you save headcount while things are going well.

CraigJPerry 5y ago
I think you’ve gone off to an extreme view. There’s a huge middle ground between “oh, it uses electricity? Yeah you want to speak to the developers if that’s not working” and “do not disturb, coding.” Having a platform engineering team, if Your company can initially afford to bootstrap such a thing, is a huge productivity accelerator for your developers but even with that in place, I’d say it’s still healthy for the dev to be first port of call for an app outage.
- adrianN 5y ago
  It of course depends on the complexity of your product. For sufficiently simple things, a single developer can handle everything themselves. But it doesn't take much to reach a point where a single developer doesn't even understand all the software, let alone all the infrastructure around it. Sysadmining is hard, it's a tall order to expect someone you hired because they where good at inverting binary trees to be a competent sysadmin.
  - Aissen 5y ago
    It seems you already know how hard ops can be, so you might already know that a little development effort can often go a long way into alleviating the ops tasks, which is a win for everyone. I'm not sure every developer knows, or understands. Yes, you could follow a rule book, or a deployment framework, but you'll be better at what you do if know why you do it.
    - adrianN 5y ago
      Imo, a good way to teach developers how they can help other roles is by having a shadow program where developers work together with a different specialized role for a couple of days but don't have to carry the responsibility. That improves cross team communication a lot, while being much less stressful for your employees.
    - lmm 5y ago
      I think it's a big win if developers and operators are the same team - sit next to each other, do the same teambuilding exercises, have responsibility for the same set of products, pair-program together where appropriate. Slice your teams vertically rather than horizontally. I'm not convinced that it's worth going all the way to having every developer be an operator and vice versa.
      - pjmlp 5y ago
        Having weared both hats multiple time, Dev and Ops, I think this is the right approach.
  - andrewstuart2 5y ago
    As a sysadmin who became a developer, I don't think I agree that it needs to be hard to operate software these days. We have tools that simplify, standardize, and aggregate a lot of the things that used to make systems administration much more complicated, such that operating the software you wrote very rarely actually requires a full set of systems administration knowledge. With good platforms, it's absolutely feasible to operate your software without needing the full knowledge of a sysadmin.
- quickthrower2 5y ago
  The joke at one place I worked was developers were good at dealing with all devices. Except the phones!
Cthulhu_ 5y ago
I think the most important part of what a developer should learn from ops is the daily struggles. Observability, performance characteristics under load, debuggability, stability, etc. If you just build shit and throw it over a hedge for an ops team to manage, you'll never learn.
- m_mueller 5y ago
  What we do is shared ops duty between the DEVs, including the lead. E.g. in a team of 5, each takes a working day a week. On-call for off hours issues is organized separately. I think this system works reasonably well to keep everyone aware of production issues.
- nicoburns 5y ago
  As someone currently leading a small team that is responsible for absolutely everything technical at our company (from development to ops to IT and even UX), I do agree that it's a good learning experience. On the other hand it sure does take a lot of time out of the day. And I think all the context switching definitely leads me to be less productive than I would otherwise be. Of note for me: while development work often requires uninterrupted time and deep concentration, ops and customer support requests are often urgent and have to be dealt with right away. I think the ideal might be to get broad experience and then specialise a little later.
- taneq 5y ago
  I think what's happening here is we're conflating "developer" with "product designer." And often, yeah, they're the same - but if you're in a big enough team that you have one person doing the designing and another person doing the software architecture and another doing coding, it's the designer hat which needs to be at the coal face watching the users use the product and understanding their pain points.
xwolfi 5y ago
Yeah I liked a lot the idea when working in a small startup as the ops team we removed was way slower than us, but now that I work in a company making actual money relying on actual regulatory deadline, I find myself wishing they'd reinstate the ops team they also removed to stop us from having to learn all the details on the gigantic infrastructure that can fail, from routers in another continent to red hat bugs to hedge fund client complaints... This move to make the dev team all-doing isn't really working as efficiently as the cost reduction made it appear.
- qaq 5y ago
  it never does. Suits want a "generic employee" that can work on any problem. Makes planing way easier.
  - pjmlp 5y ago
    Aka consultant. As I was once told, you just need to be one chapter ahead of the customer.
- draw_down 5y ago
  Isn’t it great that we’ve figured out a cool sounding phrase for what is basically cost savings? We’re full cycle developers bro, our company is definitely not just cheap!
moksly 5y ago
I think the mixture is more important. I don’t necessarily need my developers to know a whole lot about what our networking personal does. But I do need them to know how to configure a trusted partner in our ADFS, how to create and manage databases and how to operate things in Azure. We used to think differently, we used to do much more specialisation, but because operations needed to also code, they eventually became developers who could also do operations, which meant that some of them simply ended up replacing some of the developers who simply took up too much man hours to get things into production by comparison. It’s sort of a delicate line of course, because you’re right on some level. People can’t know everything. So now our split is that developers need to be capable of working with things like Azure AD, but they don’t need to know how to manage it. They need to be capable of creating databases and how to handle the security on them, but they don’t need to know how we operate our actual database clusters. And so on. On the flip side some of our operations staff need to know how to program in python to be able to handle disasters if something happens to our Azure Services or many back-ends. They also need to know how to develop our software robots. But I think the biggest challenge most people face isn’t really on the operations vs development front. It’s on the new development vs old development improvement front. This is likely not as big an issue at a place like Netflix where they don’t service a gazillion different business needs, but in more broader enterprise businesses it quickly becomes a resource issue, because nobody is really going to have updated knowledge on 400 different systems at all times. And you can’t have people sitting around waiting for update requests on systems that may not produce one in a year.
ignoramous 5y ago
> This just adds a lot of stress for your employees. The worst is when full-cycle developers (and less often, the managers) jump ship to another team, org, or company after it is clear what they've built is a big ball of unmanageable, painful mud.
- Aeolun 5y ago
  Building that is certainly not limited to full-cycle developers. At the very least, there is 1 person that understands the entire thing.
  - ignoramous 5y ago
    I meant to highlight that full-cycle developers can and do turn away from owning the thing. When you've got an SRE org, there's stricter guard rails in-place to ensure you don't end up building a frankenstein monster. For the same reason the Cloud makes sense, Serverless makes sense (offload what you can on to world-class experts who do know what they're doing)... SRE orgs would provide for better RoI than own-what-you-build teams, over the long run. That is, anyway, my armchair take.
hyperman1 5y ago
As a developer , I have less interruptions now I can partially do my own operations. The old world was one where operations did not allow any dev access on production. So you get a ticket from a user, but you have no access to logs or databases or whatever, just a screenshot from the user. Performance testing is only allowed on dev, with a dataset of 2 heavily anonymized records. Deploying to prod required devs to specify paths and server names, but devs were not allowed to know what servers ran the application or what paths were available. Detailed deployment plans are 'optimized' by ops to remove 'unnecesary' steps, then for some strange reason the new version crashes 5 seconds after release, which is promptly declared a 'programmer error'. Preproduction and production were generally configured by different people, so plenty of differences between the environments exist. Neverending migration projects mean your application might accidentally partially disappear, with only user tickets as a signal that something happened. Monitoring is ignored by ops, as it produced more alerts than they can handle. I wonder why, I never got one. So now I as a dev also do ops. I added some monitoring, and every few days I manually check basic health parameters. Problems generally get solved without users even noticing, and I can schedule time for this and work uninterrupted after that. I know what I prefer.

kennethh 5y ago

By departmentalize you get silos and reduced efficiency across teams, by going the other way and having full cycle developers you reduce the silo effect and get alignment of incentives. Again the problem is that you need people who know a lot, and the mental load is very hard and not everybody is interested or able to handle that. As an old developer, this is very much how we operated before CI/CD was a thing in the early 2000. I remember we were responsible for developing, monitoring and support. In our team we had one week each where we did support and monitoring as the primary task.

piokoch 5y ago
I think this all depends on the complexity of operational tasks. If you have hosted MySql and a couple of servers, fine, developer can handle monitoring this. What if you have, say, DB2 cluster with HADR running, do you think you can monitor it and fix something if needed? Do you know ins and outs of share drives configuration using GFS. Kubernetes configuration details, if something goes wrong should every developer have sufficient knowledge to troubleshoot this? How about A10 load balancer, what average developer knows about this, can average developer fix failed virtual IP configuration or firewall rules on A10? Solving Red Hat Satellite cache issues? I don't even mention handling some complicated network setup with third parties connections through proprietary VPN solution, AWS cloud deployments using multitude of services that AWS offers or obscure stuff like zSeries IBM or Unisys mainframes, etc. Developer can be trained in all kind technologies, but there is truly a lot of them, this is huge body of knowledge, learning this will take a lot of time and money.
- nijave 5y ago
  I think you're conflating platform engineering and operations. A DB2 cluster with HADR is another application that some engineering team has stood up. A consumer of the database isn't responsible for the implementation; they're responsible for how their code interacts with and uses the system another (engineering) team built. In that regard, if a developer sees their application degrading due to an issue with the underlying platform, they'd engage the developers that built that platform (and ideally those developers have built a system to detect problems before the "end user" system starts degrading) In the same vein, it's not expected every developer understands kernel and hardware development--there are abstractions and boundaries in place that separate responsibility. The recurring theme is the person that configured/coded/built is responsible for operating it

draw_down 5y ago

Always be thinking about what you can get out of your employer. They’re always thinking about what else they can get out of you.

5tefan 5y ago

"Eat your own dog food" is one of my guiding principles and it's very powerful.

ghostwriter 5y ago
"sure, but I'll eat it from my own plate" aka developers get to choose the tooling they will use and ops have no objections to it.

Stranger43 5y ago

The problem here as with most of the devops literature is that it's only really applicable for the rare usecase of where an entire company's worth of resources is put behind a single website/app deployment, where as the far more normal practice of having small implementation and operations teams manage a large back catalog of projects/deployment that's supposed to run stable with little need for day to day improvements/interventions, and which are often based on products that were oversold by some consultant/salesdrone 5 years ago as virtually maintenance free. And when it comes to big systemic problems with security/stability that plagues the IT industry it's usually this back catalog that's the real cause and the problem that gets the least attention.

jiggawatts 5y ago
Oh god, this. This is the real problem! Everyone wants to sell you the solution to problems where there are 10,000 servers for a single set of related apps managed by 1,000 developers. Think companies like NetFlix, where there are teams of people dedicated to specific parts of DevOps. Nobody has good solutions for legacy but still functional apps developed by some guy named Bob that was a contractor for 6 months and he's gone to some other gig. For example: Take Splunk, or Azure Log Analytics, or any similar tool. They work great if you get 100K hits per day. The graphs are beautifully smooth and the wealth of data makes it trivial to extract all sorts of insights. Approaches like A/B testing can accurately detect changes as small as 1% in many cases. But what do you do with the site that gets 5 hits per week, but is still important because what it does for each one of those hits is worth on the order of a thousands dollars? The tooling is a total letdown, basically nothing out there can help with apps like this. Trying to retrofit some legacy app into Kubernetes or whatever would cost more than app makes in revenue. Leaving it alone is just as bad, because eventually you have to upgrade the server it lives on and the load balancer in front of it. At which point it will break. Or it might not. How would you know? You only get 5 hits a week...
- flukus 5y ago
  > But what do you do with the site that gets 5 hits per week, but is still important because what it does for each one of those hits is worth on the order of a thousands dollars? Log files and few shell scripts can go an awful long way, from simple single user apps to complex corporate environments with dozens of servers. They require almost zero upfront investment and you can build on them as needed. You won't get fancy A/B testing (without a lot of work) or give you a pretty UI with smooth graphs, but grep can happily tear through 100GB worth of log files and get you some of that good stuff you want from azure log analytics.

TobySKT 5y ago

How to Build a Food Delivery app for Restaurant? Starting a food delivery service today will turn into a prosperous business in the future. This guide will help you learn how to create a food delivery app from both business and technical perspectives https://steelkiwi.com/blog/how-build-food-delivery-app-for-restaurant/ https://steelkiwi.com/blog/how-build-food-delivery-app-for-r...

musiccog 5y ago

As an older full-stack/full-cycle developer for many years, this article made sense - and gave quite a bit of hope. If you can't dog-food your product, IMHO developers need to be as close to the coal face of their 'product' as possible. Per the article, assigning a team to a feature has allowed Netflix to get better coding outcomes. With that said, I can also see a potential downside to this development model. Once a new feature is stable enough, the number of people required to support it has to reduce. Wondering how Netflix solved this?

sdevonoes 5y ago

> We mitigate this by having an on-call rotation where developers take turns handling the deployment + operations + support responsibilities Sorry, but no. All the reasons that exist for developers doing on-call rotation are reasonable, yes, I admit that, but my reason to not do on-call rotation is also reasonable: I do not want to give my employer more than 40h/week. I just do not want more money in exchange for my free time. If your company doesn't provide that, then "alright, thank you" and I will keep looking. Now, if all companies start to require (paid) on-call rotation for developers: that would be a very sad tech scenario (at least for people like me).

ghostwriter 5y ago
Charge them with a significantly higher base pay if they are willing to include on-call into your list of responsibilites, or make sure your contract mentions exemption from on-call activities when you change an employer next time. In other words, make them see on-calls in salary budgets.

ofrzeta 5y ago

If you are in the business of developing and running web applications there's a lot overlap between dev and ops in terms of knowledge. If you have to troubleshoot something you need to know HTTP Headers, CORS and related topics. Who is in charge when the latest update of Chrome breaks your clients' websites? Wearing both hats certainly helped us fixing stuff easier by inserting headers through HAProxy although our regular job is programming. What about performance optimization? Can that slow SQL query be fixed through tuning the MySQL server or maybe just rewrite the code?

deleted 5y ago

[deleted]

jacksonkmarley 5y ago

I don't know how prevalent my opinion is amongst developers, but the reason I will never take another job with a support aspect is not because of some detailed assessment of the software lifecycle. It's because I don't like it. And I can't think of anything more annoying than being on-call for work outside regular work hours. So I think it's awesome that some people want to do this work. I hope they do it full-time, and that lets me do my thing full-time. Obviously many developers like the mixed model or accept it. I'm curious about whether I'm an outlier or lots of others feel the same?

potamic 5y ago
Does having a separate on-call really shield the devs from support? I assume in most cases on-call would need to escalate or reach out to the devs for issues and for that someone needs to be reachable on phone off hours?
- jacksonkmarley 5y ago
  I mean I guess it's pretty dependent on the company and product, I just know if I see "on-call" or "support" in a job ad, I pass. Perhaps I'm just screening myself out of certain business areas. My experience was support during business hours as part of my job as a software engineer. I used to hate hearing the phone ring, and it was rarely if ever a problem directly related to what I was developing. I felt like it was just a huge disruption to my work.
leokennis 5y ago
I understand your attitude, and sort of agree with it. If I take a job with on call duty, it also means I will be woken up at night because of "your" mistakes (in code, testing etc.) and that is fine. But a very important aspect for me is that, if that happens, the developers give the correct priority to fixing the bug that woke me up at night. So, it's fine you only want to work during the day. But if I get woken up at night or in the weekend, you better start fixing the bug that caused it as soon as it's day again.
- 5y ago

deleted 5y ago

[deleted]

trestenhortz 5y ago

This is not a good idea. Developers already have an incredible amount to learn without making them devops too. This is the sort of thing than non tech management would love though. It’s a great slogan. Essentially the sign of mismanagement dressed up as innovation.

aoetalks 5y ago

100% agree with this. I work on a service deployed in Azure processing PBs of data each day. We’re a team of about 8 engineers, and we have an on-call rotation. As a result, we’re incentivized to reduce on call noise and have the system automate remediation as much as possible. Granted, our company does have centralized teams that build tooling. So that does seem to be a prerequisite to having full cycle developers. If you’re building in the cloud, I think there’s less need to be a SysAdmin these days (I used to be be one), and Azure makes it pretty easy to have automatic updates, firewall rules, etc. We rarely touch our hosting environment (Service Fabric on VMSS). The on-call sucks at times, but I’d never trade it for how fast we’re able to ship.

deleted 5y ago

[deleted]

pmlnr 5y ago

This used to be called jack of all trades, master of none.

dijit 5y ago

So, I had this argument/conversation with Jedberg (who incidentally also works at Netflix); I said that: DevOps means different things to different people; To Some: it's the evolution of "Operations" to integrate them into the team so that operational issues are automated out and that developers can get greater velocity. To Others: it's a sysadmin who can use CI/CD and fumble through scripting something. To the majority: it's a developer who can fumble through installing a server or package, and this is jedbergs definition too, and one he argues is canonical. My opinion is that specialising is important. --- To this effect: it's _far_ too much cognitive load to understand how your OS works, your application, its framework, the cloud provider, the logging tools, SLI/SLOs and error budgets, on-call policies and alerting mechanisms, application level security, authentication of systems, authorization of systems, SSO, etc;etc;etc;etc; Because to learn all of those various things and weigh them in your head every time you make a decision is fatiguing, it took me 10 years to get "great" at Ops and a decent scripter; it's possible that Ops is easier than programming but from what I understand it's not easier, it's just different. With Development (exception: Javascript) you learn primitives and they rarely change. With Ops, the tools and landscape change _frequently_. 10 years ago it was Nagios, then Zabbix, then some combination of graphite and co, now it's prometheus, maybe Thanos or is it influxDB+Kapacitor? That's just alerting, there's new and changing solutions for logging, message queues, databases, IdP tooling (keycloak, vs FreeIPA vs AD), distributed tracing systems, heck even load balancers (GCE Ingress vs Nginx vs traefik vs etc;etc;etc). Hell, understanding how Kubernetes works is basically a full time job otherwise "black magic is happening" and debugging becomes a nightmare. Frameworks change or get replaced too, but it doesn't feel like it's close to the same rate as infrastructure software. I remember at college one of the professors said to me: "It's impossible to know everything about computers, because as you're learning: new things come out, and after you've learned something: it will change. The more you learn, the more you have to keep up to date until you can no longer keep up." I think maybe people don't think like that. I think we need to have people of different disciplines.

lamontcg 5y ago

I've always been a bit confused that this wasn't what DevOps was supposed to mean from the start. Devs should be producing easy-to-operate code and their oncall should therefore not be burdensome and there should be tight cycles to drive it to near zero.

ChrisArchitect 5y ago

so many different URLs, damn Medium previous discussion, 2 years ago, 20 upvotes, https://news.ycombinator.com/item?id=19481912 https://news.ycombinator.com/item?id=19481912

sunstone 5y ago

Development and operations are two different mindsets not typically found within the same person.

shreevari 5y ago

Hope this term doesn't hit recruiters dictionaries.