18 ms·

DevOps didn’t exist when I started as a developer

267 points by tcsf 7y ago

thinkingkong 7y ago
Generally Ive found tight feedback loops to be incredibly productive. Whether its for product, release, design, development. Having the ability to make modifications with feedback from customers, systems, business partners and colleagues changes everything.
- rmetzler 7y ago
  Yeah, the faster you’re able to iterate, the faster you can improve.
  - Ididntdothis 7y ago
    Only if you make meaningful iterations. I see a lot of demand for iteration coming from above and suddenly you are in this cycle of fixed length “sprints” where you worry more about releasing on time instead of meaningful releases.
- mrkeen 7y ago
  Tight feedback loops is precisely why I dislike the current devops culture. For me, a tight feedback loop is cycling between repl/ide/compiler/tests as quickly as possible. A loose feedback loop is pull requests, docker, CI, terraform. Don't get me wrong - I appreciate a fast, smooth 'release' pipeline. I just hate that it's now become the 'development' pipeline.
peterwwillis 7y ago
Nailed it. I also got a chance to work in such an organization in the 00's, where dev and ops closely aligned with each other's work to produce truly reliable, repeatable, fast, quality products, often. It was fantastic. Nobody would say "oh but you can't do that, that's not Infrastructure as Code!" They would say, "what can we all do to make this better?"
- nickstinemates 7y ago
  Isn't it amazing that working on/as a team is seen as an outlier? We need more of this.
- jascii 7y ago
  In the 90's and 00's I was responsible for maybe 20-30 servers, now I am responsible for somewhere around the 2000 mark.. While "oh but you can't do that, that's not Infrastructure as Code!" sounds lame, and your co-worker should learn to be a better communicator, things need to be reproducible when working with more complex systems and that calls for more rigorous review practices. Please consider this: It's our pagers that go off at 3am when your code fails..
  - walshemj 7y ago
    In my experience ops call the developers when something breaks overnight in any case.
    - jascii 7y ago
      In our organisation, developers have need nor clearance to access live customer data. This means they stay out of production and we have to deal with breakage. Usually I can just revert to an earlier version and file a bug report, sometimes I have to cherry pick. Having developers "fix" things in production under time-stress at 3am without proper code review sounds like extremely poor practice.
      - walshemj 7y ago
        So you have some one every shift who is familiar with every part of a complex system and the requisite programming languages. And reverting part way through a monthy telco billing run might not be the best idea. And in my case they also looked after online services and other systems.
        peterwwillis 7y ago
dvtrn 7y ago
I found this line interesting, everything else considered: DevOps is NOT…easily achieved nor implemented Debatable, at least IMO. DevOps isn't easily achieved nor implemented if you're trying to implement ALL THE THINGS to say you did and check-off a series of "We did DevOps thing x" boxes as so many companies appear to want-at least from reading various DevOps-y job descriptions lately. It is easier implemented if, like Gene Kim tells us in "The DevOps Handbook"-we start our DevOps transformations with a small, sympathetic team and iterate outward.
zwkrt 7y ago
Sidebar: does anyone else get the feeling that infrastructure as code is just cloud vendor lock-in by another name? Especially since the output of the code ends up being wild unstructured JSON/YAML files with no spec or discernible schema. My favorite is how in the Microsoft toolchain you can build a CI pipeline in a visual UI on the right that automatically updates the YAML file on the left. “We know you don’t want YAML, but we also know your boss says you need infra-as-code...”
- kovek 7y ago
  I tried Terraform this weekend and it was awesome to use. Spinning up a new ec2 instance was very quick. Almost just as quick when you spin up multiple instances. It supports "providers", which are interfaces to the different vendors (aws, foogle cloud, etc), and "provisioners", which are frameworks for configuring a running instance (chef? Docker? Zookeeper?)
  - YawningAngel 7y ago
    Plausibly, if what I want is "start a kubernetes cluster with an ingress and a database", that's portable to every cloud provider. However, with terraform, you have to write platform-specific code to do this that isn't portable between providers.
- JBerlinsky 7y ago
  You might want to look into Terraform [1], which works across multiple cloud vendors, with pluggable open-source "providers" facilitating the communication with the backend cloud platform. [1]: https://www.terraform.io/ https://www.terraform.io/
  - LIV2 7y ago
    Terraform is great but if you want to change to a different cloud provider you pretty much have to rewrite everything
    - eropple 7y ago
otabdeveloper4 7y ago
What bunk. In the real world "DevOps" simply means "sysadmin that can write scripts".
gdulli 7y ago
Lots of things people think of as crucial today didn't exist when I started. And the overall ratio of successful work to unsuccessful work was just as high then as it is now.
- jayd16 7y ago
  But the definition of success evolved as well. Richer experiences are expected.
  - deleted 7y ago
    [deleted]
SamuelAdams 7y ago
Personally I think DevOps is like religion. It means whatever someone wants it to mean. Some companies think DevOps means having an automated pipeline for building, testing, and shipping code. Other companies think it means micro-services. Others think it means making developers do DBA / SysAdmin work. All of these things are fine for companies to do. How you run your org is on you. But I wish companies would go deeper than "DevOps" when putting up job posting requirements. That's like saying they do "security". There's a lot of different ways that can be interpreted. I (as an applicant) only know a handful of those interpretations really well, so it's important that companies clarify these terms in their job descriptions. This way they have the right candidates applying.
- JMTQp8lwXL 7y ago
  > Some companies think DevOps means having an automated pipeline for building, testing, and shipping code. This seems to be the most common interpretation, but I largely agree with you on no hard and fast rule here.
  - ownagefool 7y ago
    It's because it ultimately does mean that. I mean, sure, if we're being pedantic about it, devops is about the silos coming closer together, cross learning, culture and what have you, etc. But the outcome of that the cross learning is pretty much release engineering, that includes the above and the products giving more data for helping run them, and simplicity leading to being easier to fix, and do the aforementioned things. I also think we should just get over it and probably just adopt SRE, Release Engineer and Product. Not because I <3 google, just because the split makes a tad bit more sense.
- n42 7y ago
  my bar for "DevOps" on my (SaaS) team is that the engineers responsible for the platform the product runs on are involved in the discussions about the product we build. everything that comes after that: pipelines, testing, release engineering, migrations, post-mortems, whatever -- it's all a result of your infrastructure engineers having a stake in the thing your company is building. I have yet to see a definition of DevOps that invoked some sort of tech stack that made any sense to me. the processes and tools emerge from a cooperation between stakeholders and a shared responsibility for the delivered product.
Nursie 7y ago
> DevOps is NOT… a job title or role In which case DevOps as it exists in the real world is roughly on a par with 'agile' practices in the real world. Nothing like they were originally conceived, and nothing like the wildly optimistic descriptions of them that are shared amongst practitioners.
- MTZ-x86 7y ago
  Agile: We don't have a formalized process of developing software, just wing it. DevOps: We don't have admins, so you are reponsible to run the systems you develop. This is at least how I perceive it. But the second point is not the worst thing that could have happened. I hate to write configs as much as anyone, but it has given me a better perspective on how to design systems that actually... you know... run.
irrational 7y ago
This article was frustrating to me. When it started out in the mid-90s (which is when I started my career also), I thought our experiences would be more aligned. However, I've never worked with an operations team. In all the places I've worked I've been expected to do everything from setting up servers (originally physical servers, later cloud servers), hardening them, installing software, optimizing the software, installing and optimizing the database, creating database schemas and related objects, writing untold number of sql queries, writing server side code, writing front end code, etc. I was hoping that the article would explain what devops really means today and how I can jump on the devops wagon to hopefully make my job of doing all of the above easier.
- commandlinefan 7y ago
  > make my job of doing all of the above easier. That’s the dream… the reality in my experience is that you have to describe, in detail, the exact steps that you would perform if you were doing it yourself so that somebody else can perform them, verbatim. But you have to describe those steps, exactly, without any access to the target platform where they’ll be performed.
  - Andhurati 7y ago
    Does it get any better? I feel like I'm on thin ice whenever I deploy. Granted, I now try to make sure the SAs have to do as little as possible (because now I "understand" the infrastructure, I can build a postinstall that handles all the environment configuration I am allowed under an FID), it still feels nerve wracking to know that a single fuck up means I have to spend another 4-5 hours trying to get tickets to deploy. Fuck this.
- walshemj 7y ago
  And its also written from a very recent perspective, arguably I was formally working in devops in the 80's I was hired as development team member partially as I was a sysadmin on the hardware we were using, our team both wrote the code but also the entire JCL infrastructure that managed the compilation linking, deployment and the running of code across the cluster of 16 or 17 systems. On of my friends, an IBM guy from the operations side was shocked that we as developers where allowed to deploy and run our own code.
sigzero 7y ago
DevOps is such a cluster f*ck.
- boznz 7y ago
  As a job title I agree
trollied 7y ago
All that’s really happened is that developers have learned or taken on the responsibility to build and deploy to production, and also be knowledgeable and responsible for the development stack, hence “full stack developer”. Unfortunately, it’s incredibly dangerous if everyone involved doesn’t know what they’re doing. Then kubernites and docker turn up. To make things easy? No. They’re making the infrastructure more difficult to manage so that the developers can deploy things easier. Developers should not be deploying. Large businesses should not be doing very regular deploys to live - it’s simply too risky without a shitload of testing. Everyone praises startups with a few million customers that are open about their fuckups/downtime, but they are almost always because of their shitty modern deployments and lack of testing. Crikey, look at Monzo. “We use a shit database and fucked up scaling it, but sorry your transactions failed and you looked like a twat buying your coffee”. Try and revolutionise industries, fine. But ....
- deleted 7y ago
  [deleted]
- YawningAngel 7y ago
  If you want a database that is arbitrarily horizontally scalable, there aren't really good open source options. As a former employee it isn't clear to me that Monzo couldn't have used boring stuff for a while and switched to C* or some equivalent later when it was genuinely needed, but it's worked well enough for them that I'm not inclined to critique the decision post hoc.
  - trollied 7y ago
    Their last outage worries me greatly.
    - YawningAngel 7y ago
      I think there's a clear critique of Cassandra in there, in that it shouldn't even be possible to configure nodes to serve data they don't have. Monzo... It's very hard to say. I think based on the contents of the blog post they acted reasonably and competently (yeah, the test rollout was a badly designed test, but that's easy to say post hoc). The only fair basis for critiquing Monzo would be if they had notably more averages than is 'typical', and we don't have the data or conceptual framework required to do that. As a customer, I can't think of risks that Monzo exposes you to that aren't also present with other banks. I bank with them ¯\ _(ツ)_/¯
johngalt 7y ago
DevOps is the intersection of three things. 1. Developers learning that running code is not the same thing as reliable code. Certain things must be put in at design time to allow for operations. 2. Operations people supporting development by formalizing/streamlining the deployment process to have changes occur faster, safer, and more often. 3. Aligning goals and attitudes in such a way that prevents conflict between teams. Development doesn't get to shift the costs of outage prone unmaintainable code onto an ops team. Or that ops doesn't roadblock/veto all changes. In the bad old days, it was common for developers to act with little care for operational outcomes. The responsibility was far removed, and bugfixes didn't win many accolades. Managers want wish fulfillment and pushed developers to toss new features into the mix as fast as possible, resulting in tremendous pressure against ops teams to deploy despite increasingly crufty and debt laden code bases causing unpredictable problems. Ergo, ops teams who blocked all changes for trivial issues because they were judged only based on outages, not feature delivery.
- eternalban 7y ago
  DevOps is blue collar work. Software Engineering used to be white collar work. Think boiler room vs drafting room.
  - scarface74 7y ago
    How is Devops "blue collar work"? Provisioning servers, databases, load balancers, etc and other resources use to involve someone driving down to the colo and installing hardware. Often now, it's a yaml file you run through Cloud Formation. Devops is just as much development and software engineering when you are working in a cloud environment -- it's an API call.
    - viklove 7y ago
      That's exactly the reason. Software engineering is about writing code, devops is about writing yaml files and keying into functionality that was implemented by software engineers.
LoSboccacc 7y ago
it wasn't his own thing and there wasn't the scale yet to have it was a proper position, but when you were moving connection strings in a config file and linking it in etc you were doing devops. when you were building a shell script to fetch pinned dependencies so that the environment was repeatable, you were doing devops. weekend you were automating the package build/install and putting these files in CVS, you were doing devops. heck we had versioned openpkg scripts in 200something to quickly rotate servers during upgrades, that wasn't called with fancy names and wasn't a standalone role but the seed was there.
alexnewman 7y ago
Devops has been standard practice at every financial firm i ever went to. Back then it meant devs where a pager. Ops also wore a pager and would support us.
slowhand09 7y ago
IMHO DevOps has been around for many years. Is has just evolved as tooling became better and eventually tooling to connect tooling became better. It became META. When I started doing database work we developed and implemented. We learned to produce reports on what we'd implemented by querying system tables. We learned to use those reports to produce scripts that could recreate the database on the same or a different platform.
abjKT26nO8 7y ago
Could web developers please stop using floating top bars? It breaks the UX of the "Page Down" button. It's annoying beyond imagination. Thank you.
- Roboprog 7y ago
  I resisted. Management insisted. Good thing it was a non public app instead of a blog
aodin 7y ago
The change in job description from just "operations" to "developer operations" over the past decade is best captured by the growing number of systems that must be managed per person, often larger by a factor of 10 times or more (e.g. going from 10 systems per person to 100+). I appreciate the self-reflective views on this "new" job role, but they often seem to miss the rather basic market forces at work. More systems, more automation, new skills and demands.
awd 7y ago
I like to comment DevOps from a security perspective, a trend I noticed in my day job. Windows is/was often bashed for being insecure. Lots of that stems from the decades of development related to centralized management solutions. A default windows workstation in a domain setting will open a bunch of ports, a bunch of which can be used for command execution. The attack surface for this system includes, but is not limited to: - Remote access with local admin users via tools such as SMBExec, wmiExec, DCOM, Psexec, Powershell remoting - Remote access domain admin users access via the same - Local/domain admin access via RDP - Remote domain admin access via group policy All these have had their own associated vulnerabilities over the years. Examples are SMB relay attacks, which enabled an attacker to abuse flaws in NetNTLM and obtain access to machines by relaying other people's credentials. And then we're not even talking about the 'real' exploits, Eternal Blue, Eternal Romance, Blue Keep, MS14-068, MS08-067, and on and on. Pentesters, researchers and Microsoft have been hammering away the kinks for years now. The 'fixes' and root causes for each individual issue are well understood and each new domain functional level increases the security of a default windows Domain by leaps and bounds. When you look at the Unix/Linux side you'd see that no such attack surface ever existed. You manage your systems over SSH, and this can still be bad, an easily guessable root password shared between Dev, testing and production is still a death sentence. But by default there were no tier0 systems in your network, apart from those of sysadmins. But now with DevOps things are changing on that side. With Ansible, Puppet, Terraform, your various container management systems, the CI pipeline, jenkins and numerous development teams able to push both to infra repositories and your actual products this has changed: You use an automated CI pipeline? Any system in the chain is a tier0 system. Your developers are maintainer status or higher in your source repositories? Then they are domain admin or equivalent. They can disable protected branches, push a backdoor, and watch their attack propagate through the pipeline. Did you make it inconvenient for your developers to access various build systems? Then they are sharing credentials to these systems over your company chat. It seems, from what I've seen so far, that while the 'architecture' of modern mass centralized IT management and development is more secure. You can't relay an SSH key for example, like you can in NetNTLM. But the institutional knowledge isn't there yet. New attack surface has opened up, and infosec people have not yet completely caught up with the new 'eggs' in the basket, even if they are aware.
- 7y ago
shipit 7y ago
The goal of DevOps is to get code as developed to testing, staging, production. The deeper objective should be transfer of context- what is being deployed? do we know how? does it work as tested? and the who. As much DRY to reduce surprises and increase precision all through dev to deploy cycle. Some of the best open source projects have CI included- thats the right approach to DevOps -- its not an alien, bolt-on after the fact practice, its code that takes care of code.
deleted 7y ago
[deleted]
njharman 7y ago
[The (or any) Internet didn't exist when I started as a developer] DevOps is about how CTO organizes engineering. Is there an Infra team that supports everyone, maybe a separate Ops/security/etc teams that supports everyone. Then a bunch of App/Service dev teams. This is not devops. Or is there only* a bunch App/Service teams, each wholly self-supporting and independent. Each doing their own infra, dev, ops. Maybe some teams' "Service" is used by only other internal teams (point being you can break up scope as fine grained as you want). The key is teams are self-supporting, taking care of the entire life-cycle of an App/Service. Most "devops" sort of isn't or fails cause it's not "deployed as an organizational system, top-down (from CTO) across the whole organization. All the things people typically bike shed over re: devops don't matter. Are you independent and self-contained? Are you responsible for entire life-cycle of Product/Service/App/Etc? Does the buck stop with you? Then you are devops.
- vardump 7y ago
  > [The (or any) Internet didn't exist when I started as a developer] Internet is widely considered to have started in 1969. Although I'd consider something between 1982-1986 as the years when it truly became an international network. I've been in this business only for less than 25 years, you make me feel so young. :-) During that time, internet has just always existed. Ah, those loong boolean logic web search (hxxp://altavista.digital.com/) queries during my early years.
lanstin 7y ago
In 1997, when my first code I wrote went live, they gave me a pager and said "Welcome to Ops." Each time the code in production had an abnormal end, I got an email. I was expected to fix it. Managers and directors discussed these metrics.
suyash 7y ago
One could argue that it still doesn't exist today at most companies. I think whatever most people call DevOps is an evolution of Sys Admin role in the era of Cloud Computing.
oliverwiegers 7y ago
DevOps is pretty much not a technology but a mindset
auslander 7y ago
CircleCI trusts 8 analytics companies with your source code and API tokens https://kevin.burke.dev/kevin/circleci-is-hopelessly-insecure/ https://kevin.burke.dev/kevin/circleci-is-hopelessly-insecur...
louwrentius 7y ago
I thought that DevOps means: "doing operations with a development mindset." That captures the gist of it imho.
sytse 7y ago
I think the evolution is Waterfall to Agile to DevOps. 1. Waterfall is a cycle time of weeks 2. Agile is a cycle time of days 3. DevOps is the same cycle time as agile but delivering the the user instead of just showing a demo at the end of a sprint.
damontal 7y ago
I miss Devops Borat. https://twitter.com/devops_borat https://twitter.com/devops_borat
ConcernedCoder 7y ago
DevOps didn't exist... fella, let me tell you something, when I started, color screens were not the norm and you kept your homemade double-sided 5 1/4" floppy disks ( thank-you paper hole-punch ) in a plastic flip-top container with the contents written on the label in black sharpie.
algaeontoast 7y ago
Well, I wouldn't exactly call Dev Ops a productive career path. Most senior engineers I know seem to think it's a surefire way to drive your career into a dead end.
rurban 7y ago
It certainly did exist when I started my career a decade earlier than this guy. It was called the integration team, which was responsible for coordination of the release. In the 80ies. It was also responsible for the build servers, the SCM, and checking and prioritizing all the different parts and fixes of the various dev teams, and checking back with the PM and testers. Only hackers did cowboy coding, companies preferred processes and best practices. That's it was called "Software Engineering" and not hacking. I heard Margaret Hamilton coined that term in the 60ies for the Apollo project, which she managed.
- gaahrdner 7y ago
  Get 'em! Pretty sure it's just called "teamwork!"
gaahrdner 7y ago
It didn't exist when I started either, but talk about a let-down of an article conclusion!
caymanjim 7y ago
DevOps absolutely existed when the author started as a developer. It was just called systems administration back then. There's been a focus on developer-specific systems administration over the past decade, and a particular developer-focused role has been carved out and labeled "DevOps", but make no mistake: it is systems administration. Just a niche within it. When I mention "systems administration" to younger coworkers, they look at me like I'm a crusty old relic out of touch with modern engineering. People have no sense of history and little understanding of where the devops role came from and what it really means. I had a CTO tell me he hadn't heard the term "system administration" in ten years. What were our devops people doing? Running web servers, managing networks, configuring DNS, managing backups, configuring cloud services. Absolutely none of it in support of the dev team I was on. The developers were responsible for managing their own CI/CD pipeline and deployments. The people called the "DevOps Team" were responsible for managing production. In this case, they were systems administrators and were not in any way a devops team, but the terminology is so skewed now that they were labeled as such, and I was labeled an out of touch old greybeard. This isn't to demean devops in any way; it's a valuable role, an evolutionary step in software team organization, but by no means is it new and by no means is systems administration a dead role.
- strken 7y ago
  When I hear systems administration I assume it's pets not cattle, and that you're administering individual named servers with bash scripts rather than administering a mesos cluster with chef or puppet or whatever. I'm not sure where I picked up this impression from.
  - alchemism 7y ago
    It comes from bright, talented SWEs who do not, bless their hearts, understand that IT Operations is an entire profession unto its own, and not merely the grunt-work they are forced to do while working as interns or juniors.
- sunaurus 7y ago
  I know that "DevOps" is one of those terms that can mean whatever the user wants it to mean (like "agile" and "REST" and so on), but at least with DevOps, I've mostly only seen 2 types of common definitions: 1) "DevOps teams" are just rebranded operations teams 2) "DevOps teams" are teams that are responsible for both development and operations for their app I think the first one is the one you're frustrated with, and I agree it's a completely useless label when used like that, but it's not definitely not the only way it's used.
test111 7y ago
It's a critical skill now
omani 7y ago
cant believe so many people still get it wrong. devops means Developer/Operations. means, the sysadmin/sysops/ops guy now has to know things about development to center his daily-doing around code (ideally the code dictates what is going on [Infrastructure as Code]). a very simple example to this: back then the ops guy or sysadmin did everything by hand. today, he uses code to get things done. bash vs. a HTTP JSON API. manually by hand vs. ansible [markup language yaml], etc. so many people just dont know or understand what devops means. but it is so easy. it is the fusion of development and operations. thus, while you need two people back then, now you only need one person to do the same job. a devops is a sysadmin who ideally knows how to code. a devops is a developer who evolved into a sysadmin or is doing both because he learned it. devops is development/operations. it has nothing to do with agile, waterfall or a "mindset".
luord 7y ago
> “a developer with operational tendencies” Hmmm, I like the ring to that, might put it on my linkedin profile. On the article itself, can't say I disagree.

267 points by tcsf 7y ago

Generally Ive found tight feedback loops to be incredibly productive. Whether its for product, release, design, development. Having the ability to make modifications with feedback from customers, systems, business partners and colleagues changes everything.

rmetzler 7y ago
Yeah, the faster you’re able to iterate, the faster you can improve.
- Ididntdothis 7y ago
  Only if you make meaningful iterations. I see a lot of demand for iteration coming from above and suddenly you are in this cycle of fixed length “sprints” where you worry more about releasing on time instead of meaningful releases.
mrkeen 7y ago
Tight feedback loops is precisely why I dislike the current devops culture. For me, a tight feedback loop is cycling between repl/ide/compiler/tests as quickly as possible. A loose feedback loop is pull requests, docker, CI, terraform. Don't get me wrong - I appreciate a fast, smooth 'release' pipeline. I just hate that it's now become the 'development' pipeline.

peterwwillis 7y ago

Nailed it. I also got a chance to work in such an organization in the 00's, where dev and ops closely aligned with each other's work to produce truly reliable, repeatable, fast, quality products, often. It was fantastic. Nobody would say "oh but you can't do that, that's not Infrastructure as Code!" They would say, "what can we all do to make this better?"

nickstinemates 7y ago
Isn't it amazing that working on/as a team is seen as an outlier? We need more of this.
jascii 7y ago
In the 90's and 00's I was responsible for maybe 20-30 servers, now I am responsible for somewhere around the 2000 mark.. While "oh but you can't do that, that's not Infrastructure as Code!" sounds lame, and your co-worker should learn to be a better communicator, things need to be reproducible when working with more complex systems and that calls for more rigorous review practices. Please consider this: It's our pagers that go off at 3am when your code fails..
- walshemj 7y ago
  In my experience ops call the developers when something breaks overnight in any case.
  - jascii 7y ago
    In our organisation, developers have need nor clearance to access live customer data. This means they stay out of production and we have to deal with breakage. Usually I can just revert to an earlier version and file a bug report, sometimes I have to cherry pick. Having developers "fix" things in production under time-stress at 3am without proper code review sounds like extremely poor practice.
    - walshemj 7y ago
      So you have some one every shift who is familiar with every part of a complex system and the requisite programming languages. And reverting part way through a monthy telco billing run might not be the best idea. And in my case they also looked after online services and other systems.
      - peterwwillis 7y ago

dvtrn 7y ago

I found this line interesting, everything else considered: DevOps is NOT…easily achieved nor implemented Debatable, at least IMO. DevOps isn't easily achieved nor implemented if you're trying to implement ALL THE THINGS to say you did and check-off a series of "We did DevOps thing x" boxes as so many companies appear to want-at least from reading various DevOps-y job descriptions lately. It is easier implemented if, like Gene Kim tells us in "The DevOps Handbook"-we start our DevOps transformations with a small, sympathetic team and iterate outward.

zwkrt 7y ago

Sidebar: does anyone else get the feeling that infrastructure as code is just cloud vendor lock-in by another name? Especially since the output of the code ends up being wild unstructured JSON/YAML files with no spec or discernible schema. My favorite is how in the Microsoft toolchain you can build a CI pipeline in a visual UI on the right that automatically updates the YAML file on the left. “We know you don’t want YAML, but we also know your boss says you need infra-as-code...”

kovek 7y ago
I tried Terraform this weekend and it was awesome to use. Spinning up a new ec2 instance was very quick. Almost just as quick when you spin up multiple instances. It supports "providers", which are interfaces to the different vendors (aws, foogle cloud, etc), and "provisioners", which are frameworks for configuring a running instance (chef? Docker? Zookeeper?)
- YawningAngel 7y ago
  Plausibly, if what I want is "start a kubernetes cluster with an ingress and a database", that's portable to every cloud provider. However, with terraform, you have to write platform-specific code to do this that isn't portable between providers.
JBerlinsky 7y ago
You might want to look into Terraform [1], which works across multiple cloud vendors, with pluggable open-source "providers" facilitating the communication with the backend cloud platform. [1]: https://www.terraform.io/ https://www.terraform.io/
- LIV2 7y ago
  Terraform is great but if you want to change to a different cloud provider you pretty much have to rewrite everything
  - eropple 7y ago

otabdeveloper4 7y ago

What bunk. In the real world "DevOps" simply means "sysadmin that can write scripts".

gdulli 7y ago

Lots of things people think of as crucial today didn't exist when I started. And the overall ratio of successful work to unsuccessful work was just as high then as it is now.

jayd16 7y ago
But the definition of success evolved as well. Richer experiences are expected.
- deleted 7y ago
  [deleted]

SamuelAdams 7y ago

Personally I think DevOps is like religion. It means whatever someone wants it to mean. Some companies think DevOps means having an automated pipeline for building, testing, and shipping code. Other companies think it means micro-services. Others think it means making developers do DBA / SysAdmin work. All of these things are fine for companies to do. How you run your org is on you. But I wish companies would go deeper than "DevOps" when putting up job posting requirements. That's like saying they do "security". There's a lot of different ways that can be interpreted. I (as an applicant) only know a handful of those interpretations really well, so it's important that companies clarify these terms in their job descriptions. This way they have the right candidates applying.

JMTQp8lwXL 7y ago
> Some companies think DevOps means having an automated pipeline for building, testing, and shipping code. This seems to be the most common interpretation, but I largely agree with you on no hard and fast rule here.
- ownagefool 7y ago
  It's because it ultimately does mean that. I mean, sure, if we're being pedantic about it, devops is about the silos coming closer together, cross learning, culture and what have you, etc. But the outcome of that the cross learning is pretty much release engineering, that includes the above and the products giving more data for helping run them, and simplicity leading to being easier to fix, and do the aforementioned things. I also think we should just get over it and probably just adopt SRE, Release Engineer and Product. Not because I <3 google, just because the split makes a tad bit more sense.
n42 7y ago
my bar for "DevOps" on my (SaaS) team is that the engineers responsible for the platform the product runs on are involved in the discussions about the product we build. everything that comes after that: pipelines, testing, release engineering, migrations, post-mortems, whatever -- it's all a result of your infrastructure engineers having a stake in the thing your company is building. I have yet to see a definition of DevOps that invoked some sort of tech stack that made any sense to me. the processes and tools emerge from a cooperation between stakeholders and a shared responsibility for the delivered product.

Nursie 7y ago

> DevOps is NOT… a job title or role In which case DevOps as it exists in the real world is roughly on a par with 'agile' practices in the real world. Nothing like they were originally conceived, and nothing like the wildly optimistic descriptions of them that are shared amongst practitioners.

MTZ-x86 7y ago
Agile: We don't have a formalized process of developing software, just wing it. DevOps: We don't have admins, so you are reponsible to run the systems you develop. This is at least how I perceive it. But the second point is not the worst thing that could have happened. I hate to write configs as much as anyone, but it has given me a better perspective on how to design systems that actually... you know... run.

irrational 7y ago

This article was frustrating to me. When it started out in the mid-90s (which is when I started my career also), I thought our experiences would be more aligned. However, I've never worked with an operations team. In all the places I've worked I've been expected to do everything from setting up servers (originally physical servers, later cloud servers), hardening them, installing software, optimizing the software, installing and optimizing the database, creating database schemas and related objects, writing untold number of sql queries, writing server side code, writing front end code, etc. I was hoping that the article would explain what devops really means today and how I can jump on the devops wagon to hopefully make my job of doing all of the above easier.

commandlinefan 7y ago
> make my job of doing all of the above easier. That’s the dream… the reality in my experience is that you have to describe, in detail, the exact steps that you would perform if you were doing it yourself so that somebody else can perform them, verbatim. But you have to describe those steps, exactly, without any access to the target platform where they’ll be performed.
- Andhurati 7y ago
  Does it get any better? I feel like I'm on thin ice whenever I deploy. Granted, I now try to make sure the SAs have to do as little as possible (because now I "understand" the infrastructure, I can build a postinstall that handles all the environment configuration I am allowed under an FID), it still feels nerve wracking to know that a single fuck up means I have to spend another 4-5 hours trying to get tickets to deploy. Fuck this.
walshemj 7y ago
And its also written from a very recent perspective, arguably I was formally working in devops in the 80's I was hired as development team member partially as I was a sysadmin on the hardware we were using, our team both wrote the code but also the entire JCL infrastructure that managed the compilation linking, deployment and the running of code across the cluster of 16 or 17 systems. On of my friends, an IBM guy from the operations side was shocked that we as developers where allowed to deploy and run our own code.

sigzero 7y ago

DevOps is such a cluster f*ck.

boznz 7y ago
As a job title I agree

trollied 7y ago

All that’s really happened is that developers have learned or taken on the responsibility to build and deploy to production, and also be knowledgeable and responsible for the development stack, hence “full stack developer”. Unfortunately, it’s incredibly dangerous if everyone involved doesn’t know what they’re doing. Then kubernites and docker turn up. To make things easy? No. They’re making the infrastructure more difficult to manage so that the developers can deploy things easier. Developers should not be deploying. Large businesses should not be doing very regular deploys to live - it’s simply too risky without a shitload of testing. Everyone praises startups with a few million customers that are open about their fuckups/downtime, but they are almost always because of their shitty modern deployments and lack of testing. Crikey, look at Monzo. “We use a shit database and fucked up scaling it, but sorry your transactions failed and you looked like a twat buying your coffee”. Try and revolutionise industries, fine. But ....

deleted 7y ago
[deleted]
YawningAngel 7y ago
If you want a database that is arbitrarily horizontally scalable, there aren't really good open source options. As a former employee it isn't clear to me that Monzo couldn't have used boring stuff for a while and switched to C* or some equivalent later when it was genuinely needed, but it's worked well enough for them that I'm not inclined to critique the decision post hoc.
- trollied 7y ago
  Their last outage worries me greatly.
  - YawningAngel 7y ago
    I think there's a clear critique of Cassandra in there, in that it shouldn't even be possible to configure nodes to serve data they don't have. Monzo... It's very hard to say. I think based on the contents of the blog post they acted reasonably and competently (yeah, the test rollout was a badly designed test, but that's easy to say post hoc). The only fair basis for critiquing Monzo would be if they had notably more averages than is 'typical', and we don't have the data or conceptual framework required to do that. As a customer, I can't think of risks that Monzo exposes you to that aren't also present with other banks. I bank with them ¯\ _(ツ)_/¯

johngalt 7y ago

DevOps is the intersection of three things. 1. Developers learning that running code is not the same thing as reliable code. Certain things must be put in at design time to allow for operations. 2. Operations people supporting development by formalizing/streamlining the deployment process to have changes occur faster, safer, and more often. 3. Aligning goals and attitudes in such a way that prevents conflict between teams. Development doesn't get to shift the costs of outage prone unmaintainable code onto an ops team. Or that ops doesn't roadblock/veto all changes. In the bad old days, it was common for developers to act with little care for operational outcomes. The responsibility was far removed, and bugfixes didn't win many accolades. Managers want wish fulfillment and pushed developers to toss new features into the mix as fast as possible, resulting in tremendous pressure against ops teams to deploy despite increasingly crufty and debt laden code bases causing unpredictable problems. Ergo, ops teams who blocked all changes for trivial issues because they were judged only based on outages, not feature delivery.

eternalban 7y ago
DevOps is blue collar work. Software Engineering used to be white collar work. Think boiler room vs drafting room.
- scarface74 7y ago
  How is Devops "blue collar work"? Provisioning servers, databases, load balancers, etc and other resources use to involve someone driving down to the colo and installing hardware. Often now, it's a yaml file you run through Cloud Formation. Devops is just as much development and software engineering when you are working in a cloud environment -- it's an API call.
  - viklove 7y ago
    That's exactly the reason. Software engineering is about writing code, devops is about writing yaml files and keying into functionality that was implemented by software engineers.

LoSboccacc 7y ago

it wasn't his own thing and there wasn't the scale yet to have it was a proper position, but when you were moving connection strings in a config file and linking it in etc you were doing devops. when you were building a shell script to fetch pinned dependencies so that the environment was repeatable, you were doing devops. weekend you were automating the package build/install and putting these files in CVS, you were doing devops. heck we had versioned openpkg scripts in 200something to quickly rotate servers during upgrades, that wasn't called with fancy names and wasn't a standalone role but the seed was there.

alexnewman 7y ago

Devops has been standard practice at every financial firm i ever went to. Back then it meant devs where a pager. Ops also wore a pager and would support us.

slowhand09 7y ago

IMHO DevOps has been around for many years. Is has just evolved as tooling became better and eventually tooling to connect tooling became better. It became META. When I started doing database work we developed and implemented. We learned to produce reports on what we'd implemented by querying system tables. We learned to use those reports to produce scripts that could recreate the database on the same or a different platform.

abjKT26nO8 7y ago

Could web developers please stop using floating top bars? It breaks the UX of the "Page Down" button. It's annoying beyond imagination. Thank you.

Roboprog 7y ago
I resisted. Management insisted. Good thing it was a non public app instead of a blog

aodin 7y ago

The change in job description from just "operations" to "developer operations" over the past decade is best captured by the growing number of systems that must be managed per person, often larger by a factor of 10 times or more (e.g. going from 10 systems per person to 100+). I appreciate the self-reflective views on this "new" job role, but they often seem to miss the rather basic market forces at work. More systems, more automation, new skills and demands.

awd 7y ago

I like to comment DevOps from a security perspective, a trend I noticed in my day job. Windows is/was often bashed for being insecure. Lots of that stems from the decades of development related to centralized management solutions. A default windows workstation in a domain setting will open a bunch of ports, a bunch of which can be used for command execution. The attack surface for this system includes, but is not limited to: - Remote access with local admin users via tools such as SMBExec, wmiExec, DCOM, Psexec, Powershell remoting - Remote access domain admin users access via the same - Local/domain admin access via RDP - Remote domain admin access via group policy All these have had their own associated vulnerabilities over the years. Examples are SMB relay attacks, which enabled an attacker to abuse flaws in NetNTLM and obtain access to machines by relaying other people's credentials. And then we're not even talking about the 'real' exploits, Eternal Blue, Eternal Romance, Blue Keep, MS14-068, MS08-067, and on and on. Pentesters, researchers and Microsoft have been hammering away the kinks for years now. The 'fixes' and root causes for each individual issue are well understood and each new domain functional level increases the security of a default windows Domain by leaps and bounds. When you look at the Unix/Linux side you'd see that no such attack surface ever existed. You manage your systems over SSH, and this can still be bad, an easily guessable root password shared between Dev, testing and production is still a death sentence. But by default there were no tier0 systems in your network, apart from those of sysadmins. But now with DevOps things are changing on that side. With Ansible, Puppet, Terraform, your various container management systems, the CI pipeline, jenkins and numerous development teams able to push both to infra repositories and your actual products this has changed: You use an automated CI pipeline? Any system in the chain is a tier0 system. Your developers are maintainer status or higher in your source repositories? Then they are domain admin or equivalent. They can disable protected branches, push a backdoor, and watch their attack propagate through the pipeline. Did you make it inconvenient for your developers to access various build systems? Then they are sharing credentials to these systems over your company chat. It seems, from what I've seen so far, that while the 'architecture' of modern mass centralized IT management and development is more secure. You can't relay an SSH key for example, like you can in NetNTLM. But the institutional knowledge isn't there yet. New attack surface has opened up, and infosec people have not yet completely caught up with the new 'eggs' in the basket, even if they are aware.

7y ago

shipit 7y ago

The goal of DevOps is to get code as developed to testing, staging, production. The deeper objective should be transfer of context- what is being deployed? do we know how? does it work as tested? and the who. As much DRY to reduce surprises and increase precision all through dev to deploy cycle. Some of the best open source projects have CI included- thats the right approach to DevOps -- its not an alien, bolt-on after the fact practice, its code that takes care of code.

deleted 7y ago

[deleted]

njharman 7y ago

[The (or any) Internet didn't exist when I started as a developer] DevOps is about how CTO organizes engineering. Is there an Infra team that supports everyone, maybe a separate Ops/security/etc teams that supports everyone. Then a bunch of App/Service dev teams. This is not devops. Or is there only* a bunch App/Service teams, each wholly self-supporting and independent. Each doing their own infra, dev, ops. Maybe some teams' "Service" is used by only other internal teams (point being you can break up scope as fine grained as you want). The key is teams are self-supporting, taking care of the entire life-cycle of an App/Service. Most "devops" sort of isn't or fails cause it's not "deployed as an organizational system, top-down (from CTO) across the whole organization. All the things people typically bike shed over re: devops don't matter. Are you independent and self-contained? Are you responsible for entire life-cycle of Product/Service/App/Etc? Does the buck stop with you? Then you are devops.

vardump 7y ago
> [The (or any) Internet didn't exist when I started as a developer] Internet is widely considered to have started in 1969. Although I'd consider something between 1982-1986 as the years when it truly became an international network. I've been in this business only for less than 25 years, you make me feel so young. :-) During that time, internet has just always existed. Ah, those loong boolean logic web search (hxxp://altavista.digital.com/) queries during my early years.

lanstin 7y ago

In 1997, when my first code I wrote went live, they gave me a pager and said "Welcome to Ops." Each time the code in production had an abnormal end, I got an email. I was expected to fix it. Managers and directors discussed these metrics.

suyash 7y ago

One could argue that it still doesn't exist today at most companies. I think whatever most people call DevOps is an evolution of Sys Admin role in the era of Cloud Computing.

oliverwiegers 7y ago

DevOps is pretty much not a technology but a mindset

auslander 7y ago

CircleCI trusts 8 analytics companies with your source code and API tokens https://kevin.burke.dev/kevin/circleci-is-hopelessly-insecure/ https://kevin.burke.dev/kevin/circleci-is-hopelessly-insecur...

louwrentius 7y ago

I thought that DevOps means: "doing operations with a development mindset." That captures the gist of it imho.

sytse 7y ago

I think the evolution is Waterfall to Agile to DevOps. 1. Waterfall is a cycle time of weeks 2. Agile is a cycle time of days 3. DevOps is the same cycle time as agile but delivering the the user instead of just showing a demo at the end of a sprint.

damontal 7y ago

I miss Devops Borat. https://twitter.com/devops_borat https://twitter.com/devops_borat

ConcernedCoder 7y ago

DevOps didn't exist... fella, let me tell you something, when I started, color screens were not the norm and you kept your homemade double-sided 5 1/4" floppy disks ( thank-you paper hole-punch ) in a plastic flip-top container with the contents written on the label in black sharpie.

algaeontoast 7y ago

Well, I wouldn't exactly call Dev Ops a productive career path. Most senior engineers I know seem to think it's a surefire way to drive your career into a dead end.

rurban 7y ago

It certainly did exist when I started my career a decade earlier than this guy. It was called the integration team, which was responsible for coordination of the release. In the 80ies. It was also responsible for the build servers, the SCM, and checking and prioritizing all the different parts and fixes of the various dev teams, and checking back with the PM and testers. Only hackers did cowboy coding, companies preferred processes and best practices. That's it was called "Software Engineering" and not hacking. I heard Margaret Hamilton coined that term in the 60ies for the Apollo project, which she managed.

gaahrdner 7y ago
Get 'em! Pretty sure it's just called "teamwork!"

gaahrdner 7y ago

It didn't exist when I started either, but talk about a let-down of an article conclusion!

caymanjim 7y ago

DevOps absolutely existed when the author started as a developer. It was just called systems administration back then. There's been a focus on developer-specific systems administration over the past decade, and a particular developer-focused role has been carved out and labeled "DevOps", but make no mistake: it is systems administration. Just a niche within it. When I mention "systems administration" to younger coworkers, they look at me like I'm a crusty old relic out of touch with modern engineering. People have no sense of history and little understanding of where the devops role came from and what it really means. I had a CTO tell me he hadn't heard the term "system administration" in ten years. What were our devops people doing? Running web servers, managing networks, configuring DNS, managing backups, configuring cloud services. Absolutely none of it in support of the dev team I was on. The developers were responsible for managing their own CI/CD pipeline and deployments. The people called the "DevOps Team" were responsible for managing production. In this case, they were systems administrators and were not in any way a devops team, but the terminology is so skewed now that they were labeled as such, and I was labeled an out of touch old greybeard. This isn't to demean devops in any way; it's a valuable role, an evolutionary step in software team organization, but by no means is it new and by no means is systems administration a dead role.

strken 7y ago
When I hear systems administration I assume it's pets not cattle, and that you're administering individual named servers with bash scripts rather than administering a mesos cluster with chef or puppet or whatever. I'm not sure where I picked up this impression from.
- alchemism 7y ago
  It comes from bright, talented SWEs who do not, bless their hearts, understand that IT Operations is an entire profession unto its own, and not merely the grunt-work they are forced to do while working as interns or juniors.
sunaurus 7y ago
I know that "DevOps" is one of those terms that can mean whatever the user wants it to mean (like "agile" and "REST" and so on), but at least with DevOps, I've mostly only seen 2 types of common definitions: 1) "DevOps teams" are just rebranded operations teams 2) "DevOps teams" are teams that are responsible for both development and operations for their app I think the first one is the one you're frustrated with, and I agree it's a completely useless label when used like that, but it's not definitely not the only way it's used.

test111 7y ago

It's a critical skill now

omani 7y ago

cant believe so many people still get it wrong. devops means Developer/Operations. means, the sysadmin/sysops/ops guy now has to know things about development to center his daily-doing around code (ideally the code dictates what is going on [Infrastructure as Code]). a very simple example to this: back then the ops guy or sysadmin did everything by hand. today, he uses code to get things done. bash vs. a HTTP JSON API. manually by hand vs. ansible [markup language yaml], etc. so many people just dont know or understand what devops means. but it is so easy. it is the fusion of development and operations. thus, while you need two people back then, now you only need one person to do the same job. a devops is a sysadmin who ideally knows how to code. a devops is a developer who evolved into a sysadmin or is doing both because he learned it. devops is development/operations. it has nothing to do with agile, waterfall or a "mindset".

luord 7y ago

> “a developer with operational tendencies” Hmmm, I like the ring to that, might put it on my linkedin profile. On the article itself, can't say I disagree.