4 ms·

Yeah, the rule always seems to be "two steps forward, one step back." I'm not convinced it's a hubris thing. Many of these "two steps forward, one step back" a

Yeah, the rule always seems to be "two steps forward, one step back." I'm not convinced it's a hubris thing. Many of these "two steps forward, one step back" advancements happen because a commercial entity has intentionally blocked the tech tree from advancing in a certain direction that would undercut their position. The tree still advances, but it has to grow around the blockade, and that involves a certain amount of seemingly redundant work. It's only redundant in the sense of technological novelty, though. It's not redundant in the social context.

specialist 6y ago
The population growth of the industry is much faster than diffusion rate of best practices. People become attached to initial ideas, resist any change, regardless of relative merits. Everett Roger's The Diffusion of Innovation [1962], marketing's seminal textbook, explains all. Everything since is rehash (at best) or worse than wrong. https://wikipedia.org/wiki/Diffusion_of_innovations https://wikipedia.org/wiki/Diffusion_of_innovations
WorldMaker 6y ago
I think it's a hubris thing now, but it was clear that the "DevOps" movement/hype started as a cost-cutting measure masquerading as innovation. Who needs classic Ops/IT/sysadmins when developers can just automate all that in their spare time between working on other systems? It shouldn't be a surprise that once the cost-cutting is done you realize too late after the fact that you lost a whole bunch of institutional knowledge and have simply created a new version of your old workforce in their place. The hubris of course comes in accepting the masquerade that everything was really about innovation in the first place, rather than cost-cutting, and rather than rehiring for the old institutional knowledge, just reinventing all the wheels in the new situation with the hubris that it is vastly different than what came before (even though it isn't) because it is far more innovative (even though that is unlikely, because the businesses are still the same, the red tape the same, the expectations set by people still real familiar with the old predecessors, etc).
- robertlagrant 6y ago
  Automating repeatable things to remove the need for some institutional knowledge is a good thing.
  - WorldMaker 6y ago
    Definitely. I'm not claiming at all that it didn't necessarily come from a good place originally or that it hasn't had good ideas, just that the hype cycle around the kernels of good ideas became it's own demon that lost sight of the institutional knowledge forest for some of the most easily automated milling of institutional knowledge trees, and were surprised what happen when the forest was a clear cut field strewn with detritus and they had lost more than they expected in the culling. (I think that metaphor works? It sounds good at least.)
  - npsimons 6y ago
    This I will give to DevOps - it's hyper focus on automation, change management and repeatability are all good things. As someone with a leg in each domain (software developer by day, system administrator by night), I've really felt that both areas could learn a lot from each other. These days with version control being as quick and easy as it is, there's no reason not to use it for everything. Configuration as code is immensely powerful. And repeatability to eliminate "well, it works on my machine" is fucking brilliant. But there still needs to be documentation. The why must always be explained, even with the most readable of code, that can only layout the what and the how.
    - thu2111 6y ago
      It's gone too far though. Where I work everything is automated, even the setup of things where we only have one of them. It's all a giant pile of Ansible scripts, some of which are sort of half maintained. This kind of fanatically DevOps-y approach has a few problems: 1. It's slow and expensive to develop all these scripts. 2. It encourages "disposable infrastructure" in which people think, gee, I need a test version of {very big complicated thing}, so I'll just run these scripts and create an absurdly expensive duplicate that they then forget to shut down. You end up with a cloud filled with VMs that were used once, by one guy, for twenty minutes for a demo, and then never properly deleted. 3. There's no real discoverability or way to iteratively figure out what you want with these things. If you aren't entirely sure how to set something up, you can't really do it by coding it because you'd end up being limited by the tools. Whereas shells and vim and such are designed for interactive work. But then once you've got it set up, for many tasks you hopefully don't need to set it up again anytime soon, so at that point automation is just pointless overhead. It'd be quicker to just drop some notes in a wiki. With respect to the article, it did very much remind me of the great wheel turning. I used to be an SRE at Google and I saw there two relevant trends. One was that when I joined they still had some parts of the infrastructure managed by a giant Python script called the Babysitter. The Babysitter was in theory split into configuration and implementation aspects, but somehow not really. It's dead for perhaps 12 or 13 years now so many Googlers would never have encountered it. The replacement system (Borg) used an intentionally limited configuration language, partly due to the experience of the Babysitter in which people had full programmability, and used it! But the borgcfg language rapidly started sprouting various kinds of programmability features too, and in fact one day someone demonstrated you could build Conway's Game of Life in it. Seems like this new call to use full programming languages to create configuration is another turn of this wheel. The other was hitting the limits of what was automatable. One of the tasks I had to do in my job was to set up some services in new clusters, and shut them down in old ones. This was a multi-step process that was very time consuming and annoying, so there were many attempts to automate it. Unfortunately most were unconvincing, partly because the speed at which new clusters needed to be set up was somehow sort of aligned with the speed at which the general set of tasks was itself changing. So you could attempt to automate the task, but by the time you'd finished the steps involved were already changing, so the next time you wanted to run your script it was wrong. Also: setting up clusters of hundreds of machines isn't something that's easy to test virtually (there were no "mock datacenter" libraries back then). In the end, what I noticed was the big automation wins came from the infrastructure teams when they improved the underlying systems to be more robust. Us glorified sysadmins wrote a lot of scripts, but it was often hard to prove they really saved a lot of time when the cost of implementing and changing them was taken into account.

specialist 6y ago

The population growth of the industry is much faster than diffusion rate of best practices. People become attached to initial ideas, resist any change, regardless of relative merits. Everett Roger's The Diffusion of Innovation [1962], marketing's seminal textbook, explains all. Everything since is rehash (at best) or worse than wrong. https://wikipedia.org/wiki/Diffusion_of_innovations https://wikipedia.org/wiki/Diffusion_of_innovations

WorldMaker 6y ago

I think it's a hubris thing now, but it was clear that the "DevOps" movement/hype started as a cost-cutting measure masquerading as innovation. Who needs classic Ops/IT/sysadmins when developers can just automate all that in their spare time between working on other systems? It shouldn't be a surprise that once the cost-cutting is done you realize too late after the fact that you lost a whole bunch of institutional knowledge and have simply created a new version of your old workforce in their place. The hubris of course comes in accepting the masquerade that everything was really about innovation in the first place, rather than cost-cutting, and rather than rehiring for the old institutional knowledge, just reinventing all the wheels in the new situation with the hubris that it is vastly different than what came before (even though it isn't) because it is far more innovative (even though that is unlikely, because the businesses are still the same, the red tape the same, the expectations set by people still real familiar with the old predecessors, etc).

robertlagrant 6y ago
Automating repeatable things to remove the need for some institutional knowledge is a good thing.
- WorldMaker 6y ago
  Definitely. I'm not claiming at all that it didn't necessarily come from a good place originally or that it hasn't had good ideas, just that the hype cycle around the kernels of good ideas became it's own demon that lost sight of the institutional knowledge forest for some of the most easily automated milling of institutional knowledge trees, and were surprised what happen when the forest was a clear cut field strewn with detritus and they had lost more than they expected in the culling. (I think that metaphor works? It sounds good at least.)
- npsimons 6y ago
  This I will give to DevOps - it's hyper focus on automation, change management and repeatability are all good things. As someone with a leg in each domain (software developer by day, system administrator by night), I've really felt that both areas could learn a lot from each other. These days with version control being as quick and easy as it is, there's no reason not to use it for everything. Configuration as code is immensely powerful. And repeatability to eliminate "well, it works on my machine" is fucking brilliant. But there still needs to be documentation. The why must always be explained, even with the most readable of code, that can only layout the what and the how.
  - thu2111 6y ago
    It's gone too far though. Where I work everything is automated, even the setup of things where we only have one of them. It's all a giant pile of Ansible scripts, some of which are sort of half maintained. This kind of fanatically DevOps-y approach has a few problems: 1. It's slow and expensive to develop all these scripts. 2. It encourages "disposable infrastructure" in which people think, gee, I need a test version of {very big complicated thing}, so I'll just run these scripts and create an absurdly expensive duplicate that they then forget to shut down. You end up with a cloud filled with VMs that were used once, by one guy, for twenty minutes for a demo, and then never properly deleted. 3. There's no real discoverability or way to iteratively figure out what you want with these things. If you aren't entirely sure how to set something up, you can't really do it by coding it because you'd end up being limited by the tools. Whereas shells and vim and such are designed for interactive work. But then once you've got it set up, for many tasks you hopefully don't need to set it up again anytime soon, so at that point automation is just pointless overhead. It'd be quicker to just drop some notes in a wiki. With respect to the article, it did very much remind me of the great wheel turning. I used to be an SRE at Google and I saw there two relevant trends. One was that when I joined they still had some parts of the infrastructure managed by a giant Python script called the Babysitter. The Babysitter was in theory split into configuration and implementation aspects, but somehow not really. It's dead for perhaps 12 or 13 years now so many Googlers would never have encountered it. The replacement system (Borg) used an intentionally limited configuration language, partly due to the experience of the Babysitter in which people had full programmability, and used it! But the borgcfg language rapidly started sprouting various kinds of programmability features too, and in fact one day someone demonstrated you could build Conway's Game of Life in it. Seems like this new call to use full programming languages to create configuration is another turn of this wheel. The other was hitting the limits of what was automatable. One of the tasks I had to do in my job was to set up some services in new clusters, and shut them down in old ones. This was a multi-step process that was very time consuming and annoying, so there were many attempts to automate it. Unfortunately most were unconvincing, partly because the speed at which new clusters needed to be set up was somehow sort of aligned with the speed at which the general set of tasks was itself changing. So you could attempt to automate the task, but by the time you'd finished the steps involved were already changing, so the next time you wanted to run your script it was wrong. Also: setting up clusters of hundreds of machines isn't something that's easy to test virtually (there were no "mock datacenter" libraries back then). In the end, what I noticed was the big automation wins came from the infrastructure teams when they improved the underlying systems to be more robust. Us glorified sysadmins wrote a lot of scripts, but it was often hard to prove they really saved a lot of time when the cost of implementing and changing them was taken into account.