11 ms·
Ex Amazon here. Most grumpy system engineers did not disappear: we got hired by Google/Amazon/etc to build large-scale infrastructure... and sometimes sell it b
by ex_amazon_sde 8y ago
Ex Amazon here. Most grumpy system engineers did not disappear: we got hired by Google/Amazon/etc to build large-scale infrastructure... and sometimes sell it back to you as a service.
Believe me or not, most of the underlying infra does not run on the popular technology of the year. Far, far from it. That's why it works.
Modern devops, with its million tools that break backward compatibility every month sometimes becomes the running joke at lunch.
- icebraining 8y agoIs puppet one of those tools?
- kazen44 8y agoDepends, heck. I have seem some guys do magic with CFengine and freeBSD. (freebsd is a nice OS from an ops perspective anyhow, and has a lot more common sense and sane defaults in a lot of places).
- fb03 8y agoYou are my hero.
- spiderjerusalem 8y agoWhat fascinates me about this is, and sorry for being morbid, but what happens when y'all die? Does knowledge of the lower levels of the stack go away with your generation, or will there be enough of us young ones picking the important stuff up?
- dsr_ 8y agoThat's part of the reason why my last two hires have been at the beginning of their career. For both of them, it was their first major sysadmin responsibility after having jobs involving tech support and occasional Linux experience. The key is to pick smart people who are good at learning and find complex systems interesting. Then, of course, you need to have interesting projects for them to work on.
- user5994461 8y agoEveryone will be running on AWS, Google Cloud and Azure, who know how to operate infra that doesn't crash all the time.
- Twirrim 8y agoThere's rarely anything old school sysadmins have learned that hasn't come from experience. Been there, done that, fought that shit the first time. And the second. And the third. (it's amazing how often I find myself solving what are essentially the same problems over and over.) It's one reason why you'll find we'll push back on the "ohh shiny". There are many wonderful and fascinating things coming out. Tech is an amazing field to be working in. But it's also ridiculously frustrating because no one pays any attention to _why_ things are done the way they are, or _why_ approaches haven't worked in the past (I'm all for re-introducing past failed approaches, as long as there's evidence those reasons have been investigated) You'll find a common trend amongst us in that most of us sort of ended up in the role accidentally. Schools and college teach you to become developers. Few people tend to head to college with the view of specialising in the ops side of things. Even speaking as a comparatively old-school sysadmin, my strengths come from being flexible and adaptable. What I do today is nothing like what I was doing 5 years ago, and what I did then is nothing like what I was doing 5 years before that, and so on down the line. The field is constantly in flux. I just have the best part of two decades of experience to both anticipate the problems, and be able to get to diagnosis quicker when things do go wrong. Even as the older sysadmins die off I'm fairly confident there will be newer ones to replace them, because people are going to continue to learn from the problems they run in to.
- ams6110 8y agoAnsible is one "ohh shiny" thing that has greatly increased my productivity as a sysadmin. Before that I would automate what I could with ssh and pdsh and scripts, but it was never as well polished as Ansible. I'm even using ansible for ad-hoc stuff (tweaking a config, restarting a service) because it's easier to do that from a management server than log in to some remote host, get oriented as to the OS distribution and version, and run commends in the shell there.
- noir_lord 8y agoI like ansible with vagrant as well - it makes for a nice clean way of deploying to development environments while also been nicely 'self' documenting and not limiting (you can drop back to shell), it's a lovely tool for the most part. Edit: The thing I really like about Ansible is how unsexy it is, it's just a nice sane way of doing largely what you could do yourself with ssh and bash but in a language that doesn't make you want to cry. I've been around linux since the 90's and Ansible feels comfortable, predictable and stable - what you would want in a piece of software that can be mission critical in the most fundamental sense.
- scurvy 8y agoIt's a legit concern. There was a NANOG panel about this exact thing. I believe the quote was, "Take a look around. We're all old and greying. We have a severe pipeline problem." And then much to AWS' dude's dismay, the topic shifted towards blaming cloud services because no one takes the time to learn how any of this works any more. Want to guarantee your child's future employment? Don't just teach them to code (the machines will do that). Teach them how to build networks and truly understand network protocols.
- setquk 8y agoI’m going to teach my children how to navigate the world of insane Harry-Potter-esque rules which all IaaS/PaaS platforms enforce upon you. They will become software language lawyers and be masters of the electric Disney dollar. You know like “ahh don’t call the messaging endpoint more than 800 mega-milli-times per mega-nano-second or it will cost you three bazillion CPU credits, but only on three and a half cores which will starve all your instances, issue an invoice and proceed to melt your credit card.”
- w8rbt 8y agoDDOS is now a billing issue.
- kazen44 8y ago> Teach them how to build networks and truly understand network protocols. I don't know how the situation is in the US, but in my country network engineering is actually quite a popular field of study. (we have college level education in network engineering). The one thing that stands out though is that it's mostly done by youngsters who have either sysadmin experience, or worked in IT before that. Almost everyone who comes from high school goes into Software Engineering. I think this is mainly because networking is quite an invisible field so to speak. Many people don't even know your job exists, and many young people only see the shiny hip side of it. (being Software Engineering). Being good at network engineering is hard, especially once you get past entry level work and actually start being responsible for designing large-scale networks. Mainly because building a network is a major financial investment where garuanteeing performance is hard without either a ton of experience, or a shitton of lab time. this kind of work pays very, very well though.
- seorphates 8y agoThey'll never die. Some monkeys will just keep swinging branch to branch. Some monkeys check out the tree.
- karlkatzke 8y agoWhat makes you think there aren’t young people doing systems administration? Our last two hires in my most recent job were 23 and 27 respectively. Sure, they’re getting trained in the new hot cloud stuff... as the grumpy seniors figure it out first and set patterns... but they are still doing daily work with some rather ancient stuff.
- spiderjerusalem 8y agoI'm not saying there are no young people doing sysadmin. What I'm trying to say is that if the new 'infrastructure' that all sysadmins learn is not an open UNIXy system where you can grok all the internals if you want to, but closed systems owned by 2-3 major cloud players, then we kinda maybe have a problem in 20 years? Of course, one can argue that that will just cause a new wave of openness and the cycle continues.
- swarnie_ 8y agoMy company seem to split responsibilities based on age. 40/50 year olds deal with oracle Linux, AIX and Solaris. Under 30's hires are more focused on cloud and mobile. We're all expected to have a footing in Windows and Oracle DB.
- karlkatzke 8y agoYeah, and my point was that we start the young people that we hire on the open stuff. Then we move them up and on to the other open stuff, which runs on top of the cloud vendors. (As almost everyone else points out, the closed cloud vendor stuff is nowhere near flexible enough for most moderately complicated use cases unless you’re running at serious scale.) This is a false crisis.
- tapoxi 8y agoAs a "young" (30) sysadmin/devops dude I think that open, Unixy system is Kubernetes. I can take an application, dockerize it, write a helm chart and run it anywhere. The risk is in treating anything as a black box, whether its a managed service or a container you pull from Dockerhub. It's something you'll get burned by eventually and need to learn from experience.
- zippitydoodah68 8y agoBeen working since 86 in this industry. You lose and gain and you should always be mindful of what is coming. Humility is good. I love you young guys, your ideas keep coming and they are mostly good.
- twic 8y agoSame problem as making sure your system doesn't lose data when a server dies. Make sure you have enough copies of the knowledge by propagating it between people. Try to have some kind of offline recording (books?) for recovery from a disaster where you lose everyone. Have an idea of how to recover at a business level if you do lose the data forever. The trouble is making sure these plans actually work. This is why Netflix randomly execute some of their employees every month.
- telchar 8y ago"This is why Netflix randomly execute some of their employees every month" Would that be their King Kong application?
- jlgaddis 8y ago> This is why Netflix randomly execute some of their employees every month. I certainly hope this practice doesn't start catching on with other employers.
- alekratz 8y agoIt's a very real concern. We have a thing called a "bus plan" for all of our tech employees (6 of us - small non-tech company). It basically attempts to cover everything that we would need to know if one of us gets hit by a bus.
- brightball 8y agoWould be interested to get your opinion on Puppet/Ansible/Chef/CFEngine/SaltStack
- user5994461 8y agoCFEngine is basic text manipulation, it's not comparable to the rest. Puppet and Chef was the first generation. I wouldn't recommend. All the companies and people I know using Chef migrated away from it after many disasters. Nowadays, it's only mentioned in interviews to find out if candidates have real world fire fighting experiences. Ansible is good. Used that for managing hundreds of machines at multiple jobs (some who migrated from Chef). It's been bought by RedHat, it's well maintained and I think it will have the brightest long term future. Not sure about SaltStack. Never had the opportunity to try. I'd be a bit worried though on the long term prospect because I don't think they have much backing or user base.
- Karrot_Kream 8y ago> Ansible is great. Used that for management hundreds of machines at multiple jobs. It's been bought by RedHat, it's well maintained and I think it will have the brightest long term future. A lot of folks I know have been bitten by Ansible's performance (Ansible has a central master that runs recipes on each node, rather than having nodes "pull" from a central master).
- Florin_Andrei 8y agoAnsible has a very, very low barrier to entry. You go from 0 to 100 in a very short time. It makes a lot of sense to use it when you just begin building your infrastructure. Later on you can run Ansible Tower, deploy Ansible agents everywhere, and basically use Ansible under the same client/server model like all the other tools. Salt is eerily similar to Ansible, it's just geared towards client/server. Being experienced with Ansible, it was weird at first to use Salt because everything looked familiar, yet slightly different.
- 8y ago
- bassman9000 8y agoThat's why it works. Touché. Remember, there's value in battletested and proven, e.g. https://www.quora.com/Why-do-satellites-use-old-processors https://www.quora.com/Why-do-satellites-use-old-processors
- seorphates 8y agoGrumpykins here. I think the term "Modern devops" sort of nails it but not quite how you used it. Most departmental/enterprise sys admins/engineers of lore that had even the slightest necessity for life outside the box scaled anything resembling automation to its breaking points. Combined with knowing and serving the reasons for their existence - developers, users etc., and devops is nothing new - it is now simply the necessary manifestation of progress at scale (albeit positively devoured by managers speaking business, not entirely unlike "agile"). "Use what works" definitely presents a lot more choices these days and likely will forever more. "Use what works well" is something different where "well" implies helpful, dependable, predictable, manageable and so on that will continue to scale with your needs. Only breaking things down the "old-school" way will lead towards success, stability, security and life outside the box. Good devops is still, primarily, good engineers engineering good things, for themselves and others. Granted the article is from 2015 but my impression is author is not just cranky, but scared.
- gm-conspiracy 8y agoA good sysadmin would not look like they are doing much work (everything is humming along and can self-heal minus physical problems), but a good devops person is constantly busy.
- segmondy 8y agoGreat sysadmins get fired unless they learn to pretend to look busy. They automate everything till they have nothing to do.
- Intermernet 8y agoLogging is your friend here. You can spend days scrolling through logs, doing an occasional grep and making disapproving noises occasionally. Bonus points for developing some graphs for the next meeting.
- shakna 8y ago
- mathnode 8y agoI am genuinely curious, how much ops is hand written in just plain old bash or simple scripts?
- dsr_ 8y agoBash is everywhere but in small quantities. If it's more than a page of bash, it's probably time to rewrite in something with stricter rules and fewer surprises, or better libraries, or both. Perl or Python is quite common at that level.
- ofrzeta 8y agoThe Bash scripts are now contained in Docker files. Much better :)
- zapita 8y ago> Modern devops, with its million tools that break backward compatibility every month sometimes becomes the running joke at lunch. Ironically, modern devops and its million broken tools are a primary source of revenue for cloud providers, helping pay for your lunch in the first place.
- dredmorbius 8y agoRisk arbitrage. Present capability vs. future catastrophic risk. Technology is debt.
- zapita 8y agoI honestly can't tell whether you're making a sincere but unintelligible argument, or just trolling. Do you disagree with what I said?
- dredmorbius 8y agoYes, re: last. Tech-as-debt is a notion I'm playing with. https://mastodon.social/@natecull/99318348047974414 https://mastodon.social/@natecull/99318348047974414 https://plus.google.com/104092656004159577193/posts/jS1K9MtoKDn https://plus.google.com/104092656004159577193/posts/jS1K9Mto... Obvious antecedent: technical debt, Ward Cunningham.
- auslander 8y agoYou might be on the point, efficient designs are bad for cloud providers. On the other hand, shitty designs that get hacked are bad PR for them. Thing is, none of it matters. Bottom line matters. They care to attract company decision makers. Decision makers are engineers in small to medium businesses, and managers in big ones. Sadly, its the big ones that matter for bottom line. So target is mid management, flashy power point presentations and 'conferences' that allow for justified travel and stay. Good mid management, with tech background, exists, but is a minority. Not all is lost, truth is out there (c) Mulder :))
- auslander 8y ago> Modern devops ... million tools ... running joke So fucking true, bro