9 ms·

Ask HN: Teaching Greybeard IT

60 points by Damogran6 4y ago

I'm finding that I picked up a LOT of IT knowledge over the past 40 years...and that some of our new hires are missing out on that foundation...skills are weak on networking, Bottleneck troubleshooting, understanding what a healthy or unhealthy process looks like. There's stuff online for mutexes and race conditions and VonNeumann bottlenecks in general, but I'm having a hard time finding the concept in general. Do you have any suggestions on where I can point people, or should I just start throwing stuff together myself?

z3t4 4y ago
Write your own blog. Start with just plain text files served via lightweight web server of choise just to get over the starting hurdle.
- oxff 4y ago
  I would recommend this.
- deathanatos 4y ago
  I second this. The people who want the knowledge read.
- wil421 4y ago
  Setting up a LAMP stack used to be the starting point for learning web dev. I like the idea of starting with plain text first.
sys32768 4y ago
I was just thinking the other day that a non-profit wiki for sysadmins would be invaluable. It could cover all the basics but also feature case-studies of real-world examples of solving problems, so there could be a sort of knowledge base too that would attract seekers and lead them into the foundational content. 10 or 15 years ago, I remember being able to find detailed problem solving information on Google. Today the search results are a morass of affiliate and SEO content multiplied by plagiarism served from a mountainous dung heap of user-generated blather.
- Damogran6 4y ago
  I'd love to share security experience, suitably redacted, for experience learned from past engagements...it's easy to do at the water cooler, but that audience is limited.
  - buscoquadnary 4y ago
    Worked in info sec for a while now. Allow me to explain how every one of those stories go. "We told them that you can't be running a production service on a Windows 7 desktop underneath the developers desk. They responded there was no budget for a server this year and that they couldn't take the time to port it anyway since they were too busy. We went to management and they told us to stop bothering the developers and they didn't give a damn about security that was our job so fix it. [3 minutes later] so anyway after they exfilled with the passwords and the social security info of all the employees we managed to get things restored from our 3 month old backups, and the CEO fired everyone in security. So that's why I'm currently in the job market." Or "Ya so we got this meeting invite from some PM and get to his meeting with like 3 execs two dozen devs and enough middle management to make a B2B salesman weep for joy. They start the meeting by thanking everyone for the past two years of diligent work and the nights and weekends and promised to reimburse everyone for the legal costs associated with the ongoing divorces. Finally they ask us as security if we can give it the final approval so it can go into production at midnight. We explain we've never even heard of this project till now and what the hell is going on. Then one of the anonymous herd of grey suited PHB explains that they didn't invite us to meetings or ask for our help because they needed to "move fast and break things" and that security would've just slowed down their rockstar ninja wizard developers. Meanwhile my coworker has been poking at this for the past 10 minutes of the meeting and says there is no way in hell this thing is ready to ship. When asked why he pointed out that passwords were being sent in plaintext via a GET parameter in every request, every field was a SQL injection vulnerability and for some reason he was able to randomly kill processes by PID running on the server if he created a username that had a non ASCII character. The PM who called the meeting said we couldn't let good be the enemy of perfect and they were shipping anyway, which was met with thunderous applause. Well then you know what happened after, the lawsuits have mostly died down and the good news is my defense proved I didn't have any personal liability."
revolutukr 4y ago
As a retired sysadmin with 30 years experience, I can tell you with a good amount of certainty that nobody will care if they aren't already sysadmins. It is viewed as entirely unimportant to understand these things, and the only thing you need to get a job is good whiteboard/leet coding skills. The world changed.
- plugin-baby 4y ago
  I’m not a sysadmin, but want to at least read the contents page - after 20 years full-stack dev I’m painfully aware there’s a lot of things I don’t understand lower down the stack.
  - ilc 4y ago
    What is a "full stack" developer? This is something that always bothered me...
    - tristor 4y ago
      A developer that writes Javascript and sometimes runs it in Node as a server process or as a desktop app in Electron, but actually doesn’t know the /full/ stack at all.
    - smolder 4y ago
      IMO it should be somebody who can build a Linux kernel module or a single page web app. They can do explicit memory management in C, or query a DB via ORM. But the way it's used is usually just "can write both server and client JavaScript."
    - eddsh 4y ago
      I always assume someone who can build a small production web app from DB Schemas to the front end, including authentication and payments but nothing more specialist than that. Able to jump in anywhere along that stack and be dangerous enough. Probably abstracted via frameworks, so knowing one in-depth front-end and back-end.
    - 4y ago
Alekhine 4y ago
I think you probably have a lot of valuable knowledge that should be written down. What knowledge about networking do new hires lack? How does one get comfortable with networking? I'm an undergrad, and my plan for that topic was to learn how to read pcap files and also learn to use wireshark to monitor my network.
- Damogran6 4y ago
  it's all so ad-hoc and random. Firewalls have expensive and cheap operations...passing a packet? Cheap. Opening the packet and rewriting the header? Expensive. That can be leveraged for evil. How do you even organize that kind of random trivia?
  - Alekhine 4y ago
    I don't know much about firewalls. I wasn't aware they could alter traffic, for example. What do you mean it can be leveraged for evil? As for organization, I just dump everything I learn in a markdown file on that topic, then look for structure later. So I would just open a file, call it `firewalls.md` and write that down in there. You can accrete a lot of written knowledge over time this way.
    - ilc 4y ago
      They can be used for all types of stuff. Imagine two companies merging. They both used 10.100.0.0/16. If you were in one of the companies wouldn't it be nice if the other re-ip'd? ... Guess what.. they won't and you don't want to. So you make a box, that turns your ips into 10.101.0.0/16 and theirs into 10.102.0.0/16... and you leave your IPs alone :). Yeah, some things won't work across that bridge. But it'll get you started. Firewalls can do really, really evil stuff. DNS? Yeah, we'll answer ALL dns queries.. Even ones you send to 8.8.8.8, or 1.1.1.1... The BOFH playbook is large, and varied. :)
    - Damogran6 4y ago
      Networking, not just firewalls, is about looking at a packet, opening it up, and either passing it along to the next link in the chain, or making a change to (typically) the header and sending it along. There's a TTL or Time To Live value to make packets eventually die, otherwise, packets could potentially move from router to router forever. So at a minimum, your firewall, or router, rewrites the header to Decrement the TTL by one with every hop. Which spawns a really interesting thought experiment that results in the creation of the traceroute utility.
rr888 4y ago
Ask the new hires if they care. They probably realize they need to learn AWS, Terraform, Kubernetes and Leetcode, anything else is secondary. The guys 5-10 years of experience are likely more interested.
- Damogran6 4y ago
  But when the black box stops working, how do you find out where to look to fix it? I get that the start of the art is advancing ever onward...but there's still a MAC address involved...waaaaay down the stack.
  - deleted 4y ago
    [deleted]
  - thrown_22 4y ago
    Turn it on and off again.
    - isoprophlex 4y ago
      That will leave the system turned off?! I mean no code, no bugs, but still...
  - readingnews 4y ago
    I am in the same kind of ball park as OP. Been doing sysadmin for a long time, and I teach at the university too... see a lot of students graduating and their network skills are weak. Bad troubleshooting skills at linux/unix fundamentals. When I talk to friends doing sysadmin work about this, they kind of gravitate towards: "I throw this in AWS, need k8s, terraform, cloud-whatever-hot-tech, etc, and if it breaks, my json/ansible/CI/CD will just restart it" as the prevalent attitude now. I feel OP, but I do wonder if we are just moving to a space where people do not optimize/troubleshoot at that level? I do a LOT of hands on grunt work, performance optimizing servers / kernels / networks, but I am at a university. I think my skills are probably worthless in todays "real IT" world. (which is sad, I would love to get a new job :)
tristor 4y ago
I’ve had similar thoughts. I wrote a lot of docs on an internal wiki at a prior company, and also wrote some public docs on my site. In the end nobody really read it and it was too much effort to keep it up to date. That said, I would love to learn from you even though I’m a PM now. I have an insatiable curiosity about how things /really/ work that served me well for more than a decade as an SRE/DevOps/SysAdmin.
jollyllama 4y ago
Probably more examples, less theory. Maybe set up labs for them ex. demonstrate unhealthy processes eating up memory and what tools you can look at to see.
- Damogran6 4y ago
  That's certainly aspirational...but a long way from getting words down on a blog.
  - jollyllama 4y ago
    Fair point.
rglover 4y ago
Please do it yourself! Your level of knowledge is rare and could be helpful to a lot of people. I'd happily buy a copy of "The Greybeard IT Bible" if you wrote/published it as I'm sure many others would. Would even be cool to have a pro version where you do short podcasts/talks on certain tricks of the trade included.
glutamate 4y ago
"How Linux Works" from no starch press is very good for a bunch of stuff that is not taught in CS.
analognoise 4y ago
Why fix it? That sort of knowledge is undervalued. The new hires make more not understanding that stuff. Why not just take the knowledge with you when you retire, and make no attempt to pass it on? Why is it always up to a lone determined engineer to pass the knowledge on, and not the VPs job to listen to the recommendations to do so, and provide support and training? Because they think you love it so much, you won't be able to help yourself. Break this chain. Give them the suggestion - you provide training, time to make content, introduce it to the new people. Explain if they don't get it, what will happen... And when they say no, take the knowledge and burn it. It doesn't have to be your job to throw yourself on the fire to save their systems. The iron will endure or it won't. Use the extra time to garden instead.
stevenfoster 4y ago
Scott Simpson over at Lynda.com and now LinkedIn Learning has done a fantastic job of covering much of the entry level stuff over the past decade with some courses on intermediary concepts. It is however behind a pay wall. So yes a non-profit wiki / maybe even online academy for "Greybeard IT" would be very valuable. It seems those coming into the work force today not only lack skills to triage and troubleshoot issues across systems but then have had no exposure to why things are the way they are. From personal experience it took years to collect enough stories and insights from those who came before to get where I am today.
gw99 4y ago
Oh man no one cares any more. They fudge through it slowly when there's a problem and forget about it immediately. I've seen teams waste 3 months unable to fix a simple issue and I've walked in and seen it right away. Can I pass these skills on? No because no one gives a fuck. And quite frankly my soul died enough that I don't either. Just over 5000 days until I'm retired.
- coldpie 4y ago
  These people are at a different point in their career path than you are. They don't have the same decades of experience that you do, to immediately see the problem. Perhaps they will now, having solved it once, as you have many times before. And it may well be that they do not have the same level of passion and dedication that you had at that point in your career. The field is much larger now than it was then, meaning fewer places for people to drop out. Regardless, I hope you enjoy your retirement when it comes. You have earned it.
  - gw99 4y ago
    Most of the time it's no the answer I know, it's how to find it. All I want to do is walk them through the thought process and instil some rigour in diagnostics.
    - Entinel 4y ago
      Here is what I would suggest. Write it down in a blog post or something and make it easy to find. After that, it's out of your hands. If you think it has value there are at least a few other people who would as well.
  - oneplane 4y ago
    The people are in a different quality and time than IT used to be. We need more people, and the result wasn't more of the same, but widening the gradient of what is acceptable. It used to be that you can't get people fresh out of school without basic network knowledge, and over the past decade or so, you can. Heck, it's more likely that new candidates don't know how things work, because they have been trained to pass hiring barriers instead of actually having to know how things work. This makes sense in other areas, especially those that have existed for a long time (i.e. accounting, metal work) and aren't layered in abstractions covering 10 different disciplines. But IT is relatively young, yet still works on the same foundations it always did. Even things like ethernet that's used by mainframes, minicomputers, microcomputers but also clouds, IoT and workforce, is simply ignored because it's supported to magically work and abstracted away by 10 sys calls, 100 node packages and 20 layers of XHR promises. When it breaks, you suddenly don't know how to fix it, and because you also don't know what those 100 node packages are doing (which would be unrealistic anyway) but also don't even know what a syscall is, you will neither know how to find out if that is the problem, and even if you did, you won't know how to fix it. Some basic understanding of the ecosystem in which IT exists wouldn't be too much to ask, but because of the high demand we're just accepting overspecialised people for jobs that don't exist (no role in a company is "react-redux integration specialist"), because it used to work out in the past (but in the past you had to have the same foundational knowledge). This is the equivalent of throwing spaghetti at the wall and seeing what sticks, because it's cheaper to hire 10 new people and hoping one of them lucks out and accidentally fixes something than hiring or training one person that can actually consistently diagnose and resolve issues. And before someone assumes this means some people are "bad" because they don't know "everything" (not talking about knowing everything): this is mainly a disconnect between what is needed and what is applied for. And that in turn is because it's hard to describe what is needed, and in return is hard to prevent how as a candidate you would fulfil that need. (i.e. job postings with 15 years of senior Kubernetes experience come to mind) If you know a bit about tasks, queues, control loops, you can pretty much fill a job for legacy systems like servers running bare metal linux or windows, but also containers, container orchestrators and orchestration tooling for infrastructure as code, because at the end of the day they all are implementations of the same principles, but slightly specialised for the problem at hand. They all do the same thing: you specify what you want, and it is up to the system to make it real and report how that's going. Reading the output of invoke-rc.d, service, systemctl, kubectl, docker and nerdctl is specialised on resource names, but identical in terms of action flow. They all have a concept of a resource, a concept of something that needs to happen and a concept of telling you how it went/how it's going. But teaching people that you need to understand the principles and apply them on different implementations and abstraction levels is hard, so we just teach people the contents of a MAN page instead of actually telling them that the man command exists and you can just read it whenever you like...
jcalabro 4y ago
It's not quite aimed at IT/System admin, but Handmade Hero[0] is an excellent resource for learning how a computer works from the ground up. It starts with an excellent intro to C on Windows[1] and works its way up from there. It's really long and involved, but I've found that internalizing the general ethos is quite valuable, even if you just watch a handful of episodes. [0] https://handmadehero.org/ https://handmadehero.org/ [1] https://www.youtube.com/watch?v=F3ntGDm6hOs&list=PLEMXAbCVnmY6RverunClc_DMLNDd3ASRp https://www.youtube.com/watch?v=F3ntGDm6hOs&list=PLEMXAbCVnm...
f0e4c2f7 4y ago
I really like the resources reccomended on teachyourselfcs.com
codegeek 4y ago
I hear you. I have the same feelings on "web developers" these days. A lot of them start with some framework and cannot write a simple Form submission in HTML. Many cannot explain the difference between a regular Form submission and an AJAX submission but they surely know how to do ajax.
Damogran6 4y ago
What brought this up was a constant and unending battle with our IT department over the use of our Endpoint security tool. It indexes the local disk so that you can query for file hashes across the enterprise at scale. That means you want to be really careful on what parts of the OS you whitelist. A print queue folder has NO value being indexed and can cause performance issues. And this product persists in having a reputation for being the culprit, even though it hasn't been for more than 5 years. What we're encountering though, is that the IT staff have NO bottleneck troubleshooting skills. They can't tell you a system is running slow because it's CPU, RAM, or I/O bound...they don't know that the underlying compute could be an issue, they don't know to check /var/log for hints that something might be wrong...they just point, rigidly, at the EDR saying 'it occasionally pops to the top of the process list, therefore, it's the reason the system is slow.'
TYPE_FASTER 4y ago
At a previous employer, our software eng org migrated from on prem infrastructure supported by a systems engineering team to infrastructure in the cloud owned by the dev team. One challenge was that, while we were lucky to score some talented systems engineers who joined dev teams, it wasn't easy to communicate to leadership that owning and managing infrastructure from the public internet all the way into the database requires knowledge and experience that software engineers might not have. I tried putting together a PluralSight channel for us to use internally to help onboard new team members. We had access to AWS sandbox accounts where we had no proprietary data, and could build infra as needed to learn and/or experiment with ideas/designs/new services. You can target people with this experience when hiring, but just getting qualified devs was hard a year ago, let alone people who also had systems experience/knowledge. A lot of the candidates worked somewhere with a QA team who tested their code, a dev ops team who managed a CI/CD process and managed releasing their code, a database team who owned the database cluster, a networking team who owned networking infrastructure, etc. Really what you're looking for is somebody who likes to code, but also is interested in systems enough to at some point go read the RFC, run Wireshark, etc. I know a little about networking because I ran my own firewall on OpenBSD years ago, have been hosting e-mail and web for over 20yrs, etc. We got lucky with a few hires who lived and breathed code and systems engineering, but it will be challenging to retain people like that in a hot market. At the end of it all, I couldn't really come up with a way to effectively accelerate learning these skills and getting that experience. I started the same day as a new grad, and will never forget when, at the end of the day, he said "you mean I have to keep learning?" Yes, yes we do. Still do. edit: I'd be interested in working on putting content together if people think it would be valuable.
- orev 4y ago
  You’ve just identified the (obvious to systems engineers) problem of every movement that tries to bypass IT Operations and has Developers just do it. System engineering is actually a completely different skill than development, and past a very small size, it’s not reasonable to expect the same person to know everything. DevOps and the Cloud are just the most recent marketing campaigns trying to promise that one can get rid of “those pesky Ops people who just cost you money anyway”. Things can run fine when there are no problems, but then you realize that nobody who’s left is able to fix things when they go wrong. The mentality of “just reboot (or redeploy the container)” only take you so far.
freedude 4y ago
Have you thought of a "retirement" career as a troubleshooting consultant. Hang out your shingle and have two weeks of "fun" every other month and keep the ball rolling. Incidentally, I have learned much by closing my mouth and listening to those, like yourself, with more experience. So Thanks and keep up the good work.
THENATHE 4y ago
Maybe it is wrong of me to say it like this, but it seems like there is a trichotomy in the world of working people. You have people like you, that will voluntarily study their hearts out to find the textbook or proper way to solve things, You have people that seemingly don’t care, but learn by doing, making mistakes, learning holistically And then you have the workers that just don’t care and wanna be done with their 9 to 5 at 8 and 4. You’re going to get a wildly skewed sample looking at HN or Reddit, because, by and large, most people here are the first type. But as someone that is throughly the second type, I don’t give a flying fuck about VonNeumann and mutexes and race whatever because attaching a word doesn’t do anything to help me understand the problem and the solution. I didn’t spend years reading about theory, I spent years looking at logging software, memorizing errors, and looking at data. I have no doubt that I could look at a server with an issue and eventually figure out the nuance of and solve the issue without ever having heard of any of the proper names. I promise you I am not trying to be rude or condescending when I type all this, but the honest answer of it is true: some people aren’t academics, and any attempt to give them knowledge in the form of “book smarts” and higher level reading is gonna blow up in your face.
- rr888 4y ago
  > 9 to 5 at 8 and 4. Hey we're WFH now, its 10->4 with a few hour long breaks inbetween.
- openfuture 4y ago
  Downvote because it doesn't really have anything to do with what he is asking. You claim that someone who does the job is valuable, nobody disagrees. He asks; where can I find good explanations of concepts that I've found useful throughout my career and I want to pass on to my juniors? See how this justifies a downvote?
  - THENATHE 4y ago
    The statement is that if he cannot verbally pass that information on and have them get interested/excited, no amount of documenting and cataloging that info is going to make them suddenly want to read it. No level of explanation is going to be helpful to them when they don’t care enough to listen/read. And I claimed nothing about “someone who does the job is valuable”. My claim was that there are, generally speaking, three types of workers and the vast majority of workers are unlike OP, in the sense that they probably don’t care in the slightest because they are 9-5 workers or someone who would rather learn themselves.
reklis 4y ago
I went ahead and registered greybeardit.org I will create a github pages open source project where people can submit articles, it will be 100% open source.
- reklis 4y ago
  https://gitlab.com/greybeardit/greybeardit.org https://gitlab.com/greybeardit/greybeardit.org open to anyone that wants to contribute

60 points by Damogran6 4y ago

z3t4 4y ago

Write your own blog. Start with just plain text files served via lightweight web server of choise just to get over the starting hurdle.

oxff 4y ago
I would recommend this.
deathanatos 4y ago
I second this. The people who want the knowledge read.
wil421 4y ago
Setting up a LAMP stack used to be the starting point for learning web dev. I like the idea of starting with plain text first.

sys32768 4y ago

I was just thinking the other day that a non-profit wiki for sysadmins would be invaluable. It could cover all the basics but also feature case-studies of real-world examples of solving problems, so there could be a sort of knowledge base too that would attract seekers and lead them into the foundational content. 10 or 15 years ago, I remember being able to find detailed problem solving information on Google. Today the search results are a morass of affiliate and SEO content multiplied by plagiarism served from a mountainous dung heap of user-generated blather.

Damogran6 4y ago
I'd love to share security experience, suitably redacted, for experience learned from past engagements...it's easy to do at the water cooler, but that audience is limited.
- buscoquadnary 4y ago
  Worked in info sec for a while now. Allow me to explain how every one of those stories go. "We told them that you can't be running a production service on a Windows 7 desktop underneath the developers desk. They responded there was no budget for a server this year and that they couldn't take the time to port it anyway since they were too busy. We went to management and they told us to stop bothering the developers and they didn't give a damn about security that was our job so fix it. [3 minutes later] so anyway after they exfilled with the passwords and the social security info of all the employees we managed to get things restored from our 3 month old backups, and the CEO fired everyone in security. So that's why I'm currently in the job market." Or "Ya so we got this meeting invite from some PM and get to his meeting with like 3 execs two dozen devs and enough middle management to make a B2B salesman weep for joy. They start the meeting by thanking everyone for the past two years of diligent work and the nights and weekends and promised to reimburse everyone for the legal costs associated with the ongoing divorces. Finally they ask us as security if we can give it the final approval so it can go into production at midnight. We explain we've never even heard of this project till now and what the hell is going on. Then one of the anonymous herd of grey suited PHB explains that they didn't invite us to meetings or ask for our help because they needed to "move fast and break things" and that security would've just slowed down their rockstar ninja wizard developers. Meanwhile my coworker has been poking at this for the past 10 minutes of the meeting and says there is no way in hell this thing is ready to ship. When asked why he pointed out that passwords were being sent in plaintext via a GET parameter in every request, every field was a SQL injection vulnerability and for some reason he was able to randomly kill processes by PID running on the server if he created a username that had a non ASCII character. The PM who called the meeting said we couldn't let good be the enemy of perfect and they were shipping anyway, which was met with thunderous applause. Well then you know what happened after, the lawsuits have mostly died down and the good news is my defense proved I didn't have any personal liability."

revolutukr 4y ago

As a retired sysadmin with 30 years experience, I can tell you with a good amount of certainty that nobody will care if they aren't already sysadmins. It is viewed as entirely unimportant to understand these things, and the only thing you need to get a job is good whiteboard/leet coding skills. The world changed.

plugin-baby 4y ago
I’m not a sysadmin, but want to at least read the contents page - after 20 years full-stack dev I’m painfully aware there’s a lot of things I don’t understand lower down the stack.
- ilc 4y ago
  What is a "full stack" developer? This is something that always bothered me...
  - tristor 4y ago
    A developer that writes Javascript and sometimes runs it in Node as a server process or as a desktop app in Electron, but actually doesn’t know the /full/ stack at all.
  - smolder 4y ago
    IMO it should be somebody who can build a Linux kernel module or a single page web app. They can do explicit memory management in C, or query a DB via ORM. But the way it's used is usually just "can write both server and client JavaScript."
  - eddsh 4y ago
    I always assume someone who can build a small production web app from DB Schemas to the front end, including authentication and payments but nothing more specialist than that. Able to jump in anywhere along that stack and be dangerous enough. Probably abstracted via frameworks, so knowing one in-depth front-end and back-end.
  - 4y ago

Alekhine 4y ago

I think you probably have a lot of valuable knowledge that should be written down. What knowledge about networking do new hires lack? How does one get comfortable with networking? I'm an undergrad, and my plan for that topic was to learn how to read pcap files and also learn to use wireshark to monitor my network.

Damogran6 4y ago
it's all so ad-hoc and random. Firewalls have expensive and cheap operations...passing a packet? Cheap. Opening the packet and rewriting the header? Expensive. That can be leveraged for evil. How do you even organize that kind of random trivia?
- Alekhine 4y ago
  I don't know much about firewalls. I wasn't aware they could alter traffic, for example. What do you mean it can be leveraged for evil? As for organization, I just dump everything I learn in a markdown file on that topic, then look for structure later. So I would just open a file, call it `firewalls.md` and write that down in there. You can accrete a lot of written knowledge over time this way.
  - ilc 4y ago
    They can be used for all types of stuff. Imagine two companies merging. They both used 10.100.0.0/16. If you were in one of the companies wouldn't it be nice if the other re-ip'd? ... Guess what.. they won't and you don't want to. So you make a box, that turns your ips into 10.101.0.0/16 and theirs into 10.102.0.0/16... and you leave your IPs alone :). Yeah, some things won't work across that bridge. But it'll get you started. Firewalls can do really, really evil stuff. DNS? Yeah, we'll answer ALL dns queries.. Even ones you send to 8.8.8.8, or 1.1.1.1... The BOFH playbook is large, and varied. :)
  - Damogran6 4y ago
    Networking, not just firewalls, is about looking at a packet, opening it up, and either passing it along to the next link in the chain, or making a change to (typically) the header and sending it along. There's a TTL or Time To Live value to make packets eventually die, otherwise, packets could potentially move from router to router forever. So at a minimum, your firewall, or router, rewrites the header to Decrement the TTL by one with every hop. Which spawns a really interesting thought experiment that results in the creation of the traceroute utility.

rr888 4y ago

Ask the new hires if they care. They probably realize they need to learn AWS, Terraform, Kubernetes and Leetcode, anything else is secondary. The guys 5-10 years of experience are likely more interested.

Damogran6 4y ago
But when the black box stops working, how do you find out where to look to fix it? I get that the start of the art is advancing ever onward...but there's still a MAC address involved...waaaaay down the stack.
- deleted 4y ago
  [deleted]
- thrown_22 4y ago
  Turn it on and off again.
  - isoprophlex 4y ago
    That will leave the system turned off?! I mean no code, no bugs, but still...
- readingnews 4y ago
  I am in the same kind of ball park as OP. Been doing sysadmin for a long time, and I teach at the university too... see a lot of students graduating and their network skills are weak. Bad troubleshooting skills at linux/unix fundamentals. When I talk to friends doing sysadmin work about this, they kind of gravitate towards: "I throw this in AWS, need k8s, terraform, cloud-whatever-hot-tech, etc, and if it breaks, my json/ansible/CI/CD will just restart it" as the prevalent attitude now. I feel OP, but I do wonder if we are just moving to a space where people do not optimize/troubleshoot at that level? I do a LOT of hands on grunt work, performance optimizing servers / kernels / networks, but I am at a university. I think my skills are probably worthless in todays "real IT" world. (which is sad, I would love to get a new job :)

tristor 4y ago

I’ve had similar thoughts. I wrote a lot of docs on an internal wiki at a prior company, and also wrote some public docs on my site. In the end nobody really read it and it was too much effort to keep it up to date. That said, I would love to learn from you even though I’m a PM now. I have an insatiable curiosity about how things /really/ work that served me well for more than a decade as an SRE/DevOps/SysAdmin.

jollyllama 4y ago

Probably more examples, less theory. Maybe set up labs for them ex. demonstrate unhealthy processes eating up memory and what tools you can look at to see.

Damogran6 4y ago
That's certainly aspirational...but a long way from getting words down on a blog.
- jollyllama 4y ago
  Fair point.

rglover 4y ago

Please do it yourself! Your level of knowledge is rare and could be helpful to a lot of people. I'd happily buy a copy of "The Greybeard IT Bible" if you wrote/published it as I'm sure many others would. Would even be cool to have a pro version where you do short podcasts/talks on certain tricks of the trade included.

glutamate 4y ago

"How Linux Works" from no starch press is very good for a bunch of stuff that is not taught in CS.

analognoise 4y ago

Why fix it? That sort of knowledge is undervalued. The new hires make more not understanding that stuff. Why not just take the knowledge with you when you retire, and make no attempt to pass it on? Why is it always up to a lone determined engineer to pass the knowledge on, and not the VPs job to listen to the recommendations to do so, and provide support and training? Because they think you love it so much, you won't be able to help yourself. Break this chain. Give them the suggestion - you provide training, time to make content, introduce it to the new people. Explain if they don't get it, what will happen... And when they say no, take the knowledge and burn it. It doesn't have to be your job to throw yourself on the fire to save their systems. The iron will endure or it won't. Use the extra time to garden instead.

stevenfoster 4y ago

Scott Simpson over at Lynda.com and now LinkedIn Learning has done a fantastic job of covering much of the entry level stuff over the past decade with some courses on intermediary concepts. It is however behind a pay wall. So yes a non-profit wiki / maybe even online academy for "Greybeard IT" would be very valuable. It seems those coming into the work force today not only lack skills to triage and troubleshoot issues across systems but then have had no exposure to why things are the way they are. From personal experience it took years to collect enough stories and insights from those who came before to get where I am today.

gw99 4y ago

Oh man no one cares any more. They fudge through it slowly when there's a problem and forget about it immediately. I've seen teams waste 3 months unable to fix a simple issue and I've walked in and seen it right away. Can I pass these skills on? No because no one gives a fuck. And quite frankly my soul died enough that I don't either. Just over 5000 days until I'm retired.

coldpie 4y ago
These people are at a different point in their career path than you are. They don't have the same decades of experience that you do, to immediately see the problem. Perhaps they will now, having solved it once, as you have many times before. And it may well be that they do not have the same level of passion and dedication that you had at that point in your career. The field is much larger now than it was then, meaning fewer places for people to drop out. Regardless, I hope you enjoy your retirement when it comes. You have earned it.
- gw99 4y ago
  Most of the time it's no the answer I know, it's how to find it. All I want to do is walk them through the thought process and instil some rigour in diagnostics.
  - Entinel 4y ago
    Here is what I would suggest. Write it down in a blog post or something and make it easy to find. After that, it's out of your hands. If you think it has value there are at least a few other people who would as well.
- oneplane 4y ago
  The people are in a different quality and time than IT used to be. We need more people, and the result wasn't more of the same, but widening the gradient of what is acceptable. It used to be that you can't get people fresh out of school without basic network knowledge, and over the past decade or so, you can. Heck, it's more likely that new candidates don't know how things work, because they have been trained to pass hiring barriers instead of actually having to know how things work. This makes sense in other areas, especially those that have existed for a long time (i.e. accounting, metal work) and aren't layered in abstractions covering 10 different disciplines. But IT is relatively young, yet still works on the same foundations it always did. Even things like ethernet that's used by mainframes, minicomputers, microcomputers but also clouds, IoT and workforce, is simply ignored because it's supported to magically work and abstracted away by 10 sys calls, 100 node packages and 20 layers of XHR promises. When it breaks, you suddenly don't know how to fix it, and because you also don't know what those 100 node packages are doing (which would be unrealistic anyway) but also don't even know what a syscall is, you will neither know how to find out if that is the problem, and even if you did, you won't know how to fix it. Some basic understanding of the ecosystem in which IT exists wouldn't be too much to ask, but because of the high demand we're just accepting overspecialised people for jobs that don't exist (no role in a company is "react-redux integration specialist"), because it used to work out in the past (but in the past you had to have the same foundational knowledge). This is the equivalent of throwing spaghetti at the wall and seeing what sticks, because it's cheaper to hire 10 new people and hoping one of them lucks out and accidentally fixes something than hiring or training one person that can actually consistently diagnose and resolve issues. And before someone assumes this means some people are "bad" because they don't know "everything" (not talking about knowing everything): this is mainly a disconnect between what is needed and what is applied for. And that in turn is because it's hard to describe what is needed, and in return is hard to prevent how as a candidate you would fulfil that need. (i.e. job postings with 15 years of senior Kubernetes experience come to mind) If you know a bit about tasks, queues, control loops, you can pretty much fill a job for legacy systems like servers running bare metal linux or windows, but also containers, container orchestrators and orchestration tooling for infrastructure as code, because at the end of the day they all are implementations of the same principles, but slightly specialised for the problem at hand. They all do the same thing: you specify what you want, and it is up to the system to make it real and report how that's going. Reading the output of invoke-rc.d, service, systemctl, kubectl, docker and nerdctl is specialised on resource names, but identical in terms of action flow. They all have a concept of a resource, a concept of something that needs to happen and a concept of telling you how it went/how it's going. But teaching people that you need to understand the principles and apply them on different implementations and abstraction levels is hard, so we just teach people the contents of a MAN page instead of actually telling them that the man command exists and you can just read it whenever you like...

jcalabro 4y ago

It's not quite aimed at IT/System admin, but Handmade Hero[0] is an excellent resource for learning how a computer works from the ground up. It starts with an excellent intro to C on Windows[1] and works its way up from there. It's really long and involved, but I've found that internalizing the general ethos is quite valuable, even if you just watch a handful of episodes. [0] https://handmadehero.org/ https://handmadehero.org/ [1] https://www.youtube.com/watch?v=F3ntGDm6hOs&list=PLEMXAbCVnmY6RverunClc_DMLNDd3ASRp https://www.youtube.com/watch?v=F3ntGDm6hOs&list=PLEMXAbCVnm...

f0e4c2f7 4y ago

I really like the resources reccomended on teachyourselfcs.com

codegeek 4y ago

I hear you. I have the same feelings on "web developers" these days. A lot of them start with some framework and cannot write a simple Form submission in HTML. Many cannot explain the difference between a regular Form submission and an AJAX submission but they surely know how to do ajax.

Damogran6 4y ago

What brought this up was a constant and unending battle with our IT department over the use of our Endpoint security tool. It indexes the local disk so that you can query for file hashes across the enterprise at scale. That means you want to be really careful on what parts of the OS you whitelist. A print queue folder has NO value being indexed and can cause performance issues. And this product persists in having a reputation for being the culprit, even though it hasn't been for more than 5 years. What we're encountering though, is that the IT staff have NO bottleneck troubleshooting skills. They can't tell you a system is running slow because it's CPU, RAM, or I/O bound...they don't know that the underlying compute could be an issue, they don't know to check /var/log for hints that something might be wrong...they just point, rigidly, at the EDR saying 'it occasionally pops to the top of the process list, therefore, it's the reason the system is slow.'

TYPE_FASTER 4y ago

At a previous employer, our software eng org migrated from on prem infrastructure supported by a systems engineering team to infrastructure in the cloud owned by the dev team. One challenge was that, while we were lucky to score some talented systems engineers who joined dev teams, it wasn't easy to communicate to leadership that owning and managing infrastructure from the public internet all the way into the database requires knowledge and experience that software engineers might not have. I tried putting together a PluralSight channel for us to use internally to help onboard new team members. We had access to AWS sandbox accounts where we had no proprietary data, and could build infra as needed to learn and/or experiment with ideas/designs/new services. You can target people with this experience when hiring, but just getting qualified devs was hard a year ago, let alone people who also had systems experience/knowledge. A lot of the candidates worked somewhere with a QA team who tested their code, a dev ops team who managed a CI/CD process and managed releasing their code, a database team who owned the database cluster, a networking team who owned networking infrastructure, etc. Really what you're looking for is somebody who likes to code, but also is interested in systems enough to at some point go read the RFC, run Wireshark, etc. I know a little about networking because I ran my own firewall on OpenBSD years ago, have been hosting e-mail and web for over 20yrs, etc. We got lucky with a few hires who lived and breathed code and systems engineering, but it will be challenging to retain people like that in a hot market. At the end of it all, I couldn't really come up with a way to effectively accelerate learning these skills and getting that experience. I started the same day as a new grad, and will never forget when, at the end of the day, he said "you mean I have to keep learning?" Yes, yes we do. Still do. edit: I'd be interested in working on putting content together if people think it would be valuable.

orev 4y ago
You’ve just identified the (obvious to systems engineers) problem of every movement that tries to bypass IT Operations and has Developers just do it. System engineering is actually a completely different skill than development, and past a very small size, it’s not reasonable to expect the same person to know everything. DevOps and the Cloud are just the most recent marketing campaigns trying to promise that one can get rid of “those pesky Ops people who just cost you money anyway”. Things can run fine when there are no problems, but then you realize that nobody who’s left is able to fix things when they go wrong. The mentality of “just reboot (or redeploy the container)” only take you so far.

freedude 4y ago

Have you thought of a "retirement" career as a troubleshooting consultant. Hang out your shingle and have two weeks of "fun" every other month and keep the ball rolling. Incidentally, I have learned much by closing my mouth and listening to those, like yourself, with more experience. So Thanks and keep up the good work.

THENATHE 4y ago

Maybe it is wrong of me to say it like this, but it seems like there is a trichotomy in the world of working people. You have people like you, that will voluntarily study their hearts out to find the textbook or proper way to solve things, You have people that seemingly don’t care, but learn by doing, making mistakes, learning holistically And then you have the workers that just don’t care and wanna be done with their 9 to 5 at 8 and 4. You’re going to get a wildly skewed sample looking at HN or Reddit, because, by and large, most people here are the first type. But as someone that is throughly the second type, I don’t give a flying fuck about VonNeumann and mutexes and race whatever because attaching a word doesn’t do anything to help me understand the problem and the solution. I didn’t spend years reading about theory, I spent years looking at logging software, memorizing errors, and looking at data. I have no doubt that I could look at a server with an issue and eventually figure out the nuance of and solve the issue without ever having heard of any of the proper names. I promise you I am not trying to be rude or condescending when I type all this, but the honest answer of it is true: some people aren’t academics, and any attempt to give them knowledge in the form of “book smarts” and higher level reading is gonna blow up in your face.

rr888 4y ago
> 9 to 5 at 8 and 4. Hey we're WFH now, its 10->4 with a few hour long breaks inbetween.
openfuture 4y ago
Downvote because it doesn't really have anything to do with what he is asking. You claim that someone who does the job is valuable, nobody disagrees. He asks; where can I find good explanations of concepts that I've found useful throughout my career and I want to pass on to my juniors? See how this justifies a downvote?
- THENATHE 4y ago
  The statement is that if he cannot verbally pass that information on and have them get interested/excited, no amount of documenting and cataloging that info is going to make them suddenly want to read it. No level of explanation is going to be helpful to them when they don’t care enough to listen/read. And I claimed nothing about “someone who does the job is valuable”. My claim was that there are, generally speaking, three types of workers and the vast majority of workers are unlike OP, in the sense that they probably don’t care in the slightest because they are 9-5 workers or someone who would rather learn themselves.

reklis 4y ago

I went ahead and registered greybeardit.org I will create a github pages open source project where people can submit articles, it will be 100% open source.

reklis 4y ago
https://gitlab.com/greybeardit/greybeardit.org https://gitlab.com/greybeardit/greybeardit.org open to anyone that wants to contribute