6 ms·

Oh man no one cares any more. They fudge through it slowly when there's a problem and forget about it immediately. I've seen teams waste 3 months unable to fix

by gw99 4y ago

Oh man no one cares any more. They fudge through it slowly when there's a problem and forget about it immediately. I've seen teams waste 3 months unable to fix a simple issue and I've walked in and seen it right away. Can I pass these skills on? No because no one gives a fuck. And quite frankly my soul died enough that I don't either. Just over 5000 days until I'm retired.

coldpie 4y ago
These people are at a different point in their career path than you are. They don't have the same decades of experience that you do, to immediately see the problem. Perhaps they will now, having solved it once, as you have many times before. And it may well be that they do not have the same level of passion and dedication that you had at that point in your career. The field is much larger now than it was then, meaning fewer places for people to drop out. Regardless, I hope you enjoy your retirement when it comes. You have earned it.
- gw99 4y ago
  Most of the time it's no the answer I know, it's how to find it. All I want to do is walk them through the thought process and instil some rigour in diagnostics.
  - Entinel 4y ago
    Here is what I would suggest. Write it down in a blog post or something and make it easy to find. After that, it's out of your hands. If you think it has value there are at least a few other people who would as well.
- oneplane 4y ago
  The people are in a different quality and time than IT used to be. We need more people, and the result wasn't more of the same, but widening the gradient of what is acceptable. It used to be that you can't get people fresh out of school without basic network knowledge, and over the past decade or so, you can. Heck, it's more likely that new candidates don't know how things work, because they have been trained to pass hiring barriers instead of actually having to know how things work. This makes sense in other areas, especially those that have existed for a long time (i.e. accounting, metal work) and aren't layered in abstractions covering 10 different disciplines. But IT is relatively young, yet still works on the same foundations it always did. Even things like ethernet that's used by mainframes, minicomputers, microcomputers but also clouds, IoT and workforce, is simply ignored because it's supported to magically work and abstracted away by 10 sys calls, 100 node packages and 20 layers of XHR promises. When it breaks, you suddenly don't know how to fix it, and because you also don't know what those 100 node packages are doing (which would be unrealistic anyway) but also don't even know what a syscall is, you will neither know how to find out if that is the problem, and even if you did, you won't know how to fix it. Some basic understanding of the ecosystem in which IT exists wouldn't be too much to ask, but because of the high demand we're just accepting overspecialised people for jobs that don't exist (no role in a company is "react-redux integration specialist"), because it used to work out in the past (but in the past you had to have the same foundational knowledge). This is the equivalent of throwing spaghetti at the wall and seeing what sticks, because it's cheaper to hire 10 new people and hoping one of them lucks out and accidentally fixes something than hiring or training one person that can actually consistently diagnose and resolve issues. And before someone assumes this means some people are "bad" because they don't know "everything" (not talking about knowing everything): this is mainly a disconnect between what is needed and what is applied for. And that in turn is because it's hard to describe what is needed, and in return is hard to prevent how as a candidate you would fulfil that need. (i.e. job postings with 15 years of senior Kubernetes experience come to mind) If you know a bit about tasks, queues, control loops, you can pretty much fill a job for legacy systems like servers running bare metal linux or windows, but also containers, container orchestrators and orchestration tooling for infrastructure as code, because at the end of the day they all are implementations of the same principles, but slightly specialised for the problem at hand. They all do the same thing: you specify what you want, and it is up to the system to make it real and report how that's going. Reading the output of invoke-rc.d, service, systemctl, kubectl, docker and nerdctl is specialised on resource names, but identical in terms of action flow. They all have a concept of a resource, a concept of something that needs to happen and a concept of telling you how it went/how it's going. But teaching people that you need to understand the principles and apply them on different implementations and abstraction levels is hard, so we just teach people the contents of a MAN page instead of actually telling them that the man command exists and you can just read it whenever you like...
  - vsareto 4y ago
    TBH even the foundation has changed dramatically. It's hard to convince me we have rock solid foundations in a quickly changing industry. Most of the feedback loop is if you can get jobs of ever higher compensation. If that loop stops considering old school sysadmin foundation skills, then people will stop learning those skills because they can get jobs without them. There's been lots of progress on making things convenient for developers/IT, so that's also why: you don't need those old skills to use those, or at least your entire team doesn't.
    - oneplane 4y ago
      Well, that's just it, the foundations really haven't changed. We still use TCP and UDP, we still use Ethernet, we still use Linux (and Windows and macOS), we still have the pattern of an application needing a configuration stored somewhere where it can read them to exhibit the desired behaviour. We still have the concept of 'services' where it's the job of a (semi-)automatic management or orchestration system to ensure that they are operating within parameters. Nothing about that has changed, regardless of your daily tasks. If you write in Go, Rust, Java, C#.NET, Python or TypeScript, all of those foundations stand and are practically unchanged. Some layers might have been added, i.e. when you describe how your service is supposed to run in a Docker manifest, or in a Nomad configuration, or just a systemd unit file. But those are just formatting issues with plenty of manuals to go around. Yet still, when something goes wrong (i.e. application exits with some status integer), an engineer used to be able to understand that: - Your application or its management system detected a problem - The OS provided an indication that your application has an issue - If you don't know all codes, you know that they are perhaps called "exit code" or "return code", and you at least know how to look those up Say it turns out that your application is throwing a SEGFAULT, as an engineer, aren't you supposed to know that a segmentation fault happened, and you can suspect some shared library to be the first thing to check before going deep with some debugger? Maybe run a command to find out what libraries are needed, check if they are accessible and contained within the context of your application (i.e. if not in a complete OS, but a container or jail or chroot). While in theory only a part of an entire team might need to know this, having mostly teams where 0% of the team members know this might be a problem, don't you agree? Same goes for "I am calling this HTTP API but it doesn't work", that's just a lack of information to begin with. This question is asked plenty of times, and usually the first thing we need to know is "what is the error" because for some reason the developer doesn't understand that it could be a myriad of things and asking for help because "the thing had a booboo" doesn't mean much. Then onwards to: - Is the URL valid - Is the FQDN valid - Are you even using an FQDN or did you think everything in the world runs on your DHCP-supplied search domain (common in windows land) - Can you resolve the name via DNS - Can you connect to the IP on the port (commence lesson about netcat and similar tools, every single time, again and again) - Did you ensure the security groups or legacy firewalls are actually configured for your desired traffic - Is the service on the other end actually up and does it even exist None of this is deep systems understanding for greybeards, it's just basic client-server model programming. But still, there are more team members that don't know how to verify connectivity than teams that have at least 1 who does. Every time someone needs to reach out to that special someone they think magically knows everything in the world, that's a waste of time for two people, and a break from concentration. Advanced issues like "my packet payload get truncated and I don't know why, they are just TLS and below the MTU size", sure, can't expect everyone to understand how to check that stuff out. But we're talking about the basics here. We're running low on greybeards and we're not getting new ones because "they don't need to know" (developers, ops), and this is going to be a problem that gets bigger, not smaller.
PheonixPharts 4y ago
Depending on the org it's can be worse than "no one cares". I was at a large, growing cancerously "tech" company and this type of problem solving was seen as a threat to the large number of managers that had neither management experience nor significant IC/technical experience, whose sole purpose in life was to grow head count so they could increment their level. For these people needless complexity was a benefit. I saw complex neural networks poorly built for the most laughably simple problem, but arguing that it could be solved more simply got you a talking to, and anyone who actually did solve the problem was quickly let go. Part of the consequences of being in a tech bubble that has lasted so long is that reality stops being important, so technical skills don't matter and at the extreme threaten more important illusions. In a company where you're expected to make more money than you spend, simplicity and technical competency can be hugely valuable. In a world were 'profit' is a foreign concept, trying to save time and resources is seen as a threat to the complexity theater other people are creating to boost the importance of their work/team. Don't get me wrong, there was plenty of bs in the pre-bubble world, but there was a time when technical teams had greybeards you would consult and learn from. People like this are one of the reasons I started a career in tech to begin with.
ncr100 4y ago
I doubt that is accurate. Humans are a diverse population. Apathy is one response to facing overwhelming pressure of legacy systems, monolithic organizations. Perseverance is another. Look at George Hotz and https://comma.ai/ https://comma.ai/ - I am skeptical he and his organization would "no one gives a fuck" about broken shit. - https://www.youtube.com/watch?v=FOuCKQfybz8 https://www.youtube.com/watch?v=FOuCKQfybz8 personality snippet
EvanAnderson 4y ago
I don't know that anybody ever cared. The barrier to entry re: running an IT infrastructure has fallen so far that gross incompetence still delivers a serviceable-enough product.
- gw99 4y ago
  That's a depressingly accurate portrayal of the situation.
- mattpallissard 4y ago
  Eh, I wouldn't say it's fallen. There's just as much incompetence per capita as there ever was. I just think there are a lot more people working in the tech sector than ever before so it's more noticeable.
  - EvanAnderson 4y ago
    There are so many more tools to allow the incompetent to "deliver" a reasonable-enough result. There's a certain amount of "don't let perfect be the enemy of the good" in there, for sure, but like the grandparent poster, I've walked into many gigs where "long-standing problems" turn out to be easily fixed. Nobody with sufficient background knowledge and experience ever looked at them and they just festered.

by gw99 4y ago

coldpie 4y ago

These people are at a different point in their career path than you are. They don't have the same decades of experience that you do, to immediately see the problem. Perhaps they will now, having solved it once, as you have many times before. And it may well be that they do not have the same level of passion and dedication that you had at that point in your career. The field is much larger now than it was then, meaning fewer places for people to drop out. Regardless, I hope you enjoy your retirement when it comes. You have earned it.

gw99 4y ago
Most of the time it's no the answer I know, it's how to find it. All I want to do is walk them through the thought process and instil some rigour in diagnostics.
- Entinel 4y ago
  Here is what I would suggest. Write it down in a blog post or something and make it easy to find. After that, it's out of your hands. If you think it has value there are at least a few other people who would as well.
oneplane 4y ago
The people are in a different quality and time than IT used to be. We need more people, and the result wasn't more of the same, but widening the gradient of what is acceptable. It used to be that you can't get people fresh out of school without basic network knowledge, and over the past decade or so, you can. Heck, it's more likely that new candidates don't know how things work, because they have been trained to pass hiring barriers instead of actually having to know how things work. This makes sense in other areas, especially those that have existed for a long time (i.e. accounting, metal work) and aren't layered in abstractions covering 10 different disciplines. But IT is relatively young, yet still works on the same foundations it always did. Even things like ethernet that's used by mainframes, minicomputers, microcomputers but also clouds, IoT and workforce, is simply ignored because it's supported to magically work and abstracted away by 10 sys calls, 100 node packages and 20 layers of XHR promises. When it breaks, you suddenly don't know how to fix it, and because you also don't know what those 100 node packages are doing (which would be unrealistic anyway) but also don't even know what a syscall is, you will neither know how to find out if that is the problem, and even if you did, you won't know how to fix it. Some basic understanding of the ecosystem in which IT exists wouldn't be too much to ask, but because of the high demand we're just accepting overspecialised people for jobs that don't exist (no role in a company is "react-redux integration specialist"), because it used to work out in the past (but in the past you had to have the same foundational knowledge). This is the equivalent of throwing spaghetti at the wall and seeing what sticks, because it's cheaper to hire 10 new people and hoping one of them lucks out and accidentally fixes something than hiring or training one person that can actually consistently diagnose and resolve issues. And before someone assumes this means some people are "bad" because they don't know "everything" (not talking about knowing everything): this is mainly a disconnect between what is needed and what is applied for. And that in turn is because it's hard to describe what is needed, and in return is hard to prevent how as a candidate you would fulfil that need. (i.e. job postings with 15 years of senior Kubernetes experience come to mind) If you know a bit about tasks, queues, control loops, you can pretty much fill a job for legacy systems like servers running bare metal linux or windows, but also containers, container orchestrators and orchestration tooling for infrastructure as code, because at the end of the day they all are implementations of the same principles, but slightly specialised for the problem at hand. They all do the same thing: you specify what you want, and it is up to the system to make it real and report how that's going. Reading the output of invoke-rc.d, service, systemctl, kubectl, docker and nerdctl is specialised on resource names, but identical in terms of action flow. They all have a concept of a resource, a concept of something that needs to happen and a concept of telling you how it went/how it's going. But teaching people that you need to understand the principles and apply them on different implementations and abstraction levels is hard, so we just teach people the contents of a MAN page instead of actually telling them that the man command exists and you can just read it whenever you like...
- vsareto 4y ago
  TBH even the foundation has changed dramatically. It's hard to convince me we have rock solid foundations in a quickly changing industry. Most of the feedback loop is if you can get jobs of ever higher compensation. If that loop stops considering old school sysadmin foundation skills, then people will stop learning those skills because they can get jobs without them. There's been lots of progress on making things convenient for developers/IT, so that's also why: you don't need those old skills to use those, or at least your entire team doesn't.
  - oneplane 4y ago
    Well, that's just it, the foundations really haven't changed. We still use TCP and UDP, we still use Ethernet, we still use Linux (and Windows and macOS), we still have the pattern of an application needing a configuration stored somewhere where it can read them to exhibit the desired behaviour. We still have the concept of 'services' where it's the job of a (semi-)automatic management or orchestration system to ensure that they are operating within parameters. Nothing about that has changed, regardless of your daily tasks. If you write in Go, Rust, Java, C#.NET, Python or TypeScript, all of those foundations stand and are practically unchanged. Some layers might have been added, i.e. when you describe how your service is supposed to run in a Docker manifest, or in a Nomad configuration, or just a systemd unit file. But those are just formatting issues with plenty of manuals to go around. Yet still, when something goes wrong (i.e. application exits with some status integer), an engineer used to be able to understand that: - Your application or its management system detected a problem - The OS provided an indication that your application has an issue - If you don't know all codes, you know that they are perhaps called "exit code" or "return code", and you at least know how to look those up Say it turns out that your application is throwing a SEGFAULT, as an engineer, aren't you supposed to know that a segmentation fault happened, and you can suspect some shared library to be the first thing to check before going deep with some debugger? Maybe run a command to find out what libraries are needed, check if they are accessible and contained within the context of your application (i.e. if not in a complete OS, but a container or jail or chroot). While in theory only a part of an entire team might need to know this, having mostly teams where 0% of the team members know this might be a problem, don't you agree? Same goes for "I am calling this HTTP API but it doesn't work", that's just a lack of information to begin with. This question is asked plenty of times, and usually the first thing we need to know is "what is the error" because for some reason the developer doesn't understand that it could be a myriad of things and asking for help because "the thing had a booboo" doesn't mean much. Then onwards to: - Is the URL valid - Is the FQDN valid - Are you even using an FQDN or did you think everything in the world runs on your DHCP-supplied search domain (common in windows land) - Can you resolve the name via DNS - Can you connect to the IP on the port (commence lesson about netcat and similar tools, every single time, again and again) - Did you ensure the security groups or legacy firewalls are actually configured for your desired traffic - Is the service on the other end actually up and does it even exist None of this is deep systems understanding for greybeards, it's just basic client-server model programming. But still, there are more team members that don't know how to verify connectivity than teams that have at least 1 who does. Every time someone needs to reach out to that special someone they think magically knows everything in the world, that's a waste of time for two people, and a break from concentration. Advanced issues like "my packet payload get truncated and I don't know why, they are just TLS and below the MTU size", sure, can't expect everyone to understand how to check that stuff out. But we're talking about the basics here. We're running low on greybeards and we're not getting new ones because "they don't need to know" (developers, ops), and this is going to be a problem that gets bigger, not smaller.

PheonixPharts 4y ago

Depending on the org it's can be worse than "no one cares". I was at a large, growing cancerously "tech" company and this type of problem solving was seen as a threat to the large number of managers that had neither management experience nor significant IC/technical experience, whose sole purpose in life was to grow head count so they could increment their level. For these people needless complexity was a benefit. I saw complex neural networks poorly built for the most laughably simple problem, but arguing that it could be solved more simply got you a talking to, and anyone who actually did solve the problem was quickly let go. Part of the consequences of being in a tech bubble that has lasted so long is that reality stops being important, so technical skills don't matter and at the extreme threaten more important illusions. In a company where you're expected to make more money than you spend, simplicity and technical competency can be hugely valuable. In a world were 'profit' is a foreign concept, trying to save time and resources is seen as a threat to the complexity theater other people are creating to boost the importance of their work/team. Don't get me wrong, there was plenty of bs in the pre-bubble world, but there was a time when technical teams had greybeards you would consult and learn from. People like this are one of the reasons I started a career in tech to begin with.

ncr100 4y ago

I doubt that is accurate. Humans are a diverse population. Apathy is one response to facing overwhelming pressure of legacy systems, monolithic organizations. Perseverance is another. Look at George Hotz and https://comma.ai/ https://comma.ai/ - I am skeptical he and his organization would "no one gives a fuck" about broken shit. - https://www.youtube.com/watch?v=FOuCKQfybz8 https://www.youtube.com/watch?v=FOuCKQfybz8 personality snippet

EvanAnderson 4y ago

I don't know that anybody ever cared. The barrier to entry re: running an IT infrastructure has fallen so far that gross incompetence still delivers a serviceable-enough product.

gw99 4y ago
That's a depressingly accurate portrayal of the situation.
mattpallissard 4y ago
Eh, I wouldn't say it's fallen. There's just as much incompetence per capita as there ever was. I just think there are a lot more people working in the tech sector than ever before so it's more noticeable.
- EvanAnderson 4y ago
  There are so many more tools to allow the incompetent to "deliver" a reasonable-enough result. There's a certain amount of "don't let perfect be the enemy of the good" in there, for sure, but like the grandparent poster, I've walked into many gigs where "long-standing problems" turn out to be easily fixed. Nobody with sufficient background knowledge and experience ever looked at them and they just festered.