6 ms·

I like the article at lot, I've got a similar background. My answer to the question would be: start explaining what you're asking the OS to do when you run you

I like the article at lot, I've got a similar background. My answer to the question would be: start explaining what you're asking the OS to do when you run your program. There's an entire category of developers who view the IDE and the language as a perfect hermetically sealed environment, until they meet a leaky abstraction. Some favourites: 1. Depleting the outbound TCP ports because ports aren't infinite and the code didn't know that. 2. Any nodejs BS that involves a precompiled C lib, where homebrew or some nonsense 'just works' on the dev machine but prod runs alpine/musl. NPM sneaking arch specific blobs in is really to blame here but what do we expect from the failed state of package managers. 3. Assuming latency is not a thing. Network calls everywhere. No management of timeouts / killed connections. 4. Memory, in all shapes and forms. Most typically thinking a GC prevents memory leaks, but more generally, having no clue whatsoever what the memory usage profile of the application is. A systems engineer (IMO) has much better answers to these questions than a regular developer, and it is absolutely mostly arcane nonsense, (ulimit anyone?), until it isn't. Personally I love OS tech so maybe I'm just cut out for it. I used to be able to tell what the machine was doing from the HDD whirrs and still have a 'feel' for system latency, e.g. when something completes too quickly.

zorked 3y ago
> I used to be able to tell what the machine was doing from the HDD whirrs I guess that dimension of debugging is now lost with SSDs. I didn't run an anti-virus and noticed by machine had caught a virus (in this case, DOS/Natas) because of the extra moments of whirrs during boot and when running applications.
- beedeebeedee 3y ago
  I've been doing some audio work simulating audio equipment, and I bet you could pipe out some sound for disk writes and use that as a debug tool.
  - anotherhue 3y ago
    https://www.retrokits.de/index.php/hdd-clicker-hdd-sound-simulator-v0-1/ https://www.retrokits.de/index.php/hdd-clicker-hdd-sound-sim...
    - beedeebeedee 3y ago
      That's really cool, and neat that he uses the activity light from the hard drive as input to generate sound. I wonder how much slower processes would run if you generate sound on the computer itself from every hard drive write. I don't have deep enough knowledge about the OS to know where you would run your sound program without affecting performance (i.e., are there log files that could be read, so it is not directly interacting with the hard drive writes? would those still be timely enough to give a good indicator of what the computer is actually doing?) If that could be solved, then you could have a sound for network activity, every time a kernel module loads, etc, so users don't need to have deep knowledge of the system to know when there is a problem- they just have to learn what sounds right. I think it would be really neat to have music made by the machine itself and love the idea of adding back that dimension of debugging.
jdwithit 3y ago
The latency one hits hard, as someone approaching this from the other side who came up through Ops and learned software development over time. So much software I've had to support was written with the assumption of flawless network connectivity. No latency, no dropped packets, no fluctuation in response time. If there was any disruption or a call didn't return within a very short timeout, the app just blocked and eventually crashed. Any sort of network maintenance was a huge problem because a couple seconds of spanning tree reconverging or whatever meant a significant outage until all the applications recovered. This was of course always the network's/ops' fault, using patterns and libraries that have been around for over a decade to handle the inherently unreliable nature of computer networks was never on the table. I've seen some replies that this is basic stuff everyone should know, but the fact is, people don't. Even Senior or Principal engineers (depending on the company). You can get very far in software engineering without understanding anything outside the bubble of your domain. Hell this is basically why the field of SRE exists, because a lot of developers do not know how to write reliable software that works in the real world, not just in a perfect test environment. If you've somehow worked your entire career surrounded by engineers who have even an entry level understanding of how networks function, you're the outlier. More often I've found people invent their own bizarre mental model of what's happening that only has the vaguest grounding in reality. And honestly that's ok much of the time. Everyone can't be an expert in everything. Just have some grace about it. Don't pull a Dunning-Kruger and assume because you are a fabulous Java developer you are automatically an expert in every technical field. Most of us aren't.
gdugdjfh 3y ago
"ulimit" is discussed on the bash manual page. It isn't arcane nonsense. In fact it is incredibly concrete and understandable even by newbies. You're going to have to look elsewhere for an example of obscure, complex, useless technicalities.
tra3 3y ago
> used to be able to tell what the machine was doing from the HDD whirrs This is a tell. You’re old enough that stacks used to be shallower and you were closer to the OS.
- mjevans 3y ago
  Often the audio ground was the same as the system ground and you could literally hear activity as low static when the speakers were on.
  - anotherhue 3y ago
    and occasionally a ghostly french singer https://www.theguardian.com/technology/askjack/2010/mar/03/pc-speakers-pick-up-radio https://www.theguardian.com/technology/askjack/2010/mar/03/p...
- PhilipRoman 3y ago
  Eh, from my laptop's noises I can tell approximately what is showing up on a remote console running over X11. For some reason whenever text is updating it makes a faint buzzing noise. The matrix movie joke about seeing code is actually pretty close to the truth. Not necessarily a sign of a great programmer, but still a nice skill to have.
kqr 3y ago
The realisation that "any resource exists only in finite amounts" covers most of your points, except perhaps that about C libraries!
okdood64 3y ago
Man all this reminds me so much of my previous gig and it was so FUN. Now I have a not so fun job at Big Tech doing SRE type work; all of that mucking around is abstracted away from me.
- BirAdam 3y ago
  IMHO, the fun is that less abstracted systems are actually fixable because you can “see” the failures, as opposed to cloudy stuff where you do a lot of work to prove that the problem is on the provider’s side, and then wait months for them to implement a fix… only then to have already worked around the issue.
sph 3y ago
That's the type of job I enjoy doing! After 17 years in this career, I dislike calling myself a "software engineer" because it's a meaningless term that now is used to describe anyone that knows how to build a web page. "Full-stack engineer" has also been stolen from us by recruiter to refer to frontend devs. WTF. As a teenager I started doing software interfacing with hardware, reversing on Windows then I wrote my own mini OS, got a job and did web dev, sysadmin, backend, infrastructure, and CTO, alongside dozens of consulting contracts. I get hired to fix a problem, and soon enough I'm diving into the technicalities of the network stack, or tracking down a bottleneck with spit and strace, or scaling an infrastructure horizontally and making it fault tolerant. At this point most of the value I provide is not in the volume of code I deliver, but designing, operating and maintaining systems as a whole. A systems engineer, if you like, but as the article mentions, that's a real qualification that's got nothing to do with this. The issues is that in the job market I have to compete with everyone called a software engineer, slap a senior to it, which these days refers to anyone that has 5 years of experience, and feel it's very hard to convey to a recruiter the depth of my experience. A programmer operates on code, a "systems engineer" looks at systems holistically, horizontally (i.e. architecture) and vertically (i.e. diving down the stack)
- kabouseng 3y ago
  Funny (I am not too serious about this point :D ), but I feel systems engineer is actually an engineer that has a multi-disciplinary skill set (mechanical, electronic, software etc) that guides a multi-disciplinary team to develop a complex product. Now it is repurposed for full-stack software engineers.
  - sph 3y ago
    Is there anyone that can extend their knowledge to mechanical and electronic engineering, as well as software? I feel that my breadth in software knowledge is reaching the limits of my mental capacity and I have to shed useless weight now (i.e. I haven't kept up with frontend for 3 years now) My father was an electronic engineer, and I would love to get into it eventually as I grew up organising his components drawer, burning LEDs and licking solder wire (yeah.. I regret it now), but that's going to be something I'll have time for when I retire.
goodpoint 3y ago
The 4 things you listed are really basic knowledge.
- d0mine 3y ago
  Different people may have different basic knowledge https://xkcd.com/1053/ https://xkcd.com/1053/
- goodpoint 3y ago
  Great. Silent downvotes from HNers that don't even know TCP.
  - eropple 3y ago
    1) They're formative knowledge for understanding that part of a system. Systems are big and few people understand them entirely. I happened to know those things, but I am a pathologically T-shaped/"paint drip" developer. The middlebrow dismissal of "oh that's basic" is going to get a rise out of people. 2) You threw a one-line, no-help comment out there. That's going to get you downvoted unless it is at least interesting or funny. 3) Now you're complaining about downvotes, which is a big red sign for "downvote me more, for I am mad".
uxp100 3y ago
Perhaps the term for this is System Software engineer? (I had that title for a while, it has a sort of pleasant ring to it) As others have noted I’m more familiar with the systems engineer job title involving giant webs of requirements and specifications. Maybe something not so far off from a Product Owner in the “tech” world, though I’m not sure about that.
sergiosgc 3y ago
> Personally I love OS tech so maybe I'm just cut out for it. I used to be able to tell what the machine was doing from the HDD whirrs (...) I loved the story about how a developer of ZFS, from the Sun days, went down a rabbit hole of optimizing the filesystem because he didn't like the sound hard drives made on certain write loads. Turns out he was right, there was funky stuff going on with the I/O scheduler. Too long ago, I don't remember the source, sorry.

zorked 3y ago

> I used to be able to tell what the machine was doing from the HDD whirrs I guess that dimension of debugging is now lost with SSDs. I didn't run an anti-virus and noticed by machine had caught a virus (in this case, DOS/Natas) because of the extra moments of whirrs during boot and when running applications.

beedeebeedee 3y ago
I've been doing some audio work simulating audio equipment, and I bet you could pipe out some sound for disk writes and use that as a debug tool.
- anotherhue 3y ago
  https://www.retrokits.de/index.php/hdd-clicker-hdd-sound-simulator-v0-1/ https://www.retrokits.de/index.php/hdd-clicker-hdd-sound-sim...
  - beedeebeedee 3y ago
    That's really cool, and neat that he uses the activity light from the hard drive as input to generate sound. I wonder how much slower processes would run if you generate sound on the computer itself from every hard drive write. I don't have deep enough knowledge about the OS to know where you would run your sound program without affecting performance (i.e., are there log files that could be read, so it is not directly interacting with the hard drive writes? would those still be timely enough to give a good indicator of what the computer is actually doing?) If that could be solved, then you could have a sound for network activity, every time a kernel module loads, etc, so users don't need to have deep knowledge of the system to know when there is a problem- they just have to learn what sounds right. I think it would be really neat to have music made by the machine itself and love the idea of adding back that dimension of debugging.

jdwithit 3y ago

The latency one hits hard, as someone approaching this from the other side who came up through Ops and learned software development over time. So much software I've had to support was written with the assumption of flawless network connectivity. No latency, no dropped packets, no fluctuation in response time. If there was any disruption or a call didn't return within a very short timeout, the app just blocked and eventually crashed. Any sort of network maintenance was a huge problem because a couple seconds of spanning tree reconverging or whatever meant a significant outage until all the applications recovered. This was of course always the network's/ops' fault, using patterns and libraries that have been around for over a decade to handle the inherently unreliable nature of computer networks was never on the table. I've seen some replies that this is basic stuff everyone should know, but the fact is, people don't. Even Senior or Principal engineers (depending on the company). You can get very far in software engineering without understanding anything outside the bubble of your domain. Hell this is basically why the field of SRE exists, because a lot of developers do not know how to write reliable software that works in the real world, not just in a perfect test environment. If you've somehow worked your entire career surrounded by engineers who have even an entry level understanding of how networks function, you're the outlier. More often I've found people invent their own bizarre mental model of what's happening that only has the vaguest grounding in reality. And honestly that's ok much of the time. Everyone can't be an expert in everything. Just have some grace about it. Don't pull a Dunning-Kruger and assume because you are a fabulous Java developer you are automatically an expert in every technical field. Most of us aren't.

gdugdjfh 3y ago

"ulimit" is discussed on the bash manual page. It isn't arcane nonsense. In fact it is incredibly concrete and understandable even by newbies. You're going to have to look elsewhere for an example of obscure, complex, useless technicalities.

tra3 3y ago

> used to be able to tell what the machine was doing from the HDD whirrs This is a tell. You’re old enough that stacks used to be shallower and you were closer to the OS.

mjevans 3y ago
Often the audio ground was the same as the system ground and you could literally hear activity as low static when the speakers were on.
- anotherhue 3y ago
  and occasionally a ghostly french singer https://www.theguardian.com/technology/askjack/2010/mar/03/pc-speakers-pick-up-radio https://www.theguardian.com/technology/askjack/2010/mar/03/p...
PhilipRoman 3y ago
Eh, from my laptop's noises I can tell approximately what is showing up on a remote console running over X11. For some reason whenever text is updating it makes a faint buzzing noise. The matrix movie joke about seeing code is actually pretty close to the truth. Not necessarily a sign of a great programmer, but still a nice skill to have.

kqr 3y ago

The realisation that "any resource exists only in finite amounts" covers most of your points, except perhaps that about C libraries!

okdood64 3y ago

Man all this reminds me so much of my previous gig and it was so FUN. Now I have a not so fun job at Big Tech doing SRE type work; all of that mucking around is abstracted away from me.

BirAdam 3y ago
IMHO, the fun is that less abstracted systems are actually fixable because you can “see” the failures, as opposed to cloudy stuff where you do a lot of work to prove that the problem is on the provider’s side, and then wait months for them to implement a fix… only then to have already worked around the issue.

sph 3y ago

That's the type of job I enjoy doing! After 17 years in this career, I dislike calling myself a "software engineer" because it's a meaningless term that now is used to describe anyone that knows how to build a web page. "Full-stack engineer" has also been stolen from us by recruiter to refer to frontend devs. WTF. As a teenager I started doing software interfacing with hardware, reversing on Windows then I wrote my own mini OS, got a job and did web dev, sysadmin, backend, infrastructure, and CTO, alongside dozens of consulting contracts. I get hired to fix a problem, and soon enough I'm diving into the technicalities of the network stack, or tracking down a bottleneck with spit and strace, or scaling an infrastructure horizontally and making it fault tolerant. At this point most of the value I provide is not in the volume of code I deliver, but designing, operating and maintaining systems as a whole. A systems engineer, if you like, but as the article mentions, that's a real qualification that's got nothing to do with this. The issues is that in the job market I have to compete with everyone called a software engineer, slap a senior to it, which these days refers to anyone that has 5 years of experience, and feel it's very hard to convey to a recruiter the depth of my experience. A programmer operates on code, a "systems engineer" looks at systems holistically, horizontally (i.e. architecture) and vertically (i.e. diving down the stack)

kabouseng 3y ago
Funny (I am not too serious about this point :D ), but I feel systems engineer is actually an engineer that has a multi-disciplinary skill set (mechanical, electronic, software etc) that guides a multi-disciplinary team to develop a complex product. Now it is repurposed for full-stack software engineers.
- sph 3y ago
  Is there anyone that can extend their knowledge to mechanical and electronic engineering, as well as software? I feel that my breadth in software knowledge is reaching the limits of my mental capacity and I have to shed useless weight now (i.e. I haven't kept up with frontend for 3 years now) My father was an electronic engineer, and I would love to get into it eventually as I grew up organising his components drawer, burning LEDs and licking solder wire (yeah.. I regret it now), but that's going to be something I'll have time for when I retire.

goodpoint 3y ago

The 4 things you listed are really basic knowledge.

d0mine 3y ago
Different people may have different basic knowledge https://xkcd.com/1053/ https://xkcd.com/1053/
goodpoint 3y ago
Great. Silent downvotes from HNers that don't even know TCP.
- eropple 3y ago
  1) They're formative knowledge for understanding that part of a system. Systems are big and few people understand them entirely. I happened to know those things, but I am a pathologically T-shaped/"paint drip" developer. The middlebrow dismissal of "oh that's basic" is going to get a rise out of people. 2) You threw a one-line, no-help comment out there. That's going to get you downvoted unless it is at least interesting or funny. 3) Now you're complaining about downvotes, which is a big red sign for "downvote me more, for I am mad".

uxp100 3y ago

Perhaps the term for this is System Software engineer? (I had that title for a while, it has a sort of pleasant ring to it) As others have noted I’m more familiar with the systems engineer job title involving giant webs of requirements and specifications. Maybe something not so far off from a Product Owner in the “tech” world, though I’m not sure about that.

sergiosgc 3y ago

> Personally I love OS tech so maybe I'm just cut out for it. I used to be able to tell what the machine was doing from the HDD whirrs (...) I loved the story about how a developer of ZFS, from the Sun days, went down a rabbit hole of optimizing the filesystem because he didn't like the sound hard drives made on certain write loads. Turns out he was right, there was funky stuff going on with the I/O scheduler. Too long ago, I don't remember the source, sorry.