9 ms·

this is bound to happen. the more complicated the stack that you use becomes, the less details you understand about the lower levels. who, today, can write or

this is bound to happen. the more complicated the stack that you use becomes, the less details you understand about the lower levels. who, today, can write or optimize assembly by hand? How about understand the OS internals? How about write a compiler? How about write a library for their fav language? How about actually troubleshoot a misbehaving *nix process? All of these were table stakes at some point in time. The key is not to understand all layers perfectly. The key is to know when to stop adding layers.

abathur 5y ago
And maybe to learn the smell of a leaking layer?
ohgodplsno 5y ago
While I will not pretend to be an expert at either of those, having at least a minimal understanding of all of these is crucial if you want to pretend to be a software engineer. If you can't write a library, or figure out why your process isn't working, you're not an engineer, you're a plumber, or a code monkey. Not to say that's bad, but considering the sheer amount of mediocre devs at FAANG calling themselves engineers, it just really shines a terrible light on our profession.
- puffyvolvo 5y ago
  abstractions layers exist for this reason. as much of a sham as the 7-layer networking model is, it's the reason you can spin up an http server without knowing tcp internals, and you can write a webapp without caring (much) about if its being served over https, http/2, or SPDY.
  - dweekly 5y ago
    All true. The problems start getting gnarly when Something goes Wrong in the magic black box powering your service. That neat framework that made it trivial to spin up an HTTP/2 endpoint is emitting headers that your CDN doesn't like and now suddenly you're 14 stack layers deep in a new codebase written in a language that may not be your forte...
  - lanstin 5y ago
    I would make a big distinction between 'without knowing' and "without worrying about." Software productivity is directly proportional to the amount of the system you can ignore while you are writing the code at hand. But not knowing how stuff works makes you less of an engineer and more of a artist. Cause and effect and reason are key tools, and not knowing about TCP handshake or windows just makes it difficult to figure out how to answer fundamental questions about how your code works. It means things will be forever mysterious to you, or interesting in the sense of biology where you gather a lot of data rather than mathematics where pure thought can give you immense power.
  - exdsq 5y ago
    But this does matter to web developers! For example http/2 lets you request multiple files at once and server push support. If you don't know this you might not implement it and end up with subpar performance. http/3 is going to be built on UDP-based Quic and won't even support http:// http://, will need a `Alt-Svc:` header, and removes the http/2 prioritisation stuff. God knows how a UDP-based http is going to work but these are considerations a 'Software Engineer' who works on web systems should think about.
    - cjalmeida 5y ago
      Err, no. Look at most startups and tell me how many of them care if they’re serving optimized content over HTTP/2?
    - kaba0 5y ago
      Someone writing the framework should absolutely be intimately familiar with it, and should work on making these new capabilities easy to use from a higher level where your typical web dev can make use of it without much thought, if any.
  - ohgodplsno 5y ago
    While I wouldn't judge someone not knowing anything about layer 1 or 2, knowing something about MTUs, traffic congestion, routing is something that should be taught at any basic level of CS school. Not caring if it's served over http2? Why the hell would you? Write your software to take advantage of the platform it's on, and the stack beneath it. The simple fact of using http2 might change your organisation from one fat file served from a CDN, into many that load in parallel and quicker. By not caring about this, you just... waste it all to make yet another shitty-performing webapp. In the same way, I don't ask you to know the TCP protocol by heart, but knowing just basics means you can open up wireshark and debug things. Once again: if you don't know your stack, you're just wasting performance everywhere, and you're just a code plumber.
    - puffyvolvo 5y ago
      > knowing something about MTUs isn't that why MTU discovery exists? > Write your software to take advantage of the platform it's on, and the stack beneath it sure, but usually those bits are usually abstracted away still. otherwise cross-compatability or migrating to a different stack becomes a massive pain. > The simple fact of using http2 might change your organisation from one fat file served from a CDN, into many that load in parallel and quicker. others have pointed out things like h2push specifically, that was kind of what i meant with the "(much)" in my original comment. Even then with something like nginx supporting server-push on its end, whatever its fronting could effectively be http/2 unaware and still reap some of the benefits. I imagine it wont be long before there are smarter methods to transparently support this stuff.
  - jimbokun 5y ago
    To be an engineer, you need the ability to dive deeper into these abstractions when necessary, while most of the time you can just not think about them. Quickly getting up to speed on something you don't know yet is probably the single most critical skill to be a good engineer.
- deleted 5y ago
  [deleted]
- rantwasp 5y ago
  you know. deep down inside: we are all code monkeys. Also, as much as people like to call it software engineering, it's anything but engineering. In 95% of cases if you want to get something/anything done you will need to work at an abstraction layer where a lot of things have been decided already for you and you are just gluing them together. It's not good or bad. It is what it is.
exdsq 5y ago
Totally get your point! But I worry the industry is becoming bloated with people who can glue a few frameworks together building systems we depend on. I wish there was more of a focus on teaching and/or learning fundermentals than frameworks. Regarding your points, I actually would expect a non-junior developer to be able to write a libary in their main language and understand the basics of OS internals (to the point of debugging and profilling, which would include troubleshooting *nix processes). I don't expect them to know assembly or C, or be able to write a compiler (although I did get this for a take-home test just last week).
- fruzz 5y ago
  I think learning the fundamentals is a worthy pursuit, but in terms of getting stuff done well, you realistically only have to grok one level below whatever level of abstraction you're operating at. Being able to glue frameworks together to build systems is actually not a negative. If you're a startup, you want people to leverage what's already available.
  - bsd44 5y ago
    I agree. An ideal is far from reality. I like to get deep into low level stuff, but my employer doesn't care if I understand how a system call works or whether we can save x % of y by spending z time on performance profiling that requires good knowledge of Linux debugging and profiling tools. It's quicker, cheaper and more efficient to buy more hardware or scale up in public cloud and let me use my time to work on another project that will result in shipping a product or a service quicker and have direct impact on the business. My experience with the (startup) business world is that you need to be first to ship a feature or you lose. If you want to do something then you should use the tools that will allow you to get there as fast as possible. And to achieve that it makes sense to use technologies that other companies utilise because it's easy to find support online and easy to find qualified people that can get the job done quickly. It's a dog-eat-dog world and startups in particular have the pressure to deliver and deliver fast since they can't burn investor money indefinitely; so they pay a lot more than large and established businesses to attract talent. Those companies that develop bespoke solutions and build upon them have a hard time attracting talent because people are afraid they won't be able to change jobs easily and these companies are not willing to pay as much money. Whether you know how a boot process works or how to optimise your ELK stack to squeeze out every single atom of resource is irrelevant. What's required is to know the tools to complete a job quickly. That creates a divide in the tech world where on one side you have high-salaried people who know how to use these tools but don't really understand what goes on in the background and people who know the nitty-gritty and get paid half as much working at some XYZ company that's been trading since the 90s and is still the same size. My point is that understanding how something works underneath is extremely valuable and rewarding but isn't required to be good at something else. Nobody knows how Android works but that doesn't stop you from creating an app that you will generate revenue and earn you a living. Isn't the point of constant development of automation tools to make our jobs easier? EDIT: typo
lanstin 5y ago
But the value isn't equal. If you think of the business value implemented in code as the "picture" and the run time environment provided as the "frame" the frame has gotten much larger and the picture much smaller, as far as what people are spending their time on. (Well, not the golang folks that just push out a systemctl script and a static binary, but the k8s devops experts). I have read entire blogs on k8s and so on where the end result is just "hello world." In the old days, that was the end of the first paragraph. Now a lot of YAML and docker files and so on and so on are needed just to get to that hello world. Unix was successful initially because it was a good portable abstraction to managing hardware resources, compute, storage, memory, and network, over a variety of actual physical implementations. Many many of the problems people are addressing in k8s and running "a variety of containers efficiently on a set of hosts" are similar to problems unix solved in the 80s. I'm not really saying we should go back, Docker is certainly a solution to "depdendency control and process isolation" when you can't have a good static binary that runs a number of identical processes on a host, but the knowledge of what a socket is or how schedulers work is valuable in fixing issues in docker-based systems. (I'm actually more experienced in Mesos/docker rather than k8s/docker but the bugs are from containers spawning too many GC threads or whatever). If someone is trying to debug that LB and doesn't know what a socket is, or debug latency in apps in the cluster and not know how scheduling and perf engineering tools work, then it's going to be hard for them, and extremely likely that they will just jam 90% solution around 90% solution, enlarging the frame to do more and more, instead of actually fixing things, even if their specific problem was easy to fix and would have had a big pay off.
- coryrc 5y ago
  Kubernetes is complicated because it carries around Unix with it and then duplicates half the things and bolts some new ones on. Erlang is[0] what you can get when you try to design a coherent solution to the problem from a usability and first-principles sort of idea. But some combination of Worse is Better, Path Dependence, and randomness (hg vs git) has led us here. [0] As far as what I've read about its design philosophy.
- EdwardDiego 5y ago
  Who is using K8s for Hello World levels of complexity? Complex problems often have complex solutions, the algorithm we need to run as developers is - what's the net complexity cost of my system if I use this tool? If the tool isn't removing more complexity than it's adding, you probably shouldn't use it.
908B64B197 5y ago
> How about understand the OS internals? How about write a compiler? How about write a library for their fav language? How about actually troubleshoot a misbehaving *nix process? That's what I expect from someone who graduated from a serious CS/Engineering program.
- rantwasp 5y ago
  you're mixing having an idea of how the OS works (ie: conceptual/high level) to having working knowledge and being able to hack into the OS when needed. I know this may sound like moving the goal posts, but it really does not help me that I know conceptually that there is a file system if I don't work with it directly and/or know how to debug issues that arise from it.
  - throwawaygh 5y ago
    > having working knowledge and being able to hack into the OS when needed. I'm going to parrot the GP: "That's what I expect from someone who graduated from a serious CS/Engineering program." I know there are a lot of really bad CS programs in the US, but some experience implementing OS components in a System course so that they can "hack into the OS when needed" is exactly what I would expect out of a graduate from a good CS program.
    - Hermitian909 5y ago
      I think your expectations are out of alignment with what's happening. I know software engineers who graduated with CS degrees from schools like MIT, Urbana Champaigne, and Stanford who took Operating System classes but could not realistically "hack into the OS". If those programs aren't consistently imparting that knowledge to students without an explicit interest, I don't see how others can be expected to...
      - throwawaygh 5y ago
        By into I assume you meant on. The OS courses at UIUC (not a wine, btw :)), MIT, and Stanford def prepare you some kernel hacking if needed.
ModernMech 5y ago
> who, today, can write or optimize assembly by hand? How about understand the OS internals? How about write a compiler? How about write a library for their fav language? How about actually troubleshoot a misbehaving *nix process? Any one of the undergraduates who take the systems sequence at my University should be able to do all of this. At least the ones who earn an A!
shpongled 5y ago
disclaimer: I don't mean this to come across as arrogant or anything (I'm just ignorant). I'm totally self-taught and have never worked a programming job (only programmed for fun). Do professional SWEs not actually understand or have the capability to do these things? I've hacked on hobby operating systems, written assembly, worked on a toy compiler and written libraries... I just kind of assumed that was all par for the course
- woleium 5y ago
  yes, some intermediate devs I've worked with are unable to do almost anything except write code. e.g. unable to generate an ssh key without assistance or detailed cut and paste instructions.
  - shpongled 5y ago
    Maybe I should apply for some senior dev roles then :)
    - kanzenryu2 5y ago
      Many/most senior devs do not have the experience you described. But there are often a lot of meetings, reports, and managing other devs.
    - jimbokun 5y ago
      Yes, you absolutely should, unless you are already making a ton of money in a more fulfilling job.
  - handrous 5y ago
    Shit, I google or manpage or tealdeer ssh key generation every single time.... Pretty much any command I don't run several times a month, I look up. Unless ctrl+r finds it in my history.
- mathgladiator 5y ago
  The challenge is that lower level work doesn't always translate into value for businesses. For instance, knowledge of sockets is very interesting. On one hand, I spent my youth learning sockets. For me to bang out a new network protocol takes a few weeks. For others, it can take months. This manifested in my frustration when I lead building a new transport layer using just sockets. While the people working with me were smart, they had limited low level experience to debug things.
tovej 5y ago
Assembly aside, all the things you mention are things I would expect a software engineer to understand. As an engineer in my late twenties myself, these are exactly the things I am focusing on. I'm not saying I have a particularly deep understanding of these subjects, but I can write a recursive descent parser or a scheduler. I value this knowledge quite highly, since its applicable in many places. I think learning AWS/kubernetes/docker/pytorch/whatever framework is buzzing is easy if you understand Linux/networking/neural networks/whatever the underlying less-prone-to-change system is.
- codeisawesome 5y ago
  Is there a networking-for-developers style course that you would recommend?
  - tovej 5y ago
    The one at your local university. Either one named something like "Introduction to Networking" or "Introduction to Distributed Systems", depending on what you want to learn. You could also read some books. Rami Rosens "Linux Kernel Networking - Implementation and Theory" is quite detailed. The "UNIX and Linux System Administration Handbook" (Nemeth et al.) covers a lot superficially and will point you in the right direction to continue studying. It's very practical-minded. For low-level socket programming, you can probably read "Advanced Programming in the UNIX environment". It might be more detail than you need though. At the other extreme, if you want to study distributed systems, you could read Steen & Tanembaums "Distributed Systems"
xorcist 5y ago
I honestly can't tell if this is sarcasm or not. Which says a lot about the situation we find ourselves in, I guess.
- rantwasp 5y ago
  It's not sarcasm. A lot of things simply do not have visibility and are not rewarded at the business level - therefore the incentives to learn them are almost zero
jimbokun 5y ago
> who, today, can write or optimize assembly by hand? How about understand the OS internals? How about write a compiler? How about write a library for their fav language? How about actually troubleshoot a misbehaving *nix process? But developers should understand what assembly is and what a compiler does. Writing a library for a language you know should be a common development task. How else are you going to reuse a chunk of code needed for multiple projects? Certainly also need to have a basic understanding of unix processes to be a competent developer, too, I would think.
- rantwasp 5y ago
  there is a huge difference between understanding what something is and actually working with it / being proficient with it. huge. I understand how a car engine work. I would actually explain it to someone that does not know what is under the hood. Does that make me a car mechanic? Hell no. If my car breaks down I go to the dealership and have them fix it for me. My car/car engine is ASM/OS Internals/writing a compiler/etc.
swiftcoder 5y ago
> who, today, can write or optimize assembly by hand? How about understand the OS internals? How about write a compiler? How about write a library for their fav language? How about actually troubleshoot a misbehaving *nix process? All of these were table stakes at some point in time. All of these were still table stakes when I graduated from small CS program in 2011. I'm still a bit horrified to discover they apparently weren't table stakes at other places.
codethief 5y ago
This reminds me of Jonathan Blow's excellent talk on "Preventing the Collapse of Civilization": https://www.youtube.com/watch?v=ZSRHeXYDLko https://www.youtube.com/watch?v=ZSRHeXYDLko
ex_amazon_sde 5y ago
> who ... understand the OS internals? ... How about write a library for their fav language? How about actually troubleshoot a misbehaving *nix process? Ex-Amazon here. You are describing standard skills required to pass an interview for a SDE 2 in the teams I've been in at Amazon. Some candidates know all the popular tools and frameworks of the month but do not understand what an OS does, or how a CPU works or networking and do not get hired because they would struggle to write or debug internal software written from scratch. [added later] This was many years ago when the bar raiser thing was in full swing and in teams working on critical infrastructure.
- emerongi 5y ago
  Yes they do. There is too much software to be written. A person with adequate knowledge of higher abstractions can produce just fine code. Yes, if there is a nasty issue that needs to be debugged, understanding the lower layers is super helpful, but even without that knowledge you can figure out what's going on if you have general problem-solving abilities. I certainly have figured out a ton of issues in the internals of tools that I don't know much about. Get off your high horse.
- leesec 5y ago
  Says one guy. Sorry, there's lots of people who make a living writing software who don't know what an OS does. Gatekeeping helps nobody.
- rantwasp 5y ago
  LoL. Also Ex-Amazon here. I can tell you for a fact that most SDE2s I've worked with had zero clue on how the OS works. What you're describing may have been true 5-10 years ago, but I think is no longer true nowadays (what was that? raising the bar they called it). A typical SDE2 interview will not have questions around OS internals in it. Before jumping on your high horse again: I've done around 400 interviews during my tenure there and I don't recall ever failing anyone due to this. Also, gate-keeping is not helpful.
  - nitrogen 5y ago
chubot 5y ago
(author here) The key difference is that a C compiler is a pretty damn good abstraction (and yes Rust is even better without the undefined behavior). I have written C and C++ for decades, deployed it in production, and barely ever looked at assembly language. Kubernetes isn't a good abstraction for what's going on underneath. The blog post linked to direct evidence of that which is too long to recap here; I worked with Borg for years, etc.
- rantwasp 5y ago
  K8s may have its time and place but here is something most people are ignoring: in 80% of the time you don't need it. You don't need all that complexity. You're not Google, you don't have the scale or the problems Google has. You also don't have the discipline AND the tooling Google has to make something like this work (cough cough Borg).
- randomswede 5y ago
  For the things that are 1:1 comparable, the Borg abstraction leaks in pretty much the same places as the Kubernetes abstraction. In slightly different ways. The "kubernetes abstraction" spans a larger space than the Borg abstraction does (note, I count "Chubby" and "GSLB" as "not Borg"), so there are more abstraction leaks as a whole in Kubernetes. Source, I was a Google SRE for 5 years (Ads, Traffic). I ran the in-house kubernetes clusters at a company for 3 years (so, no, no hosted kubernetes, we stood them up either on pretty naked VMs or bare metal).

abathur 5y ago

And maybe to learn the smell of a leaking layer?

ohgodplsno 5y ago

While I will not pretend to be an expert at either of those, having at least a minimal understanding of all of these is crucial if you want to pretend to be a software engineer. If you can't write a library, or figure out why your process isn't working, you're not an engineer, you're a plumber, or a code monkey. Not to say that's bad, but considering the sheer amount of mediocre devs at FAANG calling themselves engineers, it just really shines a terrible light on our profession.

puffyvolvo 5y ago
abstractions layers exist for this reason. as much of a sham as the 7-layer networking model is, it's the reason you can spin up an http server without knowing tcp internals, and you can write a webapp without caring (much) about if its being served over https, http/2, or SPDY.
- dweekly 5y ago
  All true. The problems start getting gnarly when Something goes Wrong in the magic black box powering your service. That neat framework that made it trivial to spin up an HTTP/2 endpoint is emitting headers that your CDN doesn't like and now suddenly you're 14 stack layers deep in a new codebase written in a language that may not be your forte...
- lanstin 5y ago
  I would make a big distinction between 'without knowing' and "without worrying about." Software productivity is directly proportional to the amount of the system you can ignore while you are writing the code at hand. But not knowing how stuff works makes you less of an engineer and more of a artist. Cause and effect and reason are key tools, and not knowing about TCP handshake or windows just makes it difficult to figure out how to answer fundamental questions about how your code works. It means things will be forever mysterious to you, or interesting in the sense of biology where you gather a lot of data rather than mathematics where pure thought can give you immense power.
- exdsq 5y ago
  But this does matter to web developers! For example http/2 lets you request multiple files at once and server push support. If you don't know this you might not implement it and end up with subpar performance. http/3 is going to be built on UDP-based Quic and won't even support http:// http://, will need a `Alt-Svc:` header, and removes the http/2 prioritisation stuff. God knows how a UDP-based http is going to work but these are considerations a 'Software Engineer' who works on web systems should think about.
  - cjalmeida 5y ago
    Err, no. Look at most startups and tell me how many of them care if they’re serving optimized content over HTTP/2?
  - kaba0 5y ago
    Someone writing the framework should absolutely be intimately familiar with it, and should work on making these new capabilities easy to use from a higher level where your typical web dev can make use of it without much thought, if any.
- ohgodplsno 5y ago
  While I wouldn't judge someone not knowing anything about layer 1 or 2, knowing something about MTUs, traffic congestion, routing is something that should be taught at any basic level of CS school. Not caring if it's served over http2? Why the hell would you? Write your software to take advantage of the platform it's on, and the stack beneath it. The simple fact of using http2 might change your organisation from one fat file served from a CDN, into many that load in parallel and quicker. By not caring about this, you just... waste it all to make yet another shitty-performing webapp. In the same way, I don't ask you to know the TCP protocol by heart, but knowing just basics means you can open up wireshark and debug things. Once again: if you don't know your stack, you're just wasting performance everywhere, and you're just a code plumber.
  - puffyvolvo 5y ago
    > knowing something about MTUs isn't that why MTU discovery exists? > Write your software to take advantage of the platform it's on, and the stack beneath it sure, but usually those bits are usually abstracted away still. otherwise cross-compatability or migrating to a different stack becomes a massive pain. > The simple fact of using http2 might change your organisation from one fat file served from a CDN, into many that load in parallel and quicker. others have pointed out things like h2push specifically, that was kind of what i meant with the "(much)" in my original comment. Even then with something like nginx supporting server-push on its end, whatever its fronting could effectively be http/2 unaware and still reap some of the benefits. I imagine it wont be long before there are smarter methods to transparently support this stuff.
- jimbokun 5y ago
  To be an engineer, you need the ability to dive deeper into these abstractions when necessary, while most of the time you can just not think about them. Quickly getting up to speed on something you don't know yet is probably the single most critical skill to be a good engineer.
deleted 5y ago
[deleted]
rantwasp 5y ago
you know. deep down inside: we are all code monkeys. Also, as much as people like to call it software engineering, it's anything but engineering. In 95% of cases if you want to get something/anything done you will need to work at an abstraction layer where a lot of things have been decided already for you and you are just gluing them together. It's not good or bad. It is what it is.

exdsq 5y ago

Totally get your point! But I worry the industry is becoming bloated with people who can glue a few frameworks together building systems we depend on. I wish there was more of a focus on teaching and/or learning fundermentals than frameworks. Regarding your points, I actually would expect a non-junior developer to be able to write a libary in their main language and understand the basics of OS internals (to the point of debugging and profilling, which would include troubleshooting *nix processes). I don't expect them to know assembly or C, or be able to write a compiler (although I did get this for a take-home test just last week).

fruzz 5y ago
I think learning the fundamentals is a worthy pursuit, but in terms of getting stuff done well, you realistically only have to grok one level below whatever level of abstraction you're operating at. Being able to glue frameworks together to build systems is actually not a negative. If you're a startup, you want people to leverage what's already available.
- bsd44 5y ago
  I agree. An ideal is far from reality. I like to get deep into low level stuff, but my employer doesn't care if I understand how a system call works or whether we can save x % of y by spending z time on performance profiling that requires good knowledge of Linux debugging and profiling tools. It's quicker, cheaper and more efficient to buy more hardware or scale up in public cloud and let me use my time to work on another project that will result in shipping a product or a service quicker and have direct impact on the business. My experience with the (startup) business world is that you need to be first to ship a feature or you lose. If you want to do something then you should use the tools that will allow you to get there as fast as possible. And to achieve that it makes sense to use technologies that other companies utilise because it's easy to find support online and easy to find qualified people that can get the job done quickly. It's a dog-eat-dog world and startups in particular have the pressure to deliver and deliver fast since they can't burn investor money indefinitely; so they pay a lot more than large and established businesses to attract talent. Those companies that develop bespoke solutions and build upon them have a hard time attracting talent because people are afraid they won't be able to change jobs easily and these companies are not willing to pay as much money. Whether you know how a boot process works or how to optimise your ELK stack to squeeze out every single atom of resource is irrelevant. What's required is to know the tools to complete a job quickly. That creates a divide in the tech world where on one side you have high-salaried people who know how to use these tools but don't really understand what goes on in the background and people who know the nitty-gritty and get paid half as much working at some XYZ company that's been trading since the 90s and is still the same size. My point is that understanding how something works underneath is extremely valuable and rewarding but isn't required to be good at something else. Nobody knows how Android works but that doesn't stop you from creating an app that you will generate revenue and earn you a living. Isn't the point of constant development of automation tools to make our jobs easier? EDIT: typo

lanstin 5y ago

But the value isn't equal. If you think of the business value implemented in code as the "picture" and the run time environment provided as the "frame" the frame has gotten much larger and the picture much smaller, as far as what people are spending their time on. (Well, not the golang folks that just push out a systemctl script and a static binary, but the k8s devops experts). I have read entire blogs on k8s and so on where the end result is just "hello world." In the old days, that was the end of the first paragraph. Now a lot of YAML and docker files and so on and so on are needed just to get to that hello world. Unix was successful initially because it was a good portable abstraction to managing hardware resources, compute, storage, memory, and network, over a variety of actual physical implementations. Many many of the problems people are addressing in k8s and running "a variety of containers efficiently on a set of hosts" are similar to problems unix solved in the 80s. I'm not really saying we should go back, Docker is certainly a solution to "depdendency control and process isolation" when you can't have a good static binary that runs a number of identical processes on a host, but the knowledge of what a socket is or how schedulers work is valuable in fixing issues in docker-based systems. (I'm actually more experienced in Mesos/docker rather than k8s/docker but the bugs are from containers spawning too many GC threads or whatever). If someone is trying to debug that LB and doesn't know what a socket is, or debug latency in apps in the cluster and not know how scheduling and perf engineering tools work, then it's going to be hard for them, and extremely likely that they will just jam 90% solution around 90% solution, enlarging the frame to do more and more, instead of actually fixing things, even if their specific problem was easy to fix and would have had a big pay off.

coryrc 5y ago
Kubernetes is complicated because it carries around Unix with it and then duplicates half the things and bolts some new ones on. Erlang is[0] what you can get when you try to design a coherent solution to the problem from a usability and first-principles sort of idea. But some combination of Worse is Better, Path Dependence, and randomness (hg vs git) has led us here. [0] As far as what I've read about its design philosophy.
EdwardDiego 5y ago
Who is using K8s for Hello World levels of complexity? Complex problems often have complex solutions, the algorithm we need to run as developers is - what's the net complexity cost of my system if I use this tool? If the tool isn't removing more complexity than it's adding, you probably shouldn't use it.

908B64B197 5y ago

> How about understand the OS internals? How about write a compiler? How about write a library for their fav language? How about actually troubleshoot a misbehaving *nix process? That's what I expect from someone who graduated from a serious CS/Engineering program.

rantwasp 5y ago
you're mixing having an idea of how the OS works (ie: conceptual/high level) to having working knowledge and being able to hack into the OS when needed. I know this may sound like moving the goal posts, but it really does not help me that I know conceptually that there is a file system if I don't work with it directly and/or know how to debug issues that arise from it.
- throwawaygh 5y ago
  > having working knowledge and being able to hack into the OS when needed. I'm going to parrot the GP: "That's what I expect from someone who graduated from a serious CS/Engineering program." I know there are a lot of really bad CS programs in the US, but some experience implementing OS components in a System course so that they can "hack into the OS when needed" is exactly what I would expect out of a graduate from a good CS program.
  - Hermitian909 5y ago
    I think your expectations are out of alignment with what's happening. I know software engineers who graduated with CS degrees from schools like MIT, Urbana Champaigne, and Stanford who took Operating System classes but could not realistically "hack into the OS". If those programs aren't consistently imparting that knowledge to students without an explicit interest, I don't see how others can be expected to...
    - throwawaygh 5y ago
      By into I assume you meant on. The OS courses at UIUC (not a wine, btw :)), MIT, and Stanford def prepare you some kernel hacking if needed.

ModernMech 5y ago

> who, today, can write or optimize assembly by hand? How about understand the OS internals? How about write a compiler? How about write a library for their fav language? How about actually troubleshoot a misbehaving *nix process? Any one of the undergraduates who take the systems sequence at my University should be able to do all of this. At least the ones who earn an A!

shpongled 5y ago

disclaimer: I don't mean this to come across as arrogant or anything (I'm just ignorant). I'm totally self-taught and have never worked a programming job (only programmed for fun). Do professional SWEs not actually understand or have the capability to do these things? I've hacked on hobby operating systems, written assembly, worked on a toy compiler and written libraries... I just kind of assumed that was all par for the course

woleium 5y ago
yes, some intermediate devs I've worked with are unable to do almost anything except write code. e.g. unable to generate an ssh key without assistance or detailed cut and paste instructions.
- shpongled 5y ago
  Maybe I should apply for some senior dev roles then :)
  - kanzenryu2 5y ago
    Many/most senior devs do not have the experience you described. But there are often a lot of meetings, reports, and managing other devs.
  - jimbokun 5y ago
    Yes, you absolutely should, unless you are already making a ton of money in a more fulfilling job.
- handrous 5y ago
  Shit, I google or manpage or tealdeer ssh key generation every single time.... Pretty much any command I don't run several times a month, I look up. Unless ctrl+r finds it in my history.
mathgladiator 5y ago
The challenge is that lower level work doesn't always translate into value for businesses. For instance, knowledge of sockets is very interesting. On one hand, I spent my youth learning sockets. For me to bang out a new network protocol takes a few weeks. For others, it can take months. This manifested in my frustration when I lead building a new transport layer using just sockets. While the people working with me were smart, they had limited low level experience to debug things.

tovej 5y ago

Assembly aside, all the things you mention are things I would expect a software engineer to understand. As an engineer in my late twenties myself, these are exactly the things I am focusing on. I'm not saying I have a particularly deep understanding of these subjects, but I can write a recursive descent parser or a scheduler. I value this knowledge quite highly, since its applicable in many places. I think learning AWS/kubernetes/docker/pytorch/whatever framework is buzzing is easy if you understand Linux/networking/neural networks/whatever the underlying less-prone-to-change system is.

codeisawesome 5y ago
Is there a networking-for-developers style course that you would recommend?
- tovej 5y ago
  The one at your local university. Either one named something like "Introduction to Networking" or "Introduction to Distributed Systems", depending on what you want to learn. You could also read some books. Rami Rosens "Linux Kernel Networking - Implementation and Theory" is quite detailed. The "UNIX and Linux System Administration Handbook" (Nemeth et al.) covers a lot superficially and will point you in the right direction to continue studying. It's very practical-minded. For low-level socket programming, you can probably read "Advanced Programming in the UNIX environment". It might be more detail than you need though. At the other extreme, if you want to study distributed systems, you could read Steen & Tanembaums "Distributed Systems"

xorcist 5y ago

I honestly can't tell if this is sarcasm or not. Which says a lot about the situation we find ourselves in, I guess.

rantwasp 5y ago
It's not sarcasm. A lot of things simply do not have visibility and are not rewarded at the business level - therefore the incentives to learn them are almost zero

jimbokun 5y ago

> who, today, can write or optimize assembly by hand? How about understand the OS internals? How about write a compiler? How about write a library for their fav language? How about actually troubleshoot a misbehaving *nix process? But developers should understand what assembly is and what a compiler does. Writing a library for a language you know should be a common development task. How else are you going to reuse a chunk of code needed for multiple projects? Certainly also need to have a basic understanding of unix processes to be a competent developer, too, I would think.

rantwasp 5y ago
there is a huge difference between understanding what something is and actually working with it / being proficient with it. huge. I understand how a car engine work. I would actually explain it to someone that does not know what is under the hood. Does that make me a car mechanic? Hell no. If my car breaks down I go to the dealership and have them fix it for me. My car/car engine is ASM/OS Internals/writing a compiler/etc.

swiftcoder 5y ago

> who, today, can write or optimize assembly by hand? How about understand the OS internals? How about write a compiler? How about write a library for their fav language? How about actually troubleshoot a misbehaving *nix process? All of these were table stakes at some point in time. All of these were still table stakes when I graduated from small CS program in 2011. I'm still a bit horrified to discover they apparently weren't table stakes at other places.

codethief 5y ago

This reminds me of Jonathan Blow's excellent talk on "Preventing the Collapse of Civilization": https://www.youtube.com/watch?v=ZSRHeXYDLko https://www.youtube.com/watch?v=ZSRHeXYDLko

ex_amazon_sde 5y ago

> who ... understand the OS internals? ... How about write a library for their fav language? How about actually troubleshoot a misbehaving *nix process? Ex-Amazon here. You are describing standard skills required to pass an interview for a SDE 2 in the teams I've been in at Amazon. Some candidates know all the popular tools and frameworks of the month but do not understand what an OS does, or how a CPU works or networking and do not get hired because they would struggle to write or debug internal software written from scratch. [added later] This was many years ago when the bar raiser thing was in full swing and in teams working on critical infrastructure.

emerongi 5y ago
Yes they do. There is too much software to be written. A person with adequate knowledge of higher abstractions can produce just fine code. Yes, if there is a nasty issue that needs to be debugged, understanding the lower layers is super helpful, but even without that knowledge you can figure out what's going on if you have general problem-solving abilities. I certainly have figured out a ton of issues in the internals of tools that I don't know much about. Get off your high horse.
leesec 5y ago
Says one guy. Sorry, there's lots of people who make a living writing software who don't know what an OS does. Gatekeeping helps nobody.
rantwasp 5y ago
LoL. Also Ex-Amazon here. I can tell you for a fact that most SDE2s I've worked with had zero clue on how the OS works. What you're describing may have been true 5-10 years ago, but I think is no longer true nowadays (what was that? raising the bar they called it). A typical SDE2 interview will not have questions around OS internals in it. Before jumping on your high horse again: I've done around 400 interviews during my tenure there and I don't recall ever failing anyone due to this. Also, gate-keeping is not helpful.
- nitrogen 5y ago

chubot 5y ago

(author here) The key difference is that a C compiler is a pretty damn good abstraction (and yes Rust is even better without the undefined behavior). I have written C and C++ for decades, deployed it in production, and barely ever looked at assembly language. Kubernetes isn't a good abstraction for what's going on underneath. The blog post linked to direct evidence of that which is too long to recap here; I worked with Borg for years, etc.

rantwasp 5y ago
K8s may have its time and place but here is something most people are ignoring: in 80% of the time you don't need it. You don't need all that complexity. You're not Google, you don't have the scale or the problems Google has. You also don't have the discipline AND the tooling Google has to make something like this work (cough cough Borg).
randomswede 5y ago
For the things that are 1:1 comparable, the Borg abstraction leaks in pretty much the same places as the Kubernetes abstraction. In slightly different ways. The "kubernetes abstraction" spans a larger space than the Borg abstraction does (note, I count "Chubby" and "GSLB" as "not Borg"), so there are more abstraction leaks as a whole in Kubernetes. Source, I was a Google SRE for 5 years (Ads, Traffic). I ran the in-house kubernetes clusters at a company for 3 years (so, no, no hosted kubernetes, we stood them up either on pretty naked VMs or bare metal).