5 ms·

Possibly because so much of what happens in the tech industry has been abstracted away. The vast majority of the people responsible for the servers on the bill

Possibly because so much of what happens in the tech industry has been abstracted away. The vast majority of the people responsible for the servers on the billion-dollar company I work for have never physically touched a server, a rack, or an Ethernet cable. There's a whole generation of "point and click" admins quickly gaining prominence. I suspect cloud services have contributed to this lack of knowledge, because if all you know is cloud, you don't need to know things like this.

hinkley 4y ago
That, and the curse of working the same 18 months 4 times on the way to 'senior'. All of these little companies starting with brand new hardware as well. Far from all but a few of those 'garage startups' I knew of did limp along on used hardware for some time. It happened again a bit after the dotcom bubble burst (especially for office equipment) and again around 2008. But for drives in particular, people are pretty superstitious about giving them up.
- anitil 4y ago
  > the same 18 months 4 times This feels honestly like my whole career. Just get to a point of competence and then get an offer you can't refuse.
  - hinkley 4y ago
    The best I can say is try not to keep coming into the same point in the lifecycle of each new company, and try to pay attention to things you didn't last time instead of just jumping into your comfort zone. It can help seeing people struggle with tech debt or bad decisions you had nothing to do with, if you can put yourself into the shoes of the person who did it. The wise man learns from the mistakes of others. But it does mean you have to do a little more sleuthing than the typical postmortem that just blames everything on an engineer who quit six months ago. That's facile, and neglects the contributions of people who still work here and/or think the same way.
notacoward 4y ago
"Point and click" is unnecessarily derogatory. During my career as a developer I spent my share of time racking my own systems, diagnosing (both network and disk) cable faults, etc. I also spent time working with the kinds of software tools needed to manage dozens of clusters up to 10K machines each at 36 disks per machine. At that scale you really need to minimize the number of people who get to muck with the physical bits, and the amount of time they spend doing so. The complexity has merely shifted, not disappeared, and most developers or sysadmins from the previous generation would themselves be hard pressed to master the management of modern systems. It's no more "point and click" now than it was "lift heavy things" before. > if all you know is cloud, you don't need to know things like this. That's part of the cloud value proposition, isn't it? Leave the physical (and other low-level) stuff to people who do know that stuff, and can also reap economies of scale, while you focus on your own parts of the problem. Do you not approve of such specialization?
- pclmulqdq 4y ago
  I think the unfortunate reality of this is that even if you don't take care of the physical and OS stuff yourself, it will still hurt you when it goes wrong. Abstracting it away doesn't really take care of it for you, at least not in the same way that someone who really knows how to deal with it AND knows your system well does. And you pay through the nose for the privilege of that abstraction - even DigitalOcean is 2-3x more expensive than renting your hardware from a hosting provider, and things like managed databases and AWS lambda will run you 20x more than renting servers. When AWS fails, it fails spectacularly. It also experiences outages fairly often, and you as the developer often have nothing you can do about it, but twiddle your thumbs and hope that your application was amenable to "high-availability architecture" (which is more expensive...). Finally, this particular failure mode brought down HN's servers, rented from a reputable hosting provider, and arguably this is something that a hosting provider should know.
  - toast0 4y ago
    > I think the unfortunate reality of this is that even if you don't take care of the physical and OS stuff yourself, it will still hurt you when it goes wrong. Abstracting it away doesn't really take care of it for you, at least not in the same way that someone who really knows how to deal with it AND knows your system well does It's also really hard to fix things when they're surrounded by a black box operated by someone who doesn't care. I kind of enjoy doing it, but it would be a lot more productive if I was able to diagnose network link problems by checking the error counters on all the routers/switches rather than by sending pings that hash over the numerous lacp alternate routes between two servers to figure out how to get a traceroute that shows the issue so the network owner can look at counters and fix the underlying issue.
pragmatic 4y ago
ClickOps is a term I heard second hand for this.
majewsky 4y ago
Once you cross a certain size, how else is it supposed to work, realistically speaking? My team administers servers in Brazil, Australia, Japan, Netherlands, the US, and several other countries. We can't do racking, cabling and hardware maintenance by ourselves. If we did, we'd spend 50% of our work hours just commuting on international flights. The only realistic option is to rely on local DC Ops teams to perform the manual steps involved in hardware buildup, maintenance and decom.

hinkley 4y ago

That, and the curse of working the same 18 months 4 times on the way to 'senior'. All of these little companies starting with brand new hardware as well. Far from all but a few of those 'garage startups' I knew of did limp along on used hardware for some time. It happened again a bit after the dotcom bubble burst (especially for office equipment) and again around 2008. But for drives in particular, people are pretty superstitious about giving them up.

anitil 4y ago
> the same 18 months 4 times This feels honestly like my whole career. Just get to a point of competence and then get an offer you can't refuse.
- hinkley 4y ago
  The best I can say is try not to keep coming into the same point in the lifecycle of each new company, and try to pay attention to things you didn't last time instead of just jumping into your comfort zone. It can help seeing people struggle with tech debt or bad decisions you had nothing to do with, if you can put yourself into the shoes of the person who did it. The wise man learns from the mistakes of others. But it does mean you have to do a little more sleuthing than the typical postmortem that just blames everything on an engineer who quit six months ago. That's facile, and neglects the contributions of people who still work here and/or think the same way.

notacoward 4y ago

"Point and click" is unnecessarily derogatory. During my career as a developer I spent my share of time racking my own systems, diagnosing (both network and disk) cable faults, etc. I also spent time working with the kinds of software tools needed to manage dozens of clusters up to 10K machines each at 36 disks per machine. At that scale you really need to minimize the number of people who get to muck with the physical bits, and the amount of time they spend doing so. The complexity has merely shifted, not disappeared, and most developers or sysadmins from the previous generation would themselves be hard pressed to master the management of modern systems. It's no more "point and click" now than it was "lift heavy things" before. > if all you know is cloud, you don't need to know things like this. That's part of the cloud value proposition, isn't it? Leave the physical (and other low-level) stuff to people who do know that stuff, and can also reap economies of scale, while you focus on your own parts of the problem. Do you not approve of such specialization?

pclmulqdq 4y ago
I think the unfortunate reality of this is that even if you don't take care of the physical and OS stuff yourself, it will still hurt you when it goes wrong. Abstracting it away doesn't really take care of it for you, at least not in the same way that someone who really knows how to deal with it AND knows your system well does. And you pay through the nose for the privilege of that abstraction - even DigitalOcean is 2-3x more expensive than renting your hardware from a hosting provider, and things like managed databases and AWS lambda will run you 20x more than renting servers. When AWS fails, it fails spectacularly. It also experiences outages fairly often, and you as the developer often have nothing you can do about it, but twiddle your thumbs and hope that your application was amenable to "high-availability architecture" (which is more expensive...). Finally, this particular failure mode brought down HN's servers, rented from a reputable hosting provider, and arguably this is something that a hosting provider should know.
- toast0 4y ago
  > I think the unfortunate reality of this is that even if you don't take care of the physical and OS stuff yourself, it will still hurt you when it goes wrong. Abstracting it away doesn't really take care of it for you, at least not in the same way that someone who really knows how to deal with it AND knows your system well does It's also really hard to fix things when they're surrounded by a black box operated by someone who doesn't care. I kind of enjoy doing it, but it would be a lot more productive if I was able to diagnose network link problems by checking the error counters on all the routers/switches rather than by sending pings that hash over the numerous lacp alternate routes between two servers to figure out how to get a traceroute that shows the issue so the network owner can look at counters and fix the underlying issue.

pragmatic 4y ago

ClickOps is a term I heard second hand for this.

majewsky 4y ago

Once you cross a certain size, how else is it supposed to work, realistically speaking? My team administers servers in Brazil, Australia, Japan, Netherlands, the US, and several other countries. We can't do racking, cabling and hardware maintenance by ourselves. If we did, we'd spend 50% of our work hours just commuting on international flights. The only realistic option is to rely on local DC Ops teams to perform the manual steps involved in hardware buildup, maintenance and decom.