5 ms·

I think people who have no experience managing servers dramatically overestimate how much time it takes to manage servers. Depending on your team, it can defini

I think people who have no experience managing servers dramatically overestimate how much time it takes to manage servers. Depending on your team, it can definitely be easier to manage your own hardware than to manage your cloud infrastructure.

lykr0n 7y ago
Yep. A lot of issues come from legacy products and configurations. You can do a greenfield infrastructure with OpenStack, VMware, or oVirt/RHEV and it's a pleasure to work with.
wooly_bully 7y ago
In my experience, it's not the time required but that a lot of development teams don't have a sysadmin or ops skillset.
- sergiotapia 7y ago
  This is accurate in my experience. Also keeping things up and being there in case shit hits the fan is a full-time job. You can't write features and also manage servers equally well. Unless you have no life I guess.
  - wpietri 7y ago
    Totally. I've had personal colo'd servers for 20 years at this point. But I'm tired of knowing that at any point I might have to wake up and haul my ass down to San Jose to swear at some piece of failing gear. I'm excitedly moving it all into the cloud.
    - mcny 7y ago
      How far are you from you collocation? Is moving that closer an option?
      - wpietri 7y ago
        Colocation in SF proper is very spendy, which is how we ended up in San Jose. But waking up to go anywhere at 3 am to swear at gear is no longer on my list of fun activities. And there's no colo that will move the gear along with me when I go on vacation.
        speleding 7y ago
        > there's no colo that will move the gear along with me when I go on vacation This is actually the main one for me. I've managed our own servers for a decade with almost zero downtime, and very little time spent at the colo. But you cannot safely go out of town without having someone else around who is familiar enough with your setup to deal with an outage. So moving stuff to AWS now almost entirely for that reason.
    - krab 7y ago
      There is a middle ground. There are smaller companies where you can rent a bare metal server, usually with unmetered connection (priced by bandwidth). There is an on-call support 24x7 to replace any failing part when you call them. They can build custom servers and also give long term or bulk discounts. I have a good experience with [1] (a smaller local company) and Hetzner [2], a bigger provider. Compare the prices with cloud. Especially if you need something RAM intensive. [1]: https://www.superhosting.net/dedicated-servers https://www.superhosting.net/dedicated-servers [2]: https://www.hetzner.com/dedicated-rootserver https://www.hetzner.com/dedicated-rootserver
- jcrawfordor 7y ago
  I live in a software engineering world professionally but my background is in traditional "neckbeard" Linux system administration. This ends up making me "DevOps" but honestly a lot of what I've ended up doing in my career is basic sysadmin for organizations that get a remarkably long ways before realizing they need it - things like telephony and video surveillance become really unreasonably expensive when you end up relying on a cloud service because you don't have the skillset to manage them in-house. This is purely my opinion, but I think that 1) there is a strange shortage of IT professionals (people who are not software engineers but instead understand systems) in much of the industry today, and 2) a lot of tech companies, even those that are currently well functioning, might be able to save a lot of money if they hired someone with a conventional IT background. This is a little self-serving of course, but it really does astound me when I see the bills that some companies are paying cloud services to do something that is traditionally done in-house by an IT department. And not everything can readily be outsourced to some "aaS" provider, so on top of that you end up with things like software companies with multi-million budgets running an office network that consists of a consumer WiFi router someone picked up at Fry's - not realizing that they are losing a lot of time to dealing with how poorly that ends up working. I think part of the problem rests in academia - at least in my area a lot of universities seem to have really backed off on IT programs in favor of CS. I went through an undergraduate program that involved project management, decision analysis, and finance courses because these were considered by the college (I would say accurately) critical skills for the IT field. But that program had an incredible two students and was widely considered inferior to the CS program with hundreds. Another part of the problem though seems to rest in industry. The salary differential between "DevOps Engineer" and "IT Analyst" is incredible when in practice they end up doing mostly the same thing in a lot of small orgs. So I end up walking sort of an odd line of "I have a long background in IaC but I also know about conference room equipment." And I'm not saying that everything with a Cisco/Tandberg badge isn't overpriced, but Zoom rooms can end up costing just as much and seem to be less reliable - not surprising for a platform which, by practical necessity of the lack of IT support in many orgs, is built on the Silicon Valley time-tested architecture of "five apple consumer products taped together."
  - chillfox 7y ago
    From my experience, large enterprises sabotage the effectiveness of internal IT with bureaucracy and politics in a misguided attempt to eliminate all possibility of mistakes being made. It's usually done with the "let's pretend it is ITIL" process. Let me give two examples where if I had been the client then I would absolutely have sprinted for the cloud if I could, or at the very least start talking it up as much better. 1) System outage, time to fix 5 hours and 3 minutes. The 5 hours was me sitting in front of my computer with screens open showing the problem and waiting for various managers/decision-makers to fly by and take a look as they were ping-ponging around the office panicking about what would be impacted by the fix. Everything that was going to get impacted was already impacted by the system not working, and I had to explain that to them multiple times. Towards the end of the day, I eventually got the go-ahead to do the 3 minutes of work to fix the system. This system being down had prevented another team from doing any work for the entire afternoon. 2) Two full days of politics and paperwork to get approval to do 30 minutes of work, all while the client was impatiently asking "is it done yet" every few hours.
- pmlnr 7y ago
  Learn it. Just like a new framework. It's really not that hard.
foobiekr 7y ago
There has also been an incredible propaganda campaign by the cloud providers to make this seem incredibly hard.
- dazzawazza 7y ago
  Agreed, it's just become part of the group think of our times.
- doublerabbit 7y ago
  The only hard part is finding a provider who won't screw you over. But then again this is the same as if you are to buy a car. You buy rack space, they give you a network cable and IP details. You configure the OS with that IP and your online.
user5994461 7y ago
Counterpoint: The OP is saying they spend 1 day a month managing servers. It's ridiculously little. The servers might as well be unmanaged. That's not enough time to keep OS or software up to date, or to monitor hardware usage, or to replace failing hardware and deal with spare parts.
- dahfizz 7y ago
  This is just not true. A properly set up system requires very little manual work. Updated, monitoring, and alerting are all automated. I help manage ~50 on prem servers plus NAS, switches, etc. Over the past three years, I have had to replace one hard drive and one power supply. And neither of those brought anything down because of redundancy. We get alerts when something goes wrong, but that is a rare occurrence. This is my original point: servers won't just explode the minute you look away. It is not a full time job to "manage" them, unless you are at a real datacenter scale where a 1/1000 event happens twice a day. You set them up and then they continue to do what they do for a very long time.
upcode 7y ago
Do you have any resources you could link that go through buying and setting up servers?
pickle-wizard 7y ago
We have about 170 servers in a colo about 30 minutes from my home. I can go months without going to the colo. Usually when I go out there it is to replace a disk. We have a bunch of hardware coming up on end of life. I looked at moving us to cloud. The new hardware will cost about what 3 years of AWS would cost. Considering that the hardware I am replacing is over 6 years old, we are better off sticking with on-prem.

lykr0n 7y ago

Yep. A lot of issues come from legacy products and configurations. You can do a greenfield infrastructure with OpenStack, VMware, or oVirt/RHEV and it's a pleasure to work with.

wooly_bully 7y ago

In my experience, it's not the time required but that a lot of development teams don't have a sysadmin or ops skillset.

sergiotapia 7y ago
This is accurate in my experience. Also keeping things up and being there in case shit hits the fan is a full-time job. You can't write features and also manage servers equally well. Unless you have no life I guess.
- wpietri 7y ago
  Totally. I've had personal colo'd servers for 20 years at this point. But I'm tired of knowing that at any point I might have to wake up and haul my ass down to San Jose to swear at some piece of failing gear. I'm excitedly moving it all into the cloud.
  - mcny 7y ago
    How far are you from you collocation? Is moving that closer an option?
    - wpietri 7y ago
      Colocation in SF proper is very spendy, which is how we ended up in San Jose. But waking up to go anywhere at 3 am to swear at gear is no longer on my list of fun activities. And there's no colo that will move the gear along with me when I go on vacation.
      - speleding 7y ago
        > there's no colo that will move the gear along with me when I go on vacation This is actually the main one for me. I've managed our own servers for a decade with almost zero downtime, and very little time spent at the colo. But you cannot safely go out of town without having someone else around who is familiar enough with your setup to deal with an outage. So moving stuff to AWS now almost entirely for that reason.
  - krab 7y ago
    There is a middle ground. There are smaller companies where you can rent a bare metal server, usually with unmetered connection (priced by bandwidth). There is an on-call support 24x7 to replace any failing part when you call them. They can build custom servers and also give long term or bulk discounts. I have a good experience with [1] (a smaller local company) and Hetzner [2], a bigger provider. Compare the prices with cloud. Especially if you need something RAM intensive. [1]: https://www.superhosting.net/dedicated-servers https://www.superhosting.net/dedicated-servers [2]: https://www.hetzner.com/dedicated-rootserver https://www.hetzner.com/dedicated-rootserver
jcrawfordor 7y ago
I live in a software engineering world professionally but my background is in traditional "neckbeard" Linux system administration. This ends up making me "DevOps" but honestly a lot of what I've ended up doing in my career is basic sysadmin for organizations that get a remarkably long ways before realizing they need it - things like telephony and video surveillance become really unreasonably expensive when you end up relying on a cloud service because you don't have the skillset to manage them in-house. This is purely my opinion, but I think that 1) there is a strange shortage of IT professionals (people who are not software engineers but instead understand systems) in much of the industry today, and 2) a lot of tech companies, even those that are currently well functioning, might be able to save a lot of money if they hired someone with a conventional IT background. This is a little self-serving of course, but it really does astound me when I see the bills that some companies are paying cloud services to do something that is traditionally done in-house by an IT department. And not everything can readily be outsourced to some "aaS" provider, so on top of that you end up with things like software companies with multi-million budgets running an office network that consists of a consumer WiFi router someone picked up at Fry's - not realizing that they are losing a lot of time to dealing with how poorly that ends up working. I think part of the problem rests in academia - at least in my area a lot of universities seem to have really backed off on IT programs in favor of CS. I went through an undergraduate program that involved project management, decision analysis, and finance courses because these were considered by the college (I would say accurately) critical skills for the IT field. But that program had an incredible two students and was widely considered inferior to the CS program with hundreds. Another part of the problem though seems to rest in industry. The salary differential between "DevOps Engineer" and "IT Analyst" is incredible when in practice they end up doing mostly the same thing in a lot of small orgs. So I end up walking sort of an odd line of "I have a long background in IaC but I also know about conference room equipment." And I'm not saying that everything with a Cisco/Tandberg badge isn't overpriced, but Zoom rooms can end up costing just as much and seem to be less reliable - not surprising for a platform which, by practical necessity of the lack of IT support in many orgs, is built on the Silicon Valley time-tested architecture of "five apple consumer products taped together."
- chillfox 7y ago
  From my experience, large enterprises sabotage the effectiveness of internal IT with bureaucracy and politics in a misguided attempt to eliminate all possibility of mistakes being made. It's usually done with the "let's pretend it is ITIL" process. Let me give two examples where if I had been the client then I would absolutely have sprinted for the cloud if I could, or at the very least start talking it up as much better. 1) System outage, time to fix 5 hours and 3 minutes. The 5 hours was me sitting in front of my computer with screens open showing the problem and waiting for various managers/decision-makers to fly by and take a look as they were ping-ponging around the office panicking about what would be impacted by the fix. Everything that was going to get impacted was already impacted by the system not working, and I had to explain that to them multiple times. Towards the end of the day, I eventually got the go-ahead to do the 3 minutes of work to fix the system. This system being down had prevented another team from doing any work for the entire afternoon. 2) Two full days of politics and paperwork to get approval to do 30 minutes of work, all while the client was impatiently asking "is it done yet" every few hours.
pmlnr 7y ago
Learn it. Just like a new framework. It's really not that hard.

foobiekr 7y ago

There has also been an incredible propaganda campaign by the cloud providers to make this seem incredibly hard.

dazzawazza 7y ago
Agreed, it's just become part of the group think of our times.
doublerabbit 7y ago
The only hard part is finding a provider who won't screw you over. But then again this is the same as if you are to buy a car. You buy rack space, they give you a network cable and IP details. You configure the OS with that IP and your online.

user5994461 7y ago

Counterpoint: The OP is saying they spend 1 day a month managing servers. It's ridiculously little. The servers might as well be unmanaged. That's not enough time to keep OS or software up to date, or to monitor hardware usage, or to replace failing hardware and deal with spare parts.

dahfizz 7y ago
This is just not true. A properly set up system requires very little manual work. Updated, monitoring, and alerting are all automated. I help manage ~50 on prem servers plus NAS, switches, etc. Over the past three years, I have had to replace one hard drive and one power supply. And neither of those brought anything down because of redundancy. We get alerts when something goes wrong, but that is a rare occurrence. This is my original point: servers won't just explode the minute you look away. It is not a full time job to "manage" them, unless you are at a real datacenter scale where a 1/1000 event happens twice a day. You set them up and then they continue to do what they do for a very long time.

upcode 7y ago

Do you have any resources you could link that go through buying and setting up servers?

pickle-wizard 7y ago

We have about 170 servers in a colo about 30 minutes from my home. I can go months without going to the colo. Usually when I go out there it is to replace a disk. We have a bunch of hardware coming up on end of life. I looked at moving us to cloud. The new hardware will cost about what 3 years of AWS would cost. Considering that the hardware I am replacing is over 6 years old, we are better off sticking with on-prem.