5 ms·

Funny, I managed to run a globally-dispersed on-prem system with over 300 individual physical machines (well, blades in racks) with one other guy no problem for

Funny, I managed to run a globally-dispersed on-prem system with over 300 individual physical machines (well, blades in racks) with one other guy no problem for many years. I was dealing with four different hosting providers doing remote hands and never had any issues whatsoever. About the most complicated tooling we had was Anaconda. I think I lost three hard drives in that time and all three were hotswapped with zero downtime under warranty. No other failures. And that's not an outlier - I've been doing this 25+ years and the hardware failure rate has been pretty consistently low.

user5994461 8y ago
Well, you're either incredibly lucky or you have zero monitoring and failures have no impact. I couldn't imagine having to manage 300 physical servers in a team of two. The hardware, OS and networking alone are more than a full time job with 24/7 oncall expectations. In fact that's why I don't do this anymore. Pretty sure my current company had more failed hard drives this month than you all these years. I remember a shipment of servers once with more dead motherboards than that.
- nasalgoat 8y ago
  I bought all brand-new Dell blade servers so that's probably the main reason it was fine. IP-based kickstart boot, anaconda provisioning, it only took one person to monitor a server's setup. The point is that managing on-prem equipment is the easiest its ever been. When I started the ratio of admin to box was 1:50, now it's easily 1:300.
  - deleted 8y ago
    [deleted]
eastern 8y ago
Similar experience here roughly, albeit mine is at a smaller scale. The key phrase in your post is 'doing this 25+ years'. What you're providing is the /confidence/ that you can do it. Most outfits nowadays probably have no one who has that.
- speedplane 8y ago
  I think there's a sliding scale with cloud services. At the ultra-small scale (1 person company), it's arguably cheaper to run from a single server you bought. Mid-sized companies (~2-1000 people) probably benefit the most from cloud computing because they need fewer dedicated salaried employees to manage them. Beyond ~1000 employees however, you're going to need that expertise regardless of whether you use cloud or on-prem. And the markup of those thousands of cloud machines will start to become significant. You'll need to do a cost/benefit analysis and I can see it going in either direction.
speedplane 8y ago
The cloud's cost savings are not in your 300 individual machines. In fact, I'd bet that your 300 machines are much cheaper than the equivalent on AWS. Rather, the cost savings come from the salaries of you and the "one other guy" you mentioned. If cloud hardware can bring your staff of 2 down to 1, then the cost savings are huge. More-so if they can bring it down to 0 and have devs manage the hosted infrastructure. Each of your salaries is likely near or higher than the cost of running all those machines.
- nasalgoat 8y ago
  In my experience with cloud - pushing 8 years now - it's a total fallacy that you can just run without admins. Every time I've come in to clean up a giant mess left by "devops" who were actually just devs who knew nothing about administration of servers I've found massive amounts of waste, duplication, zero optimizations and poorly operating systems that even a intermediate admin would have helped to mitigate. One place, just doing a bit of tuning and moving a few services on-prem resulted in a $200K/year savings. Developers and Administrators have different priorities and I can count on one hand the number of true "DevOps" people I know - the rest are either decent admins and poor devs or decent devs but poor admins. The idea is nice but the reality isn't quite what its made out to be.
  - user5994461 8y ago
    It's a fallacy to save money on personnel. The time saved by not managing low level hardware will be assigned to other tasks. The savings are actually done on instance costs and flexibility. Physical instances have too much over provisioning and zero flexibility. You can easily be paying for an order of magnitude above the actually used capacity. (This can be helped with VmWare virtualization instead of bare metal but the amount of new companies using VmWare is shrinking and licenses are expensive.) Whereas with the cloud, you can half any instance CPU/memory to save half the money. There is even build in monitoring to analyze usage. There is little waste.
    - speedplane 8y ago

user5994461 8y ago

Well, you're either incredibly lucky or you have zero monitoring and failures have no impact. I couldn't imagine having to manage 300 physical servers in a team of two. The hardware, OS and networking alone are more than a full time job with 24/7 oncall expectations. In fact that's why I don't do this anymore. Pretty sure my current company had more failed hard drives this month than you all these years. I remember a shipment of servers once with more dead motherboards than that.

nasalgoat 8y ago
I bought all brand-new Dell blade servers so that's probably the main reason it was fine. IP-based kickstart boot, anaconda provisioning, it only took one person to monitor a server's setup. The point is that managing on-prem equipment is the easiest its ever been. When I started the ratio of admin to box was 1:50, now it's easily 1:300.
- deleted 8y ago
  [deleted]

eastern 8y ago

Similar experience here roughly, albeit mine is at a smaller scale. The key phrase in your post is 'doing this 25+ years'. What you're providing is the /confidence/ that you can do it. Most outfits nowadays probably have no one who has that.

speedplane 8y ago
I think there's a sliding scale with cloud services. At the ultra-small scale (1 person company), it's arguably cheaper to run from a single server you bought. Mid-sized companies (~2-1000 people) probably benefit the most from cloud computing because they need fewer dedicated salaried employees to manage them. Beyond ~1000 employees however, you're going to need that expertise regardless of whether you use cloud or on-prem. And the markup of those thousands of cloud machines will start to become significant. You'll need to do a cost/benefit analysis and I can see it going in either direction.

speedplane 8y ago

The cloud's cost savings are not in your 300 individual machines. In fact, I'd bet that your 300 machines are much cheaper than the equivalent on AWS. Rather, the cost savings come from the salaries of you and the "one other guy" you mentioned. If cloud hardware can bring your staff of 2 down to 1, then the cost savings are huge. More-so if they can bring it down to 0 and have devs manage the hosted infrastructure. Each of your salaries is likely near or higher than the cost of running all those machines.

nasalgoat 8y ago
In my experience with cloud - pushing 8 years now - it's a total fallacy that you can just run without admins. Every time I've come in to clean up a giant mess left by "devops" who were actually just devs who knew nothing about administration of servers I've found massive amounts of waste, duplication, zero optimizations and poorly operating systems that even a intermediate admin would have helped to mitigate. One place, just doing a bit of tuning and moving a few services on-prem resulted in a $200K/year savings. Developers and Administrators have different priorities and I can count on one hand the number of true "DevOps" people I know - the rest are either decent admins and poor devs or decent devs but poor admins. The idea is nice but the reality isn't quite what its made out to be.
- user5994461 8y ago
  It's a fallacy to save money on personnel. The time saved by not managing low level hardware will be assigned to other tasks. The savings are actually done on instance costs and flexibility. Physical instances have too much over provisioning and zero flexibility. You can easily be paying for an order of magnitude above the actually used capacity. (This can be helped with VmWare virtualization instead of bare metal but the amount of new companies using VmWare is shrinking and licenses are expensive.) Whereas with the cloud, you can half any instance CPU/memory to save half the money. There is even build in monitoring to analyze usage. There is little waste.
  - speedplane 8y ago