5 ms·

Same here - CTO of medium sized company. Our IT infra costs are 1/10th the cost of cloud, simply because I happen to be comfortable having on-premise machine a

Same here - CTO of medium sized company. Our IT infra costs are 1/10th the cost of cloud, simply because I happen to be comfortable having on-premise machine and working on them (sometime myself). We have two dozen servers in two locations. It's more time to setup, but maintenance is actually quite low.

dkdk8283 7y ago
> maintenance is actually quite low. I do hope you regularly patch and reboot your systems.
- biztos 7y ago
  I hope so too, but it's not like a fleet of fine-tuned EC2 instances running CentOS 6.5 is going to be patched and rebooted regularly without a similar amount of sysadmin attention. (Not picking on CentOS, just using it as a placeholder for $REQUIRED_OS_BECAUSE_OF_CUSTOM_SOFTWARE.)
  - mr_toad 7y ago
    Automatic unattended updates are totally possible (I haven’t tried it in CentOS but it works fine on Ubuntu). Even kernel updates can now be done without a reboot. And with the proper setup even reboots can be automated.
    - mrweasel 7y ago
      But that's not limited to cloud, you can do that for on-premise stuff.
      - twic 7y ago
        There's a cognitive failure i see a lot where people incorrectly conflate two different new hotnesses like this. Cloud? Automatically updating operating systems. On-prem? Must be stuck on an old version. Microservices? You can release to production in minutes. Monolith? Must take you weeks to get a release out. Go? Simple self-contained deployables. Java? Must need a JDK and Tomcat installed on the server. It always seem to come up when the leading term is something dubious (cloud, microservices, Go), but the following term is something unequivocally good. Or maybe that's just when i notice it.
    - Conan_Kudo 7y ago
      This is totally possible on CentOS. Up to CentOS 7, yum-cron lets you do this. With CentOS 8, this has been replaced with dnf-automatic, which offers a lot more flexibility on how to configure automatic updates.
    - m01 7y ago
      How do you do kernel updates without a reboot? I've looked several times and found different technologies over time (kexec, (Oracle's) ksplice, kGraft, kpatch, livepatch). They do appear to have some use-cases, e.g. delaying the need for a reboot by being able to install a critical vulnerability fix/workaround so that the reboot can be done at a more convenient time. Because many of the patch mechanisms are function-based, they don't appear to solve the general problem in such a way that reboots can be avoided all together for arbitrary large kernel changes. From my reading of the solutions none are at the level of unattended upgrades using apt/yum-cron or similar in a way that "most" can benefit from them without worrying too much about it (ksplice might do it, but not sure how much you need to pay for it for server use and therefore how accessible it is). kexec helps with skipping the bootloader/BIOS, but I'm not sure if it ends up restaring all the systemd services or going up/down the runlevels, some places suggest it reduces downtime but doesn't eliminate it. I've not experimented with any of these myself yet... so I'd be happy to be proven wrong and in any case learn more! References: - http://jensd.be/651/linux/linux-live-kernel-patching-with-kpatch-on-centos-7 http://jensd.be/651/linux/linux-live-kernel-patching-with-kp... - https://linux-audit.com/livepatch-linux-kernel-updates-without-rebooting/ https://linux-audit.com/livepatch-linux-kernel-updates-witho... - https://wiki.archlinux.org/index.php/Kernel_live_patching https://wiki.archlinux.org/index.php/Kernel_live_patching - https://wiki.archlinux.org/index.php/Kexec https://wiki.archlinux.org/index.php/Kexec EDIT: forgot to mention livepatch
      - heavenlyblue 7y ago
        You don’t need to avoid reboot if you have enough machines
  - close04 7y ago
    The main difference is in admin effort even with the same amount of automation. It doesn't scale linearly because of economy of scales. With reasonable automation a relatively small team can handle a pretty large cloud fleet. For smaller on-prem fleets the scale down might not be ideal though.
wheelerwj 7y ago
24 servers, interesting. That seems like a small enough setup that I would definitely have left it in the cloud. Much more than that and i think it makes sense to start moving to physical servers. But 24 I would have guessed would be cheaper to maintain in EC2. Are you setup with one or two racks in each location and maybe one full time IT/Sys admin at each location?
- deleted 7y ago
  [deleted]
- deleted 7y ago
  [deleted]
- deleted 7y ago
  [deleted]
- t0mas88 7y ago
  24 dedicated dual Xeon boxes can easily be comparable to 100 to 200 m5.2xlarge EC2 instances. So it's relatively small but not tiny, you're going to be paying at least 25k / month for that (inc reserved instance discounts) So if you use a lot of bandwidth on top of it (can be another 25-50k at AWS) this could reach into to levels where it's worth it to hire two devops guys to run your own in combination with some SLA/management agreements with a colo.
  - wheelerwj 7y ago
    I might need to re-look at this. That's much more efficient at an earlier stage than I was expecting. Thanks!
- bsenftner 7y ago
  At my own startup we ran 17 colocated servers, each a high compute resource, at an monthly expense of $600, using purchased hardware that cost $55K. That is versus $96K per month that would be the same infrastructure at AWS.
  - 7y ago
saltking112 7y ago
Is this the all in cost? Including lease on land/building, staff, electricity etc...
- deleted 7y ago
  [deleted]
- koheripbal 7y ago
  Yes. Electricity is cheap and it's housed in an existing office(s). The real cost is IT talent and time. I happen to have homelab experience with HP servers, so (for us) those costs are extremely low. It's honestly not as complex or costly as cloud providers make it sound. Replication offsite is easy via s2s vpn and veeam - as is spinning up new VMs for dev or testing. Building already has good a/c and backup power, and hardware/drive failures are rare.
- woofcat 7y ago
  It's likely colocation, getting to the physically building data-centers size is another size up.
wwn_se 7y ago
On factor for small companies that I have noticed at the places where I have been is that (as a dev consultant). The places that use the cloud often scale up to much. Since the cloud don't limit developers its very easy to just "spinn up a new xx" and not think about the long term costs. Unless you are really small, have variable workloads then the cloud is maybe not for you. Unless the cost is a small part of the total cost of the plattform like not really related to how many users/sales you have.
- kaydub 7y ago
  I've noticed small companies are more worried about the actual bills coming in where as big companies are more worried about the perceived lack of speed/agility in their development. We could spend time making sure we use the smallest aurora instances possible in AWS or we could standardize on a version to support and use the smallest size of that version that's available. We spend a lot on this extra capacity in some places, but it greatly increases our speed in other places. It's all about trade-offs and what your organization values.
- degenerate 7y ago
  I see this a lot in my industry (health). More often than not, my fellow devs are happy throwing extra servers at a problem instead of tackling the problem :(
fragmede 7y ago
Agreed. And it sucks to have to do this, but replacements can be deferred. During a global recession, they may have to be. Servers will eventually degrade (ram goes bad, SSDs fail, spinning rust rusts), but arguably thats better than going poof in the cloud. I'd rather not get put in that position. Pre-pandemic, business was pretty good, but that seems like an unwise assumption as we slide into a recession. Maintenance burden that's avoided has two inflection points rather than one. In a large enough data center, there's enough work simply replacing hard drives that justifies having a team that manages it. From the other end, that's just not true if there's a small number of hard drives (like O(1)). There's a middle area though, where there's enough work for more people but the overhead of there being a team is too expensive.

dkdk8283 7y ago

> maintenance is actually quite low. I do hope you regularly patch and reboot your systems.

biztos 7y ago
I hope so too, but it's not like a fleet of fine-tuned EC2 instances running CentOS 6.5 is going to be patched and rebooted regularly without a similar amount of sysadmin attention. (Not picking on CentOS, just using it as a placeholder for $REQUIRED_OS_BECAUSE_OF_CUSTOM_SOFTWARE.)
- mr_toad 7y ago
  Automatic unattended updates are totally possible (I haven’t tried it in CentOS but it works fine on Ubuntu). Even kernel updates can now be done without a reboot. And with the proper setup even reboots can be automated.
  - mrweasel 7y ago
    But that's not limited to cloud, you can do that for on-premise stuff.
    - twic 7y ago
      There's a cognitive failure i see a lot where people incorrectly conflate two different new hotnesses like this. Cloud? Automatically updating operating systems. On-prem? Must be stuck on an old version. Microservices? You can release to production in minutes. Monolith? Must take you weeks to get a release out. Go? Simple self-contained deployables. Java? Must need a JDK and Tomcat installed on the server. It always seem to come up when the leading term is something dubious (cloud, microservices, Go), but the following term is something unequivocally good. Or maybe that's just when i notice it.
  - Conan_Kudo 7y ago
    This is totally possible on CentOS. Up to CentOS 7, yum-cron lets you do this. With CentOS 8, this has been replaced with dnf-automatic, which offers a lot more flexibility on how to configure automatic updates.
  - m01 7y ago
    How do you do kernel updates without a reboot? I've looked several times and found different technologies over time (kexec, (Oracle's) ksplice, kGraft, kpatch, livepatch). They do appear to have some use-cases, e.g. delaying the need for a reboot by being able to install a critical vulnerability fix/workaround so that the reboot can be done at a more convenient time. Because many of the patch mechanisms are function-based, they don't appear to solve the general problem in such a way that reboots can be avoided all together for arbitrary large kernel changes. From my reading of the solutions none are at the level of unattended upgrades using apt/yum-cron or similar in a way that "most" can benefit from them without worrying too much about it (ksplice might do it, but not sure how much you need to pay for it for server use and therefore how accessible it is). kexec helps with skipping the bootloader/BIOS, but I'm not sure if it ends up restaring all the systemd services or going up/down the runlevels, some places suggest it reduces downtime but doesn't eliminate it. I've not experimented with any of these myself yet... so I'd be happy to be proven wrong and in any case learn more! References: - http://jensd.be/651/linux/linux-live-kernel-patching-with-kpatch-on-centos-7 http://jensd.be/651/linux/linux-live-kernel-patching-with-kp... - https://linux-audit.com/livepatch-linux-kernel-updates-without-rebooting/ https://linux-audit.com/livepatch-linux-kernel-updates-witho... - https://wiki.archlinux.org/index.php/Kernel_live_patching https://wiki.archlinux.org/index.php/Kernel_live_patching - https://wiki.archlinux.org/index.php/Kexec https://wiki.archlinux.org/index.php/Kexec EDIT: forgot to mention livepatch
    - heavenlyblue 7y ago
      You don’t need to avoid reboot if you have enough machines
- close04 7y ago
  The main difference is in admin effort even with the same amount of automation. It doesn't scale linearly because of economy of scales. With reasonable automation a relatively small team can handle a pretty large cloud fleet. For smaller on-prem fleets the scale down might not be ideal though.

wheelerwj 7y ago

24 servers, interesting. That seems like a small enough setup that I would definitely have left it in the cloud. Much more than that and i think it makes sense to start moving to physical servers. But 24 I would have guessed would be cheaper to maintain in EC2. Are you setup with one or two racks in each location and maybe one full time IT/Sys admin at each location?

deleted 7y ago
[deleted]
deleted 7y ago
[deleted]
deleted 7y ago
[deleted]
t0mas88 7y ago
24 dedicated dual Xeon boxes can easily be comparable to 100 to 200 m5.2xlarge EC2 instances. So it's relatively small but not tiny, you're going to be paying at least 25k / month for that (inc reserved instance discounts) So if you use a lot of bandwidth on top of it (can be another 25-50k at AWS) this could reach into to levels where it's worth it to hire two devops guys to run your own in combination with some SLA/management agreements with a colo.
- wheelerwj 7y ago
  I might need to re-look at this. That's much more efficient at an earlier stage than I was expecting. Thanks!
bsenftner 7y ago
At my own startup we ran 17 colocated servers, each a high compute resource, at an monthly expense of $600, using purchased hardware that cost $55K. That is versus $96K per month that would be the same infrastructure at AWS.
- 7y ago

saltking112 7y ago

Is this the all in cost? Including lease on land/building, staff, electricity etc...

deleted 7y ago
[deleted]
koheripbal 7y ago
Yes. Electricity is cheap and it's housed in an existing office(s). The real cost is IT talent and time. I happen to have homelab experience with HP servers, so (for us) those costs are extremely low. It's honestly not as complex or costly as cloud providers make it sound. Replication offsite is easy via s2s vpn and veeam - as is spinning up new VMs for dev or testing. Building already has good a/c and backup power, and hardware/drive failures are rare.
woofcat 7y ago
It's likely colocation, getting to the physically building data-centers size is another size up.

wwn_se 7y ago

On factor for small companies that I have noticed at the places where I have been is that (as a dev consultant). The places that use the cloud often scale up to much. Since the cloud don't limit developers its very easy to just "spinn up a new xx" and not think about the long term costs. Unless you are really small, have variable workloads then the cloud is maybe not for you. Unless the cost is a small part of the total cost of the plattform like not really related to how many users/sales you have.

kaydub 7y ago
I've noticed small companies are more worried about the actual bills coming in where as big companies are more worried about the perceived lack of speed/agility in their development. We could spend time making sure we use the smallest aurora instances possible in AWS or we could standardize on a version to support and use the smallest size of that version that's available. We spend a lot on this extra capacity in some places, but it greatly increases our speed in other places. It's all about trade-offs and what your organization values.
degenerate 7y ago
I see this a lot in my industry (health). More often than not, my fellow devs are happy throwing extra servers at a problem instead of tackling the problem :(

fragmede 7y ago

Agreed. And it sucks to have to do this, but replacements can be deferred. During a global recession, they may have to be. Servers will eventually degrade (ram goes bad, SSDs fail, spinning rust rusts), but arguably thats better than going poof in the cloud. I'd rather not get put in that position. Pre-pandemic, business was pretty good, but that seems like an unwise assumption as we slide into a recession. Maintenance burden that's avoided has two inflection points rather than one. In a large enough data center, there's enough work simply replacing hard drives that justifies having a team that manages it. From the other end, that's just not true if there's a small number of hard drives (like O(1)). There's a middle area though, where there's enough work for more people but the overhead of there being a team is too expensive.