6 ms·

I often fantasize about moving a bunch of my company's crap off of ECS/EKS and onto a managed colo with just old-school ansible deployments. I've even spec'd o

I often fantasize about moving a bunch of my company's crap off of ECS/EKS and onto a managed colo with just old-school ansible deployments. I've even spec'd out some bare metal from our local colocation facility and the servers they offer are so ridiculously powerful and CHEAP. We could run our entire production system in a half-rack for about 10% of what we pay AWS. But alas, nobody will ever go for it and I'm left to dream.

tiffanyh 4y ago
> "We could run our entire production system in a half-rack for about 10% of what we pay AWS" Performance. You'd also get way faster hardware too.
- glintik 4y ago
  And very wide choice of hardware and tuning abilities. Also network config can speed up overall performance greatly.
  - Nextgrid 4y ago
    Direct-attach storage is a major improvement for things like databases, etc which you pretty much never have in the cloud, at least not in a persistent form.
    - glintik 4y ago
      Don’t agree. Multiple database nodes with enterprise grade SSDs is a way better in many terms.
      - Nextgrid 4y ago
        For conventional relational databases like Postgres, multiple nodes only give you reliability, not performance (ignoring things like read-only replicas which your application explicitly has to choose to query), so horizontal scaling doesn't help there. Enterprise-grade SSDs is what I meant by direct-attach storage - I was comparing against network storage which is what all cloud providers use (your EBS volume is accessed over the network internally, which incurs some latency, ultimately limits IOPS and destroys random access performance which can't be cached or read-ahead by the underlying hypervisor).
        glintik 4y ago
        «is what all cloud providers use» - many cloud providers can offer local ssds, i.e.: https://cloud.google.com/local-ssd https://cloud.google.com/local-ssd “multiple nodes only give you reliability, not performance” - this, actually, is not true. Even mysql can offer ways to tune and gain performance improvements.
        Nextgrid 4y ago
        Those SSDs are ephemeral, so unsuitable for long-term storage. If you wanted to use them you’d need a task that copies the entire persistent volume into the ephemeral SSD at boot, and then the reverse on shutdown (and pray that the machine doesn’t die abruptly as that would mean you lose the SSD without doing that reverse copy operation). Presumably they would also lack the durability guarantees of the normal block storage offered by the provider.
hotpotamus 4y ago
I had to look it up - Ansible's initial release was just over 11 years ago. Calling it "old-school" sure took me by surprise, though I suppose that is a long time in the tech world. To me, old school is doing everything via Bash or even by hand.
- klodolph 4y ago
  To me, old-school is the approach of updating servers so they match the desired configuration, rather than wiping the servers and deploying new versions of your containers.
  - acedTrex 4y ago
    If you manage to run on prem infra with vms and clusters in a true cattle mindset then cloud doesnt offer you as much for sure
    - anonymouse008 4y ago
      I feel like you have a unique perspective to offer here - though at the moment I disagree given the lack of detail. What about the specialized services from AWS provides leverage outside of the “cattle mindset”? In my experience, AWS’s primary value and leverage is based on intermittent burst compute. It’s why you rarely get a hard answer on the Hz of each vCPU and have credits to the overuse or underuse of said instances. The cpu is then used to arbitrage value in the dedicated services like lambda and others - so your mindset and infra needs to match AWS’s incentives to get the same value, correct?
      - Symbiote 4y ago
        The main disadvantage of the colo servers I administrate is the lack of flexibility for periods of hours to days. We use ~50 fairly high performance (RAM, CPU, disc) servers pretty much continuously, so the savings compared to AWS etc are considerable. Every couple of years there's a software upgrade required, and it would be convenient to have another 50 servers to use during the transition, just for a few days. Compared to AWS prices, we can easily justify keeping some old servers around to help with this sort of thing. A managed server company (Hetzner etc) would be somewhere in-between; presumably able to rent us 50 extra machines for the duration, but (last time we checked) still more expensive than managing the servers ourselves.
manv1 4y ago
"We could run our entire production system in a half-rack for about 10% of what we pay AWS" It sounds like you should make a business case. 90% savings is a big deal, unless it's the difference between $1k and $100.
turtlebits 4y ago
Who’s going to be on call for hardware failures? Who is going to build/manage your monitoring/alerting stack? Will you hire a network engineer? Will you have redundant hardware running to failover to? What about disaster recovery, are you going to colo in more than 1 physical data center?
- fidgewidge 4y ago
  None of these are the gotchas cloud people seem to think they are. The answers look like this. Who's going to be on-call: the same people who are on call now. HW failures don't need special on-call people. Who is going to build/manage your monitoring/alerting stack. Same people as now. Your servers are monitored, right? Adding RAID into the mix is hardly difficult. Will you hire a network engineer? No, colos will often offer to manage the network for you up to the switch and at that point it's just a matter of plugging the server in. You don't need any expertise to do that and remote hands can do it for you. Redundant hardware? Only if you need it currently. HW is reliable these days and you'd going to have spare capacity anyway, so this really doesn't change much. Disaster recovery? Take whatever answer you currently have in the cloud for multi-AZ.

tiffanyh 4y ago

> "We could run our entire production system in a half-rack for about 10% of what we pay AWS" Performance. You'd also get way faster hardware too.

glintik 4y ago
And very wide choice of hardware and tuning abilities. Also network config can speed up overall performance greatly.
- Nextgrid 4y ago
  Direct-attach storage is a major improvement for things like databases, etc which you pretty much never have in the cloud, at least not in a persistent form.
  - glintik 4y ago
    Don’t agree. Multiple database nodes with enterprise grade SSDs is a way better in many terms.
    - Nextgrid 4y ago
      For conventional relational databases like Postgres, multiple nodes only give you reliability, not performance (ignoring things like read-only replicas which your application explicitly has to choose to query), so horizontal scaling doesn't help there. Enterprise-grade SSDs is what I meant by direct-attach storage - I was comparing against network storage which is what all cloud providers use (your EBS volume is accessed over the network internally, which incurs some latency, ultimately limits IOPS and destroys random access performance which can't be cached or read-ahead by the underlying hypervisor).
      - glintik 4y ago
        «is what all cloud providers use» - many cloud providers can offer local ssds, i.e.: https://cloud.google.com/local-ssd https://cloud.google.com/local-ssd “multiple nodes only give you reliability, not performance” - this, actually, is not true. Even mysql can offer ways to tune and gain performance improvements.
        Nextgrid 4y ago
        Those SSDs are ephemeral, so unsuitable for long-term storage. If you wanted to use them you’d need a task that copies the entire persistent volume into the ephemeral SSD at boot, and then the reverse on shutdown (and pray that the machine doesn’t die abruptly as that would mean you lose the SSD without doing that reverse copy operation). Presumably they would also lack the durability guarantees of the normal block storage offered by the provider.

hotpotamus 4y ago

I had to look it up - Ansible's initial release was just over 11 years ago. Calling it "old-school" sure took me by surprise, though I suppose that is a long time in the tech world. To me, old school is doing everything via Bash or even by hand.

klodolph 4y ago
To me, old-school is the approach of updating servers so they match the desired configuration, rather than wiping the servers and deploying new versions of your containers.
- acedTrex 4y ago
  If you manage to run on prem infra with vms and clusters in a true cattle mindset then cloud doesnt offer you as much for sure
  - anonymouse008 4y ago
    I feel like you have a unique perspective to offer here - though at the moment I disagree given the lack of detail. What about the specialized services from AWS provides leverage outside of the “cattle mindset”? In my experience, AWS’s primary value and leverage is based on intermittent burst compute. It’s why you rarely get a hard answer on the Hz of each vCPU and have credits to the overuse or underuse of said instances. The cpu is then used to arbitrage value in the dedicated services like lambda and others - so your mindset and infra needs to match AWS’s incentives to get the same value, correct?
    - Symbiote 4y ago
      The main disadvantage of the colo servers I administrate is the lack of flexibility for periods of hours to days. We use ~50 fairly high performance (RAM, CPU, disc) servers pretty much continuously, so the savings compared to AWS etc are considerable. Every couple of years there's a software upgrade required, and it would be convenient to have another 50 servers to use during the transition, just for a few days. Compared to AWS prices, we can easily justify keeping some old servers around to help with this sort of thing. A managed server company (Hetzner etc) would be somewhere in-between; presumably able to rent us 50 extra machines for the duration, but (last time we checked) still more expensive than managing the servers ourselves.

manv1 4y ago

"We could run our entire production system in a half-rack for about 10% of what we pay AWS" It sounds like you should make a business case. 90% savings is a big deal, unless it's the difference between $1k and $100.

turtlebits 4y ago

Who’s going to be on call for hardware failures? Who is going to build/manage your monitoring/alerting stack? Will you hire a network engineer? Will you have redundant hardware running to failover to? What about disaster recovery, are you going to colo in more than 1 physical data center?

fidgewidge 4y ago
None of these are the gotchas cloud people seem to think they are. The answers look like this. Who's going to be on-call: the same people who are on call now. HW failures don't need special on-call people. Who is going to build/manage your monitoring/alerting stack. Same people as now. Your servers are monitored, right? Adding RAID into the mix is hardly difficult. Will you hire a network engineer? No, colos will often offer to manage the network for you up to the switch and at that point it's just a matter of plugging the server in. You don't need any expertise to do that and remote hands can do it for you. Redundant hardware? Only if you need it currently. HW is reliable these days and you'd going to have spare capacity anyway, so this really doesn't change much. Disaster recovery? Take whatever answer you currently have in the cloud for multi-AZ.