3 ms·

Where you using cloud scale style purchasing in house or were you on enterprise servers with enterprise switches with enterprise storage? The cloud is good for

Where you using cloud scale style purchasing in house or were you on enterprise servers with enterprise switches with enterprise storage? The cloud is good for many users, especially if they migrate to cloud native system design, but as a telco you would probably have facilities and connectivity which helps out a lot. Companies like Mirantis, choosing a technology completely inappropriate for distributed systems (puppet) put a bad taste in the mouths of many people. I implemented OpenStack at one previous employer to just convince them that they could run VMs, intending it to allow for a cloud migration in the future. As they ran a lot of long lived large instances it was trivial to make it cheaper to run in our own datacenter. Well until I moved roles and the IT team tried to implement it with expensive enterprise gear and in an attempt to save money used FEXs despite the fact I had documented they wouldn't work for our traffic patterns. Same thing during the .com crash. I remember our cage was next to one of the old Hotmail cages with their motherboards on boards. We were installing dozens and dozens of Netras and Yahoo was down the hall with a cage full of DEC gear...we went under because we couldn't right-size(cost) our costs. A lot of the companies who save a lot in cloud migrations were the same, having decked out enterprise servers, SANs, and network gear that was wasted in a private cloud context. Enterprise _ is often a euphemism for we are fiercely defending very expensive CYA strategy irrespective of value to the company or material risk.

e12e 3y ago
> in an attempt to save money used FEXs despite the fact I had documented they wouldn't work for our traffic patterns. What is a FEX? Feline Expedited eXchange?
- wmf 3y ago
  Cisco Fabric EXtender. Like a TOR switch but dumber.
doctor_eval 3y ago
We provided SaaS billing services to telcos. We were very successful in our market, but not very big. Just a couple of racks of gear. Our production workload was pretty homogenous, and we were super cheeky - we’d use previous generation servers to keep capex down. We didn’t even use VMs, just containers; docker-swarm was good enough (barely). Our bottleneck was always iops, so we’d have a few decked-out machines to run our redundant databases. It worked fine, but I have subsequently enjoyed working with k8s a lot more. We did use enterprise gear, but previous gen stuff is so much cheaper than current gen. So our perf per watt was not great, and we’d often make the decision to upgrade a rack when we hit our power limits, since we rarely had other constraints. As mentioned elsewhere, we did use AWS spot instances for fractional loads like build and test. It’s not that we didn’t use cloud, it’s that we used it when it made sense. All of that said, I do suspect the equation has changed - not with AWS, but with Vultr. I’ve deployed some complex production systems there (Nomad, Kafka+ZK, PG) and the costs are much closer to in house. They have also avoided the complexity of binding all their different services together. They also now provide K8s and PG out of the box, charging only for the cost of the VMs - as opposed to the wild complexity of AWS billing. So maybe I’m coming around.

e12e 3y ago

> in an attempt to save money used FEXs despite the fact I had documented they wouldn't work for our traffic patterns. What is a FEX? Feline Expedited eXchange?

wmf 3y ago
Cisco Fabric EXtender. Like a TOR switch but dumber.

doctor_eval 3y ago

We provided SaaS billing services to telcos. We were very successful in our market, but not very big. Just a couple of racks of gear. Our production workload was pretty homogenous, and we were super cheeky - we’d use previous generation servers to keep capex down. We didn’t even use VMs, just containers; docker-swarm was good enough (barely). Our bottleneck was always iops, so we’d have a few decked-out machines to run our redundant databases. It worked fine, but I have subsequently enjoyed working with k8s a lot more. We did use enterprise gear, but previous gen stuff is so much cheaper than current gen. So our perf per watt was not great, and we’d often make the decision to upgrade a rack when we hit our power limits, since we rarely had other constraints. As mentioned elsewhere, we did use AWS spot instances for fractional loads like build and test. It’s not that we didn’t use cloud, it’s that we used it when it made sense. All of that said, I do suspect the equation has changed - not with AWS, but with Vultr. I’ve deployed some complex production systems there (Nomad, Kafka+ZK, PG) and the costs are much closer to in house. They have also avoided the complexity of binding all their different services together. They also now provide K8s and PG out of the box, charging only for the cost of the VMs - as opposed to the wild complexity of AWS billing. So maybe I’m coming around.