4 ms·

I've done large colo deployments. When you're provisioning your own hardware it's cloud-init, ansible, K8s, whatever. Nothing even knows you're in colos operate

I've done large colo deployments. When you're provisioning your own hardware it's cloud-init, ansible, K8s, whatever. Nothing even knows you're in colos operated by different providers in different regions. Nothing knows or cares you're using five different upstream internet providers (whether your own - complicated or IP blends from the colo). Have a falling out or big issue? No reputable colo provider is going to hold your hardware hostage (it's the physical world and legal and reputational issues heat up quickly). They'll be happy to ship it out (get rid of you) to another provider, who will be even happier to re-rack. Meanwhile if your architecture is even remotely well designed (load balancer, multiple facilities) you're never down and it's not even that much of a "fire drill". With the capabilities of modern hardware this is often a single 1U redundant HW machine. All of this, by the way, is a fraction of the cost of cloud solution (a $25k bonkers 1U is ~$700 a month with a three year lease and 1U hosting + bandwidth + power is ~$200 a month). So for $2k a month (and it's actually predictable) you're up in two facilities and have capabilities an order of magnitude greater than a big cloud at equivalent spend. Companies screaming on twitter and profusely apologizing to their customers due to the most recent "name a cloud" outage demonstrate that multi-cloud (and even multi-region) is extremely difficult if not impossible to do in practice.

dman 4y ago
Could I contact you to pick your brains about this? I spend much more than $2k for dev environments at my startup and a setup like this would work beautifully for us.
- kkielhofner 4y ago
  Sure! Contact info in profile.
sofixa 4y ago
> All of this, by the way, is a fraction of the cost of cloud solution (a $25k bonkers 1U is ~$700 a month with a three year lease and 1U hosting + bandwidth + power is ~$200 a month). So for $2k a month (and it's actually predictable) you're up in two facilities and have capabilities an order of magnitude greater than a big cloud at equivalent spend. You mean you have raw power an order of magnitude greater than a big cloud at equivalent spend. Capabilities are all DIY - if you want a database, object storage, replicated storage between your two locations, message queue, etc. etc. you have to set it all (and think of the backups, patching, monitoring, etc.) on your own. That's a non-insignificant costs that will easily dwarf that $2k. Source: I've done this at an MSP/MHP, handling a decent chunk of the infra side.
- gspencley 4y ago
  Yeah your mileage will vary greatly depending on your requirements. A couple of years ago I owned a very high-traffic, media-oriented website. Bandwidth costs were my highest cost, not compute or storage. I explored moving to something cloud-based and the cost increase just on bandwidth per GB would have been enough to put me out of business. Yes this was including a paid Cloudflare / CDN account sitting in front of my servers. I considered a "hybrid" model where I would keep bare metal servers for distributing media but move compute and persistence to "the cloud" but just couldn't really justify it on cost alone. The replication and auto-scaling would have been nice, and were the major selling points, but wasn't a high enough ROI for me.
- kkielhofner 4y ago
  When you turn $20k/mo of cloud spend into $2k/mo that’s $216k/yr - $648k over three years. Postgres, minio, rabbit, whatever absolutely does not even come close for initial implementation and ongoing support. If you spend more than a tenth of that you’re doing it wrong. Source: CTO who’s done all these things.

dman 4y ago

Could I contact you to pick your brains about this? I spend much more than $2k for dev environments at my startup and a setup like this would work beautifully for us.

kkielhofner 4y ago
Sure! Contact info in profile.

sofixa 4y ago

> All of this, by the way, is a fraction of the cost of cloud solution (a $25k bonkers 1U is ~$700 a month with a three year lease and 1U hosting + bandwidth + power is ~$200 a month). So for $2k a month (and it's actually predictable) you're up in two facilities and have capabilities an order of magnitude greater than a big cloud at equivalent spend. You mean you have raw power an order of magnitude greater than a big cloud at equivalent spend. Capabilities are all DIY - if you want a database, object storage, replicated storage between your two locations, message queue, etc. etc. you have to set it all (and think of the backups, patching, monitoring, etc.) on your own. That's a non-insignificant costs that will easily dwarf that $2k. Source: I've done this at an MSP/MHP, handling a decent chunk of the infra side.

gspencley 4y ago
Yeah your mileage will vary greatly depending on your requirements. A couple of years ago I owned a very high-traffic, media-oriented website. Bandwidth costs were my highest cost, not compute or storage. I explored moving to something cloud-based and the cost increase just on bandwidth per GB would have been enough to put me out of business. Yes this was including a paid Cloudflare / CDN account sitting in front of my servers. I considered a "hybrid" model where I would keep bare metal servers for distributing media but move compute and persistence to "the cloud" but just couldn't really justify it on cost alone. The replication and auto-scaling would have been nice, and were the major selling points, but wasn't a high enough ROI for me.
kkielhofner 4y ago
When you turn $20k/mo of cloud spend into $2k/mo that’s $216k/yr - $648k over three years. Postgres, minio, rabbit, whatever absolutely does not even come close for initial implementation and ongoing support. If you spend more than a tenth of that you’re doing it wrong. Source: CTO who’s done all these things.