4 ms·

I sort of tend to agree with you, but the counter-point is that there is always going to be some critical piece of the infrastructure that you are dependent upo

I sort of tend to agree with you, but the counter-point is that there is always going to be some critical piece of the infrastructure that you are dependent upon that someone else controls. If you colocate, it's the data centre. If you host on-prem, it's the physical telco infrastructure. There are no shortages of incompetent people and institutions that will screw you. It's typically incompetence, not malice. But you will run into malicious actors as well. The reason I sort of agree with you is that, being someone who is very comfortable with bare metal solutions and came from the 90s mentality of writing portable code, I have started to see "The Cloud" as a return to mainframe programming. There is nothing wrong with that, but you are programming for "a machine" that often leads to code that is exceedingly hard to port. "Cloud Agnosticism" is the industry buzzword for not locking yourself into one vendor, but you need to make that decision on day one and set engineering policies enforcing a cloud services abstraction layer to be maintained. Most startups in the process of scaling are not going to see the immediate business value in incurring that cost. (though, with more horror stories like this one maybe that will change... but I doubt it).

kkielhofner 4y ago
I've done large colo deployments. When you're provisioning your own hardware it's cloud-init, ansible, K8s, whatever. Nothing even knows you're in colos operated by different providers in different regions. Nothing knows or cares you're using five different upstream internet providers (whether your own - complicated or IP blends from the colo). Have a falling out or big issue? No reputable colo provider is going to hold your hardware hostage (it's the physical world and legal and reputational issues heat up quickly). They'll be happy to ship it out (get rid of you) to another provider, who will be even happier to re-rack. Meanwhile if your architecture is even remotely well designed (load balancer, multiple facilities) you're never down and it's not even that much of a "fire drill". With the capabilities of modern hardware this is often a single 1U redundant HW machine. All of this, by the way, is a fraction of the cost of cloud solution (a $25k bonkers 1U is ~$700 a month with a three year lease and 1U hosting + bandwidth + power is ~$200 a month). So for $2k a month (and it's actually predictable) you're up in two facilities and have capabilities an order of magnitude greater than a big cloud at equivalent spend. Companies screaming on twitter and profusely apologizing to their customers due to the most recent "name a cloud" outage demonstrate that multi-cloud (and even multi-region) is extremely difficult if not impossible to do in practice.
- dman 4y ago
  Could I contact you to pick your brains about this? I spend much more than $2k for dev environments at my startup and a setup like this would work beautifully for us.
  - kkielhofner 4y ago
    Sure! Contact info in profile.
- sofixa 4y ago
  > All of this, by the way, is a fraction of the cost of cloud solution (a $25k bonkers 1U is ~$700 a month with a three year lease and 1U hosting + bandwidth + power is ~$200 a month). So for $2k a month (and it's actually predictable) you're up in two facilities and have capabilities an order of magnitude greater than a big cloud at equivalent spend. You mean you have raw power an order of magnitude greater than a big cloud at equivalent spend. Capabilities are all DIY - if you want a database, object storage, replicated storage between your two locations, message queue, etc. etc. you have to set it all (and think of the backups, patching, monitoring, etc.) on your own. That's a non-insignificant costs that will easily dwarf that $2k. Source: I've done this at an MSP/MHP, handling a decent chunk of the infra side.
  - gspencley 4y ago
    Yeah your mileage will vary greatly depending on your requirements. A couple of years ago I owned a very high-traffic, media-oriented website. Bandwidth costs were my highest cost, not compute or storage. I explored moving to something cloud-based and the cost increase just on bandwidth per GB would have been enough to put me out of business. Yes this was including a paid Cloudflare / CDN account sitting in front of my servers. I considered a "hybrid" model where I would keep bare metal servers for distributing media but move compute and persistence to "the cloud" but just couldn't really justify it on cost alone. The replication and auto-scaling would have been nice, and were the major selling points, but wasn't a high enough ROI for me.
  - kkielhofner 4y ago

kkielhofner 4y ago

I've done large colo deployments. When you're provisioning your own hardware it's cloud-init, ansible, K8s, whatever. Nothing even knows you're in colos operated by different providers in different regions. Nothing knows or cares you're using five different upstream internet providers (whether your own - complicated or IP blends from the colo). Have a falling out or big issue? No reputable colo provider is going to hold your hardware hostage (it's the physical world and legal and reputational issues heat up quickly). They'll be happy to ship it out (get rid of you) to another provider, who will be even happier to re-rack. Meanwhile if your architecture is even remotely well designed (load balancer, multiple facilities) you're never down and it's not even that much of a "fire drill". With the capabilities of modern hardware this is often a single 1U redundant HW machine. All of this, by the way, is a fraction of the cost of cloud solution (a $25k bonkers 1U is ~$700 a month with a three year lease and 1U hosting + bandwidth + power is ~$200 a month). So for $2k a month (and it's actually predictable) you're up in two facilities and have capabilities an order of magnitude greater than a big cloud at equivalent spend. Companies screaming on twitter and profusely apologizing to their customers due to the most recent "name a cloud" outage demonstrate that multi-cloud (and even multi-region) is extremely difficult if not impossible to do in practice.

dman 4y ago
Could I contact you to pick your brains about this? I spend much more than $2k for dev environments at my startup and a setup like this would work beautifully for us.
- kkielhofner 4y ago
  Sure! Contact info in profile.
sofixa 4y ago
> All of this, by the way, is a fraction of the cost of cloud solution (a $25k bonkers 1U is ~$700 a month with a three year lease and 1U hosting + bandwidth + power is ~$200 a month). So for $2k a month (and it's actually predictable) you're up in two facilities and have capabilities an order of magnitude greater than a big cloud at equivalent spend. You mean you have raw power an order of magnitude greater than a big cloud at equivalent spend. Capabilities are all DIY - if you want a database, object storage, replicated storage between your two locations, message queue, etc. etc. you have to set it all (and think of the backups, patching, monitoring, etc.) on your own. That's a non-insignificant costs that will easily dwarf that $2k. Source: I've done this at an MSP/MHP, handling a decent chunk of the infra side.
- gspencley 4y ago
  Yeah your mileage will vary greatly depending on your requirements. A couple of years ago I owned a very high-traffic, media-oriented website. Bandwidth costs were my highest cost, not compute or storage. I explored moving to something cloud-based and the cost increase just on bandwidth per GB would have been enough to put me out of business. Yes this was including a paid Cloudflare / CDN account sitting in front of my servers. I considered a "hybrid" model where I would keep bare metal servers for distributing media but move compute and persistence to "the cloud" but just couldn't really justify it on cost alone. The replication and auto-scaling would have been nice, and were the major selling points, but wasn't a high enough ROI for me.
- kkielhofner 4y ago