5 ms·

1) It's not like cloud databases are problem free. You can have very non-trivial issues, and they especially have nonstandard (and very nontrivial) transaction

1) It's not like cloud databases are problem free. You can have very non-trivial issues, and they especially have nonstandard (and very nontrivial) transaction semantics. I'm not saying this is necessarily a problem, but unless you're on top of it. You need someone with real database administration experience. You especially CANNOT "START TRANSACTION; SELECT * FROM products; SELECT * FROM five_other_tables; UPDATE sales ...; COMMIT TRANSACTION" (so all product prices and ... are included in the transaction "and nothing goes wrong"). It does not scale, it does not scale on postgres, it does not scale on Aurora, it does not scale on Spanner, it does not scale anywhere. It will scale on MySQL and MariaDB because it will just ignore your SELECT statements, which will cause a disaster at some point. 2) The big problem with cloud databases is the same as with normal databases: let bad developers just do what they want, and you'll have 40 queries per pageload, each returning 20-30 Mb of data. You NEED to optimize as soon as you hit even medium scale. And, sorry, "just slap a cache in front of it" is very likely to make the problem worse. Cloud Hyperscalers salivate seeing this, have simply said: "don't pay an annoying database/system admin to tell you to FIX IT, simply pay us 10x what you pay that guy". There's 2 kinds of people. There's people who look at the $10 pageload as both a technical embarassement and a potential business disaster. And there's business people who build hyperscalers, dine the CTO and hold a competition for a $100 pageload, while ordering their next Ferrari. 3) There's nothing automatic about automatic sharding. And if you "just shard UUID", your query performance will be limited. It will not scale. It just won't, you need to THINK about WHAT you shard, and HOW. No exceptions. 4) you'd be surprised how far ansible + k8s + postgres operator goes (for both CI/CD and distributed databases)

lumost 3y ago
Once in my career, I did a high 7 figure migration from cloud to colo. Our startups lead investor, was invested in a few firms that aimed to make a cloud experience on-prem. So we enjoyed strong board support, and had a strong team. On paper, the migration saved 10-20x its TCO over 3 years. We weren’t flying by night, and used high quality colo facilities near the cloud. Everything went smooth enough at first, the hardware was late by 12 weeks - and there were some minor hiccups in the build. Time to turn-off of the original infra was maybe 2 months longer than planned. All in all 3 months of net delay where we were running both stacks and the financial clock was ticking. We even had industry press and tech talks on the success of the whole effort. Now here is where problems actually started, 1 year post-build - folks wanted SSDs. While I fought to get SSDs in the initial build, I was overruled by a consultant for the main db vendor we had at the time. It’s tough to ask for 500k+ in SSDs with 1/4 the storage volume of spinners if everyone else thinks the drives are a waste. 2 years in, the main cloud vendor dropped prices by north of 50% for the hardware we had bought. To make matters worse, the service team wasn’t using the hardware we had to full effect - usage was around 25%. By year four contracts weren’t renewed and the workload was migrated to cloud again. In total, we had probably saved 10-25% vs. aggressive negotiations with the cloud vendor. We could probably have beaten the Colo price if we had aggressively tuned the workload on the cloud. However there were political reasons not to do this. If you add in opportunity cost and the use of VC funds… we were probably negative ROI. Given this backdrop, I never worked on colos again (this all went down 10+ years ago).
- vidarh 3y ago
  So you had a badly planned transition, did it in a single big switch (why?), overprovisioned when already having cloud experience instead makes you perfectly placed for under provisioning, and it sounds like you paid up front for no good reason? Sounds like lots of valuable lessons there, but assuming from this that cloud was better for you seems flawed. 1. Don't do migrations at once. In a large deployment you likely have multiple locations anyway, so do one at a time, and/or rack by rack. 2. Verify usage levels, and take advantage of your ability to scale into cloud instances to ensure you only provision tiny headroom on your non-cloud environment. Doing so makes the cost gap to a cloud environment much bigger. 3. The advantage is being able to tune servers to your workload. That you had to fight for SSDs is already a big red flag - if you don't have a culture of testing and picking based on what actually work, yeah, you may actually be best off in the cloud because you'll overpay by a massive factor no matter what you choose and in a cloud you can at least rapidly rectify the repeated major mistakes that will be made. 4. Don't pay upfront unless you're at a scale where it brings you discounts at a return that makes it worth it. Rent or lease to own. For smaller environments, rent managed servers. The cloud provider dropping prices 2 years in shouldn't matter because by year 3 you should have paid off the hardware at an average cost well below anyway, and start cycling out hardware as your load requires and as/when hardware fails and/or it's not cost effective to have it take up space in your racks any more. Depending on your setup, that might be soon, or you might find yourself still running 6+ year old hardware you paid off 3 years ago for many tasks. If you can't beat cloud on TCO, you're doing something very wrong. I use cloud for lots of things that aren't cost sensitive, but there are extremely few scenarios where it is the cost effective option.
  - lumost 3y ago
    Hardware beyond 3 years starts to be inefficient due to the space and power cost. paid off servers aren't free :) Otherwise, we did mortgage the servers in the backend - buying upfront gave us around a 50% discount on top of the steep discounts you get from buying direct from an integrator vs. Dell et. al. On the rent-to-own topic, the people time cost of the migration was budgeted at around 10-20% of total TCO if I recall correctly, but we were in the realm of "we will have to hire people to maintain this project at 24/7 uptime". As the people time was relatively fixed to manage a sane on-call rotation - shrinking the infra footprint would have simply increased the people time portion of the TCO. If the footprint had shrunk by 75% - the maintenance cost would have become problematic. Due to the higher intensity work of build outs vs. maintenance, the people time factor would have risen had we started in more experimental amounts. As it happened, tech salaries also rose ~2x over the 3-5 years that Colo existed. On the workload front, as happens with many large organizations - there are teams who have the attitude of "don't touch my stuff". At the time that the buy was initiated, there were several estimates indicating that we would need 4x the hardware we were buying in 5 years time. Whenever someone says that they are beating cloud TCO, I'd suggest you do the following math. - Sum up all costs related to the on-prem hardware, include smart-hands, power, Racks/network gear, power adaptors. - Calculate a depreciation schedule for the existing hardware targeted to 3 years. - Add a 10% cost of capital to depreciate future savings - Amortize all "non-server" hardware costs onto your principal bottleneck (be it CPU/Network/Memory/Storage) - Project how your savings compare against cloud costs under different assumptions of future cloud discounts. Both negotiated and public. - Project how your savings compare under different utilization assumptions. - Add a bus factor into the investment, what happens if your team leaves/hardware gets smashed/Workload becomes more efficient/Other crises occurs Next, and more controversially - Add in the people cost of everyone involved in the maintenance of your on-prem infrastructure. - Project what happens under varying assumptions for how much of a raise they will ask for. What happens if tech compensation rises by another 2x over the next 3 years? - Explore what happens if on cloud you needed X% fewer engineers. Depending on what you are doing X could be 0%, or even a negative percentage - but for many shops some fraction of existing work can be automated or avoided. It was this math that made me turn away from non-cloud offerings. I still use cloud alternatives for some personal projects - but I don't bill myself in that setting.
vidarh 3y ago
I wrote an orchestrator from scratch in the pre-K8S days, and even factoring in the opportunity cost of that colo and eventually Hetzner was far cheaper than cloud, to the point where - and we costed this out yearly - the company wouldn't have survived a year if we had to pay AWS level prices for infrastructure. Cloud is great for many things, but not cutting cost.

lumost 3y ago

Once in my career, I did a high 7 figure migration from cloud to colo. Our startups lead investor, was invested in a few firms that aimed to make a cloud experience on-prem. So we enjoyed strong board support, and had a strong team. On paper, the migration saved 10-20x its TCO over 3 years. We weren’t flying by night, and used high quality colo facilities near the cloud. Everything went smooth enough at first, the hardware was late by 12 weeks - and there were some minor hiccups in the build. Time to turn-off of the original infra was maybe 2 months longer than planned. All in all 3 months of net delay where we were running both stacks and the financial clock was ticking. We even had industry press and tech talks on the success of the whole effort. Now here is where problems actually started, 1 year post-build - folks wanted SSDs. While I fought to get SSDs in the initial build, I was overruled by a consultant for the main db vendor we had at the time. It’s tough to ask for 500k+ in SSDs with 1/4 the storage volume of spinners if everyone else thinks the drives are a waste. 2 years in, the main cloud vendor dropped prices by north of 50% for the hardware we had bought. To make matters worse, the service team wasn’t using the hardware we had to full effect - usage was around 25%. By year four contracts weren’t renewed and the workload was migrated to cloud again. In total, we had probably saved 10-25% vs. aggressive negotiations with the cloud vendor. We could probably have beaten the Colo price if we had aggressively tuned the workload on the cloud. However there were political reasons not to do this. If you add in opportunity cost and the use of VC funds… we were probably negative ROI. Given this backdrop, I never worked on colos again (this all went down 10+ years ago).

vidarh 3y ago
So you had a badly planned transition, did it in a single big switch (why?), overprovisioned when already having cloud experience instead makes you perfectly placed for under provisioning, and it sounds like you paid up front for no good reason? Sounds like lots of valuable lessons there, but assuming from this that cloud was better for you seems flawed. 1. Don't do migrations at once. In a large deployment you likely have multiple locations anyway, so do one at a time, and/or rack by rack. 2. Verify usage levels, and take advantage of your ability to scale into cloud instances to ensure you only provision tiny headroom on your non-cloud environment. Doing so makes the cost gap to a cloud environment much bigger. 3. The advantage is being able to tune servers to your workload. That you had to fight for SSDs is already a big red flag - if you don't have a culture of testing and picking based on what actually work, yeah, you may actually be best off in the cloud because you'll overpay by a massive factor no matter what you choose and in a cloud you can at least rapidly rectify the repeated major mistakes that will be made. 4. Don't pay upfront unless you're at a scale where it brings you discounts at a return that makes it worth it. Rent or lease to own. For smaller environments, rent managed servers. The cloud provider dropping prices 2 years in shouldn't matter because by year 3 you should have paid off the hardware at an average cost well below anyway, and start cycling out hardware as your load requires and as/when hardware fails and/or it's not cost effective to have it take up space in your racks any more. Depending on your setup, that might be soon, or you might find yourself still running 6+ year old hardware you paid off 3 years ago for many tasks. If you can't beat cloud on TCO, you're doing something very wrong. I use cloud for lots of things that aren't cost sensitive, but there are extremely few scenarios where it is the cost effective option.
- lumost 3y ago
  Hardware beyond 3 years starts to be inefficient due to the space and power cost. paid off servers aren't free :) Otherwise, we did mortgage the servers in the backend - buying upfront gave us around a 50% discount on top of the steep discounts you get from buying direct from an integrator vs. Dell et. al. On the rent-to-own topic, the people time cost of the migration was budgeted at around 10-20% of total TCO if I recall correctly, but we were in the realm of "we will have to hire people to maintain this project at 24/7 uptime". As the people time was relatively fixed to manage a sane on-call rotation - shrinking the infra footprint would have simply increased the people time portion of the TCO. If the footprint had shrunk by 75% - the maintenance cost would have become problematic. Due to the higher intensity work of build outs vs. maintenance, the people time factor would have risen had we started in more experimental amounts. As it happened, tech salaries also rose ~2x over the 3-5 years that Colo existed. On the workload front, as happens with many large organizations - there are teams who have the attitude of "don't touch my stuff". At the time that the buy was initiated, there were several estimates indicating that we would need 4x the hardware we were buying in 5 years time. Whenever someone says that they are beating cloud TCO, I'd suggest you do the following math. - Sum up all costs related to the on-prem hardware, include smart-hands, power, Racks/network gear, power adaptors. - Calculate a depreciation schedule for the existing hardware targeted to 3 years. - Add a 10% cost of capital to depreciate future savings - Amortize all "non-server" hardware costs onto your principal bottleneck (be it CPU/Network/Memory/Storage) - Project how your savings compare against cloud costs under different assumptions of future cloud discounts. Both negotiated and public. - Project how your savings compare under different utilization assumptions. - Add a bus factor into the investment, what happens if your team leaves/hardware gets smashed/Workload becomes more efficient/Other crises occurs Next, and more controversially - Add in the people cost of everyone involved in the maintenance of your on-prem infrastructure. - Project what happens under varying assumptions for how much of a raise they will ask for. What happens if tech compensation rises by another 2x over the next 3 years? - Explore what happens if on cloud you needed X% fewer engineers. Depending on what you are doing X could be 0%, or even a negative percentage - but for many shops some fraction of existing work can be automated or avoided. It was this math that made me turn away from non-cloud offerings. I still use cloud alternatives for some personal projects - but I don't bill myself in that setting.

vidarh 3y ago

I wrote an orchestrator from scratch in the pre-K8S days, and even factoring in the opportunity cost of that colo and eventually Hetzner was far cheaper than cloud, to the point where - and we costed this out yearly - the company wouldn't have survived a year if we had to pay AWS level prices for infrastructure. Cloud is great for many things, but not cutting cost.