5 ms·

One thing we should all have a subtle understanding of: if you are a company selling infrastructure tooling as a service (Vercel, for example). You should proba

by dijit 3y ago

One thing we should all have a subtle understanding of: if you are a company selling infrastructure tooling as a service (Vercel, for example). You should probably try to avoid hyperscalers. The adage is that you should outsource the things that are not your core competence. However if your product is infrastructure for developers then outsourcing that is going to be painful. You are forced to pass down costs from your upstream supplier, and headroom is necessary (somewhat) because your provider can dictate terms essentially on a whim. I feel like this is such an obvious statement, but we do seem to be in the gold-rush of avoiding responsibility so I'm not certain.

lumost 3y ago
Practically, this means that if you are an infrastructure tolling company - then you must reinvent all pieces of the cloud supply chain before building your product. That’s a very expensive proposition. While the paper cost of a few racks in a colo is low - the people time to get where you want to be is high. If you mess up in this evolution - there is a risk that others outcompete you on something that isn’t your core product.
- vidarh 3y ago
  The people time of setup in a colo or managed provider is trivial if you hire someone who actually know what they're doing, and usually lower than a cloud setup. It's stupidly easy to mess up a cloud setup in ways that are hard with a rack. I find this is mostly an issue with a lot of shops simply having nobody with actual ops experience.
  - lumost 3y ago
    It’s not so much the setup time for the hardware/networking. This takes some time, but can reasonably be done in a few months even for complex setups. The problem comes when you then wish to have things like CI/CD, distributed databases etc.
    - vidarh 3y ago
      The vast majority of companies never reach a scale where wiring up the infra needs to take more than days. And CI/CD and distributed databases are equally not an issue of you actually have someone experienced do it. Done this many times.
      - candiodari 3y ago
        1) It's not like cloud databases are problem free. You can have very non-trivial issues, and they especially have nonstandard (and very nontrivial) transaction semantics. I'm not saying this is necessarily a problem, but unless you're on top of it. You need someone with real database administration experience. You especially CANNOT "START TRANSACTION; SELECT * FROM products; SELECT * FROM five_other_tables; UPDATE sales ...; COMMIT TRANSACTION" (so all product prices and ... are included in the transaction "and nothing goes wrong"). It does not scale, it does not scale on postgres, it does not scale on Aurora, it does not scale on Spanner, it does not scale anywhere. It will scale on MySQL and MariaDB because it will just ignore your SELECT statements, which will cause a disaster at some point. 2) The big problem with cloud databases is the same as with normal databases: let bad developers just do what they want, and you'll have 40 queries per pageload, each returning 20-30 Mb of data. You NEED to optimize as soon as you hit even medium scale. And, sorry, "just slap a cache in front of it" is very likely to make the problem worse. Cloud Hyperscalers salivate seeing this, have simply said: "don't pay an annoying database/system admin to tell you to FIX IT, simply pay us 10x what you pay that guy". There's 2 kinds of people. There's people who look at the $10 pageload as both a technical embarassement and a potential business disaster. And there's business people who build hyperscalers, dine the CTO and hold a competition for a $100 pageload, while ordering their next Ferrari. 3) There's nothing automatic about automatic sharding. And if you "just shard UUID", your query performance will be limited. It will not scale. It just won't, you need to THINK about WHAT you shard, and HOW. No exceptions. 4) you'd be surprised how far ansible + k8s + postgres operator goes (for both CI/CD and distributed databases)
        lumost 3y ago
        Once in my career, I did a high 7 figure migration from cloud to colo. Our startups lead investor, was invested in a few firms that aimed to make a cloud experience on-prem. So we enjoyed strong board support, and had a strong team. On paper, the migration saved 10-20x its TCO over 3 years. We weren’t flying by night, and used high quality colo facilities near the cloud. Everything went smooth enough at first, the hardware was late by 12 weeks - and there were some minor hiccups in the build. Time to turn-off of the original infra was maybe 2 months longer than planned. All in all 3 months of net delay where we were running both stacks and the financial clock was ticking. We even had industry press and tech talks on the success of the whole effort. Now here is where problems actually started, 1 year post-build - folks wanted SSDs. While I fought to get SSDs in the initial build, I was overruled by a consultant for the main db vendor we had at the time. It’s tough to ask for 500k+ in SSDs with 1/4 the storage volume of spinners if everyone else thinks the drives are a waste. 2 years in, the main cloud vendor dropped prices by north of 50% for the hardware we had bought. To make matters worse, the service team wasn’t using the hardware we had to full effect - usage was around 25%. By year four contracts weren’t renewed and the workload was migrated to cloud again. In total, we had probably saved 10-25% vs. aggressive negotiations with the cloud vendor. We could probably have beaten the Colo price if we had aggressively tuned the workload on the cloud. However there were political reasons not to do this. If you add in opportunity cost and the use of VC funds… we were probably negative ROI. Given this backdrop, I never worked on colos again (this all went down 10+ years ago).

by dijit 3y ago

lumost 3y ago

Practically, this means that if you are an infrastructure tolling company - then you must reinvent all pieces of the cloud supply chain before building your product. That’s a very expensive proposition. While the paper cost of a few racks in a colo is low - the people time to get where you want to be is high. If you mess up in this evolution - there is a risk that others outcompete you on something that isn’t your core product.

vidarh 3y ago
The people time of setup in a colo or managed provider is trivial if you hire someone who actually know what they're doing, and usually lower than a cloud setup. It's stupidly easy to mess up a cloud setup in ways that are hard with a rack. I find this is mostly an issue with a lot of shops simply having nobody with actual ops experience.
- lumost 3y ago
  It’s not so much the setup time for the hardware/networking. This takes some time, but can reasonably be done in a few months even for complex setups. The problem comes when you then wish to have things like CI/CD, distributed databases etc.
  - vidarh 3y ago
    The vast majority of companies never reach a scale where wiring up the infra needs to take more than days. And CI/CD and distributed databases are equally not an issue of you actually have someone experienced do it. Done this many times.
    - candiodari 3y ago
      1) It's not like cloud databases are problem free. You can have very non-trivial issues, and they especially have nonstandard (and very nontrivial) transaction semantics. I'm not saying this is necessarily a problem, but unless you're on top of it. You need someone with real database administration experience. You especially CANNOT "START TRANSACTION; SELECT * FROM products; SELECT * FROM five_other_tables; UPDATE sales ...; COMMIT TRANSACTION" (so all product prices and ... are included in the transaction "and nothing goes wrong"). It does not scale, it does not scale on postgres, it does not scale on Aurora, it does not scale on Spanner, it does not scale anywhere. It will scale on MySQL and MariaDB because it will just ignore your SELECT statements, which will cause a disaster at some point. 2) The big problem with cloud databases is the same as with normal databases: let bad developers just do what they want, and you'll have 40 queries per pageload, each returning 20-30 Mb of data. You NEED to optimize as soon as you hit even medium scale. And, sorry, "just slap a cache in front of it" is very likely to make the problem worse. Cloud Hyperscalers salivate seeing this, have simply said: "don't pay an annoying database/system admin to tell you to FIX IT, simply pay us 10x what you pay that guy". There's 2 kinds of people. There's people who look at the $10 pageload as both a technical embarassement and a potential business disaster. And there's business people who build hyperscalers, dine the CTO and hold a competition for a $100 pageload, while ordering their next Ferrari. 3) There's nothing automatic about automatic sharding. And if you "just shard UUID", your query performance will be limited. It will not scale. It just won't, you need to THINK about WHAT you shard, and HOW. No exceptions. 4) you'd be surprised how far ansible + k8s + postgres operator goes (for both CI/CD and distributed databases)
      - lumost 3y ago
        Once in my career, I did a high 7 figure migration from cloud to colo. Our startups lead investor, was invested in a few firms that aimed to make a cloud experience on-prem. So we enjoyed strong board support, and had a strong team. On paper, the migration saved 10-20x its TCO over 3 years. We weren’t flying by night, and used high quality colo facilities near the cloud. Everything went smooth enough at first, the hardware was late by 12 weeks - and there were some minor hiccups in the build. Time to turn-off of the original infra was maybe 2 months longer than planned. All in all 3 months of net delay where we were running both stacks and the financial clock was ticking. We even had industry press and tech talks on the success of the whole effort. Now here is where problems actually started, 1 year post-build - folks wanted SSDs. While I fought to get SSDs in the initial build, I was overruled by a consultant for the main db vendor we had at the time. It’s tough to ask for 500k+ in SSDs with 1/4 the storage volume of spinners if everyone else thinks the drives are a waste. 2 years in, the main cloud vendor dropped prices by north of 50% for the hardware we had bought. To make matters worse, the service team wasn’t using the hardware we had to full effect - usage was around 25%. By year four contracts weren’t renewed and the workload was migrated to cloud again. In total, we had probably saved 10-25% vs. aggressive negotiations with the cloud vendor. We could probably have beaten the Colo price if we had aggressively tuned the workload on the cloud. However there were political reasons not to do this. If you add in opportunity cost and the use of VC funds… we were probably negative ROI. Given this backdrop, I never worked on colos again (this all went down 10+ years ago).