5 ms·

There are basically three tiers of startup infrastructure needs, dependent business model. Your decision process should be driven by which one you are operating

by calcsam 10y ago

There are basically three tiers of startup infrastructure needs, dependent business model. Your decision process should be driven by which one you are operating in. (1) Consumer ad-supported. Pinterest, Instagram, Buzzfeed, etc. Your CPM is going to be pretty low, so you probably want to run your own infra -- it's all gonna come down to the margins. Dropbox notably transitioned to running their own infra after years on AWS. (2) Freemium software with heavy data ingestion needs, eg enterprise messaging or CRM. Slack, Streak, etc. You have pretty high value per customer, but you still have a ton of data streaming in to your system all the time. Probably use a public cloud provider, but monitor your bill somewhat carefully lest it get out of control. (3) Typical B2B workflow SaaS, or very high CPM consumer site. Airbnb, Zenefits, Gusto, etc. You store a relatively low amount of data, on the order of megabytes per customer if not less. Use public cloud infra and make it widely available. Eliminate "how much will this increase our AWS bill" from discussions about eg event sourcing, proposed ML experiments, etc.

skuhn 10y ago
Agreed, there are different solutions that make sense for different businesses. The smaller your margins, the more important it can be to run your own stuff. Or maybe you just want to cut out a layer of someone else's profit from your own revenue to give you more flexibility in your business. If you can get costs to $5/unit at the same margin when everyone else in your industry is $10/unit, that can be a big motivator. It also depends on how your service is positioned to grow. If you have any sort of reliable, foreseeable growth then you can save money by building infrastructure yourself. You are paying for Amazon's overhead to allow you to spin up any instance anywhere at any time (and there's no guarantee it will really be there) -- even if you're buying hardware 6 months ahead of when you actually need it, as long as you're growing and not missing the mark too badly you can do a better job on cost. Most importantly: are you planning for success? If you sign a giant customer, will it help you grow your margins by getting you to better economies of scale with your vendors and your expenses? Or will it crush you because now your burn rate just doubled? There are economies of scale available with AWS, but they aren't great and you have to be pretty huge to get anything meaningful. Two of your examples stuck out to me: Dropbox was never entirely run in AWS. That complicated things operationally to a degree, but ultimately made it a lot easier to move pieces of the service in or out of AWS as it made financial and technical sense. Pinterest was built in AWS. Since they waited so long to start seriously considering alternatives, they're in a tough spot. Migrating out will require a substantial capital commitment (not to mention engineering and product commitments), and they'll have to pay for both environments during what is liable to be a lengthy cutover period. You have more resources to leverage, but the scale of the problem just gets harder over time.
- xorgar831 10y ago
  The problem with onsite data centers is that it's rare for the total cost to be tracked effectively, they often don't include salaries for example. Additionally to say that building something outside of the cloud ensures you're cloud agnostic is too simplistic. There are many more shops that would like to move to the cloud to reduce their costs, but can't due to on-site lock-in. They then run into much more expensive problems around expanding datacenters, network or power limitations, or staffing to maintain physical machines which have real world MTBFs you have to do something about when they fail. I think the issue isn't as much lock-in as much as apps being designed poorly, reliant on a specific feature that's not necessary, but then means their app will only work in one cloud as a result. However in the cases where there is no way around that feature, then you're getting a huge benefit of an instant solution so you can focus on your product rather than infrastructure. And if that cloud does provide a single killer feature you need, then just run that feature there.
  - skuhn 10y ago
    I agree that there is no one right answer for all cases. Companies have definitely implemented their own datacenters poorly and been trapped as a result. You can avoid that fate, but you have to hire right and plan right. In my experience, the difference in personnel between a service built in AWS versus one built in datacenters isn't wildly different. If you have a five person team for the former, you might need a six or seven person team for the latter. You aren't dealing with hardware and vendor problems in AWS necessarily, but it's also not as simple as 'just run your app and forget the rest' (at least once you start to scale up). Again though, this is if you're smart about how you build the datacenter. There are a lot of tricks of the trade and a bunch of gotchas that aren't necessarily apparent until you've experienced them for yourself. Hiring experience here is the key. While you don't necessarily want to limit yourself to the lowest common denominator of cloud providers or your own hardware, any time you make use of something unique you need to make a decision. Is it worth getting locked in? What will you do if that solution goes poof in the night? To use a real recent example: what do you do if AWS runs out of X1 instances and you really really need some more right now to do your work? It sucks to get into a situation where you have no control over the outcome and even money can't fix it, either in your own datacenter or in AWS.
reitzensteinm 10y ago
At my day job, we receive a real money for processing each transaction, which adds maybe 1kb of information to the database. This makes the scaling story laughably easy. By the time we're maxing out the biggest database you can buy, it's IPO time. For most of my life I worked in games, which is almost the exact opposite problem. Tiny CPMs, insane traffic. Multiple providers of cheap bandwidth have gone bankrupt and left me high and dry; but the savings were worth it, and I continued to chase those deals. It was the right call. Engineers love to design architecture that (they believe/hope/pray) will scale to Uber sizes. But if you're not having a conversation with the business as to whether that's a sensible goal, it's negligence bordering on fraud. You aren't being paid to needlessly teach yourself cool new technology to solve imaginary problems. Which is all to say: ignore parent's advice at your peril.
- davedx 10y ago
  > it's negligence bordering on fraud That's the first time I've heard designing scalable software called fraudulent. I'm not sure it means what you think it does. ;)
  - adamzochowski 10y ago
    Parent spoke about 'sensible goal' that is needed by business. If engineer delivers something that is not sensible but overengineered, then maybe it could be called fraudulent. Business needs wheelbarrow, receives a tesla instead.

by calcsam 10y ago

skuhn 10y ago

Agreed, there are different solutions that make sense for different businesses. The smaller your margins, the more important it can be to run your own stuff. Or maybe you just want to cut out a layer of someone else's profit from your own revenue to give you more flexibility in your business. If you can get costs to $5/unit at the same margin when everyone else in your industry is $10/unit, that can be a big motivator. It also depends on how your service is positioned to grow. If you have any sort of reliable, foreseeable growth then you can save money by building infrastructure yourself. You are paying for Amazon's overhead to allow you to spin up any instance anywhere at any time (and there's no guarantee it will really be there) -- even if you're buying hardware 6 months ahead of when you actually need it, as long as you're growing and not missing the mark too badly you can do a better job on cost. Most importantly: are you planning for success? If you sign a giant customer, will it help you grow your margins by getting you to better economies of scale with your vendors and your expenses? Or will it crush you because now your burn rate just doubled? There are economies of scale available with AWS, but they aren't great and you have to be pretty huge to get anything meaningful. Two of your examples stuck out to me: Dropbox was never entirely run in AWS. That complicated things operationally to a degree, but ultimately made it a lot easier to move pieces of the service in or out of AWS as it made financial and technical sense. Pinterest was built in AWS. Since they waited so long to start seriously considering alternatives, they're in a tough spot. Migrating out will require a substantial capital commitment (not to mention engineering and product commitments), and they'll have to pay for both environments during what is liable to be a lengthy cutover period. You have more resources to leverage, but the scale of the problem just gets harder over time.

xorgar831 10y ago
The problem with onsite data centers is that it's rare for the total cost to be tracked effectively, they often don't include salaries for example. Additionally to say that building something outside of the cloud ensures you're cloud agnostic is too simplistic. There are many more shops that would like to move to the cloud to reduce their costs, but can't due to on-site lock-in. They then run into much more expensive problems around expanding datacenters, network or power limitations, or staffing to maintain physical machines which have real world MTBFs you have to do something about when they fail. I think the issue isn't as much lock-in as much as apps being designed poorly, reliant on a specific feature that's not necessary, but then means their app will only work in one cloud as a result. However in the cases where there is no way around that feature, then you're getting a huge benefit of an instant solution so you can focus on your product rather than infrastructure. And if that cloud does provide a single killer feature you need, then just run that feature there.
- skuhn 10y ago
  I agree that there is no one right answer for all cases. Companies have definitely implemented their own datacenters poorly and been trapped as a result. You can avoid that fate, but you have to hire right and plan right. In my experience, the difference in personnel between a service built in AWS versus one built in datacenters isn't wildly different. If you have a five person team for the former, you might need a six or seven person team for the latter. You aren't dealing with hardware and vendor problems in AWS necessarily, but it's also not as simple as 'just run your app and forget the rest' (at least once you start to scale up). Again though, this is if you're smart about how you build the datacenter. There are a lot of tricks of the trade and a bunch of gotchas that aren't necessarily apparent until you've experienced them for yourself. Hiring experience here is the key. While you don't necessarily want to limit yourself to the lowest common denominator of cloud providers or your own hardware, any time you make use of something unique you need to make a decision. Is it worth getting locked in? What will you do if that solution goes poof in the night? To use a real recent example: what do you do if AWS runs out of X1 instances and you really really need some more right now to do your work? It sucks to get into a situation where you have no control over the outcome and even money can't fix it, either in your own datacenter or in AWS.

reitzensteinm 10y ago

At my day job, we receive a real money for processing each transaction, which adds maybe 1kb of information to the database. This makes the scaling story laughably easy. By the time we're maxing out the biggest database you can buy, it's IPO time. For most of my life I worked in games, which is almost the exact opposite problem. Tiny CPMs, insane traffic. Multiple providers of cheap bandwidth have gone bankrupt and left me high and dry; but the savings were worth it, and I continued to chase those deals. It was the right call. Engineers love to design architecture that (they believe/hope/pray) will scale to Uber sizes. But if you're not having a conversation with the business as to whether that's a sensible goal, it's negligence bordering on fraud. You aren't being paid to needlessly teach yourself cool new technology to solve imaginary problems. Which is all to say: ignore parent's advice at your peril.

davedx 10y ago
> it's negligence bordering on fraud That's the first time I've heard designing scalable software called fraudulent. I'm not sure it means what you think it does. ;)
- adamzochowski 10y ago
  Parent spoke about 'sensible goal' that is needed by business. If engineer delivers something that is not sensible but overengineered, then maybe it could be called fraudulent. Business needs wheelbarrow, receives a tesla instead.