4 ms·

If I followed this advice, it would hurt my startup badly. We would be investing far more time into system operations than I do now thanks to AWS's automation

If I followed this advice, it would hurt my startup badly. We would be investing far more time into system operations than I do now thanks to AWS's automation of standard stuff. My hosting bill would increase - partly because we can run on t2.micro instances, but that awfully glib advice about overprovisioning is most definitely asking for trouble; and we'd lose curated services like RDS and OpsWorks - which, by the way, are PostgreSQL and Chef, hardly a "ugly vendor lock" but simply well-designed infrastructure services based on standard parts. Oh, and I'd have to spend more money and time on auditors, because bare-metal providers don't have compliance programmes like AWS/Azure/GCP do. And I've haven't even started to think about securing the resulting systems to the same level I get for minimal effort from a major public cloud. You're not a dinosaur, no, but these claims don't match my reality, nor that of many other projects besides. This is compounded in my view because you've claimed specific expertise to promote your opinion. Any manager bringing me such a poorly considered and bombastically argued business case would be sent away with an ear bashing about TCO and opportunity cost.

_Codemonkeyism 9y ago
I think people need to make a conscious tradeoff between ~5x AWS cost compared to rented server vs. OPS costs. Some startups (A) I work with have basically no OPS costs beyond setup, integrating Docker deployements and getting automatic backup working. Most simple technology just works and devs easily can do operations. The largest pain point still is VPN. Machines today are very very fast and load of many startups is very low. This are mostly simple marketplace startups etc. without any rocket science aka "web ui frontend to database". They often have <10 servers and are still over provisioned mostly due to HA requirements. Some startups (B) I work with have high OPS costs due to demanding technology needs for throughput, load peaks, amount of data with innovative technology at their core, aka "rocket science". I have not seen any correlation between AWS usage and A/B types. From my experience with startups the only way to successfully use AWS is deep integration and using lots of services. If you use AWS and do everything on your own people are doing it wrong.
- pizza234 9y ago
  I managed a metal to AWS transition, and 5x definitely doesn't match the costs I experienced (in this specific case, it was around the lines of 1.2x). I don't mean this case to be universal, in particular, I think cloud services force applications to have a particularly good/modular design (which is a cost in itself) - where, with metal, as you wrote, you can relatively cheaply overprovision. I think the analysis you're making overlooks some important characteristics of the infrastructure engineering aspect. Some typical network/infrastructure elements, in particular firewalling, load balancing, and network management don't necessarily belong to the "rocket science" type of application; they are easy to overlook in "type A" services, ending with a "kind-of-HA-but-not-really" infrastructure, which is ok, but it makes the comparison cloud <> metal not really meaningful, as in the cloud, those features are baked in ("almost" for free). I'm very skeptical for example, that the 5x figure includes hardware for the above network equipment and management. To summarize, it's perfectly fine not to have an "advanced" infrastructure but it must be highlighted that such conditions make a direct comparison incorrect.
  - vidarh 9y ago
    I think the "5x" (though my experience is closer to 2x-3x) tends to come in if you already do the good/modular design and are set up to scale into hybrid setups as needed to handle traffic spikes. In those instances you don't overprovision, or overprovision maybe 20%, knowing that extending your virtualized setup into a part on premises or manage hosting setup, part public cloud is fast and seamless. E.g. I had a setup that spanned on-demand instances, rented managed servers, racks in two separate colos and racks on premises. We expanded resources whenever it was cost effective at the time. Generally the colos won out, with on-demand instances handling traffic spikes, and managed servers primarily used for locations we did not have staff. When your infrastructure is designed so that adding a new one of any of those is just a matter of assigning IP space to the new satellite network and deploy the first instances - whatever they're physically on - your utilisation of all the resources can be far higher. E.g. in this setup we have instances where we move containers seamlessly between the UK, New Zealand and Germany currently depending on load, available resources, and which instances need low latency (Germany vs. UK makes a roughly 8ms latency difference despite going over an encrypted VPN connection, so we've even had times where client traffic hits load-balancers in the UK while the web servers were temporarily in Germany because it happened to be cheaper to expand there for a while (and contrary to with AWS, our bandwidth costs in both locations are trivial). If you're comparing to "lets throw a bunch of servers somewhere", then, yes, AWS probably won't be that much more expensive, and presumably that is a big part of why so many people gets caught out by AWS costs once they start scaling up.
vidarh 9y ago
I do contract devops for both physical servers (that includes me occasionally travelling to a data center), managed servers in a full service colo, and AWS, and the overall cost per server consistently ends up higher including all devops time for the AWS instance in my experience. I love it when customers pick AWS (though I usually advice not to, unless they have very specific needs), as my billable hours are way higher for those clients, though it is annoying having to deal with the inevitable "why is my AWS bill so big?" after I'd told them exactly why it'd be expensive in the first place. This is particularly true with bandwidth heavy setups, where AWS charges tens of times more per TB transferred than e.g. Hetzner. Often I even end up getting paid to help them get off AWS again down the line when they realise how expensive it is. Overall the idea that managed servers and even bare metal colocated servers take up so much more operations time is just not what I experience. Even if it did, if you're even moderate successful it'd need to save you a crazy amount of operational time per server to make up for the price differences in the hosting. Further my experience is that you don't need to over-provision much exactly because setting up hybrid setups where you spin up AWS instances (or any other cloud) to handle spikes works well. The trick is to treat managed servers exactly as cloud instances, apart from at most the initial hardware provisioning (though many hosting companies provides APIs for this so you can abstract away that too), so that it doesn't matter where you provision. As for services like RDS etc., they're a very mixed bag. When they work for you, great, though they're expensive, but very often I end up with clients having to move off them because they need some plugin or other that isn't available. Very few of my clients in the end stay on them for very long. Their biggest benefit is to defer the initial setup of a self-managed cluster. That doesn't mean there are no cases where AWS is the right choice - for starters having it there for traffic spikes is great (though we usually end up needing it very rarely), and if you need large batch jobs or other environments you spin up/down frequently, it may be cost effective. But it's extremely rare I see cases where it's cost effective for base load. The typical argument then is that big companies wouldn't have picked them if it wasn't. But big companies don't pay the prices mere mortals pay. I know concrete examples of large negotiated discounts for a couple of larger companies, and they are steep. > and I'd have to spend more money and time on auditors That might be true, but most companies are not in a position where they need their IT infrastructure audited. This might very well be a niche that makes it worth using AWS. > And I've haven't even started to think about securing the resulting systems to the same level I get for minimal effort from a major public cloud. My experience is that getting security right on the public clouds is harder than bare metal. If you take the effort to do it properly, the end results can be good. But a lot of that is simplified a lot in a colo'ed environment by simply physically separating resources into different network segments and the like. For people with very complex requirements, you might even get better results, but I could never agree with "minimal effort" - it's very common to see people badly misconfiguring their IAM setup for example, because it was too hard for them to figure out how to open up just the specific things they needed to open.

_Codemonkeyism 9y ago

I think people need to make a conscious tradeoff between ~5x AWS cost compared to rented server vs. OPS costs. Some startups (A) I work with have basically no OPS costs beyond setup, integrating Docker deployements and getting automatic backup working. Most simple technology just works and devs easily can do operations. The largest pain point still is VPN. Machines today are very very fast and load of many startups is very low. This are mostly simple marketplace startups etc. without any rocket science aka "web ui frontend to database". They often have <10 servers and are still over provisioned mostly due to HA requirements. Some startups (B) I work with have high OPS costs due to demanding technology needs for throughput, load peaks, amount of data with innovative technology at their core, aka "rocket science". I have not seen any correlation between AWS usage and A/B types. From my experience with startups the only way to successfully use AWS is deep integration and using lots of services. If you use AWS and do everything on your own people are doing it wrong.

pizza234 9y ago
I managed a metal to AWS transition, and 5x definitely doesn't match the costs I experienced (in this specific case, it was around the lines of 1.2x). I don't mean this case to be universal, in particular, I think cloud services force applications to have a particularly good/modular design (which is a cost in itself) - where, with metal, as you wrote, you can relatively cheaply overprovision. I think the analysis you're making overlooks some important characteristics of the infrastructure engineering aspect. Some typical network/infrastructure elements, in particular firewalling, load balancing, and network management don't necessarily belong to the "rocket science" type of application; they are easy to overlook in "type A" services, ending with a "kind-of-HA-but-not-really" infrastructure, which is ok, but it makes the comparison cloud <> metal not really meaningful, as in the cloud, those features are baked in ("almost" for free). I'm very skeptical for example, that the 5x figure includes hardware for the above network equipment and management. To summarize, it's perfectly fine not to have an "advanced" infrastructure but it must be highlighted that such conditions make a direct comparison incorrect.
- vidarh 9y ago
  I think the "5x" (though my experience is closer to 2x-3x) tends to come in if you already do the good/modular design and are set up to scale into hybrid setups as needed to handle traffic spikes. In those instances you don't overprovision, or overprovision maybe 20%, knowing that extending your virtualized setup into a part on premises or manage hosting setup, part public cloud is fast and seamless. E.g. I had a setup that spanned on-demand instances, rented managed servers, racks in two separate colos and racks on premises. We expanded resources whenever it was cost effective at the time. Generally the colos won out, with on-demand instances handling traffic spikes, and managed servers primarily used for locations we did not have staff. When your infrastructure is designed so that adding a new one of any of those is just a matter of assigning IP space to the new satellite network and deploy the first instances - whatever they're physically on - your utilisation of all the resources can be far higher. E.g. in this setup we have instances where we move containers seamlessly between the UK, New Zealand and Germany currently depending on load, available resources, and which instances need low latency (Germany vs. UK makes a roughly 8ms latency difference despite going over an encrypted VPN connection, so we've even had times where client traffic hits load-balancers in the UK while the web servers were temporarily in Germany because it happened to be cheaper to expand there for a while (and contrary to with AWS, our bandwidth costs in both locations are trivial). If you're comparing to "lets throw a bunch of servers somewhere", then, yes, AWS probably won't be that much more expensive, and presumably that is a big part of why so many people gets caught out by AWS costs once they start scaling up.

vidarh 9y ago

I do contract devops for both physical servers (that includes me occasionally travelling to a data center), managed servers in a full service colo, and AWS, and the overall cost per server consistently ends up higher including all devops time for the AWS instance in my experience. I love it when customers pick AWS (though I usually advice not to, unless they have very specific needs), as my billable hours are way higher for those clients, though it is annoying having to deal with the inevitable "why is my AWS bill so big?" after I'd told them exactly why it'd be expensive in the first place. This is particularly true with bandwidth heavy setups, where AWS charges tens of times more per TB transferred than e.g. Hetzner. Often I even end up getting paid to help them get off AWS again down the line when they realise how expensive it is. Overall the idea that managed servers and even bare metal colocated servers take up so much more operations time is just not what I experience. Even if it did, if you're even moderate successful it'd need to save you a crazy amount of operational time per server to make up for the price differences in the hosting. Further my experience is that you don't need to over-provision much exactly because setting up hybrid setups where you spin up AWS instances (or any other cloud) to handle spikes works well. The trick is to treat managed servers exactly as cloud instances, apart from at most the initial hardware provisioning (though many hosting companies provides APIs for this so you can abstract away that too), so that it doesn't matter where you provision. As for services like RDS etc., they're a very mixed bag. When they work for you, great, though they're expensive, but very often I end up with clients having to move off them because they need some plugin or other that isn't available. Very few of my clients in the end stay on them for very long. Their biggest benefit is to defer the initial setup of a self-managed cluster. That doesn't mean there are no cases where AWS is the right choice - for starters having it there for traffic spikes is great (though we usually end up needing it very rarely), and if you need large batch jobs or other environments you spin up/down frequently, it may be cost effective. But it's extremely rare I see cases where it's cost effective for base load. The typical argument then is that big companies wouldn't have picked them if it wasn't. But big companies don't pay the prices mere mortals pay. I know concrete examples of large negotiated discounts for a couple of larger companies, and they are steep. > and I'd have to spend more money and time on auditors That might be true, but most companies are not in a position where they need their IT infrastructure audited. This might very well be a niche that makes it worth using AWS. > And I've haven't even started to think about securing the resulting systems to the same level I get for minimal effort from a major public cloud. My experience is that getting security right on the public clouds is harder than bare metal. If you take the effort to do it properly, the end results can be good. But a lot of that is simplified a lot in a colo'ed environment by simply physically separating resources into different network segments and the like. For people with very complex requirements, you might even get better results, but I could never agree with "minimal effort" - it's very common to see people badly misconfiguring their IAM setup for example, because it was too hard for them to figure out how to open up just the specific things they needed to open.