6 ms·

Do you actually use AWS or are you just parroting something you've read? Do you understand that with AWS you still have to setup instances, manage security upd

Do you actually use AWS or are you just parroting something you've read? Do you understand that with AWS you still have to setup instances, manage security updates, permission and deal with disaster response (US-East-1 outage?) If you've run anything at any scale for any length of time you'll know that EC2 hosts frequently get rebooted with short notice.

Rantenki 6y ago
I have extensive infrastructure running across 3 separate org accounts for blast radius, with multiple separate VPCs per org, RDS databases across multiple AZs, and a large variety of instances. Everything is built up and torn down via Terraform, and ZERO maintenance tasks are done via the console. Changes to infrastructure configuration are managed via CAB which approves PRs on Github which are automatically deployed via TF cloud. We do monthly disaster simulations where we replace a _production_ environment. I'll agree that AWS paints a very rosy picture of their own services, but there is substance underneath that PR.
- ProAm 6y ago
  We see just as much downtown with AWS with DR/HA/MTTR as with colo. You still have to set it all up and maintain it, its just different setup and maintenance than if you actually have a servers in DCs. Most people still have the learning curve with AWS and usually do it incorrectly, or have to wait for Amazon to admit degradation of a service is occurring. It's really a matter of who within the company wants to know whats behind the curtain. I'd almost always vote for self hosting but its a hard argument internally when everyone thinks the cloud eliminates these issues magically.
  - outworlder 6y ago
    > or have to wait for Amazon to admit degradation of a service is occurring. Please compare apples to apples. If you are comparing to colo, I don't know of any colocation facilities which provide higher level services (say, Kinesis). They might provide some features like CDNs, but nothing very fancy. You are left with a EC2 vs servers racked in some colocation facility comparison. At that point, it's not even fair. I can run a terraform script and spin up a copy of the production environment in minutes. I don't have to create purchase orders and wait for hardware. I don't have to issue tickets. If any machine goes down it can be recovered in minutes, and it will spin up in another hypervisor, maybe even a different datacenter. You don't have to spend time and resources with discussions on how the 'network topology' is going to look like, and then send people off to implement and wire up stuff. You don't have to spend cycles diagnosing issues just to find out that there's a bad transceiver in a NIC somewhere. YOU DONT CARE. AWS cares and, even if they can't find an issue, stop/start, new hardware in minutes. You cannot compare the two in good faith. Sure, it might make sense to host some workloads in a colo, but you are giving up a lot.
    - ProAm 6y ago
      There is a lot there, and some of it is apples to apples and some is definitely apples to oranges. I'm just saying overall you have the same number of headaches, they are just different headaches. Just last week there was significant degradation that took quite some time to be admitted (admitted might be the wrong word but everyone seemed to know there was an issue with AWS before AWS said their was an issue) > I don't have to create purchase orders and wait for hardware. Why would this need to be done if initial implementation was planned and done correctly? Baring catastrophic hardware failures you shouldn't need to do this? Im nitpicking your comment and I think is tangential to my original point (And dont want to get into a planning/design/implementation discussion either). > You don't have to spend time and resources with discussions on how the 'network topology' is going to look like, and then send people off to implement and wire up stuff. This isn't that hard if you still have the expertise in house. Yes it is work and knowhow, but so is AWS networking/wiring, service interop. > AWS cares I think this is arguable and YMMV. Sure it depends on who you are and who you talk to. Im not saying AWS isn't viable tech, and there are 30 ways to skin a cat at the end of the day if you end up with a skinned cat you like you're good to go. I believe most people don't need AWS to accomplish what they are looking for (subjective for sure). I also find it not the savings or experience that is in the marketing materials or sales pitches. This isn't a binary right or wrong decision, all we need is a skinned cat at the end of the day.
      - marcinzm 6y ago
        >Why would this need to be done if initial implementation was planned and done correctly? Anyone who thinks they planned everything correctly is just deluding themselves. Admitting you've fucked up, don't know how you fucked up yet but will one day need to fix the fuckup is very important in engineering.
        ProAm 6y ago
        It's possible but not to think enough through the process that you need to create purchases orders large enough to need approvals? However if you don't have a basic understanding of hardware you'll need you probably shouldn't be working/in charge of this proejct anyhow. If you fked up this bad you're in for a bad time regardless of where you've decided to host.
  - apple4ever 6y ago
    That's the thing about this argument- it gets sold as not having to worry about something (racking etc) except the reality is it's just a different worry.
- strogonoff 6y ago
  Setting up and maintaining Terraform scripts to manage your AWS infrastructure takes time and effort; besides, there’s this nagging feeling that should something happens to Terraform you’d suddenly be left trying to keep track of hundreds of resources (which cost you money each hour, or millisecond in case of Lambda) and their access policies “by hand”. On the other hand, it could be a very powerful combo.
- moltar 6y ago
  What’s CAB?
drchopchop 6y ago
They don't get force-rebooted that often, I have EC2 instances that have been up for over 365 days with no issues. Setting up instances, managing updates, adding IAM perms is orders of magnitude faster/easier than dealing with rack-and-stack data centers. Full downtime isn't that frequent, either. The Kinesis outage on Wednesday didn't affect any of our USE1 functionality, with the exception of maybe Cloudfront propagation.
- mekster 6y ago
  I don't remember them force rebooting for nearly 10 years. A server is easily surviving over 1000 days and it's usually our side that needs to reboot.
  - sokoloff 6y ago
    I agree in general, though many were affected by the 2014 Xen security issue.
Zach_the_Lizard 6y ago
If you're at the scale where a data center going down is a problem and thus you need to run in multiple coasts for disaster recovery, you're going to have to be able to handle the kinds of distributed computing issues that'll crop up in both cases. The advantage of AWS is that in theory you can script your infrastructure. Deploy to multiple zones, add hosts, remove hosts, deploy artifact, etc. Makes all kinds of things very easy. Dedicated SQL / Cassandra / DynamoDB offerings make that a few clicks to get going. This saves people costs, but just so happens to be more expensive in terms of hardware costs. You can replicate all of this in your own DC or whatever. You do need to invest in tooling, hardware, etc. which is worth it at the highest scales. Or worth it if you don't need everything and thus don't need to pay for it. Personally, I think many large organizations can optimize by running with a mix of on-prem and cloud services. Demand may be elastic but not totally elastic, so your base load could be cheaper in house. Storage is cheaper on premise for huge datasets. You can afford enough staff to rack servers, but you aren't dominated by the lack of servers. You buy more EC2 instances to scale your business, letting the product team deliver the product, and deploy on-prem to squeeze out more efficiency behind them. Yes, you probably don't use the full offerings of a cloud provider, but that also means less lock in at a cost of a little lower convenience and more staff.
- TravelPiglet 6y ago
  The cost of being down might be high and happens seldom enough that hiring people to sit idle is really expensive.
- t_sawyer 6y ago
  Idk why this take isn’t more popular. Nothing is stopping you from spinning up ec2s for massive spikes in traffic just because your app is currently on-prem. Nothing is stopping you from scaling first to AWS, realizing oh this is normal now, and then procuring hardware.
busterarm 6y ago
I have several million dollars in infrastructure that I'm responsible for in multiple cloud providers and on our own metal in multiple physical datacenters. In terms of management burden, I will take the cloud infrastructure a thousand times out of ten. Automation work is easily an order of magnitude of less effort in the cloud than on-prem. Labor dollars spent go exponentially further.
unethical_ban 6y ago
Everything you said still has to happen with colo, as well as everything the parent mentioned. "The cloud is just someone else's computer" - I used to say that with contempt, and now I say it with understanding.
TravelPiglet 6y ago
Things got rebooted a lot due to Specter and Meltdown mitigations a few years ago, but rarely now. We’ve had more issues with the collocation data center, like removing one of the good drives in a degraded raid-array and extremely expensive bandwidth.
darth_avocado 6y ago
clearly you've never had to deal with physical servers. A dead mouse once caused an outage (fire) that took an entire dev team days to fix. I'd rather not deal with that in my lifetime.
outworlder 6y ago
> If you've run anything at any scale for any length of time you'll know that EC2 hosts frequently get rebooted with short notice. They do not. We run many thousands of instances on AWS. Sure, given our scale, every week we get a couple of instance retirement emails. Usually they are issued many days in advance(sometimes, weeks). Per year, we may get a handful of instances that are suddenly unresponsive. And we don't care. You know why? Because it's just a matter of issuing stop/start. Done! Server is back up, potentially even in a different datacenter, but it is none the wiser. It looks like a reboot. Even better, add an auto-recovery alert and AWS will do this for you, automatically. If part of an ASG, add health checks. For the most part, we don't even notice when instances go down. Our workloads are engineered to be fault-tolerant. If a meteor destroys one AWS datacenter, it might temporarily take out some instances. So what? New ones will be back very shortly, all the while databases will fail-over, etc. If these were physical instances, someone would have to do the maintenance work, purchase orders, wait for hardware to arrive, and so on and so forth. And, for most "co-location" scenarios, if your datacenter has issues, everything will go down. A single AZ in AWS has multiple datacenters, you might not even be affected if one goes up in flames. But let's say you run a massive pet server farm and none of then can go down for any period of time. You have given names and everything, and you celebrate their birthdays. Cool. Run that on GCP then. They do auto-migration. I've never seen an instance go down. > Do you understand that with AWS you still have to setup instances, manage security updates, permission and deal with disaster response This is true. However, if you are running your own hardware, you have to do that IN ADDITION TO dealing with hardware and datacenter shenanigans, with either a specialized (and expensive) workforce, or a barely capable one that's shoehorned and doing double duty, with zero economies of scale, probably in a single data-center.
- busterarm 6y ago
  I have to echo everything you're saying here. Anyone putting anything in "the cloud" and not designing for failure as a first principle is shooting themselves in the foot. I've encountered a lot of folks trying to do this and building systems that can't tolerate downtime and it starts long, difficult conversations always. But I've never once needed to actually pay attention to those instance retirement emails or GCP auto-migrations. We've built to expect it and we let the provider handle that for us. Time saved for everyone.
manquer 6y ago
EC2 is hardly the main service that people decide over. EKS, ECS, Fargate and Lamba, Aurora and other managed DBs, S3 eliminates a lot of the setup skills and time required to setup something even remotely reliable and on par in features. Sure if you need a reliable VM there are cheaper options perhaps.

Rantenki 6y ago

I have extensive infrastructure running across 3 separate org accounts for blast radius, with multiple separate VPCs per org, RDS databases across multiple AZs, and a large variety of instances. Everything is built up and torn down via Terraform, and ZERO maintenance tasks are done via the console. Changes to infrastructure configuration are managed via CAB which approves PRs on Github which are automatically deployed via TF cloud. We do monthly disaster simulations where we replace a _production_ environment. I'll agree that AWS paints a very rosy picture of their own services, but there is substance underneath that PR.

ProAm 6y ago
We see just as much downtown with AWS with DR/HA/MTTR as with colo. You still have to set it all up and maintain it, its just different setup and maintenance than if you actually have a servers in DCs. Most people still have the learning curve with AWS and usually do it incorrectly, or have to wait for Amazon to admit degradation of a service is occurring. It's really a matter of who within the company wants to know whats behind the curtain. I'd almost always vote for self hosting but its a hard argument internally when everyone thinks the cloud eliminates these issues magically.
- outworlder 6y ago
  > or have to wait for Amazon to admit degradation of a service is occurring. Please compare apples to apples. If you are comparing to colo, I don't know of any colocation facilities which provide higher level services (say, Kinesis). They might provide some features like CDNs, but nothing very fancy. You are left with a EC2 vs servers racked in some colocation facility comparison. At that point, it's not even fair. I can run a terraform script and spin up a copy of the production environment in minutes. I don't have to create purchase orders and wait for hardware. I don't have to issue tickets. If any machine goes down it can be recovered in minutes, and it will spin up in another hypervisor, maybe even a different datacenter. You don't have to spend time and resources with discussions on how the 'network topology' is going to look like, and then send people off to implement and wire up stuff. You don't have to spend cycles diagnosing issues just to find out that there's a bad transceiver in a NIC somewhere. YOU DONT CARE. AWS cares and, even if they can't find an issue, stop/start, new hardware in minutes. You cannot compare the two in good faith. Sure, it might make sense to host some workloads in a colo, but you are giving up a lot.
  - ProAm 6y ago
    There is a lot there, and some of it is apples to apples and some is definitely apples to oranges. I'm just saying overall you have the same number of headaches, they are just different headaches. Just last week there was significant degradation that took quite some time to be admitted (admitted might be the wrong word but everyone seemed to know there was an issue with AWS before AWS said their was an issue) > I don't have to create purchase orders and wait for hardware. Why would this need to be done if initial implementation was planned and done correctly? Baring catastrophic hardware failures you shouldn't need to do this? Im nitpicking your comment and I think is tangential to my original point (And dont want to get into a planning/design/implementation discussion either). > You don't have to spend time and resources with discussions on how the 'network topology' is going to look like, and then send people off to implement and wire up stuff. This isn't that hard if you still have the expertise in house. Yes it is work and knowhow, but so is AWS networking/wiring, service interop. > AWS cares I think this is arguable and YMMV. Sure it depends on who you are and who you talk to. Im not saying AWS isn't viable tech, and there are 30 ways to skin a cat at the end of the day if you end up with a skinned cat you like you're good to go. I believe most people don't need AWS to accomplish what they are looking for (subjective for sure). I also find it not the savings or experience that is in the marketing materials or sales pitches. This isn't a binary right or wrong decision, all we need is a skinned cat at the end of the day.
    - marcinzm 6y ago
      >Why would this need to be done if initial implementation was planned and done correctly? Anyone who thinks they planned everything correctly is just deluding themselves. Admitting you've fucked up, don't know how you fucked up yet but will one day need to fix the fuckup is very important in engineering.
      - ProAm 6y ago
        It's possible but not to think enough through the process that you need to create purchases orders large enough to need approvals? However if you don't have a basic understanding of hardware you'll need you probably shouldn't be working/in charge of this proejct anyhow. If you fked up this bad you're in for a bad time regardless of where you've decided to host.
- apple4ever 6y ago
  That's the thing about this argument- it gets sold as not having to worry about something (racking etc) except the reality is it's just a different worry.
strogonoff 6y ago
Setting up and maintaining Terraform scripts to manage your AWS infrastructure takes time and effort; besides, there’s this nagging feeling that should something happens to Terraform you’d suddenly be left trying to keep track of hundreds of resources (which cost you money each hour, or millisecond in case of Lambda) and their access policies “by hand”. On the other hand, it could be a very powerful combo.
moltar 6y ago
What’s CAB?

drchopchop 6y ago

They don't get force-rebooted that often, I have EC2 instances that have been up for over 365 days with no issues. Setting up instances, managing updates, adding IAM perms is orders of magnitude faster/easier than dealing with rack-and-stack data centers. Full downtime isn't that frequent, either. The Kinesis outage on Wednesday didn't affect any of our USE1 functionality, with the exception of maybe Cloudfront propagation.

mekster 6y ago
I don't remember them force rebooting for nearly 10 years. A server is easily surviving over 1000 days and it's usually our side that needs to reboot.
- sokoloff 6y ago
  I agree in general, though many were affected by the 2014 Xen security issue.

Zach_the_Lizard 6y ago

If you're at the scale where a data center going down is a problem and thus you need to run in multiple coasts for disaster recovery, you're going to have to be able to handle the kinds of distributed computing issues that'll crop up in both cases. The advantage of AWS is that in theory you can script your infrastructure. Deploy to multiple zones, add hosts, remove hosts, deploy artifact, etc. Makes all kinds of things very easy. Dedicated SQL / Cassandra / DynamoDB offerings make that a few clicks to get going. This saves people costs, but just so happens to be more expensive in terms of hardware costs. You can replicate all of this in your own DC or whatever. You do need to invest in tooling, hardware, etc. which is worth it at the highest scales. Or worth it if you don't need everything and thus don't need to pay for it. Personally, I think many large organizations can optimize by running with a mix of on-prem and cloud services. Demand may be elastic but not totally elastic, so your base load could be cheaper in house. Storage is cheaper on premise for huge datasets. You can afford enough staff to rack servers, but you aren't dominated by the lack of servers. You buy more EC2 instances to scale your business, letting the product team deliver the product, and deploy on-prem to squeeze out more efficiency behind them. Yes, you probably don't use the full offerings of a cloud provider, but that also means less lock in at a cost of a little lower convenience and more staff.

TravelPiglet 6y ago
The cost of being down might be high and happens seldom enough that hiring people to sit idle is really expensive.
t_sawyer 6y ago
Idk why this take isn’t more popular. Nothing is stopping you from spinning up ec2s for massive spikes in traffic just because your app is currently on-prem. Nothing is stopping you from scaling first to AWS, realizing oh this is normal now, and then procuring hardware.

busterarm 6y ago

I have several million dollars in infrastructure that I'm responsible for in multiple cloud providers and on our own metal in multiple physical datacenters. In terms of management burden, I will take the cloud infrastructure a thousand times out of ten. Automation work is easily an order of magnitude of less effort in the cloud than on-prem. Labor dollars spent go exponentially further.

unethical_ban 6y ago

Everything you said still has to happen with colo, as well as everything the parent mentioned. "The cloud is just someone else's computer" - I used to say that with contempt, and now I say it with understanding.

TravelPiglet 6y ago

Things got rebooted a lot due to Specter and Meltdown mitigations a few years ago, but rarely now. We’ve had more issues with the collocation data center, like removing one of the good drives in a degraded raid-array and extremely expensive bandwidth.

darth_avocado 6y ago

clearly you've never had to deal with physical servers. A dead mouse once caused an outage (fire) that took an entire dev team days to fix. I'd rather not deal with that in my lifetime.

outworlder 6y ago

> If you've run anything at any scale for any length of time you'll know that EC2 hosts frequently get rebooted with short notice. They do not. We run many thousands of instances on AWS. Sure, given our scale, every week we get a couple of instance retirement emails. Usually they are issued many days in advance(sometimes, weeks). Per year, we may get a handful of instances that are suddenly unresponsive. And we don't care. You know why? Because it's just a matter of issuing stop/start. Done! Server is back up, potentially even in a different datacenter, but it is none the wiser. It looks like a reboot. Even better, add an auto-recovery alert and AWS will do this for you, automatically. If part of an ASG, add health checks. For the most part, we don't even notice when instances go down. Our workloads are engineered to be fault-tolerant. If a meteor destroys one AWS datacenter, it might temporarily take out some instances. So what? New ones will be back very shortly, all the while databases will fail-over, etc. If these were physical instances, someone would have to do the maintenance work, purchase orders, wait for hardware to arrive, and so on and so forth. And, for most "co-location" scenarios, if your datacenter has issues, everything will go down. A single AZ in AWS has multiple datacenters, you might not even be affected if one goes up in flames. But let's say you run a massive pet server farm and none of then can go down for any period of time. You have given names and everything, and you celebrate their birthdays. Cool. Run that on GCP then. They do auto-migration. I've never seen an instance go down. > Do you understand that with AWS you still have to setup instances, manage security updates, permission and deal with disaster response This is true. However, if you are running your own hardware, you have to do that IN ADDITION TO dealing with hardware and datacenter shenanigans, with either a specialized (and expensive) workforce, or a barely capable one that's shoehorned and doing double duty, with zero economies of scale, probably in a single data-center.

busterarm 6y ago
I have to echo everything you're saying here. Anyone putting anything in "the cloud" and not designing for failure as a first principle is shooting themselves in the foot. I've encountered a lot of folks trying to do this and building systems that can't tolerate downtime and it starts long, difficult conversations always. But I've never once needed to actually pay attention to those instance retirement emails or GCP auto-migrations. We've built to expect it and we let the provider handle that for us. Time saved for everyone.

manquer 6y ago

EC2 is hardly the main service that people decide over. EKS, ECS, Fargate and Lamba, Aurora and other managed DBs, S3 eliminates a lot of the setup skills and time required to setup something even remotely reliable and on par in features. Sure if you need a reliable VM there are cheaper options perhaps.