9 ms·

AWS is more often than not a better solution than colo when factoring in the on-site engineers, techs, and operational complexity costs a company will pay to mo

AWS is more often than not a better solution than colo when factoring in the on-site engineers, techs, and operational complexity costs a company will pay to monitor and respond to hardware-related events. One could build out a datacenter management team with on-call engineers, or, they could pay AWS to handle all that, and focus on innovation and products that make their company unique and (hopefully) profitable. AWS makes a lot of sense for companies that wish to inoculate themselves from the hardware layer, and it would probably take a company many magnitudes larger than Heap to realize any real benefits from self-hosting at a colo. This isn't even considering the fact that uptime matters, and you'll need more than 1 colo to really do it right. I say this as someone who built, manages and operates datacenters and colo spaces.

runlevel1 5y ago
I'm not saying it never happens, but I've never seen moving to AWS (or Azure, GCP, etc.) save in people costs at any tech company with a large resource footprint. It just shifted where the time is spent and who had to spend it. The public cloud and managed services work great for the most common use cases, but go outside those and you start having to engineer around limitations. If you have a sizable footprint in any given dimension you're trading one complexity for another.
- bombcar 5y ago
  The "who had to spend it" is huge - companies love paying providers and hate paying people/depreciating costs.
  - markus_zhang 5y ago
    Curiously our company moved from AWS to on-premise a couple of years ago. Something about CAPEX -> OPEX was mentioned back then.
    - bombcar 5y ago
      That's the second part of the loop, when the cost of AWS is high enough that you can show immediate dollar savings by bringing it in-house.
      - markus_zhang 5y ago
        Yeah, that was a bit more than one year before we filed for IPO :)
  - walrus01 5y ago
    > hate paying depreciating costs. One possible solution for this if you want to do it as bare metal you control, is leased equipment (even with $1 buyout at end of term), which can be accounted for differently than purchasing it up front.
nickstinemates 5y ago
Our company is not many magnitudes larger than any company and it is not remotely cost competitive for us to run any of our stack on AWS (heavy data ingest (hundreds of billions of inserts a day,) many disks/'big data' backend, hundreds of customers accessing the data. Even 1 time deals to get us into the door are not cost competitive, let alone long tail economics. Cloud for fast storage and high bandwidth usecases is extremely expensive.
walrus01 5y ago
If you have petabytes of data in AWS you already have a number of on-staff engineers with significant six figure salaries. If the problem is that your group of six-figure salary people only know how to put data into AWS, or other cloud services, and not design/engineer/maintain your own bare metal infrastructure as well, then that would definitely be a limitation. For reference, a few petabytes of data is not actually that many systems these days, if you have something like a bunch of 72-drive supermicros or equivalent with 14-16TB drives in them. Set up properly this can be administered by one FTE (of course with additional staffing/tech resources for when that FTE is on vacation/unavailable, and appropriate training for other persons who might have admin on the setup). my very rough calculation here says that a 36-drive ZFS RAIDZ2 composed of 16TB drives is something like 492TB (447TiB) usable storage capacity. so five such arrays would be 2460TB. compared to the monthly AWS bill for 2.0 to 2.5TB of data you could probably afford to entirely duplicate the whole setup in a twin identical set of hardware at a geographically diverse off-site location.
- amluto 5y ago
  You have to factor in the cost of egress from AWS to your nice colocated drive.
  - walrus01 5y ago
    welcome to the hotel california... Last thing I remember, I was Running for the door I had to find the passage back To the place I was before "Relax, " said the night man, "We are programmed to receive. You can check-out any time you like, But you can never leave! "
    - amluto 5y ago
      And this is why I don’t think AWS will lower egress fees in response to R2. AWS may be more interested in discouraging people from using egress than in capturing the revenue from egress. I predict that, at most, we’ll see a narrowly tailored reduction in egress fees that is designed to be entirely useless for communication between server applications.
_3u10 5y ago
It isn’t. I have the same overhead running my server as I do a VM. What I don’t get is a $3000 bill for $100 worth of server. You can generally buy whatever you are renting from AWS for 1 to 3 months of an AWS bill. The only thing I don’t get from colo is a bunch of other customers thrashing the cache on my CPUs Databases are not web servers there’s no possible way to not run a database on smaller / fewer instances when running at non-peak times. Instant scaling is the only possible advantage AWS could bring. However with the prices they charge it’s simpler and cheaper to just buy/rent your own hardware. Especially if you have to pay egress fees. (bandwidth is really the biggest ripoff)
- midasuni 5y ago
  The argument isn’t about using AWS to run a VM (which can be cheaper that coloing your own kit, depending how many you want and for how long), it’s all the extra stuff. Start an aws load balancer rather than run and maintain your own for example. I don’t like lock-in, but the prevailing view has always been in favour of lock-in, be it IBM mainframes, oracle databases, windows servers etc, and if you swing that way aws has tempting offers. Oh and databases do scale. Say you want to run end of quarter financials that require a lot of processing for a day, you bring up tons of read replicas and away you go
  - _3u10 5y ago
    If you bought hardware with what you pay for AWS RDS you could run your entire DB in RAM. Hell you could probably put the data in memory on a GPU. Also, this is generally why you run financials overnight. If your hardware is serving transactions during the day it can easily run your quarterlies at night. nginx is far easier to maintain than AWS load balancers which is what load balancers their load balancers are. The best part about nginx configs? They are cloud agnostic and will work on everything from a Raspberry Pi to a 128 core EPYC server. I'll tell you something about RDS reliabilty, your monthly maintenance window brings your DB down far more often than a single unreplicated server ever fails. EBS (like the entire thing) has failed more times in the last year on us-east-1 than my colo RAID. The selling point of AWS is that if you pick AWS and it fails you can say, well the richest guy in the world can't figure this stuff out so it must be impossible, when in reality high school kids could make a more reliable system. If you pick AWS you have the unreliability of the base software / hardware of their systems plus whatever the AWS engineers fuck up. At this point it's pretty clear that they can't even keep a SAN working.
saiya-jin 5y ago
That's a nice statement sounding like straight from Amazon sales reps, and it can actually work for some companies, maybe. But for our bank (top 10 globally), the only way to get even equal financially to our own farms is to have aggressive downtime every night that negatively affect productivity of our global teams. Pricing is really not that great if you deal in scale. You wanna push 1 evening a bit late to deliver something valuable for the project? Sorry, no can do. I don't even factor in horribly expensive migration projects that brought actually 0 added business value for the type of apps we use. We still have to keep our Network, Windows and Unix admins, various App support personnel etc., there is plenty of work for them with AWS. Not 1 single IT guy was made redundant. No cost savings, in contrary.
IgorPartola 5y ago
That argument doesn’t hold up. If how could AWS be cheaper than doing it yourself at that scale? Like if you are a tiny company that can’t afford your own DC, your own engineers, etc. then yes AWS is cheaper in absolute costs but not in per-byte costs. But at scale you should be able to hire engineers and build out a DC at which point you aren’t paying the AWS margin, which is how you save money. In other words your assessment would only be true if AWS had a 0% or a negative margin.
- kevincox 5y ago
  There is still an economy of scale. You are right, as you use more resources your economy of scale will increase, but it will never match AWS's (unless you are huge). So the math is if the AWS margin is less than the difference between the two different economies of scale then it makes sense to run your own datacenters (ignoring the opportunity cost of the transition costs). Of course at some point the margin will exceed that difference, but depending on what type of infrastructure you need it can be at a very high point.
  - IgorPartola 5y ago
    Your last point is the important part: depending on the type of infrastructure you need you might be able to save money. If you want a cheap place to dump your files, B2 is cheaper than S3 and raw storage hardware pays for itself in about a year. If you need a sophisticated CDN then yeah you’ll need to be huge before it pays for itself. I would consider ditching S3 at the point where I can hire two full time engineers to worry about my storage layer.
deleted 5y ago
[deleted]
gsliepen 5y ago
For a while I did maintain a storage cluster that had close to 0.5 PB of data, and had a capacity of up to 3 PB (if you filled all the slots with the largest disks you can buy). You want to ensure you have a lot of redundancy and spare capacity if you are managing it yourself. Luckily, the hardware is relatively cheap. It's the manpower that costs a lot. Still, I think it was only 0.1 FTE to manage this storage cluster, including the network, file systems, user access, swapping out bad harddisks and storage pods (but granted, it's I/O load was very light). Also, while AWS takes away the burden of handling the physical drives and the filesystem for you, now you have to handle interfacing with AWS. That means you need an engineer that knows how to integrate your application with AWS. If you can leverage more AWS services, maybe even avoid needing your own server rooms because everything is running there, it might pay off. European vs. American salaries might also change the equation. But if you just use it for storage, I don't think it's worth it at any scale.
eigenvalue 5y ago
A lot responses here seem to make the embedded assumption that users pay the "advertised price" for storage and compute instances from AWS. When you are at that scale, there are presumably closely negotiated agreements that result in much lower than normal pricing but require large volume commitments and clear ramping projections. Thus the arb between doing it all yourself versus AWS is probably quite different than it might seem from the outside.
- lamnk 5y ago
  Hardware is cheaper at scale too

runlevel1 5y ago

I'm not saying it never happens, but I've never seen moving to AWS (or Azure, GCP, etc.) save in people costs at any tech company with a large resource footprint. It just shifted where the time is spent and who had to spend it. The public cloud and managed services work great for the most common use cases, but go outside those and you start having to engineer around limitations. If you have a sizable footprint in any given dimension you're trading one complexity for another.

bombcar 5y ago
The "who had to spend it" is huge - companies love paying providers and hate paying people/depreciating costs.
- markus_zhang 5y ago
  Curiously our company moved from AWS to on-premise a couple of years ago. Something about CAPEX -> OPEX was mentioned back then.
  - bombcar 5y ago
    That's the second part of the loop, when the cost of AWS is high enough that you can show immediate dollar savings by bringing it in-house.
    - markus_zhang 5y ago
      Yeah, that was a bit more than one year before we filed for IPO :)
- walrus01 5y ago
  > hate paying depreciating costs. One possible solution for this if you want to do it as bare metal you control, is leased equipment (even with $1 buyout at end of term), which can be accounted for differently than purchasing it up front.

nickstinemates 5y ago

Our company is not many magnitudes larger than any company and it is not remotely cost competitive for us to run any of our stack on AWS (heavy data ingest (hundreds of billions of inserts a day,) many disks/'big data' backend, hundreds of customers accessing the data. Even 1 time deals to get us into the door are not cost competitive, let alone long tail economics. Cloud for fast storage and high bandwidth usecases is extremely expensive.

walrus01 5y ago

If you have petabytes of data in AWS you already have a number of on-staff engineers with significant six figure salaries. If the problem is that your group of six-figure salary people only know how to put data into AWS, or other cloud services, and not design/engineer/maintain your own bare metal infrastructure as well, then that would definitely be a limitation. For reference, a few petabytes of data is not actually that many systems these days, if you have something like a bunch of 72-drive supermicros or equivalent with 14-16TB drives in them. Set up properly this can be administered by one FTE (of course with additional staffing/tech resources for when that FTE is on vacation/unavailable, and appropriate training for other persons who might have admin on the setup). my very rough calculation here says that a 36-drive ZFS RAIDZ2 composed of 16TB drives is something like 492TB (447TiB) usable storage capacity. so five such arrays would be 2460TB. compared to the monthly AWS bill for 2.0 to 2.5TB of data you could probably afford to entirely duplicate the whole setup in a twin identical set of hardware at a geographically diverse off-site location.

amluto 5y ago
You have to factor in the cost of egress from AWS to your nice colocated drive.
- walrus01 5y ago
  welcome to the hotel california... Last thing I remember, I was Running for the door I had to find the passage back To the place I was before "Relax, " said the night man, "We are programmed to receive. You can check-out any time you like, But you can never leave! "
  - amluto 5y ago
    And this is why I don’t think AWS will lower egress fees in response to R2. AWS may be more interested in discouraging people from using egress than in capturing the revenue from egress. I predict that, at most, we’ll see a narrowly tailored reduction in egress fees that is designed to be entirely useless for communication between server applications.

_3u10 5y ago

It isn’t. I have the same overhead running my server as I do a VM. What I don’t get is a $3000 bill for $100 worth of server. You can generally buy whatever you are renting from AWS for 1 to 3 months of an AWS bill. The only thing I don’t get from colo is a bunch of other customers thrashing the cache on my CPUs Databases are not web servers there’s no possible way to not run a database on smaller / fewer instances when running at non-peak times. Instant scaling is the only possible advantage AWS could bring. However with the prices they charge it’s simpler and cheaper to just buy/rent your own hardware. Especially if you have to pay egress fees. (bandwidth is really the biggest ripoff)

midasuni 5y ago
The argument isn’t about using AWS to run a VM (which can be cheaper that coloing your own kit, depending how many you want and for how long), it’s all the extra stuff. Start an aws load balancer rather than run and maintain your own for example. I don’t like lock-in, but the prevailing view has always been in favour of lock-in, be it IBM mainframes, oracle databases, windows servers etc, and if you swing that way aws has tempting offers. Oh and databases do scale. Say you want to run end of quarter financials that require a lot of processing for a day, you bring up tons of read replicas and away you go
- _3u10 5y ago
  If you bought hardware with what you pay for AWS RDS you could run your entire DB in RAM. Hell you could probably put the data in memory on a GPU. Also, this is generally why you run financials overnight. If your hardware is serving transactions during the day it can easily run your quarterlies at night. nginx is far easier to maintain than AWS load balancers which is what load balancers their load balancers are. The best part about nginx configs? They are cloud agnostic and will work on everything from a Raspberry Pi to a 128 core EPYC server. I'll tell you something about RDS reliabilty, your monthly maintenance window brings your DB down far more often than a single unreplicated server ever fails. EBS (like the entire thing) has failed more times in the last year on us-east-1 than my colo RAID. The selling point of AWS is that if you pick AWS and it fails you can say, well the richest guy in the world can't figure this stuff out so it must be impossible, when in reality high school kids could make a more reliable system. If you pick AWS you have the unreliability of the base software / hardware of their systems plus whatever the AWS engineers fuck up. At this point it's pretty clear that they can't even keep a SAN working.

saiya-jin 5y ago

That's a nice statement sounding like straight from Amazon sales reps, and it can actually work for some companies, maybe. But for our bank (top 10 globally), the only way to get even equal financially to our own farms is to have aggressive downtime every night that negatively affect productivity of our global teams. Pricing is really not that great if you deal in scale. You wanna push 1 evening a bit late to deliver something valuable for the project? Sorry, no can do. I don't even factor in horribly expensive migration projects that brought actually 0 added business value for the type of apps we use. We still have to keep our Network, Windows and Unix admins, various App support personnel etc., there is plenty of work for them with AWS. Not 1 single IT guy was made redundant. No cost savings, in contrary.

IgorPartola 5y ago

That argument doesn’t hold up. If how could AWS be cheaper than doing it yourself at that scale? Like if you are a tiny company that can’t afford your own DC, your own engineers, etc. then yes AWS is cheaper in absolute costs but not in per-byte costs. But at scale you should be able to hire engineers and build out a DC at which point you aren’t paying the AWS margin, which is how you save money. In other words your assessment would only be true if AWS had a 0% or a negative margin.

kevincox 5y ago
There is still an economy of scale. You are right, as you use more resources your economy of scale will increase, but it will never match AWS's (unless you are huge). So the math is if the AWS margin is less than the difference between the two different economies of scale then it makes sense to run your own datacenters (ignoring the opportunity cost of the transition costs). Of course at some point the margin will exceed that difference, but depending on what type of infrastructure you need it can be at a very high point.
- IgorPartola 5y ago
  Your last point is the important part: depending on the type of infrastructure you need you might be able to save money. If you want a cheap place to dump your files, B2 is cheaper than S3 and raw storage hardware pays for itself in about a year. If you need a sophisticated CDN then yeah you’ll need to be huge before it pays for itself. I would consider ditching S3 at the point where I can hire two full time engineers to worry about my storage layer.

deleted 5y ago

[deleted]

gsliepen 5y ago

For a while I did maintain a storage cluster that had close to 0.5 PB of data, and had a capacity of up to 3 PB (if you filled all the slots with the largest disks you can buy). You want to ensure you have a lot of redundancy and spare capacity if you are managing it yourself. Luckily, the hardware is relatively cheap. It's the manpower that costs a lot. Still, I think it was only 0.1 FTE to manage this storage cluster, including the network, file systems, user access, swapping out bad harddisks and storage pods (but granted, it's I/O load was very light). Also, while AWS takes away the burden of handling the physical drives and the filesystem for you, now you have to handle interfacing with AWS. That means you need an engineer that knows how to integrate your application with AWS. If you can leverage more AWS services, maybe even avoid needing your own server rooms because everything is running there, it might pay off. European vs. American salaries might also change the equation. But if you just use it for storage, I don't think it's worth it at any scale.

eigenvalue 5y ago

A lot responses here seem to make the embedded assumption that users pay the "advertised price" for storage and compute instances from AWS. When you are at that scale, there are presumably closely negotiated agreements that result in much lower than normal pricing but require large volume commitments and clear ramping projections. Thus the arb between doing it all yourself versus AWS is probably quite different than it might seem from the outside.

lamnk 5y ago
Hardware is cheaper at scale too