11 ms·

The cost of egress traffic is a very good reason for many organizations to not fully migrate to a cloud provider anytime soon. And since, unlike with storage co

The cost of egress traffic is a very good reason for many organizations to not fully migrate to a cloud provider anytime soon. And since, unlike with storage costs, there doesn't seem to be an actual reason (other than: it makes migrating to competitors cost-prohibitive in a subset of cases), that seems kind of... weird? Small example: an actual company I do some work for is in the business of delivering creative assets to distributors. This results in an egress of around 180TB per month, which is, on average just, around 500Mb/s. So, this company currently operates 2 racks in commercial data centers, linked via 10Gb/s Ethernet-over-DWDM, with 2x512Mb/s and 1x1Gb/s Internet uplinks per DC. Each rack has 2 generic-OEM servers with ~64 AMD Zen cores, 1/2TB RAM, ~8TB NVMe and ~100TB SAS RAID6 storage per node. Just the cost-savings over egress on AWS is enough to justify that setup, including the cost of an engineer to keep it all up and running (even though the effort required for that turns out to be minimal). So, are cloud providers ignoring a significant market here, or is the markup on their current customers lucrative enough?

jonatron 3y ago
The AWS Enterprise Discount Program apparently requires $1M per year spend. 180TB is about $13k on AWS so presumably not enough to be interesting to them. Hopefully someone who works at AWS can share some info.
- tky 3y ago
  EDP can be great if you can meet their required year over year growth requirement and if your spend is high enough to make the discounts (which are variable and negotiated often at a product offering level, not blanket) offset the required top-shelf support contract. For smaller orgs even at the $1-2M/mo level, it can often be a risk not worth taking vs other cost savings mechanisms and locating high-egress use cases elsewhere. Egress bandwidth pricing has been the sacred cow money firehose forever, despite transit costs continuing to shrink.
_cenw 3y ago
It really depends on the quality of the peering you expect. It doesn't matter, until it does. Consumer ISPs sometimes do their utmost to not peer with open exchanges, and the entire thing gets even more complex when you go to places where bandwidth is more expensive (i.e. Oceania). There's a reason the favorite chart to exemplify value Cloudflare reps like to show is Argo Smart Routing, and why it costs about $100 per TB just like AWS and GCP.
- naiv 3y ago
  We are using Argo for our xhr search traffic as it makes more sense than setting up different servers/vms in parts of the world. Each request is only 1kb max. But I would not use it for static assets. For this we use Bunny edge storage to provide faster response times at very reasonable prices.
- Fripplebubby 3y ago
  I agree, and I would also put forward that most people don't understand what peering is or how it works. When people (usually developers who are not network engineers and have not worked at that level of the stack) talk about "egress", they mean delivering bits from your network (cloud or otherwise) to any other network on the internet. How can you put just one price on delivering a bit either to a host within the same datacenter or one on the opposite side of the planet? Physics still mean that one is more expensive than the other. The existence of the world wide web has tricked us into thinking that sending traffic anywhere is/should be the same, but of course it is not. So while the price you (a cloud customer) pay for egress pricing is (often) indiscriminate on where that traffic is going, using common sense, we can understand that some traffic is more expensive than others, and the price we pay is a blended price with that aspect "baked in" or "priced in".
  - wmf 3y ago
    Transit is also the same price regardless of distance.
  - pests 3y ago
    USPS manages to do it.
toast0 3y ago
My feeling is that egress is easily measured, so it's where costs that are hard to assess get moved to. It doesn't feel great to be line item billed for stuff at 10x the rate of credible other offers. I think there is also some geo-specific pricing that gets hidden in a global price; bandwidth can be a lot more expensive in some locations than others and if you are charged 5x for egress in south america, nobody will use the south america locations and that's not good for business.
- raid2000 3y ago
  Right. Egress is an imperfect, but reasonable metric for overall utilization. If they started charging for CPU hertz above a certain threshold, that'd be a harder sell.
  - zokier 3y ago
    > If they started charging for CPU hertz above a certain threshold, that'd be a harder sell. You mean like EC2 T series burstable CPU credits?
  - felbane 3y ago
    I mean, they kinda already do depending on your plan (see: AWS EC2 spot instances)
- amluto 3y ago
  I don’t believe this. Operating an internal cloud network is expensive, but it’s expensive because of internal traffic, and they don’t charge for that internal traffic. Egress is just like traffic to any other system, and AWS doesn’t charge for that. Also: > It doesn't feel great to be line item billed for stuff at 10x the rate of credible other offers. It’s quite a bit worse than 10x
  - zokier 3y ago
    AWS charges for internal traffic too, both cross-az and cross-region.
thimp 3y ago
I saw a hilarious fuck up a few months ago. Company sets up an AWS hosted always on VPN solution. Connects 1000 staff through it. Celebrates how they saved $50k on the VPN solution. Gets $25k AWS bill for the just the first month of egress traffic. Turns out the data was leaving AWS egress three separate times.
- ailurooo 3y ago
  how were they spending that 50k previously? a bespoke saas thing or self hosting?
  - dijit 3y ago
    Likely they are mixing timelines (one time cost or yearly cost with AWS’s monthly charges). Cisco anyconnect VPN capable appliances (that can do 10GBE) are very expensive, and licenses are per user- so if an appliance needed an upgrade it is conceivable that it could cost $50k in the first year.
    - CaliforniaKarl 3y ago
      I agree. You're probably looking at: • The cost of two hardware gateways (the depreciation cost, that is) • The cost of 24x7xNBD (next business day) hardware support • The cost of the user/session licenses (which might be depreciated also) • The cost of software support/upgrades
    - thimp 3y ago
      This. It wasn’t Cisco. Far crappier vendor! I don’t want to name names but they had a few high profile incidents that suggested their software was written by idiots. There was a panic move by the security people to switch away and still tick the box but they didn’t really understand the billing and architectural models of AWS. Pricing was around 75k for 1000 seats for a year. They thought it was going to be $25k a year but it turned out to be that a month.
- constrain5795 3y ago
  AWS VPN base cost is $72/mo + $36/u/mo. So they were spending $80k/mo before?
ozr 3y ago
Definitely lucrative enough. The use case you've described isn't particularly uncommon, but lots of companies just pay for the egress. The problem is that there are now multiple generations of software engineers that do not know how bandwidth is priced. They've only used managed providers that charge per unit of ingress/egress, at some fractional dollar per GB.
- api 3y ago
  I’ve had people refuse to believe that bandwidth is actually very cheap and cloud markup is insane (hundreds or even thousands of times cost). I show them bare metal providers and colo that bills by size of pipe rather than transfer. They refuse to believe it or assume there must be a catch. There usually isn’t, though sometimes the very cheapest skimp on things like rich peering and can be slightly slower or less reliable. But still cheapest is relative here. Expensive bare metal or colo bandwidth is still usually hundreds of times less than big three cloud egress. It’s just nuts. It’s a subset of a wider problem of multiple generations of developers being fully brainwashed by “cloud native” in lots of ways. What an amazing racket this all has been for providers…
  - kkielhofner 3y ago
    Cloud providers have done an incredible job creating an entire generation of people who have no knowledge of such things. $0.09/GB? I guess that’s just what it costs. Expected when you’ve never looked at, considered, or even heard of things like peering and buying transit or even co-location. Enter 95th percentile billing on a gig port for $500/mo or whatever… Same goes for hardware. Want to watch something glorious? Take someone who has only ever used VMs, etc in big cloud and give them even a moderate spec new bare metal server. The performance melts their brains. Then they realize the entire thing - colo, servers, bandwidth, even staff is a fraction of the cost.
    - api 3y ago
      A big part of the problem is that software development has wrapped itself around cloud native. Sure that metal server has brain melting performance for nothing, but try provisioning it or running it. Nobody knows how and we are dependent on stacks like K8S that require an ops team to run. The industry really found a way to stop the downward cost spiral of personal computing and bring everyone back into mainframe. Moore’s Law is not dead but you no longer see the benefits. The difference is just pocketed by cloud providers who will use the gains to pack more low performing VMs onto less hardware.
amluto 3y ago
> other than: it makes migrating to competitors cost-prohibitive in a subset of cases My theory: it forces third party services into the same cloud. Suppose you use AWS and you want to pay a third party SaaS provider for some service involving moderate-to-large amounts of data. Here’s one of many examples: https://www.snowflake.com/en/data-cloud/pricing-options/ https://www.snowflake.com/en/data-cloud/pricing-options/ And look at this remarkable choice: you get to pick AWS, Azure, or GCP! Snowflake is paying a lot of money to host on those clouds, and they’re passing those costs on to customers. Snowflake is big. They have lots of engineers. They are obviously cloud-agnostic: they already support three clouds. It would surely be much cheaper to operate a physical facility, and they could plausibly offer better performance (because NVMe is amazing), and they could split the cost savings with customers. But they don’t, and my theory is that egress from customers to Snowflake would negate any cost savings, and the variable nature of the costs would scare away customers. So my theory is that the ways that customers avoid egress fees makes the major clouds a lot of money. IMO regulators should take a very careful look at this, but it’s an excellent business decision on the parts of the clouds.
- Eridrus 3y ago
  Snowflake's margins are like 90%+ on top of the compute they sell, and they pass on all these costs including egress directly to customers.
  - amluto 3y ago
    Their list price for storage capacity is only on the order of 2x what S3 charges, and Snowflake and S3 likely both offer discounts. Comparing compute costs is harder. If I were running a service like Snowflake, I would certainly appreciate the effortless scaling that the major clouds offer. But I also know what I pay for actual servers and roughly what I would pay in a major cloud, and I much prefer the former.
    - Eridrus 3y ago
      Yeah, storage is much more competitive, but there have been leaks on what hardware they run on, their margins are incredible, which makes sense since they are selling software, not hardware! Anyway, there is an explicit egress charge: https://docs.snowflake.com/en/user-guide/cost-understanding-data-transfer https://docs.snowflake.com/en/user-guide/cost-understanding-...
kansi 3y ago
Are you able to share which provider do they use for their rack setup?
xyst 3y ago
just 1 engineer? lol. Let's hope it's not business critical for the servers to be up and running at all times if you only need 1 engineer.
- pinkgolem 3y ago
  you need an oncall team be it cloud or not, taking one fte position and dedicating it to managing 2 servers(documentation/updates/backup & restore procedures testing) seems rather.. high
electroly 3y ago
If you still want to use some AWS services, you can get an AWS Direct Connect fiber cross connect from your data center rack to AWS, just like you do with your Internet connections. They operate Direct Connect in lots of third party carrier-neutral data centers. AWS egress over Direct Connect is $0.02/GB instead of $0.09/GB over the public Internet. You can serve customers through your unmetered Internet connections while accessing S3 (or whatever) via Direct Connect on the backend.
- amluto 3y ago
  I can pay overpriced cross-connect rates in giant name brand datacenters, with or without terminating one end at Direct Connect. (AFAICT the $1600/mo or so for 10Gbps doesn’t actually cover the cost of the cross-connect.) But that extra $65k/mo to fully utilize the link is utterly and completely nuts. My mind boggles when someone calls that a good deal. I can buy and fully depreciate the equipment needed to utilize that in a couple of days. (Well, I can’t actually buy equipment on the AWS end, but I can _rent_ it, from AWS, for a lot less money than $64k/mo.) And I don’t believe at all that it costs AWS anything close to this much to operate the thing. I can, after all, send 10Gbps between two EC2 instances (or between S3 and EC2) for a lot less money. That $65k is simply a punitive charge IMO. AWS wants me to avoid paying it by not doing this or, in cases where I can’t avoid it (e.g. the thing is actually a link to my office or a factory or whatever) to collect as much money as they can without driving me off AWS entirely.
- ricktdotorg 3y ago
  i'm a customer of the GCP equivalent: partner interconnect. our DC is in an equinix facility, they wire up drops for us that layer 3 straight into GCP. unmetered 1Gbps for about 250 bucks a month per (paid to EQX not GCP). are AWS really charging you per Gb for data egress from AWS into your own DC over an AWS direct connect??
  - electroly 3y ago
    I'm not familiar with GCP but this page below seems basically identical to AWS Direct Connect's pricing structure, including $0.02/GB for egress. Are you sure nobody is paying egress on your behalf? https://cloud.google.com/network-connectivity/docs/interconnect/pricing https://cloud.google.com/network-connectivity/docs/interconn... Ctrl+F "Data transfer from a VPC network through a Cloud Interconnect connection"
belter 3y ago
Context matters here. How critical is that workload? What the economic and reputational impact for the company, if one of the physical connections or some technical problem with the data center causes a downtime of hours or days?
- pinkgolem 3y ago
  looking at the uptime from aws and from most big outtage notices i have read in the last few years, there does not seem to be a benefit in regards to reliability when using cloud. see reddit, see amazon/aws outages taking with them netflix/disney plus etc honestly its a lot better to keep your architecture cloud agnostics and test restores regulary on a different provider/region also: store your backups somewhere else, not on your primary cloud provider, not connected to your auth system, not overwritable from anywhere
  - belter 3y ago
    I am not aware of an AWS outage in their 15-16 years of existence, that an architecture, built according to the recommended best practices, of distribution across availability zones and regions, would not be able to withstand. I am willing to be proven wrong. Can you provide one example? Of course, these come with increased cost, but I am thinking web retail on a large scale, or airline companies, where for example a downtime of a few hours will easily wipe out any savings made by relying on a local data center. It might not be the solution for a smaller company.
    - pinkgolem 3y ago
      I guess you did not have any service running on aws winter 2021? Does not really matter which region, with certain services you could get lucky but basically everything depending on fresh iam creds was down And if netflix/disney/slack/ring amazon themself can not do it, with a multiple with the ressources i have access to, good luck to you. there are also times where your company might rely on a single aws product, and it might simply break an integration(aws iot mqtt ->sqs for example) took them 2 days to fix
andix 3y ago
180 TB is still a small customer in the scale of big cloud providers, so they probably just don't care. If the customers are willing to pay the price they are taking their money, if they go to one of the smaller providers they are fine with it too. Also it could be possible to set up some hybrid solution and offload the egress heavy assets serving to another provider and only run the "brains" inside AWS/etc.
rsync 3y ago
Not every cloud provider has egress charges…
hatsix 3y ago
Our 20tb/m costs us $414... if you're saying that you can amortize those servers, rent the space and pay for upkeep for less than $4k a month? We have another service that has 20x the bandwidth, but it's a legacy GAE that has super cheap premium egress... But I'm told that AWS says that their pre-pay discount will be competitive in about a year, the rate that we're growing. The negotiated egress prices are much lower, so long as you are buying future egress. If they're not worried about you jumping ship (you use a lot of their aws-native services), you can get a great deal.
- pinkgolem 3y ago
  >If they're not worried about you jumping ship (you use a lot of their aws-native services), you can get a great deal. Aka as long as you are not using the cloud as cloud the cloud can be cheap? >rent the space and pay for upkeep for less than $4k a month? I mean, you can just rent servers somewhere else.. or do a colocation rack. Also I would strongly suggest going with the first one
- re-thc 3y ago
  > If they're not worried about you jumping ship (you use a lot of their aws-native services), you can get a great deal. Shouldn't it be the other way around? Why are they giving you are discount if you're not likely to jump?
- shiftpgdn 3y ago
  I could buy a whole off lease fully deprcriated server and a year of colo with a gig link for under $4,000, yes. I would almost certainly see better performance than any cloud provider can give me too.
deleted 3y ago
[deleted]
mgaunard 3y ago
There is no scenario in which cloud is cheaper than bare metal to operate. What's cheaper with cloud is that you don't need upfront costs, but here, you already have the whole infrastructure already there, up and running. Why would you even consider getting rid of it and replacing it with AWS, makes no sense.
- kiitos 3y ago
  It's weird that AWS is just absurdly profitable, then, isn't it? (You're wrong, there are many such scenarios)
  - mgaunard 3y ago
    It's absurdly profitable precisely because it resells things way above their real price. People use AWS for the same reason they rent houses: less investment required and you can change things more quickly.
    - kiitos 3y ago
      > People use AWS for the same reason they rent houses: less investment required and you can change things more quickly. Yes, exactly. And it's very often the case that these capabilities are more valuable to users than "real price" as you mean. You... understand this, right? That value is measured contextually, not objectively?
vidarh 3y ago
Yeah, I've built several setups where total cost of ownership was lower than egress cost alone would have been on AWS. Both w/physical hardware in colos and with rented managed servers at providers like OVH and Hetzner.
cocothem 3y ago
Are you a Google Cloud customer looking to exit Google Cloud? If so, you are eligible for free data transfer when you migrate all of your Google Cloud workloads and data from Google Cloud to another cloud provider or an on-premises data center. https://cloud.google.com/exit-cloud https://cloud.google.com/exit-cloud
intelVISA 3y ago
Between the vendor lock-in and egregious costs it's a sad state for so many in our profession to call this 'the future'.

jonatron 3y ago

The AWS Enterprise Discount Program apparently requires $1M per year spend. 180TB is about $13k on AWS so presumably not enough to be interesting to them. Hopefully someone who works at AWS can share some info.

tky 3y ago
EDP can be great if you can meet their required year over year growth requirement and if your spend is high enough to make the discounts (which are variable and negotiated often at a product offering level, not blanket) offset the required top-shelf support contract. For smaller orgs even at the $1-2M/mo level, it can often be a risk not worth taking vs other cost savings mechanisms and locating high-egress use cases elsewhere. Egress bandwidth pricing has been the sacred cow money firehose forever, despite transit costs continuing to shrink.

_cenw 3y ago

It really depends on the quality of the peering you expect. It doesn't matter, until it does. Consumer ISPs sometimes do their utmost to not peer with open exchanges, and the entire thing gets even more complex when you go to places where bandwidth is more expensive (i.e. Oceania). There's a reason the favorite chart to exemplify value Cloudflare reps like to show is Argo Smart Routing, and why it costs about $100 per TB just like AWS and GCP.

naiv 3y ago
We are using Argo for our xhr search traffic as it makes more sense than setting up different servers/vms in parts of the world. Each request is only 1kb max. But I would not use it for static assets. For this we use Bunny edge storage to provide faster response times at very reasonable prices.
Fripplebubby 3y ago
I agree, and I would also put forward that most people don't understand what peering is or how it works. When people (usually developers who are not network engineers and have not worked at that level of the stack) talk about "egress", they mean delivering bits from your network (cloud or otherwise) to any other network on the internet. How can you put just one price on delivering a bit either to a host within the same datacenter or one on the opposite side of the planet? Physics still mean that one is more expensive than the other. The existence of the world wide web has tricked us into thinking that sending traffic anywhere is/should be the same, but of course it is not. So while the price you (a cloud customer) pay for egress pricing is (often) indiscriminate on where that traffic is going, using common sense, we can understand that some traffic is more expensive than others, and the price we pay is a blended price with that aspect "baked in" or "priced in".
- wmf 3y ago
  Transit is also the same price regardless of distance.
- pests 3y ago
  USPS manages to do it.

toast0 3y ago

My feeling is that egress is easily measured, so it's where costs that are hard to assess get moved to. It doesn't feel great to be line item billed for stuff at 10x the rate of credible other offers. I think there is also some geo-specific pricing that gets hidden in a global price; bandwidth can be a lot more expensive in some locations than others and if you are charged 5x for egress in south america, nobody will use the south america locations and that's not good for business.

raid2000 3y ago
Right. Egress is an imperfect, but reasonable metric for overall utilization. If they started charging for CPU hertz above a certain threshold, that'd be a harder sell.
- zokier 3y ago
  > If they started charging for CPU hertz above a certain threshold, that'd be a harder sell. You mean like EC2 T series burstable CPU credits?
- felbane 3y ago
  I mean, they kinda already do depending on your plan (see: AWS EC2 spot instances)
amluto 3y ago
I don’t believe this. Operating an internal cloud network is expensive, but it’s expensive because of internal traffic, and they don’t charge for that internal traffic. Egress is just like traffic to any other system, and AWS doesn’t charge for that. Also: > It doesn't feel great to be line item billed for stuff at 10x the rate of credible other offers. It’s quite a bit worse than 10x
- zokier 3y ago
  AWS charges for internal traffic too, both cross-az and cross-region.

thimp 3y ago

I saw a hilarious fuck up a few months ago. Company sets up an AWS hosted always on VPN solution. Connects 1000 staff through it. Celebrates how they saved $50k on the VPN solution. Gets $25k AWS bill for the just the first month of egress traffic. Turns out the data was leaving AWS egress three separate times.

ailurooo 3y ago
how were they spending that 50k previously? a bespoke saas thing or self hosting?
- dijit 3y ago
  Likely they are mixing timelines (one time cost or yearly cost with AWS’s monthly charges). Cisco anyconnect VPN capable appliances (that can do 10GBE) are very expensive, and licenses are per user- so if an appliance needed an upgrade it is conceivable that it could cost $50k in the first year.
  - CaliforniaKarl 3y ago
    I agree. You're probably looking at: • The cost of two hardware gateways (the depreciation cost, that is) • The cost of 24x7xNBD (next business day) hardware support • The cost of the user/session licenses (which might be depreciated also) • The cost of software support/upgrades
  - thimp 3y ago
    This. It wasn’t Cisco. Far crappier vendor! I don’t want to name names but they had a few high profile incidents that suggested their software was written by idiots. There was a panic move by the security people to switch away and still tick the box but they didn’t really understand the billing and architectural models of AWS. Pricing was around 75k for 1000 seats for a year. They thought it was going to be $25k a year but it turned out to be that a month.
constrain5795 3y ago
AWS VPN base cost is $72/mo + $36/u/mo. So they were spending $80k/mo before?

ozr 3y ago

Definitely lucrative enough. The use case you've described isn't particularly uncommon, but lots of companies just pay for the egress. The problem is that there are now multiple generations of software engineers that do not know how bandwidth is priced. They've only used managed providers that charge per unit of ingress/egress, at some fractional dollar per GB.

api 3y ago
I’ve had people refuse to believe that bandwidth is actually very cheap and cloud markup is insane (hundreds or even thousands of times cost). I show them bare metal providers and colo that bills by size of pipe rather than transfer. They refuse to believe it or assume there must be a catch. There usually isn’t, though sometimes the very cheapest skimp on things like rich peering and can be slightly slower or less reliable. But still cheapest is relative here. Expensive bare metal or colo bandwidth is still usually hundreds of times less than big three cloud egress. It’s just nuts. It’s a subset of a wider problem of multiple generations of developers being fully brainwashed by “cloud native” in lots of ways. What an amazing racket this all has been for providers…
- kkielhofner 3y ago
  Cloud providers have done an incredible job creating an entire generation of people who have no knowledge of such things. $0.09/GB? I guess that’s just what it costs. Expected when you’ve never looked at, considered, or even heard of things like peering and buying transit or even co-location. Enter 95th percentile billing on a gig port for $500/mo or whatever… Same goes for hardware. Want to watch something glorious? Take someone who has only ever used VMs, etc in big cloud and give them even a moderate spec new bare metal server. The performance melts their brains. Then they realize the entire thing - colo, servers, bandwidth, even staff is a fraction of the cost.
  - api 3y ago
    A big part of the problem is that software development has wrapped itself around cloud native. Sure that metal server has brain melting performance for nothing, but try provisioning it or running it. Nobody knows how and we are dependent on stacks like K8S that require an ops team to run. The industry really found a way to stop the downward cost spiral of personal computing and bring everyone back into mainframe. Moore’s Law is not dead but you no longer see the benefits. The difference is just pocketed by cloud providers who will use the gains to pack more low performing VMs onto less hardware.

amluto 3y ago

> other than: it makes migrating to competitors cost-prohibitive in a subset of cases My theory: it forces third party services into the same cloud. Suppose you use AWS and you want to pay a third party SaaS provider for some service involving moderate-to-large amounts of data. Here’s one of many examples: https://www.snowflake.com/en/data-cloud/pricing-options/ https://www.snowflake.com/en/data-cloud/pricing-options/ And look at this remarkable choice: you get to pick AWS, Azure, or GCP! Snowflake is paying a lot of money to host on those clouds, and they’re passing those costs on to customers. Snowflake is big. They have lots of engineers. They are obviously cloud-agnostic: they already support three clouds. It would surely be much cheaper to operate a physical facility, and they could plausibly offer better performance (because NVMe is amazing), and they could split the cost savings with customers. But they don’t, and my theory is that egress from customers to Snowflake would negate any cost savings, and the variable nature of the costs would scare away customers. So my theory is that the ways that customers avoid egress fees makes the major clouds a lot of money. IMO regulators should take a very careful look at this, but it’s an excellent business decision on the parts of the clouds.

Eridrus 3y ago
Snowflake's margins are like 90%+ on top of the compute they sell, and they pass on all these costs including egress directly to customers.
- amluto 3y ago
  Their list price for storage capacity is only on the order of 2x what S3 charges, and Snowflake and S3 likely both offer discounts. Comparing compute costs is harder. If I were running a service like Snowflake, I would certainly appreciate the effortless scaling that the major clouds offer. But I also know what I pay for actual servers and roughly what I would pay in a major cloud, and I much prefer the former.
  - Eridrus 3y ago
    Yeah, storage is much more competitive, but there have been leaks on what hardware they run on, their margins are incredible, which makes sense since they are selling software, not hardware! Anyway, there is an explicit egress charge: https://docs.snowflake.com/en/user-guide/cost-understanding-data-transfer https://docs.snowflake.com/en/user-guide/cost-understanding-...

kansi 3y ago

Are you able to share which provider do they use for their rack setup?

xyst 3y ago

just 1 engineer? lol. Let's hope it's not business critical for the servers to be up and running at all times if you only need 1 engineer.

pinkgolem 3y ago
you need an oncall team be it cloud or not, taking one fte position and dedicating it to managing 2 servers(documentation/updates/backup & restore procedures testing) seems rather.. high

electroly 3y ago

If you still want to use some AWS services, you can get an AWS Direct Connect fiber cross connect from your data center rack to AWS, just like you do with your Internet connections. They operate Direct Connect in lots of third party carrier-neutral data centers. AWS egress over Direct Connect is $0.02/GB instead of $0.09/GB over the public Internet. You can serve customers through your unmetered Internet connections while accessing S3 (or whatever) via Direct Connect on the backend.

amluto 3y ago
I can pay overpriced cross-connect rates in giant name brand datacenters, with or without terminating one end at Direct Connect. (AFAICT the $1600/mo or so for 10Gbps doesn’t actually cover the cost of the cross-connect.) But that extra $65k/mo to fully utilize the link is utterly and completely nuts. My mind boggles when someone calls that a good deal. I can buy and fully depreciate the equipment needed to utilize that in a couple of days. (Well, I can’t actually buy equipment on the AWS end, but I can _rent_ it, from AWS, for a lot less money than $64k/mo.) And I don’t believe at all that it costs AWS anything close to this much to operate the thing. I can, after all, send 10Gbps between two EC2 instances (or between S3 and EC2) for a lot less money. That $65k is simply a punitive charge IMO. AWS wants me to avoid paying it by not doing this or, in cases where I can’t avoid it (e.g. the thing is actually a link to my office or a factory or whatever) to collect as much money as they can without driving me off AWS entirely.
ricktdotorg 3y ago
i'm a customer of the GCP equivalent: partner interconnect. our DC is in an equinix facility, they wire up drops for us that layer 3 straight into GCP. unmetered 1Gbps for about 250 bucks a month per (paid to EQX not GCP). are AWS really charging you per Gb for data egress from AWS into your own DC over an AWS direct connect??
- electroly 3y ago
  I'm not familiar with GCP but this page below seems basically identical to AWS Direct Connect's pricing structure, including $0.02/GB for egress. Are you sure nobody is paying egress on your behalf? https://cloud.google.com/network-connectivity/docs/interconnect/pricing https://cloud.google.com/network-connectivity/docs/interconn... Ctrl+F "Data transfer from a VPC network through a Cloud Interconnect connection"

belter 3y ago

Context matters here. How critical is that workload? What the economic and reputational impact for the company, if one of the physical connections or some technical problem with the data center causes a downtime of hours or days?

pinkgolem 3y ago
looking at the uptime from aws and from most big outtage notices i have read in the last few years, there does not seem to be a benefit in regards to reliability when using cloud. see reddit, see amazon/aws outages taking with them netflix/disney plus etc honestly its a lot better to keep your architecture cloud agnostics and test restores regulary on a different provider/region also: store your backups somewhere else, not on your primary cloud provider, not connected to your auth system, not overwritable from anywhere
- belter 3y ago
  I am not aware of an AWS outage in their 15-16 years of existence, that an architecture, built according to the recommended best practices, of distribution across availability zones and regions, would not be able to withstand. I am willing to be proven wrong. Can you provide one example? Of course, these come with increased cost, but I am thinking web retail on a large scale, or airline companies, where for example a downtime of a few hours will easily wipe out any savings made by relying on a local data center. It might not be the solution for a smaller company.
  - pinkgolem 3y ago
    I guess you did not have any service running on aws winter 2021? Does not really matter which region, with certain services you could get lucky but basically everything depending on fresh iam creds was down And if netflix/disney/slack/ring amazon themself can not do it, with a multiple with the ressources i have access to, good luck to you. there are also times where your company might rely on a single aws product, and it might simply break an integration(aws iot mqtt ->sqs for example) took them 2 days to fix

andix 3y ago

180 TB is still a small customer in the scale of big cloud providers, so they probably just don't care. If the customers are willing to pay the price they are taking their money, if they go to one of the smaller providers they are fine with it too. Also it could be possible to set up some hybrid solution and offload the egress heavy assets serving to another provider and only run the "brains" inside AWS/etc.

rsync 3y ago

Not every cloud provider has egress charges…

hatsix 3y ago

Our 20tb/m costs us $414... if you're saying that you can amortize those servers, rent the space and pay for upkeep for less than $4k a month? We have another service that has 20x the bandwidth, but it's a legacy GAE that has super cheap premium egress... But I'm told that AWS says that their pre-pay discount will be competitive in about a year, the rate that we're growing. The negotiated egress prices are much lower, so long as you are buying future egress. If they're not worried about you jumping ship (you use a lot of their aws-native services), you can get a great deal.

pinkgolem 3y ago
>If they're not worried about you jumping ship (you use a lot of their aws-native services), you can get a great deal. Aka as long as you are not using the cloud as cloud the cloud can be cheap? >rent the space and pay for upkeep for less than $4k a month? I mean, you can just rent servers somewhere else.. or do a colocation rack. Also I would strongly suggest going with the first one
re-thc 3y ago
> If they're not worried about you jumping ship (you use a lot of their aws-native services), you can get a great deal. Shouldn't it be the other way around? Why are they giving you are discount if you're not likely to jump?
shiftpgdn 3y ago
I could buy a whole off lease fully deprcriated server and a year of colo with a gig link for under $4,000, yes. I would almost certainly see better performance than any cloud provider can give me too.

deleted 3y ago

[deleted]

mgaunard 3y ago

There is no scenario in which cloud is cheaper than bare metal to operate. What's cheaper with cloud is that you don't need upfront costs, but here, you already have the whole infrastructure already there, up and running. Why would you even consider getting rid of it and replacing it with AWS, makes no sense.

kiitos 3y ago
It's weird that AWS is just absurdly profitable, then, isn't it? (You're wrong, there are many such scenarios)
- mgaunard 3y ago
  It's absurdly profitable precisely because it resells things way above their real price. People use AWS for the same reason they rent houses: less investment required and you can change things more quickly.
  - kiitos 3y ago
    > People use AWS for the same reason they rent houses: less investment required and you can change things more quickly. Yes, exactly. And it's very often the case that these capabilities are more valuable to users than "real price" as you mean. You... understand this, right? That value is measured contextually, not objectively?

vidarh 3y ago

Yeah, I've built several setups where total cost of ownership was lower than egress cost alone would have been on AWS. Both w/physical hardware in colos and with rented managed servers at providers like OVH and Hetzner.

cocothem 3y ago

Are you a Google Cloud customer looking to exit Google Cloud? If so, you are eligible for free data transfer when you migrate all of your Google Cloud workloads and data from Google Cloud to another cloud provider or an on-premises data center. https://cloud.google.com/exit-cloud https://cloud.google.com/exit-cloud

intelVISA 3y ago

Between the vendor lock-in and egregious costs it's a sad state for so many in our profession to call this 'the future'.