6 ms·

Egress costs at those major clouds are ridiculous. Once you start having some traffic it can easily make 50% of all costs or more. That is not justifiable! Mean

by neop1x 7y ago

Egress costs at those major clouds are ridiculous. Once you start having some traffic it can easily make 50% of all costs or more. That is not justifiable! Meanwhile hardware costs are going down. We need more k8s providers with more resonable pricing. Unfortunatelly both Digital Ocean and Oracle Cloud don't have proper network load balancer implementations which is a must for elastic regional clusters and to forward TCP in a way that client IP is preserved and be able to add nodes without downtimes or TCP resets. OVH cloud doesn't implement LoadBalancer service type at all. So the choice in 2020 is really just Google, Amazon, Azure with their rolls-royce pricing. The cost difference between them is neglible. And then there are confidential free credits for startups. So sad...

atombender 7y ago
For HTTP, an external load balancer/CDN such as CloudFlare or Fastly could easily fulfill that role, though you obviously have to jump through a few more hoops to get there, since it's not built in. But with some of them you also get things like TLS termination, DNS hosting, advanced rewriting, caching, and DDOS protection.
ikiris 7y ago
You seem to think theres some kind of disconnect between egress pricing, and if the network has a functional backbone with load balancing infrastrucutre. If anything, egress pricing is to low in the vast majority of cases, and it really shows when customers start using it.
vasco 7y ago
All of AWS's VPC features are free. You get Security Groups, Subnets, Route Tables, all kinds of shenanigans in a very stable way, with basically no incidents that I can remember. Except those are not free, they're just charged for separately under "egress transfer costs". End of the day, the engineers in the networking teams at AWS still have to get paid and that service probably also needs to run at a profit. When seen under this light, the costs make more sense to me than the usual simple view of "but bandwidth transfer costs are cheaper everywhere else!!!"
- jugg1es 7y ago
  NAT gateways (a required component if you want real security) cost like $30/month each and you need at least 2 for HA.
  - scarface74 7y ago
    Well, I’ve got bad news for you, if you think NAT Gateways are required for “real security”, just wait until you enable IPv6 on AWS.....
    - kawsper 7y ago
      Could you elaborate on this?
      - scarface74 7y ago
        When you enable IPv6, there is no NATing. All of the IP addresses are public. You can use an Egress only Internet Gateway. https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat.html https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat.htm... But technically, your EC2 instance never has a public IP address. You can see for yourself by assigning a public IP address to an ENI attached to your instance and doing an ifconfig. An Internet Gateway is a one to one NAT.
        jugg1es 7y ago
        Interesting, I did not know this is how IPv6 worked. Need to investigate this more.
- nine_k 7y ago
  The amount of work required from AWS engineers to set up security groups, routing, etc is not proportional to the amount of traffic your server produces, but is charged as if it were. This requires every egress gigabyte to generate a lot of money to be worth it. If you serve high-traffic applications, like video, you have to be Netflix.
geuis 7y ago
I've been a Linode customer for the better part of a decade. Primary reason is that their egress is quite generous compared to any other provider I've looked at. Also, their customer service is exceptional and over the years the hardware has been upgraded without being charged more money. They're now rolling out a Kubernetes service which I'm in the beta of. Been working on migrating a high egress service over to k8s for the last month or so. So far so good. I'd recommend you take a look at Linode's offerings.
solatic 7y ago
Not everyone cares about egress costs. Our product is a log processing/analytics tool. Customers pay for a certain period of time for which we'll hold onto their logs for search and analytics, after which point the logs are expired and then deleted. So our ingress-to-egress ratio is very, very high, and egress is practically a rounding error on our AWS bill. I've seen some stupid shit. Like security architects requiring employees to always but always be tunneled in with a VPN, without considering what the egress cost is of an employee surfing YouTube through the VPN. At the end of the day, you are responsible for your architecture choices. As part of your architecture planning, you should design an architecture with maintainable costs, in particular, one where the cost of adding additional users doesn't outweigh the revenue those users bring.
- remus 7y ago
  > I've seen some stupid shit. Like security architects requiring employees to always but always be tunneled in with a VPN, without considering what the egress cost is of an employee surfing YouTube through the VPN. Actually the cost of this is pretty small. I tend to spend a big part of my day tunneling through a VPN hosted on GCP and almost always have a mix of some sort playing in the background on youtube i.e. fairly high use. Total cost is ~£5/month. It's not free, but I suspect most companies have bigger fish to fry.
  - solatic 7y ago
    I have a feeling (no evidence) that YouTube's systems are smart enough to figure out when a given video clip is a static background for the entire clip, or when playback is in a background tab, and drastically cut down on the amount of video data sent. After all, YouTube has high enough egress costs of its own, and anything to optimize down the egress would, in aggregate, save YouTube a bundle of money.
lonelappde 7y ago
Fun fact: a major reason Google survived in the early days was that they negotiated much cheaper ingress than egress, and then ingressed the whole web while serving "10 blue links", at a time when data center clients were expected to run and pay for high egress:ingress ratio servers.
neop1x 7y ago
EDIT: As almostdigital mentioned here, Scaleway supports Kubernetes, their load balancing works the way it should, based in EU (France) and pricing is very nice!

by neop1x 7y ago

atombender 7y ago

For HTTP, an external load balancer/CDN such as CloudFlare or Fastly could easily fulfill that role, though you obviously have to jump through a few more hoops to get there, since it's not built in. But with some of them you also get things like TLS termination, DNS hosting, advanced rewriting, caching, and DDOS protection.

ikiris 7y ago

You seem to think theres some kind of disconnect between egress pricing, and if the network has a functional backbone with load balancing infrastrucutre. If anything, egress pricing is to low in the vast majority of cases, and it really shows when customers start using it.

vasco 7y ago

All of AWS's VPC features are free. You get Security Groups, Subnets, Route Tables, all kinds of shenanigans in a very stable way, with basically no incidents that I can remember. Except those are not free, they're just charged for separately under "egress transfer costs". End of the day, the engineers in the networking teams at AWS still have to get paid and that service probably also needs to run at a profit. When seen under this light, the costs make more sense to me than the usual simple view of "but bandwidth transfer costs are cheaper everywhere else!!!"

jugg1es 7y ago
NAT gateways (a required component if you want real security) cost like $30/month each and you need at least 2 for HA.
- scarface74 7y ago
  Well, I’ve got bad news for you, if you think NAT Gateways are required for “real security”, just wait until you enable IPv6 on AWS.....
  - kawsper 7y ago
    Could you elaborate on this?
    - scarface74 7y ago
      When you enable IPv6, there is no NATing. All of the IP addresses are public. You can use an Egress only Internet Gateway. https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat.html https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat.htm... But technically, your EC2 instance never has a public IP address. You can see for yourself by assigning a public IP address to an ENI attached to your instance and doing an ifconfig. An Internet Gateway is a one to one NAT.
      - jugg1es 7y ago
        Interesting, I did not know this is how IPv6 worked. Need to investigate this more.
nine_k 7y ago
The amount of work required from AWS engineers to set up security groups, routing, etc is not proportional to the amount of traffic your server produces, but is charged as if it were. This requires every egress gigabyte to generate a lot of money to be worth it. If you serve high-traffic applications, like video, you have to be Netflix.

geuis 7y ago

I've been a Linode customer for the better part of a decade. Primary reason is that their egress is quite generous compared to any other provider I've looked at. Also, their customer service is exceptional and over the years the hardware has been upgraded without being charged more money. They're now rolling out a Kubernetes service which I'm in the beta of. Been working on migrating a high egress service over to k8s for the last month or so. So far so good. I'd recommend you take a look at Linode's offerings.

solatic 7y ago

Not everyone cares about egress costs. Our product is a log processing/analytics tool. Customers pay for a certain period of time for which we'll hold onto their logs for search and analytics, after which point the logs are expired and then deleted. So our ingress-to-egress ratio is very, very high, and egress is practically a rounding error on our AWS bill. I've seen some stupid shit. Like security architects requiring employees to always but always be tunneled in with a VPN, without considering what the egress cost is of an employee surfing YouTube through the VPN. At the end of the day, you are responsible for your architecture choices. As part of your architecture planning, you should design an architecture with maintainable costs, in particular, one where the cost of adding additional users doesn't outweigh the revenue those users bring.

remus 7y ago
> I've seen some stupid shit. Like security architects requiring employees to always but always be tunneled in with a VPN, without considering what the egress cost is of an employee surfing YouTube through the VPN. Actually the cost of this is pretty small. I tend to spend a big part of my day tunneling through a VPN hosted on GCP and almost always have a mix of some sort playing in the background on youtube i.e. fairly high use. Total cost is ~£5/month. It's not free, but I suspect most companies have bigger fish to fry.
- solatic 7y ago
  I have a feeling (no evidence) that YouTube's systems are smart enough to figure out when a given video clip is a static background for the entire clip, or when playback is in a background tab, and drastically cut down on the amount of video data sent. After all, YouTube has high enough egress costs of its own, and anything to optimize down the egress would, in aggregate, save YouTube a bundle of money.

lonelappde 7y ago

Fun fact: a major reason Google survived in the early days was that they negotiated much cheaper ingress than egress, and then ingressed the whole web while serving "10 blue links", at a time when data center clients were expected to run and pay for high egress:ingress ratio servers.

neop1x 7y ago

EDIT: As almostdigital mentioned here, Scaleway supports Kubernetes, their load balancing works the way it should, based in EU (France) and pricing is very nice!