12 ms·

Why cloud bandwidth is so obscenely expensive and what you can do about that

154 points by randomint64 3y ago

varsketiz 3y ago
Serious Q for big Cloud spenders - how much have you negiotiated egress down from the list price?
- AaronM 3y ago
  Im not sure you can. At least at the scale we use AWS its a flat % off the top based on our spend
  - dilyevsky 3y ago
    Ime you can when you apply some creative negotiation coupled with a multi-million $ spends. Usually to their peered (“partner”) networks though not generic transit
- nmjohn 3y ago
  If you can sign a yearly bandwidth commit (not sure what the minimum bandwidth requirement is, but 1PB / year may be in the ballpark) - you will get prices that are extremely competitive (maybe like 90%+ off base list pricing?)
  - dilyevsky 3y ago
    You can get it down to ~1c/g at best then if you want cheaper you gotta build your own
    - Aachen 3y ago
      Did you see the sibling comment posted 45 minutes before you saying they pay 0.1c/g at aws when committing to spending 1500$/month?
      - dilyevsky 3y ago
        Did you see my comment in a sibling thread? If this was common then you could earn $$$ via egress arbitrage…
        nmjohn 3y ago
        What would your hypothetical egress arbitrage look like? Keep in mind we are specifically talking about cloudfront bandwidth - so being able to route to an upstream without that upstream also paying for bandwidth likely is not possible.
        dilyevsky 3y ago
        1. Run a proxy in your VPC and redirect all CF requests to someone elses VPC 2. ???? 3. Profit! (assuming you charged them >2.1c/g)
    - nmjohn 3y ago
      In my experience pricing can go far lower than that. The sibling commenter saying $0.001/GB on an 18PB /year commit seems reasonable to me.
      - dilyevsky 3y ago
        You said 90% off which i’ve seen personally and amount to roughly 1c (most of the list starts at like 8-10c in the US). The sibling says they somehow got 90% off that original 90% which seems silly and def not something i would count on
- speedgoose 3y ago
  Not a extreme cloud spender but we have basically free egress on Azure. Well, "waived" for up to 15% of the total monthly bill so we can’t go too crazy. It’s not secret information: https://azure.microsoft.com/en-us/blog/azure-egress-fee-waiver-for-the-academic-community/ https://azure.microsoft.com/en-us/blog/azure-egress-fee-waiv...
- mannyv 3y ago
  $1500/mo commit got our bandwidth down to < $.001/GB in AWS via CloudFront.
  - omarfarooq 3y ago
    So the $23K DDOS bill at AWS from the article would be only $476. Not bad.
AaronM 3y ago
Summary: Use a provider that doesnt charge egress Optimize your egress usage
- ushakov 3y ago
  Hetzner comes to mind…
  - CodesInChaos 3y ago
    Hetzner's peerings aren't the best, especially outside EU/US. Plus they recently intercepted traffic to one of their servers, an attack likely perpetrated by the German state.
    - withinboredom 3y ago
      They have regions outside of Germany too. And IIRC, the people being wire tapped were doing some illegal stuff.
      - qeternity 3y ago
        Also it was a court order. All legit providers have to comply with pen register / trap and trace warrants.
      - radiator 3y ago
        Exactly, they were communicating in the russian language.
        withinboredom 3y ago
        I mean, I have no idea what they were doing but they literally said (in a deleted HN comment) that what they were doing was illegal.
onlyrealcuzzo 3y ago
TL;DR - author claims the hardest part of running a cloud PaaS is networking, thus the cloud providers charge you a ton for egress to outsource the optimization problem to you. It's not clear to me this claim is true - they're in the business of solving exactly this optimization problem - and they have, they're just capturing maximum profit. If it was really the hardest / most costly - then other providers wouldn't be able to offer it so much cheaper. It's clearly a solved problem. This is just the knob the pricing departments found to be the one to dial to juice profits to the max. The article seems like a puff piece to promote other providers. But it could be useful to people shopping for cheap egress.
- zeroCalories 3y ago
  ??? This is obviously true. The bandwidth available in a geographic location is, practically speaking, limited. Roads are cheap to pave, but if there was congestion 24/7 and you paved all you could, then you would need to start rationing road usage. For a business the best way to ration is to charge more. Solving this problem requires a massive amount of engineering work, which would also justify the increased cost.
  - onlyrealcuzzo 3y ago
    And you're not physically limited by how many machines you can put in a data center?
  - weird-eye-issue 3y ago
    How does Cloudflare offer it for free?
    - withinboredom 3y ago
      Um. It’s not. They have a free plan, but it’s subsidized by their paying customers.
      - Syonyk 3y ago
goldinfra 3y ago
Beware of cloud providers that don't "charge for egress" but do charge on a metric that is effectively a proxy for egress. For example, a cloud streaming service that charges for video delivery by per minute streamed is just charging for egress with extra steps.
- buggeryorkshire 3y ago
  Eh? If you're delivering video at scale there's no way anybody is going to do that for free? Whereas with AWS they charge you for traffic between AWS regions last time I checked?
  - Dylan16807 3y ago
    They're not asking for free service, they're just noting that a claim of "free egress" might be a lie.
  - omnimus 3y ago
    Any CDN that doesnt charge for egress (like cloudflare) will deliver your videos without charges.
    - dilyevsky 3y ago
      Delivering video will instantly promote you to their business plan where you are charged
      - omnimus 3y ago
        Afaik its allowed with R2.
      - andrewstuart 3y ago
        The CTO of Cloudflare denied that.
- baq 3y ago
  You see egress with extra steps, I see billing for work relevant to my business and my customers
CodesInChaos 3y ago
I think egress would benefit from some regulation. This would enable other companies to compete with specific AWS services, without competing with AWS as a whole. Though I probably wouldn't price cap egress itself, but rather force amazon to offer cheap peering.
- adventured 3y ago
  Amazon & Co. will always rule for the exact same reason Microsoft Office has/will. It's the combination of services effect creating a natural moat. There's nothing you can do about that other than to break up the large entities, it forms by default unless you do that. It requires a comically enormous burden placed on the entity to break it because of how fantastically strong the combination effect is for customers, cheap peering won't touch it. So long as they don't put too many barriers up (raising the cost through unnecessary regulation to cut out the competition, which would benefit AWS), we'll always have competitive alternatives, including running your own infrastructure inexpensively with companies like Hetzner or Cloudflare (depending on what you need).
  - KronisLV 3y ago
    > So long as they don't put too many barriers up (raising the cost through unnecessary regulation to cut out the competition, which would benefit AWS), we'll always have competitive alternatives, including running your own infrastructure inexpensively with companies like Hetzner or Cloudflare (depending on what you need). I mean, if the article mentions small businesses in particular, their needs are probably met by software running on a few VPSes. As long as they don't want to leverage any advanced features that the PaaS/SaaS providers would give them, most of the affordable providers should be enough: Hetzner, Contabo, Time4VPS, or even Scaleway, DigitalOcean, Vultr and so on. Probably pick a company that has been around for >5 or >10 years instead of something on LowEndBox that might disappear a year or two down the line, but other than that it's likely to be fine (as long as you have backups figured out and have either some redundancy/failover, or just are okay with downtime).
- bnprks 3y ago
  Agree 100%. I've always found it surprising there isn't a requirement for symmetric bandwidth charges. I think it's fine if AWS wants to have expensive bandwidth, but having free ingress + exorbitant egress seems like a transparent attempt at preventing customers from partial migrations to competitors.
api 3y ago
Based on the pricing you see at VPS providers, bare metal providers, and colocation, I have long called BS on large cloud provider bandwidth pricing. It's completely insane, like more than 10X higher than VPSes and more than 100-1000X higher than wholesale (bare metal and colo). Take this for example: https://www.fdcservers.com/ https://www.fdcservers.com/ That company is profitable, which means they have to be paying less than this for bandwidth or at least selling bandwidth at cost and paying less for other things. I'm sure they are overselling but even then this model couldn't work if bandwidth were anywhere even close to what cloud charges for it. (We run a few things there for high bandwidth simple services and it works great and seems to be as advertised. There are many other companies with insanely low pricing on bandwidth, but they're one of the cheapest.) The key to understanding why I really think AWS/GCP/etc. price bandwidth the way they do is that ingress is free and egress is absurdly expensive. I call this "roach motel pricing" after the old roach motel ad: "roaches go in, but they don't come out." The obvious objective is to get data to flow effortlessly in but make it costly to either relay data for edge or endpoint first computation or move data out. The whole pricing structure is to encourage cloud providers to suck in the entire computing ecosystem and host it internally within large data centers, which is their business model. It works very well. The instant I saw this pricing structure when cloud first started to be introduced I immediately understood what was happening here.
- tyingq 3y ago
  Searching for terms like "wholesale bandwidth transit" might be a better reference for comparison of actual cost. The providers you mention are depending somewhat on customers that statistically underutilize what they have and using obscure TOS clauses to chase away heavy users.
  - api 3y ago
    That's the list price. You don't pay the list price if you purchase at high volume. Bare metal and VPS providers are surely very high volume customers. This is perhaps an argument against pure DIY colo, but most small scale colo isn't connected "naked" to the Internet but gets racked in a data center that offers a bandwidth package and does peering for you. The only people who peer directly tend to be clouds, bare metal hosts, CDNs, and ISPs. Even if you have your own IP space and an ASN you are usually behind someone who is bulk purchasing transit and handling peering. To be fair very large customers can negotiate costs down at cloud providers too, but you have to be spending tens of thousands a month to even have a chance at getting their attention for any kind of special deal.
candiodari 3y ago
I love the term "denial of wallet attack".
tedivm 3y ago
I'm kind of amazed that AWS is the cheapest on this list, as they're typically the first example people give when they talk about overcharging for bandwidth. When I worked at Malwarebytes between 2008 and 2014 we measured our bandwidth in petabytes, but it was still only about $25k/month. This was obviously a custom contract, but it really is interesting how bandwidth pricing has remained relatively static over the last decade.
- andrenotgiant 3y ago
  Yeah I think the list is only highlighting the worst offenders. It would be more useful to also include the cheap VPS providers mentioned later, that would be: - Scaleway: $0 - Hetzner: ~$490 _(article says bandwidth is free but I think above 20TB it's charged at 1 Euro/TB)_ - DigitalOcean/Vultr: ~$5k depending on compute usage _(you get a free bandwidth allowance based on compute)_
  - Aachen 3y ago
    How far does the FUP go on that 0$ offer, though? It doesn't say on the pricing page, VPS info page, or the various pdf documents with terms and conditions. Maybe you need to sign up and get to the ordering page before you can see the conditions applying to your VPS or maybe I overlooked it, but this is where I stopped looking
    - andrenotgiant 3y ago
      Great question. "Free Bandwidth" is almost worse than cheap bandwidth because in reality it means: "Secret bandwidth pricing/limiting"
- hodgesrm 3y ago
  Just to be clear, was Malwarebytes using AWS?
  - tedivm 3y ago
    We used a combination of CDNs and balanced according to region (some were better in Australia than others, for instance).
CathalMullan 3y ago
A word of warning when it comes to Vultr. While they are listed as a member of the "Bandwidth Alliance", they don't discount egress automatically. You need to request the discount from support. See: https://www.answeroverflow.com/m/1118148928915918990 https://www.answeroverflow.com/m/1118148928915918990
Syonyk 3y ago
What you can do about it is go back to the early 2000s and rack up your own boxes in a colo, with some reasonable unmetered bandwidth allocations, and skip the whole "cloud" nonsense. My personal box is a 6C/12T Xeon from a few years back, 128GB ECC RAM, and about 24TB usable disk plus 2TB NVMe, for running a lot of other VMs that I run actual services in (remote backup, my web presence, communications, game servers, the works). Then a 25Mbit/100Mbit burst connection at a local datacenter for a bit over $100/mo. A 12vCPU/96GB RAM GCE N2 box with similar specs as my box is about $1800/mo. So... given that "a few months of cloud spend" buys my system, I genuinely don't see the case for "cloud" if you're using a lot of data, or a lot of RAM, or a lot of CPU. Yes, it's managed and convenient, but it's horridly expensive for what you get compared to "owning your own hardware." I'd been spending quite a bit more a month on various cloud instances to do the same thing I do with my server.
- jauntywundrkind 3y ago
  If you want bandwidth, vps are quite cheap. 100Mbps averages to a a little over 32TB/mo. There there's tons of low CPU low ram systems with 32TB/mo transfer for cheap. Now under $10/mo: https://lowendbox.com/blog/hosteons-they-survived-dedipath-and-are-thriving-in-six-locations/ https://lowendbox.com/blog/hosteons-they-survived-dedipath-a... People should rack their own rpi5 or nucs, at least. There's various providers for both. And consider moving up from there. The remaining folks who will accept & rack a mid tower box for under $75 are saints.
  - Syonyk 3y ago
    Sure, I'll use a $5/mo DO Droplet (maybe $6/mo now for the new ones?) and their 1TB transfer pool if I just need raw bandwidth, but, tbh, not a lot of stuff is just pure bandwidth needs. And "capable cloud boxes for various purposes" start adding up. If I just wanted to shovel out a static website, sure. But I can do that a variety of other ways too. Once you start running "real" things, CPU and RAM use start climbing - mostly RAM in my experience. I run a Matrix homeserver - Synapse really struggle with <4GB RAM on the system, so it's one of a few things on a VM with 16GB. I've got a Minecraft and Valheim server running for friends and family - 8-12GB is reasonable for both. You certainly can do it in less, and I've done it, but a lot of problems go away when you can just throw RAM at the problem. Also, most of the cheap VPS boxes are badly IOPS limited on disk - unless you start paying a good bit more for faster disk. I split my box between a 2TB NVMe mirror for VM root drives and "high write traffic" stuff (the gameserver drives and the like), and my large, slow, spinning rust array for backup storage and "big stuff." Again, you can do it with cloud, but storage adds up in a hurry - 24TB of "slow" disk is a grand a month. It just depends on what you need. I really like cloud boxes for light use "toy" boxes, and if that's what you want, they're great. I just cannot wrap my head around the merits of spending a lot of money for "serious cloud compute" when I can build and rack a similarly capable box for 2-3 months of the cloud spend on a similar system. I understand opex vs capex issues for business, but, man, it sure seems stupid to me. Then again, I'm the kind of person who's had a box racked up a colo (or somewhere similar - random basements on a good connection are nice) for the majority of my adult life. So I'm pretty clearly biased. But I did do a few years of "Hey, cloud, this is cool" - and eventually sat down, evaluated my needs and the costs, and went back to a colo'd box.
mannyv 3y ago
Cloud bandwidth is cheap if you commit to spending a certain amount per month. Our bandwidth is less than .001/GB with like a $1500 monthly commitment in AWS. Fastly was even cheaper (which is why we're using fastly). Call you AM and talk. It's not hard, and it'll save you a ton of money.
- kiwijamo 3y ago
  Is that something generally avaliable to customers? I've done some searches and can't find anything other than savings plans for EC2 etc.
  - mannyv 3y ago
    On AWS you have to call and ask them. We got quotes from AWS, Fastly, and Akamai. All of them are substantially cheaper than the public rates.
- dilyevsky 3y ago
  We had a commit that was like three orders of magnitude more and our price was 1-2c for cdn and 2c for tier1 cloud (but only “partner” networks, the rest was list). That was after us threatening to leave (and having capability to do so). So either you are mistaken or your am really really likes you.
  - mannyv 3y ago
    No, it's your people suck at negotiating. Again, we got quotes from Fastly, Akamai, and AWS. Our commit is/was $1500/month (we're on fastly now). Fastly gave us like 6 month free to switch, both bandwidth and requests.
    - dilyevsky 3y ago
      So you mean to tell me you negotiated roughly -99% their list with a 1.5k commit?
      - 3y ago
proteal 3y ago
One thing to keep in mind - if you accidentally run up a bill because of a mistake, there’s a good chance you can reach out to support and they will credit your account. $5k may be a lot to an individual, but the cloud provider’s costs for that service are significantly less and they shouldn’t mind forgiving the charge. Case in point, when the author deleted his account and switched providers, the old host didn’t chase him for the charges.
- swatcoder 3y ago
  That's been true during recent flush years, where money was cheap and nobody really cared, but there are signs all over the place that the attitude is changing. Everything that used to be free is being monetized and everything that was monetized is seeing increased prices. We should expect that these sort of implicit affordances are being trimmed as well, or at least that they're likely to be soon.
  - _jal 3y ago
    Yeah, I rather suspect they'll be much less forgiving of this sort of thing in the future. All that takes is a lean quarter and some MBA to coin a term for rookie mistake-windfalls, and they will come to feel entitled to revenue from it.
    - bcrosby95 3y ago
      Bob, I see the collateral spend of your clients is down this quarter. Can we see that bump up a bit? Mmmmkay that would be great. Thanks. Good talk.
- 0cf8612b2e1e 3y ago
  That’s still a lot of hopes and prayers you do not end in financial ruin. Instead, the cloud providers could offer prepaid credits/billing maximums to let individuals sleep at night. I would love to run my side project off of AWS to gain the experience, but no way I feel comfortable with a potentially unlimited liability because I configured something poorly. My joke project can take the uptime hit if it means I know that I will never spend more than $X to host it.
tinrab 3y ago
Is there a way to protect against the "Denial of wallets" attack?
- mcherm 3y ago
  Yes, there is a way to protect against that attack! All you need to do is have triggers in place that will shut down or slow down your service when the costs are exceeding some amount. (If you get a moment of viral growth, you can always disable the trigger.) Unfortunately, AWS makes it extremely difficult to build such a trigger, and I'm not sure about other cloud providers.
  - mlhpdx 3y ago
    There’s always rate limiting. Cloudfront supports it, API gateway support it, and it’s super easy to set up.
stickfigure 3y ago
Cloudflare R2 (their S3 clone) has free egress. We are in the middle of migrating about 5TB of fairly active data from Google Cloud Storage to R2. For years now, most of our cloud bill has been egress from GCS. Activity spiked recently and doing something about it became urgent. R2 has an S3-compatible API (we're using the Amazon-provided S3 Java SDK). Storage is 25% cheaper than GCS standard. Making our code multi-provider was pretty easy. I'm actually looking forward to next month's hosting bill. I feel like I'm slowly getting pushed off of GCP due to pricing. Last year we moved some stateless image processing infrastructure to AWS Elastic Beanstalk because RAM is much, much cheaper there.
- chpatrick 3y ago
  It's a bit worrying though that one day they might decide to not leave the money on the table.
- Aachen 3y ago
  General question, can we just call object storage object storage? If there's going to be another provider after already remembering S3, B2, now R2, on top of the various aliases for products like virtual servers, shared hosting, VLANs, and other things people are presenting as new products with the most random of abbreviations, I'm really going to lose track.
  - namtab00 3y ago
    we need something like [this](https://www.azureperiodictable.com/ https://www.azureperiodictable.com/), but across all cloud providers
- andersa 3y ago
  You kinda pay in reduced performance. For example, civitai uses r2 for delivering the images, and it frequently doesn't load at all for me or very slowly. When your users have to use a VPN to move their traffic to another Cloudflare data center so the site actually loads, something is wrong...
  - 3y ago
tekkk 3y ago
I wonder is There also is Hetzner and their dedicated servers. You pay for a server with a 1Gb/s connection and all egress is free. Unfortunately, I can't recommend them as each time I've tried to create an account, their abuse detection systems banned my account even before I had the occasion to enter a payment method. related to I've myself won the jackpot in the past on Netlify where I forgot to put a sleep in a while (true) loop, making my script flood my website and generate a lot of traffic. For the story, I quickly migrated the website to another provider, deleted my account, and never had to pay the bill :) Is there some kind of blacklist if you do something like that? But I do know Hetzner has some extremely strict policies regarding account creation.
- hatware 3y ago
  [dead]
- derefr 3y ago
  > Is there some kind of blacklist if you do something like that? No; but oddly enough, the people who "do things like that" generally tend to come from certain countries — I guess countries with cultures that don't place much weight on the concept of "incurring a debt of honor" by consuming someone else's resources without them ever knowing about it. So most systems don't generally need a big, manually curated and ever-growing blacklist; they just need to block registrations from IP addresses / ASNs of ISPs headquartered in these countries; and/or block payment attempts from credit cards issued by banks headquartered in these countries. That immediately stops 90% of such abuse. And of the remaining 10%, half of it is still people from those same countries — just using foreign IPs through residential-botnet VPNs, and stolen credit cards they purchased on scammer forums. Blocking these is a bit of an art, but it's possible: there's always patterns to the requests themselves, often because those same scammers try to solve all their problems with money, and so have also purchased scam-site kits to run on the hosting they acquire — things like cryptocurrency "drainers." If you're a VPS hosting provider, you can just detect these by the SHAs of the files; if you're a dedicated hosting provider with no access to customers' disks, it's still pretty easy to pick these out by the outbound signature of the network traffic they generate — as they almost always rely on making requests to particular third-party SaaS information systems, that you can turn into an IDS detection fingerprint. (I'm personally in a different position in this ecosystem — my company operates one of the public informational SaaS services that these scam-site kits like to use. From my company's perspective, these scammers are perfectly normal paying customers, not intent to scam us... but we don't want these people as customers, so we still detect this fraudware by the fingerprint it makes in our API request logs, and permaban the users who deploy such kits by every fingerprinting metric we can.) --- Though, on another note, I suppose you could call the observational "IP reputation" metrics gathered by providers like https://www.ipqualityscore.com/ https://www.ipqualityscore.com/ something like a blacklist — and I'm sure hosting providers like Hetzner check your "IP score" before letting you register. But these aren't blacklists in the sense of being manually curated. Instead, what these providers curate is something like a distributed version of an SSHGuard blocklist: a bunch of the provider's own "observer nodes", all over the world, observe what IPs are hitting them with DDoSes and other botnet-like activities, and consider these IPs temporarily compromised for as long as that activity persists (because any device infected by a botnet can potentially be repurposed as a part of a residential-proxy VPN network — and that means that any traffic observed to come from such a device, can't be trusted to be originating from that device.) IIRC, these providers will also "go undercover" to buy access to both commercial and residential-botnet VPNs; cycle through them to find out what all the available exit-node IP addresses are from a client's perspective — and then mark all these as compromised as well.
lowbloodsugar 3y ago
Do you have time, or do you have money? If you have time, then build some boxes and put them in a colo. If you have money then negotiate a cloud contract. Even if you have time, it might be better to start prototyping on a free tier on the cloud, and then maybe go colo, and then back to cloud.
hlandau 3y ago
>The second, less known, reason is that between compute, storage and network, network is actually the hardest part to scale for cloud providers. Thus they need to find a way to incentive their customers to optimize their network usage by themselves. And what better motivation to optimize something than taking a huge chunk of your profits? This doesn't really make sense as an explanation. Cloud networking has two components: 1. The network connecting the datacentres to the internet (the datacentre edge infrastructure). 2. The network connecting the servers and virtual machines to that edge. 1 is a solved problem and the variable costs are going to be based on any transit connections used. What cloud providers charge for bandwidth here is out of proportion to that, and they don't bill for ingress anyway, so this makes no sense. 2 is a technically more complex area, except it's very clear that getting people to use this infrastructure less is not the motivation because cloud providers don't charge for intra-DC traffic. You can ferry data around between servers in the same AZ as much as you like and not pay anything. The same is true of transferring data to and from e.g. S3 and EC2. In other words, the more technically complex and sophisticated network component is the one which isn't billed at all. Meanwhile, boring internet connectivity is billed at an exorbitant rate, but only in one direction. It's really quite obvious at this point that the motivation is vendor lock-in. You are penalised if you take data out of a big cloud provider but not to put data in. This means as soon as you put anything in AWS you are suddenly motivated to do as much in AWS as possible. This is fundamentally anti-competitive. If AWS starts offering some infrastructure service, it can offer free bandwidth to that service for its own customers, but some third party providing a comparable service can't, at least unless it also starts using AWS. But that's not even the best of it. If you want even better proof that these charges are BS, just look at Snowball. This is a service where you can transfer data to S3 by having AWS mail you a hard drive. Amazingly, they have the gall to charge per GB for this in addition to the fixed fee you pay... but only for exporting data. In other words, I can pay $150 and get loaned a SSD, fill it up with 14TB of data and import that data to S3. If I pay $300 to get data out of S3, I also pay $0.03/GB. Internet transit is not even involved here! It is pure anti-competitive lock-in. It's also really against the principles of the internet, in that the internet couldn't have developed as it did if we accepted different billing for communicating with different IPs (can you imagine "I don't want to send traffic to a foreign IP, I'll pay international rates!"). Yet cloud providers work on this principle. If you think about it it is effectively a net neutrality violation, just not on the part of residential ISPs - they charge you less to communicate with their own services than with others (zero-rating).
Obscurity4340 3y ago
Can anyone take a crack at why the hell you need a company and its computer farm 3000 miles away to merey sync your calendar, CalDav or whatever it is between two devices that are oerfectly capable of taking to each other or with a local-first mediating process or tool? I don't get what is about syncing things (particularly the basics like Contacts, Calendar/Reminders that requires you to send the data thousands of miles away and back when its all in the same room
- deleted 3y ago
  [deleted]
- ihaveajob 3y ago
  Put simply: There's no incentive for most organizations to implement this like this.
- bombcar 3y ago
  Because it's simpler for them to develop, because they make the server the source of truth, and then they can sell it to you. And for most people, it works fine, but when you hit edge cases you're dead in the water.
- crazygringo 3y ago
  Here's the answer: because it's not written for just 2 devices. It's written to sync however many devices you want -- 3 or 4 or 5 or whatever. Both your phones, your tablet, your laptop, and your watch. And it has to sync when your devices might never actually be turned on at the same time. And you want it to sync when you're traveling, and you don't want to open your home firewall for security reasons. So it requires a single, centralized, always-on server as the source of truth. And given the speed of light, whether that server is 3 miles or 3,000 miles away is largely irrelevant. And writing additional code to sync directly when devices are on the same LAN and turned on at the same time, as a special case, is not only entirely redundant -- but you still have to sync with the server, since you generally want the information backed up as well in case you lose your devices, or they're stolen, or damaged. Using your examples of Contacts and Calendar, those are often some of the most important things in someone's life to keep synced remotely. Does that answer your question?
mlhpdx 3y ago
I’ve just never seen bandwidth be a significant part of the bill for any sass I’ve built/worked for. I’m super curious about this. What kind of sass businesses use huge amounts of bandwdith but don’t provide enough commercial value to the customers to make it viable? Can folks with first-hand experience please share here?
- danpalmer 3y ago
  Bandwidth is rarely an issue if you're transferring mostly text, often an issue if you're transferring images (except for highly cacheable static images), and always an issue if you're transferring video or game assets. That's just how it works because the orders of magnitude differ so much. An image resource can easily be 100x some text (or HTML or API response), and a video can easily be 1000x an image. Another factor is cacheability. If you're a B2B SaaS business you'll have few clients and can take advantage of caching much more typically. If you're B2C that flips and caching might not provide much benefit. Anecdotally, on a medium sized ecommerce store, AWS egress for product images was a huge cost.
  - mlhpdx 3y ago
    Thank you, the e-commerce example is definitely something I hadn’t thought about. Having high quality imagery helps make sales but comes at a cost. Looking around at some sites I buy from, I think I can squint and see this “out in plain sight”.
    - danpalmer 3y ago
      In fact an interesting detail of the ecommerce image situation is that you can somewhat trade-off bandwidth and compute. Creating resized images is expensive (we had ~2m unique images viewed per month), but if you can resize an image down and compress it well you might save 80% of the bandwidth on it, and deliver a better UX due to faster images. We did a bit of modelling to figure out how many sizes we wanted to deliver in order to get the best compute/bandwidth trade-off. This can be tricky for many companies but we were in a weird position because we had a large catalog compared to order volume, because of how the business worked. Video is an even better one. While we didn't have to deal with it in ecommerce, so many social applications are built around video. That doesn't necessarily mean YouTube/TikTok/etc, but also smaller things like BeReal who I'd guess are built entirely on cloud infra, even embedded videos in Mastodon posts. A similar thing could even be said about Podcasting/audio – not as bad as video, but there are specialist podcast hosts for a reason, and it's not that they do a great job, it's because they have cheap bandwidth.
simonsarris 3y ago
I'm surprised this about egress without mentioning the new Sippy: https://developers.cloudflare.com/r2/data-migration/sippy/ https://developers.cloudflare.com/r2/data-migration/sippy/ It allows you to incrementally migrate off of providers like S3 and onto the egress-free Cloudflare R2. Very clever idea.
lofaszvanitt 3y ago
It's the idiot tax.
CSSer 3y ago
There are a couple things this article glosses over. With Vercel vs. Netlify, be mindful of the many other fees they charge besides bandwidth. I wouldn’t be surprised if the costs are comparable when that’s accounted for. Sometimes the answer with these things is just to be diligent. Don’t be surprised at what you get if you stick your hand in a meat grinder. Put up guards in your CI process if you have to. The other thing is that I’m not sure how willing I am to jump to endorse an efficient cache policy as a blanket prescription like they seem to. I suppose I’m not opposed to the advice, but it has very strong “draw the owl” vibes. Project requirements vary, and I’d even argue that sometimes minimal or no caching is wise when requirements are subject to change or you’re doing a lot of static deployments because the savings can sometimes not be worth the headache. End-users are very demanding nowadays. Sometimes “it’ll roll over soon” isn’t good enough. Caching is tricky. Make sure you really get it right in addition to doing the math on the cost-savings if you go down that road.
TheIronYuppie 3y ago
Hey everyone! I'm David Aronchick - co-founder of Bacalhau[1] - and we are specifically building an open source platform to help with this. Basically, we allow you to do compute over data, making it much easier to schedule jobs (container, WASM, arbitrary binary) to where the data is being stored, even over irregular/unreliable networks. We were able to show 90%+ savings in things like log aggregations. Happy to answer any questions if you'd like! David Aronchick aronchick@expanso.io [1] https://github.com/bacalhau-project/bacalhau https://github.com/bacalhau-project/bacalhau
deleted 3y ago
[deleted]
TacticalCoder 3y ago
> There also is Hetzner and their dedicated servers. You pay for a server with a 1Gb/s connection and all egress is free. There's also OVH and they have things like 1 Gbit/s guaranteed, up to 5 Gbit/s (not guaranteed) EPYC 7313 servers with 25,50 or 100 Gbit/s guaranteed private (from OVH servers to OVH servers) for... 200 EUR / month. Re-using the example from TFA, you could max the bandwith of such a server 24/7, be "downgraded" from 5 Gbit/s to 1 Gbit/s guaranteed and yet it'd cost you... 200 EUR/month. Versus hundreds of thousands of egress bill in the cloud. 200 EUR / month vs hundred of thousands. But yay. Go cloud!
- tutfbhuf 3y ago
  Combine the two. Set up a large nginx reverse proxy with cache on the 200 EUR/month machine in Hetzner or OVH and serve 99.9% of the traffic from this location (js, css, images, videos, etc.). Use the cloud for convenience in managing the complex parts in the backend.
  - tiernano 3y ago
    Why not just go Cloudflare then? In our sites, most of what Cloudflare caches is js, cs, images, etc...
nikolqy 3y ago
Oracle Cloud offers 10TB free bandwidth per month and is part of the bandwidth alliance I think with Cloudflare. Edgecast (Edgio) is free as far as I can tell, except you pay for support if you want it and some extra features. Datapacket has unmetered bandwidth for example. There are plenty of solutions, but they aren’t very well known because of the dominance AWS and others have. Not to mention, it might be cheaper to just use direct connect with a cloud provider or just grab an ASN and host stuff in a local datacenter.

154 points by randomint64 3y ago

varsketiz 3y ago

Serious Q for big Cloud spenders - how much have you negiotiated egress down from the list price?

AaronM 3y ago
Im not sure you can. At least at the scale we use AWS its a flat % off the top based on our spend
- dilyevsky 3y ago
  Ime you can when you apply some creative negotiation coupled with a multi-million $ spends. Usually to their peered (“partner”) networks though not generic transit
nmjohn 3y ago
If you can sign a yearly bandwidth commit (not sure what the minimum bandwidth requirement is, but 1PB / year may be in the ballpark) - you will get prices that are extremely competitive (maybe like 90%+ off base list pricing?)
- dilyevsky 3y ago
  You can get it down to ~1c/g at best then if you want cheaper you gotta build your own
  - Aachen 3y ago
    Did you see the sibling comment posted 45 minutes before you saying they pay 0.1c/g at aws when committing to spending 1500$/month?
    - dilyevsky 3y ago
      Did you see my comment in a sibling thread? If this was common then you could earn $$$ via egress arbitrage…
      - nmjohn 3y ago
        What would your hypothetical egress arbitrage look like? Keep in mind we are specifically talking about cloudfront bandwidth - so being able to route to an upstream without that upstream also paying for bandwidth likely is not possible.
        dilyevsky 3y ago
        1. Run a proxy in your VPC and redirect all CF requests to someone elses VPC 2. ???? 3. Profit! (assuming you charged them >2.1c/g)
  - nmjohn 3y ago
    In my experience pricing can go far lower than that. The sibling commenter saying $0.001/GB on an 18PB /year commit seems reasonable to me.
    - dilyevsky 3y ago
      You said 90% off which i’ve seen personally and amount to roughly 1c (most of the list starts at like 8-10c in the US). The sibling says they somehow got 90% off that original 90% which seems silly and def not something i would count on
speedgoose 3y ago
Not a extreme cloud spender but we have basically free egress on Azure. Well, "waived" for up to 15% of the total monthly bill so we can’t go too crazy. It’s not secret information: https://azure.microsoft.com/en-us/blog/azure-egress-fee-waiver-for-the-academic-community/ https://azure.microsoft.com/en-us/blog/azure-egress-fee-waiv...
mannyv 3y ago
$1500/mo commit got our bandwidth down to < $.001/GB in AWS via CloudFront.
- omarfarooq 3y ago
  So the $23K DDOS bill at AWS from the article would be only $476. Not bad.

AaronM 3y ago

Summary: Use a provider that doesnt charge egress Optimize your egress usage

ushakov 3y ago
Hetzner comes to mind…
- CodesInChaos 3y ago
  Hetzner's peerings aren't the best, especially outside EU/US. Plus they recently intercepted traffic to one of their servers, an attack likely perpetrated by the German state.
  - withinboredom 3y ago
    They have regions outside of Germany too. And IIRC, the people being wire tapped were doing some illegal stuff.
    - qeternity 3y ago
      Also it was a court order. All legit providers have to comply with pen register / trap and trace warrants.
    - radiator 3y ago
      Exactly, they were communicating in the russian language.
      - withinboredom 3y ago
        I mean, I have no idea what they were doing but they literally said (in a deleted HN comment) that what they were doing was illegal.

onlyrealcuzzo 3y ago

TL;DR - author claims the hardest part of running a cloud PaaS is networking, thus the cloud providers charge you a ton for egress to outsource the optimization problem to you. It's not clear to me this claim is true - they're in the business of solving exactly this optimization problem - and they have, they're just capturing maximum profit. If it was really the hardest / most costly - then other providers wouldn't be able to offer it so much cheaper. It's clearly a solved problem. This is just the knob the pricing departments found to be the one to dial to juice profits to the max. The article seems like a puff piece to promote other providers. But it could be useful to people shopping for cheap egress.

zeroCalories 3y ago
??? This is obviously true. The bandwidth available in a geographic location is, practically speaking, limited. Roads are cheap to pave, but if there was congestion 24/7 and you paved all you could, then you would need to start rationing road usage. For a business the best way to ration is to charge more. Solving this problem requires a massive amount of engineering work, which would also justify the increased cost.
- onlyrealcuzzo 3y ago
  And you're not physically limited by how many machines you can put in a data center?
- weird-eye-issue 3y ago
  How does Cloudflare offer it for free?
  - withinboredom 3y ago
    Um. It’s not. They have a free plan, but it’s subsidized by their paying customers.
    - Syonyk 3y ago

goldinfra 3y ago

Beware of cloud providers that don't "charge for egress" but do charge on a metric that is effectively a proxy for egress. For example, a cloud streaming service that charges for video delivery by per minute streamed is just charging for egress with extra steps.

buggeryorkshire 3y ago
Eh? If you're delivering video at scale there's no way anybody is going to do that for free? Whereas with AWS they charge you for traffic between AWS regions last time I checked?
- Dylan16807 3y ago
  They're not asking for free service, they're just noting that a claim of "free egress" might be a lie.
- omnimus 3y ago
  Any CDN that doesnt charge for egress (like cloudflare) will deliver your videos without charges.
  - dilyevsky 3y ago
    Delivering video will instantly promote you to their business plan where you are charged
    - omnimus 3y ago
      Afaik its allowed with R2.
    - andrewstuart 3y ago
      The CTO of Cloudflare denied that.
baq 3y ago
You see egress with extra steps, I see billing for work relevant to my business and my customers

CodesInChaos 3y ago

I think egress would benefit from some regulation. This would enable other companies to compete with specific AWS services, without competing with AWS as a whole. Though I probably wouldn't price cap egress itself, but rather force amazon to offer cheap peering.

adventured 3y ago
Amazon & Co. will always rule for the exact same reason Microsoft Office has/will. It's the combination of services effect creating a natural moat. There's nothing you can do about that other than to break up the large entities, it forms by default unless you do that. It requires a comically enormous burden placed on the entity to break it because of how fantastically strong the combination effect is for customers, cheap peering won't touch it. So long as they don't put too many barriers up (raising the cost through unnecessary regulation to cut out the competition, which would benefit AWS), we'll always have competitive alternatives, including running your own infrastructure inexpensively with companies like Hetzner or Cloudflare (depending on what you need).
- KronisLV 3y ago
  > So long as they don't put too many barriers up (raising the cost through unnecessary regulation to cut out the competition, which would benefit AWS), we'll always have competitive alternatives, including running your own infrastructure inexpensively with companies like Hetzner or Cloudflare (depending on what you need). I mean, if the article mentions small businesses in particular, their needs are probably met by software running on a few VPSes. As long as they don't want to leverage any advanced features that the PaaS/SaaS providers would give them, most of the affordable providers should be enough: Hetzner, Contabo, Time4VPS, or even Scaleway, DigitalOcean, Vultr and so on. Probably pick a company that has been around for >5 or >10 years instead of something on LowEndBox that might disappear a year or two down the line, but other than that it's likely to be fine (as long as you have backups figured out and have either some redundancy/failover, or just are okay with downtime).
bnprks 3y ago
Agree 100%. I've always found it surprising there isn't a requirement for symmetric bandwidth charges. I think it's fine if AWS wants to have expensive bandwidth, but having free ingress + exorbitant egress seems like a transparent attempt at preventing customers from partial migrations to competitors.

api 3y ago

Based on the pricing you see at VPS providers, bare metal providers, and colocation, I have long called BS on large cloud provider bandwidth pricing. It's completely insane, like more than 10X higher than VPSes and more than 100-1000X higher than wholesale (bare metal and colo). Take this for example: https://www.fdcservers.com/ https://www.fdcservers.com/ That company is profitable, which means they have to be paying less than this for bandwidth or at least selling bandwidth at cost and paying less for other things. I'm sure they are overselling but even then this model couldn't work if bandwidth were anywhere even close to what cloud charges for it. (We run a few things there for high bandwidth simple services and it works great and seems to be as advertised. There are many other companies with insanely low pricing on bandwidth, but they're one of the cheapest.) The key to understanding why I really think AWS/GCP/etc. price bandwidth the way they do is that ingress is free and egress is absurdly expensive. I call this "roach motel pricing" after the old roach motel ad: "roaches go in, but they don't come out." The obvious objective is to get data to flow effortlessly in but make it costly to either relay data for edge or endpoint first computation or move data out. The whole pricing structure is to encourage cloud providers to suck in the entire computing ecosystem and host it internally within large data centers, which is their business model. It works very well. The instant I saw this pricing structure when cloud first started to be introduced I immediately understood what was happening here.

tyingq 3y ago
Searching for terms like "wholesale bandwidth transit" might be a better reference for comparison of actual cost. The providers you mention are depending somewhat on customers that statistically underutilize what they have and using obscure TOS clauses to chase away heavy users.
- api 3y ago
  That's the list price. You don't pay the list price if you purchase at high volume. Bare metal and VPS providers are surely very high volume customers. This is perhaps an argument against pure DIY colo, but most small scale colo isn't connected "naked" to the Internet but gets racked in a data center that offers a bandwidth package and does peering for you. The only people who peer directly tend to be clouds, bare metal hosts, CDNs, and ISPs. Even if you have your own IP space and an ASN you are usually behind someone who is bulk purchasing transit and handling peering. To be fair very large customers can negotiate costs down at cloud providers too, but you have to be spending tens of thousands a month to even have a chance at getting their attention for any kind of special deal.

candiodari 3y ago

I love the term "denial of wallet attack".

tedivm 3y ago

I'm kind of amazed that AWS is the cheapest on this list, as they're typically the first example people give when they talk about overcharging for bandwidth. When I worked at Malwarebytes between 2008 and 2014 we measured our bandwidth in petabytes, but it was still only about $25k/month. This was obviously a custom contract, but it really is interesting how bandwidth pricing has remained relatively static over the last decade.

andrenotgiant 3y ago
Yeah I think the list is only highlighting the worst offenders. It would be more useful to also include the cheap VPS providers mentioned later, that would be: - Scaleway: $0 - Hetzner: ~$490 _(article says bandwidth is free but I think above 20TB it's charged at 1 Euro/TB)_ - DigitalOcean/Vultr: ~$5k depending on compute usage _(you get a free bandwidth allowance based on compute)_
- Aachen 3y ago
  How far does the FUP go on that 0$ offer, though? It doesn't say on the pricing page, VPS info page, or the various pdf documents with terms and conditions. Maybe you need to sign up and get to the ordering page before you can see the conditions applying to your VPS or maybe I overlooked it, but this is where I stopped looking
  - andrenotgiant 3y ago
    Great question. "Free Bandwidth" is almost worse than cheap bandwidth because in reality it means: "Secret bandwidth pricing/limiting"
hodgesrm 3y ago
Just to be clear, was Malwarebytes using AWS?
- tedivm 3y ago
  We used a combination of CDNs and balanced according to region (some were better in Australia than others, for instance).

CathalMullan 3y ago

A word of warning when it comes to Vultr. While they are listed as a member of the "Bandwidth Alliance", they don't discount egress automatically. You need to request the discount from support. See: https://www.answeroverflow.com/m/1118148928915918990 https://www.answeroverflow.com/m/1118148928915918990

Syonyk 3y ago

What you can do about it is go back to the early 2000s and rack up your own boxes in a colo, with some reasonable unmetered bandwidth allocations, and skip the whole "cloud" nonsense. My personal box is a 6C/12T Xeon from a few years back, 128GB ECC RAM, and about 24TB usable disk plus 2TB NVMe, for running a lot of other VMs that I run actual services in (remote backup, my web presence, communications, game servers, the works). Then a 25Mbit/100Mbit burst connection at a local datacenter for a bit over $100/mo. A 12vCPU/96GB RAM GCE N2 box with similar specs as my box is about $1800/mo. So... given that "a few months of cloud spend" buys my system, I genuinely don't see the case for "cloud" if you're using a lot of data, or a lot of RAM, or a lot of CPU. Yes, it's managed and convenient, but it's horridly expensive for what you get compared to "owning your own hardware." I'd been spending quite a bit more a month on various cloud instances to do the same thing I do with my server.

jauntywundrkind 3y ago
If you want bandwidth, vps are quite cheap. 100Mbps averages to a a little over 32TB/mo. There there's tons of low CPU low ram systems with 32TB/mo transfer for cheap. Now under $10/mo: https://lowendbox.com/blog/hosteons-they-survived-dedipath-and-are-thriving-in-six-locations/ https://lowendbox.com/blog/hosteons-they-survived-dedipath-a... People should rack their own rpi5 or nucs, at least. There's various providers for both. And consider moving up from there. The remaining folks who will accept & rack a mid tower box for under $75 are saints.
- Syonyk 3y ago
  Sure, I'll use a $5/mo DO Droplet (maybe $6/mo now for the new ones?) and their 1TB transfer pool if I just need raw bandwidth, but, tbh, not a lot of stuff is just pure bandwidth needs. And "capable cloud boxes for various purposes" start adding up. If I just wanted to shovel out a static website, sure. But I can do that a variety of other ways too. Once you start running "real" things, CPU and RAM use start climbing - mostly RAM in my experience. I run a Matrix homeserver - Synapse really struggle with <4GB RAM on the system, so it's one of a few things on a VM with 16GB. I've got a Minecraft and Valheim server running for friends and family - 8-12GB is reasonable for both. You certainly can do it in less, and I've done it, but a lot of problems go away when you can just throw RAM at the problem. Also, most of the cheap VPS boxes are badly IOPS limited on disk - unless you start paying a good bit more for faster disk. I split my box between a 2TB NVMe mirror for VM root drives and "high write traffic" stuff (the gameserver drives and the like), and my large, slow, spinning rust array for backup storage and "big stuff." Again, you can do it with cloud, but storage adds up in a hurry - 24TB of "slow" disk is a grand a month. It just depends on what you need. I really like cloud boxes for light use "toy" boxes, and if that's what you want, they're great. I just cannot wrap my head around the merits of spending a lot of money for "serious cloud compute" when I can build and rack a similarly capable box for 2-3 months of the cloud spend on a similar system. I understand opex vs capex issues for business, but, man, it sure seems stupid to me. Then again, I'm the kind of person who's had a box racked up a colo (or somewhere similar - random basements on a good connection are nice) for the majority of my adult life. So I'm pretty clearly biased. But I did do a few years of "Hey, cloud, this is cool" - and eventually sat down, evaluated my needs and the costs, and went back to a colo'd box.

mannyv 3y ago

Cloud bandwidth is cheap if you commit to spending a certain amount per month. Our bandwidth is less than .001/GB with like a $1500 monthly commitment in AWS. Fastly was even cheaper (which is why we're using fastly). Call you AM and talk. It's not hard, and it'll save you a ton of money.

kiwijamo 3y ago
Is that something generally avaliable to customers? I've done some searches and can't find anything other than savings plans for EC2 etc.
- mannyv 3y ago
  On AWS you have to call and ask them. We got quotes from AWS, Fastly, and Akamai. All of them are substantially cheaper than the public rates.
dilyevsky 3y ago
We had a commit that was like three orders of magnitude more and our price was 1-2c for cdn and 2c for tier1 cloud (but only “partner” networks, the rest was list). That was after us threatening to leave (and having capability to do so). So either you are mistaken or your am really really likes you.
- mannyv 3y ago
  No, it's your people suck at negotiating. Again, we got quotes from Fastly, Akamai, and AWS. Our commit is/was $1500/month (we're on fastly now). Fastly gave us like 6 month free to switch, both bandwidth and requests.
  - dilyevsky 3y ago
    So you mean to tell me you negotiated roughly -99% their list with a 1.5k commit?
    - 3y ago

proteal 3y ago

One thing to keep in mind - if you accidentally run up a bill because of a mistake, there’s a good chance you can reach out to support and they will credit your account. $5k may be a lot to an individual, but the cloud provider’s costs for that service are significantly less and they shouldn’t mind forgiving the charge. Case in point, when the author deleted his account and switched providers, the old host didn’t chase him for the charges.

swatcoder 3y ago
That's been true during recent flush years, where money was cheap and nobody really cared, but there are signs all over the place that the attitude is changing. Everything that used to be free is being monetized and everything that was monetized is seeing increased prices. We should expect that these sort of implicit affordances are being trimmed as well, or at least that they're likely to be soon.
- _jal 3y ago
  Yeah, I rather suspect they'll be much less forgiving of this sort of thing in the future. All that takes is a lean quarter and some MBA to coin a term for rookie mistake-windfalls, and they will come to feel entitled to revenue from it.
  - bcrosby95 3y ago
    Bob, I see the collateral spend of your clients is down this quarter. Can we see that bump up a bit? Mmmmkay that would be great. Thanks. Good talk.
0cf8612b2e1e 3y ago
That’s still a lot of hopes and prayers you do not end in financial ruin. Instead, the cloud providers could offer prepaid credits/billing maximums to let individuals sleep at night. I would love to run my side project off of AWS to gain the experience, but no way I feel comfortable with a potentially unlimited liability because I configured something poorly. My joke project can take the uptime hit if it means I know that I will never spend more than $X to host it.

tinrab 3y ago

Is there a way to protect against the "Denial of wallets" attack?

mcherm 3y ago
Yes, there is a way to protect against that attack! All you need to do is have triggers in place that will shut down or slow down your service when the costs are exceeding some amount. (If you get a moment of viral growth, you can always disable the trigger.) Unfortunately, AWS makes it extremely difficult to build such a trigger, and I'm not sure about other cloud providers.
- mlhpdx 3y ago
  There’s always rate limiting. Cloudfront supports it, API gateway support it, and it’s super easy to set up.

stickfigure 3y ago

Cloudflare R2 (their S3 clone) has free egress. We are in the middle of migrating about 5TB of fairly active data from Google Cloud Storage to R2. For years now, most of our cloud bill has been egress from GCS. Activity spiked recently and doing something about it became urgent. R2 has an S3-compatible API (we're using the Amazon-provided S3 Java SDK). Storage is 25% cheaper than GCS standard. Making our code multi-provider was pretty easy. I'm actually looking forward to next month's hosting bill. I feel like I'm slowly getting pushed off of GCP due to pricing. Last year we moved some stateless image processing infrastructure to AWS Elastic Beanstalk because RAM is much, much cheaper there.

chpatrick 3y ago
It's a bit worrying though that one day they might decide to not leave the money on the table.
Aachen 3y ago
General question, can we just call object storage object storage? If there's going to be another provider after already remembering S3, B2, now R2, on top of the various aliases for products like virtual servers, shared hosting, VLANs, and other things people are presenting as new products with the most random of abbreviations, I'm really going to lose track.
- namtab00 3y ago
  we need something like [this](https://www.azureperiodictable.com/ https://www.azureperiodictable.com/), but across all cloud providers
andersa 3y ago
You kinda pay in reduced performance. For example, civitai uses r2 for delivering the images, and it frequently doesn't load at all for me or very slowly. When your users have to use a VPN to move their traffic to another Cloudflare data center so the site actually loads, something is wrong...
- 3y ago

tekkk 3y ago

I wonder is There also is Hetzner and their dedicated servers. You pay for a server with a 1Gb/s connection and all egress is free. Unfortunately, I can't recommend them as each time I've tried to create an account, their abuse detection systems banned my account even before I had the occasion to enter a payment method. related to I've myself won the jackpot in the past on Netlify where I forgot to put a sleep in a while (true) loop, making my script flood my website and generate a lot of traffic. For the story, I quickly migrated the website to another provider, deleted my account, and never had to pay the bill :) Is there some kind of blacklist if you do something like that? But I do know Hetzner has some extremely strict policies regarding account creation.

hatware 3y ago
[dead]
derefr 3y ago
> Is there some kind of blacklist if you do something like that? No; but oddly enough, the people who "do things like that" generally tend to come from certain countries — I guess countries with cultures that don't place much weight on the concept of "incurring a debt of honor" by consuming someone else's resources without them ever knowing about it. So most systems don't generally need a big, manually curated and ever-growing blacklist; they just need to block registrations from IP addresses / ASNs of ISPs headquartered in these countries; and/or block payment attempts from credit cards issued by banks headquartered in these countries. That immediately stops 90% of such abuse. And of the remaining 10%, half of it is still people from those same countries — just using foreign IPs through residential-botnet VPNs, and stolen credit cards they purchased on scammer forums. Blocking these is a bit of an art, but it's possible: there's always patterns to the requests themselves, often because those same scammers try to solve all their problems with money, and so have also purchased scam-site kits to run on the hosting they acquire — things like cryptocurrency "drainers." If you're a VPS hosting provider, you can just detect these by the SHAs of the files; if you're a dedicated hosting provider with no access to customers' disks, it's still pretty easy to pick these out by the outbound signature of the network traffic they generate — as they almost always rely on making requests to particular third-party SaaS information systems, that you can turn into an IDS detection fingerprint. (I'm personally in a different position in this ecosystem — my company operates one of the public informational SaaS services that these scam-site kits like to use. From my company's perspective, these scammers are perfectly normal paying customers, not intent to scam us... but we don't want these people as customers, so we still detect this fraudware by the fingerprint it makes in our API request logs, and permaban the users who deploy such kits by every fingerprinting metric we can.) --- Though, on another note, I suppose you could call the observational "IP reputation" metrics gathered by providers like https://www.ipqualityscore.com/ https://www.ipqualityscore.com/ something like a blacklist — and I'm sure hosting providers like Hetzner check your "IP score" before letting you register. But these aren't blacklists in the sense of being manually curated. Instead, what these providers curate is something like a distributed version of an SSHGuard blocklist: a bunch of the provider's own "observer nodes", all over the world, observe what IPs are hitting them with DDoSes and other botnet-like activities, and consider these IPs temporarily compromised for as long as that activity persists (because any device infected by a botnet can potentially be repurposed as a part of a residential-proxy VPN network — and that means that any traffic observed to come from such a device, can't be trusted to be originating from that device.) IIRC, these providers will also "go undercover" to buy access to both commercial and residential-botnet VPNs; cycle through them to find out what all the available exit-node IP addresses are from a client's perspective — and then mark all these as compromised as well.

lowbloodsugar 3y ago

Do you have time, or do you have money? If you have time, then build some boxes and put them in a colo. If you have money then negotiate a cloud contract. Even if you have time, it might be better to start prototyping on a free tier on the cloud, and then maybe go colo, and then back to cloud.

hlandau 3y ago

>The second, less known, reason is that between compute, storage and network, network is actually the hardest part to scale for cloud providers. Thus they need to find a way to incentive their customers to optimize their network usage by themselves. And what better motivation to optimize something than taking a huge chunk of your profits? This doesn't really make sense as an explanation. Cloud networking has two components: 1. The network connecting the datacentres to the internet (the datacentre edge infrastructure). 2. The network connecting the servers and virtual machines to that edge. 1 is a solved problem and the variable costs are going to be based on any transit connections used. What cloud providers charge for bandwidth here is out of proportion to that, and they don't bill for ingress anyway, so this makes no sense. 2 is a technically more complex area, except it's very clear that getting people to use this infrastructure less is not the motivation because cloud providers don't charge for intra-DC traffic. You can ferry data around between servers in the same AZ as much as you like and not pay anything. The same is true of transferring data to and from e.g. S3 and EC2. In other words, the more technically complex and sophisticated network component is the one which isn't billed at all. Meanwhile, boring internet connectivity is billed at an exorbitant rate, but only in one direction. It's really quite obvious at this point that the motivation is vendor lock-in. You are penalised if you take data out of a big cloud provider but not to put data in. This means as soon as you put anything in AWS you are suddenly motivated to do as much in AWS as possible. This is fundamentally anti-competitive. If AWS starts offering some infrastructure service, it can offer free bandwidth to that service for its own customers, but some third party providing a comparable service can't, at least unless it also starts using AWS. But that's not even the best of it. If you want even better proof that these charges are BS, just look at Snowball. This is a service where you can transfer data to S3 by having AWS mail you a hard drive. Amazingly, they have the gall to charge per GB for this in addition to the fixed fee you pay... but only for exporting data. In other words, I can pay $150 and get loaned a SSD, fill it up with 14TB of data and import that data to S3. If I pay $300 to get data out of S3, I also pay $0.03/GB. Internet transit is not even involved here! It is pure anti-competitive lock-in. It's also really against the principles of the internet, in that the internet couldn't have developed as it did if we accepted different billing for communicating with different IPs (can you imagine "I don't want to send traffic to a foreign IP, I'll pay international rates!"). Yet cloud providers work on this principle. If you think about it it is effectively a net neutrality violation, just not on the part of residential ISPs - they charge you less to communicate with their own services than with others (zero-rating).

Obscurity4340 3y ago

Can anyone take a crack at why the hell you need a company and its computer farm 3000 miles away to merey sync your calendar, CalDav or whatever it is between two devices that are oerfectly capable of taking to each other or with a local-first mediating process or tool? I don't get what is about syncing things (particularly the basics like Contacts, Calendar/Reminders that requires you to send the data thousands of miles away and back when its all in the same room

deleted 3y ago
[deleted]
ihaveajob 3y ago
Put simply: There's no incentive for most organizations to implement this like this.
bombcar 3y ago
Because it's simpler for them to develop, because they make the server the source of truth, and then they can sell it to you. And for most people, it works fine, but when you hit edge cases you're dead in the water.
crazygringo 3y ago
Here's the answer: because it's not written for just 2 devices. It's written to sync however many devices you want -- 3 or 4 or 5 or whatever. Both your phones, your tablet, your laptop, and your watch. And it has to sync when your devices might never actually be turned on at the same time. And you want it to sync when you're traveling, and you don't want to open your home firewall for security reasons. So it requires a single, centralized, always-on server as the source of truth. And given the speed of light, whether that server is 3 miles or 3,000 miles away is largely irrelevant. And writing additional code to sync directly when devices are on the same LAN and turned on at the same time, as a special case, is not only entirely redundant -- but you still have to sync with the server, since you generally want the information backed up as well in case you lose your devices, or they're stolen, or damaged. Using your examples of Contacts and Calendar, those are often some of the most important things in someone's life to keep synced remotely. Does that answer your question?

mlhpdx 3y ago

I’ve just never seen bandwidth be a significant part of the bill for any sass I’ve built/worked for. I’m super curious about this. What kind of sass businesses use huge amounts of bandwdith but don’t provide enough commercial value to the customers to make it viable? Can folks with first-hand experience please share here?

danpalmer 3y ago
Bandwidth is rarely an issue if you're transferring mostly text, often an issue if you're transferring images (except for highly cacheable static images), and always an issue if you're transferring video or game assets. That's just how it works because the orders of magnitude differ so much. An image resource can easily be 100x some text (or HTML or API response), and a video can easily be 1000x an image. Another factor is cacheability. If you're a B2B SaaS business you'll have few clients and can take advantage of caching much more typically. If you're B2C that flips and caching might not provide much benefit. Anecdotally, on a medium sized ecommerce store, AWS egress for product images was a huge cost.
- mlhpdx 3y ago
  Thank you, the e-commerce example is definitely something I hadn’t thought about. Having high quality imagery helps make sales but comes at a cost. Looking around at some sites I buy from, I think I can squint and see this “out in plain sight”.
  - danpalmer 3y ago
    In fact an interesting detail of the ecommerce image situation is that you can somewhat trade-off bandwidth and compute. Creating resized images is expensive (we had ~2m unique images viewed per month), but if you can resize an image down and compress it well you might save 80% of the bandwidth on it, and deliver a better UX due to faster images. We did a bit of modelling to figure out how many sizes we wanted to deliver in order to get the best compute/bandwidth trade-off. This can be tricky for many companies but we were in a weird position because we had a large catalog compared to order volume, because of how the business worked. Video is an even better one. While we didn't have to deal with it in ecommerce, so many social applications are built around video. That doesn't necessarily mean YouTube/TikTok/etc, but also smaller things like BeReal who I'd guess are built entirely on cloud infra, even embedded videos in Mastodon posts. A similar thing could even be said about Podcasting/audio – not as bad as video, but there are specialist podcast hosts for a reason, and it's not that they do a great job, it's because they have cheap bandwidth.

simonsarris 3y ago

I'm surprised this about egress without mentioning the new Sippy: https://developers.cloudflare.com/r2/data-migration/sippy/ https://developers.cloudflare.com/r2/data-migration/sippy/ It allows you to incrementally migrate off of providers like S3 and onto the egress-free Cloudflare R2. Very clever idea.

lofaszvanitt 3y ago

It's the idiot tax.

CSSer 3y ago

There are a couple things this article glosses over. With Vercel vs. Netlify, be mindful of the many other fees they charge besides bandwidth. I wouldn’t be surprised if the costs are comparable when that’s accounted for. Sometimes the answer with these things is just to be diligent. Don’t be surprised at what you get if you stick your hand in a meat grinder. Put up guards in your CI process if you have to. The other thing is that I’m not sure how willing I am to jump to endorse an efficient cache policy as a blanket prescription like they seem to. I suppose I’m not opposed to the advice, but it has very strong “draw the owl” vibes. Project requirements vary, and I’d even argue that sometimes minimal or no caching is wise when requirements are subject to change or you’re doing a lot of static deployments because the savings can sometimes not be worth the headache. End-users are very demanding nowadays. Sometimes “it’ll roll over soon” isn’t good enough. Caching is tricky. Make sure you really get it right in addition to doing the math on the cost-savings if you go down that road.

TheIronYuppie 3y ago

Hey everyone! I'm David Aronchick - co-founder of Bacalhau[1] - and we are specifically building an open source platform to help with this. Basically, we allow you to do compute over data, making it much easier to schedule jobs (container, WASM, arbitrary binary) to where the data is being stored, even over irregular/unreliable networks. We were able to show 90%+ savings in things like log aggregations. Happy to answer any questions if you'd like! David Aronchick aronchick@expanso.io [1] https://github.com/bacalhau-project/bacalhau https://github.com/bacalhau-project/bacalhau

deleted 3y ago

[deleted]

TacticalCoder 3y ago

> There also is Hetzner and their dedicated servers. You pay for a server with a 1Gb/s connection and all egress is free. There's also OVH and they have things like 1 Gbit/s guaranteed, up to 5 Gbit/s (not guaranteed) EPYC 7313 servers with 25,50 or 100 Gbit/s guaranteed private (from OVH servers to OVH servers) for... 200 EUR / month. Re-using the example from TFA, you could max the bandwith of such a server 24/7, be "downgraded" from 5 Gbit/s to 1 Gbit/s guaranteed and yet it'd cost you... 200 EUR/month. Versus hundreds of thousands of egress bill in the cloud. 200 EUR / month vs hundred of thousands. But yay. Go cloud!

tutfbhuf 3y ago
Combine the two. Set up a large nginx reverse proxy with cache on the 200 EUR/month machine in Hetzner or OVH and serve 99.9% of the traffic from this location (js, css, images, videos, etc.). Use the cloud for convenience in managing the complex parts in the backend.
- tiernano 3y ago
  Why not just go Cloudflare then? In our sites, most of what Cloudflare caches is js, cs, images, etc...

nikolqy 3y ago

Oracle Cloud offers 10TB free bandwidth per month and is part of the bandwidth alliance I think with Cloudflare. Edgecast (Edgio) is free as far as I can tell, except you pay for support if you want it and some extra features. Datapacket has unmetered bandwidth for example. There are plenty of solutions, but they aren’t very well known because of the dominance AWS and others have. Not to mention, it might be cheaper to just use direct connect with a cloud provider or just grab an ASN and host stuff in a local datacenter.