8 ms·

For certain apps, bandwidth cost is stupid expensive on AWS. For example I've been following the development of an online manga reading site (mangadex.org) tha

by zawerf 7y ago

For certain apps, bandwidth cost is stupid expensive on AWS. For example I've been following the development of an online manga reading site (mangadex.org) that is now pushing over 1PB/month of images. If they had built it on aws, even at the lowest rate of $0.02/GB with cloudfront/S3, it would be $20,000 per month. But they ended up paying nothing by using cloudflare who gave them unmetered bandwidth. (well, until they got throttled earlier this year but they got out of that by upgrading to a measly $200/month plan)

kasey_junk 7y ago
I thought most of the clouds offered free ingress?
- dharmab 7y ago
  MangaDex serves very high resolution scans of comic pages (egress)
  - kasey_junk 7y ago
    Ah. I misunderstood the “push” part of the comment.
- contras1970 7y ago
  yes, but the topic is the price of egress. i thought this was obvious to everybody: public clouds want to lock you in, so they make it very cheap and easy to get data in, but charge stupendous amounts for getting your data out.
stingraycharles 7y ago
Bandwidth is insanely expensive on all clouds. If you’re a large consumer, however, you can usually negotiate a much better rate.
- krn 7y ago
  > Bandwidth is insanely expensive on all clouds. OVH Public Cloud[1] offers unmetered bandwidth (250-500 Mbps) in all regions, apart from Asia-Pacific. Hetzner Cloud[2] offers 20 TB (1 Gbps) for each cloud instance, but has locations only in Europe. Both of them offer dedicated servers with up to 1-3 Gbps unmetered bandwidth that could be used as exit nodes. [1] https://www.ovh.com/world/public-cloud/instances/prices/ https://www.ovh.com/world/public-cloud/instances/prices/ [2] https://www.hetzner.com/cloud https://www.hetzner.com/cloud
  - chipperyman573 7y ago
    Is this actually unmetered? Or is it unmetered until you hit a secret limit that they don't tell you, and then demand you pay them, like Cloudflare? (I'm not saying this is a bad thing, it's reasonable for CF to want their highest-usage customers to pay, I'm just curious)
    - derefr 7y ago
      I believe it’s unmetered by them personally. But there’s always a “secret limit”—the point where your bandwidth usage looks like a DDoS attack, and the tier-1 exchange feeding the cloud provider’s DC decides to blackhole your traffic for the sake of the network.
      - deleted 7y ago
        [deleted]
    - krn 7y ago
      Their bandwidth limits are based on Mbps / Gbps, so the bandwidth is unmetered, but not unlimited. And unlike CDNs, public cloud companies make money from the computational power (CPU / RAM) they provide.
    - StavrosK 7y ago
      They're going to limit you if you spin up a bunch of cheap instances to use up their bandwidth, but if you pay $300/mo for their 3 Gbps connection you can saturate that and they don't care.
awinder 7y ago
Yes I came here to talk about bandwidth too, which is not mentioned in the article at all but can be the biggest engineering cost focus depending on app/scale :-). Over 10tb of transfer you can (and should) negotiate some cheaper pre-committed pricing on bandwidth. There’s also engineering around the network involved in your application. A lot of architectures internally reflect traffic around (moreso if you’re also accepting some ridiculous amount of traffic rather than just serving it). In those cases you can save a lot of money by figuring out ways to pass traffic through unmetered connections like internal albs and amazon hosted dbs. Otherwise you can either rack up a lot of cross-az traffic charges or be on the hook for engineering your way to minimize those costs.
scrollaway 7y ago
Just for everyone's clarity, you can use Cloudflare with AWS just fine. A previous site I built (https://hearthstonejson.com/ https://hearthstonejson.com/) is pushing on the order of 100TB / month in images and JSON data, with 96% cached requests and 99.9% cloudflare-cached bandwidth. Almost nothing comes out of S3 itself. Cloudfront is the ridiculously-priced offering IMO. I don't really understand why people use it.
- dpq 7y ago
  I'm afraid you are in for a nasty surprise: CloudFlare specifically forbids using it mostly as an image/video CDN unless you are on the Enterprise tier (clause 2.8 in the Terms). I found it out the hard way a couple of years ago when I was helping with maintenance of a pretty large image hosting website (we got to 1-2Pb/mo range). CF may not pay attention to you while you are small, but when they do, it's going to cost you.
  - ianai 7y ago
    What’s the enterprise cost per month?
    - dpq 7y ago
      Subject for negotiation. But it's going to be in the ballpark of many $k/mo
    - joshmn 7y ago
      $Contact us
      - qes 7y ago
        That. We ended up at $3500/month for ~200 TB/month. Reading this thread I might have negotiate harder when renewal time comes.
    - user5994461 7y ago
      Pro is $200, enterprise is $2000. This is probably customized per client and could be more for big companies.
deleted 7y ago
[deleted]
hnaccy 7y ago
Didn't you say some manga re-uploaders used Blogger for free image hosing bandwidth? Did that get shut down?
- zawerf 7y ago
  I don't know how you remember that comment, it was from over a year ago: https://news.ycombinator.com/item?id=16784267 https://news.ycombinator.com/item?id=16784267 But as far as I know, kissmanga.com is still abusing blogspot.com/tumblr.com by using them as free image cdns.
qaq 7y ago
I think I remember seeing ads for HE 100Gbit @ 10K/mon. the cloud markup for bandwidth is massive.
- mfjordvald 7y ago
  To be fair, there's a world of difference between a single POP HE connection and the globally distributed AWS CDN. Understand your use-case and what factors are important because HE has significant drawbacks in some cases.
  - qaq 7y ago
    that's the marketing side of the equation in reality cloud provides have such an amazingly complex control planes that you will see comparable amount of downtime. Notice the 2 google cloud outages were at network layer.
chaitanya 7y ago
Right. At a previous company our Cloudfront bill ran up to $2000/mo because someone ran a test script a few times every minute that fetched a few resources from CF. Once we killed that script our CF bill dropped down to $200/mo. Cloudfront’s pricing can absolutely bite you if you are not careful. Which is one of the reasons why I prefer Cloudflare over it.
bezoisevil 7y ago
This is also why companies should always evaluate using on-prem cloud instead of letting the big 3 host. This is not talked about enough, I think it's because there aren't a lot of people on here that have scaled for Fortune companies
dizzih 7y ago
Thanks for reminding me that I forgot to include the bandwidth costs! I've added them to the article.

by zawerf 7y ago

kasey_junk 7y ago

I thought most of the clouds offered free ingress?

dharmab 7y ago
MangaDex serves very high resolution scans of comic pages (egress)
- kasey_junk 7y ago
  Ah. I misunderstood the “push” part of the comment.
contras1970 7y ago
yes, but the topic is the price of egress. i thought this was obvious to everybody: public clouds want to lock you in, so they make it very cheap and easy to get data in, but charge stupendous amounts for getting your data out.

stingraycharles 7y ago

Bandwidth is insanely expensive on all clouds. If you’re a large consumer, however, you can usually negotiate a much better rate.

krn 7y ago
> Bandwidth is insanely expensive on all clouds. OVH Public Cloud[1] offers unmetered bandwidth (250-500 Mbps) in all regions, apart from Asia-Pacific. Hetzner Cloud[2] offers 20 TB (1 Gbps) for each cloud instance, but has locations only in Europe. Both of them offer dedicated servers with up to 1-3 Gbps unmetered bandwidth that could be used as exit nodes. [1] https://www.ovh.com/world/public-cloud/instances/prices/ https://www.ovh.com/world/public-cloud/instances/prices/ [2] https://www.hetzner.com/cloud https://www.hetzner.com/cloud
- chipperyman573 7y ago
  Is this actually unmetered? Or is it unmetered until you hit a secret limit that they don't tell you, and then demand you pay them, like Cloudflare? (I'm not saying this is a bad thing, it's reasonable for CF to want their highest-usage customers to pay, I'm just curious)
  - derefr 7y ago
    I believe it’s unmetered by them personally. But there’s always a “secret limit”—the point where your bandwidth usage looks like a DDoS attack, and the tier-1 exchange feeding the cloud provider’s DC decides to blackhole your traffic for the sake of the network.
    - deleted 7y ago
      [deleted]
  - krn 7y ago
    Their bandwidth limits are based on Mbps / Gbps, so the bandwidth is unmetered, but not unlimited. And unlike CDNs, public cloud companies make money from the computational power (CPU / RAM) they provide.
  - StavrosK 7y ago
    They're going to limit you if you spin up a bunch of cheap instances to use up their bandwidth, but if you pay $300/mo for their 3 Gbps connection you can saturate that and they don't care.

awinder 7y ago

Yes I came here to talk about bandwidth too, which is not mentioned in the article at all but can be the biggest engineering cost focus depending on app/scale :-). Over 10tb of transfer you can (and should) negotiate some cheaper pre-committed pricing on bandwidth. There’s also engineering around the network involved in your application. A lot of architectures internally reflect traffic around (moreso if you’re also accepting some ridiculous amount of traffic rather than just serving it). In those cases you can save a lot of money by figuring out ways to pass traffic through unmetered connections like internal albs and amazon hosted dbs. Otherwise you can either rack up a lot of cross-az traffic charges or be on the hook for engineering your way to minimize those costs.

scrollaway 7y ago

Just for everyone's clarity, you can use Cloudflare with AWS just fine. A previous site I built (https://hearthstonejson.com/ https://hearthstonejson.com/) is pushing on the order of 100TB / month in images and JSON data, with 96% cached requests and 99.9% cloudflare-cached bandwidth. Almost nothing comes out of S3 itself. Cloudfront is the ridiculously-priced offering IMO. I don't really understand why people use it.

dpq 7y ago
I'm afraid you are in for a nasty surprise: CloudFlare specifically forbids using it mostly as an image/video CDN unless you are on the Enterprise tier (clause 2.8 in the Terms). I found it out the hard way a couple of years ago when I was helping with maintenance of a pretty large image hosting website (we got to 1-2Pb/mo range). CF may not pay attention to you while you are small, but when they do, it's going to cost you.
- ianai 7y ago
  What’s the enterprise cost per month?
  - dpq 7y ago
    Subject for negotiation. But it's going to be in the ballpark of many $k/mo
  - joshmn 7y ago
    $Contact us
    - qes 7y ago
      That. We ended up at $3500/month for ~200 TB/month. Reading this thread I might have negotiate harder when renewal time comes.
  - user5994461 7y ago
    Pro is $200, enterprise is $2000. This is probably customized per client and could be more for big companies.

deleted 7y ago

[deleted]

hnaccy 7y ago

Didn't you say some manga re-uploaders used Blogger for free image hosing bandwidth? Did that get shut down?

zawerf 7y ago
I don't know how you remember that comment, it was from over a year ago: https://news.ycombinator.com/item?id=16784267 https://news.ycombinator.com/item?id=16784267 But as far as I know, kissmanga.com is still abusing blogspot.com/tumblr.com by using them as free image cdns.

qaq 7y ago

I think I remember seeing ads for HE 100Gbit @ 10K/mon. the cloud markup for bandwidth is massive.

mfjordvald 7y ago
To be fair, there's a world of difference between a single POP HE connection and the globally distributed AWS CDN. Understand your use-case and what factors are important because HE has significant drawbacks in some cases.
- qaq 7y ago
  that's the marketing side of the equation in reality cloud provides have such an amazingly complex control planes that you will see comparable amount of downtime. Notice the 2 google cloud outages were at network layer.

chaitanya 7y ago

Right. At a previous company our Cloudfront bill ran up to $2000/mo because someone ran a test script a few times every minute that fetched a few resources from CF. Once we killed that script our CF bill dropped down to $200/mo. Cloudfront’s pricing can absolutely bite you if you are not careful. Which is one of the reasons why I prefer Cloudflare over it.

bezoisevil 7y ago

This is also why companies should always evaluate using on-prem cloud instead of letting the big 3 host. This is not talked about enough, I think it's because there aren't a lot of people on here that have scaled for Fortune companies

dizzih 7y ago

Thanks for reminding me that I forgot to include the bandwidth costs! I've added them to the article.