6 ms·

Bandwidth costs are really not that bad. 95th billing, adaptive, progressive playing and just cap buffer to the minimum to keep playing. Equals ~$1M/month for

Bandwidth costs are really not that bad. 95th billing, adaptive, progressive playing and just cap buffer to the minimum to keep playing. Equals ~$1M/month for +10 tbit/s egress. Source: Worked at one of the largest bandwidth consumers in the world.

lopkeny12ko 2y ago
Anyone who says "bandwidth costs are not really that bad" should spend 2 minutes playing with the AWS cost calculator. You would think the VMs are the expensive part, but no, egress is easily multiple times the cost of the compute.
- NBJack 2y ago
  To be fair, that is the penalty of being in a shared cloud. They are incetivized to keep their customers from using everything, everywhere, all at once. Jump into a 'bare metal' datacenter and things can get much different.
- vundercind 2y ago
  Not even AWS. Any CDN. It won’t be AWS-bad (way, way, way under) but it ain’t good.
- manquer 2y ago
  Cloud bandwidth pricing has nothing with do with costs and everything to do with lock in . You can get 100x cheaper and unmetered at a low cost provider like OVH or hetzner or similar bare metal data centers . It doesn’t even need significant monthly commits to get that pricing if you are running video streaming at scale you are not running on AWS or even tier 2 like OVH for sure
  - Aerroon 2y ago
    Are they actually unmetered or are they unmetered*? * up to 2TB per month
    - immibis 2y ago
      It's unmetered* * if our upstream is saturated we're going to look at our biggest users and if the number is really big we'll send them a polite email to please reduce it or pay more. There are reports of people getting emails from Hetzner after sending multiple Gbps continuously for several months. That's the level you have to reach before the * kicks in. Only 1Gbps servers are unmetered, so you'd have to have several. If you want to know a better approximation of their true cost just look at their non-unlimited plans: 20TB/month included for free; 1€/TB (excl VAT) after that. I have one more interesting data point to add: I was quoted 950€/month for a dedicated 10Gbps between Berlin and Amsterdam (about 600km) plus peering at AMS-IX, or 300€ for 1Gbps. (They're not secretive and you can just ask for a quote using their sales contact form). Extrapolating, it seems that 1€ is worth about 2.5 petabyte-kilometers, at least within the dense interconnections of continental Europe. About twice the price of shipping a petabyte of hard drives the same distance.
      - Dylan16807 2y ago
        > If you want to know a better approximation of their true cost just look at their non-unlimited plans: 20TB/month included for free; 1€/TB (excl VAT) after that. I will point out that this is still about 50x-80x cheaper than Amazon. Not far off the claim of 100x.
        immibis 2y ago
        Mhm. Save literally 98% of your egress bill by avoiding AWS, GCP and Azure.
        manquer 2y ago
        AWS is particularly bad at this for S3. Cloudflare and most other Object Storage Providers either fully free egress for all users or at-least for inter-cloud transit so you can then put a free/cheap CDN like Cloudflare in front and not pay all that much for b/w. AWS refuses to participate. Costs of retrieval of all data plus associated bandwidth is a so high for many people that they stick to S3 including me.
  - mtnGoat 2y ago
    But anyone doing things at great scale, isn’t going with OVH. You could use it as an origin I goes but you’d still need CDN for decent content delivery.
    - manquer 2y ago
      Yes, at certain scale you likely have deals with many CDNs. At even higher scale (YouTube, Facebook or even Netflix ) you are going to be putting content caching servers at the local ISP PoPs : it is mutually beneficial to do so .
      - mtnGoat 2y ago
        So you can piggy back on the CDNs that do the same until you can afford to do so, with the same performance. This isn’t a privilege only given to those cloud providers, anyone with a checkbook and a pulse can do it.
        manquer 2y ago
        The key point is video streaming products are not impaired because of high cost of bandwidth, that only YouTube can only afford because Google subsidizes it from other revenue. YouTube is profitable by itself, the combination of premium and ads is more than enough to pay for it. It is hard to directly compete on long-form video because of audience and content depth advantage Google has with YouTube. There are successful niche players who are fairly large (like Vimeo or Twitch or even OnlyFans) who focus on specific markets that don't require social network advantage like corporate or smaller segments etc. For general purpose media, creators are going to focus on the platform with most audience and vice-versa, very hard to break that.
        mtnGoat 2y ago
        Agreed, I guess my point in my original comment is; yes, bandwidth is cheap but you aren’t going to compete at any scale hosting at OVH as was proposed. And you’ll need a lot more than cheap VMs too.
- wolfendin 2y ago
  You’re getting scammed with the AWS cost calculator or anything more than $20/mbps
mattmaroon 2y ago
If bandwidth costs were very high a whole lot of the services we love would cost more or not exist.
- vundercind 2y ago
  I am so confused here. Either I’ve been doing high-bandwidth bit slinging extremely wrong for quite a while or a lot of HN has never done it at all and is opining on it anyway. It’s real money, IME.
Jyaif 2y ago
My home connection uploads at 681mbit/s (just did the test over wifi) for 40 euros/months. At that price, I'd get 13tbit/s for 800k euros. It's a bit surprising that you were not getting significantly better prices than individuals.
- vundercind 2y ago
  Price business pipes where they won’t cut you off for saturating that 24/7. [edit] and that have good deliverability worldwide, no weird paths to other consumer IPs that intermittently fail to route or inexplicably have dial-up transfer speeds. And have anything like a real SLA.
  - swiftcoder 2y ago
    I don't think they legally can in number of European jurisdictions. This isn't like Comcast overselling their copper cables - brand new fiber paid for by the EU comes with certain obligations.
- vel0city 2y ago
  Try uploading at that rate for a few months non-stop and see how long your residential ISP will continue doing business with you.
- callalex 2y ago
  That can be taken out by a particularly unfortunate shovel or truck though, and isn’t sufficient to run a business on.
  - Jyaif 2y ago
    For 800k I would have 20000 different connections. This would allow me to survive many unfortunate shovels per month.
jononor 2y ago
Assuming 10 Mbit/s per stream that would serve 1 million concurrent streams 24/7. If we assume people watch 2.4 hours per day on average, it could support 10 million active users. Or a cost to serve for bandwidth alone of 1 USD/user per year.
- 55555 2y ago
  Yeah I'm coming to similar numbers. I guess this is why the Cloudflare CTO has said several times on hackernews that at scale bandwidth is free.
  - jgrahamc 2y ago
    Pretty sure I didn't say that, but I know who did: Cloudflare's CEO (eastdakota). https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=eastdakota%20bandwidth%20free&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
vundercind 2y ago
> Source: Worked at one of the largest bandwidth consumers in the world. Mindgeek? :-)
- Egrodo 2y ago
  Just checked his LinkedIn and yes, literally lol
- Egrodo 2y ago
  Just checked his LinkedIn and yes, literally lol
  - vundercind 2y ago
    Oh wow, I didn’t even snoop, just a guess.
- E39M5S62 2y ago
  They're no longer Mindgeek - they go by Aylo now. I worked for the company that hosted PornHub.
55555 2y ago
Could anyone help me understand what this means in practice, at scale? I keep doing the math and getting numbers either way to small or way too big. If each users streams 200MB over a 10 minute session, and they arrive on the site uniformly distributed over time (false premises, I know), how much would 1,000,000 sessions a month cost? I get confused by terabits and terabytes and how "per second" bandwidth metering actually works. I keep getting to the conclusion that for $1,000,000 you could handle 16,200,000,000 sessions per month, which is more traffic than phub gets and less money than I assume they spend.

lopkeny12ko 2y ago

Anyone who says "bandwidth costs are not really that bad" should spend 2 minutes playing with the AWS cost calculator. You would think the VMs are the expensive part, but no, egress is easily multiple times the cost of the compute.

NBJack 2y ago
To be fair, that is the penalty of being in a shared cloud. They are incetivized to keep their customers from using everything, everywhere, all at once. Jump into a 'bare metal' datacenter and things can get much different.
vundercind 2y ago
Not even AWS. Any CDN. It won’t be AWS-bad (way, way, way under) but it ain’t good.
manquer 2y ago
Cloud bandwidth pricing has nothing with do with costs and everything to do with lock in . You can get 100x cheaper and unmetered at a low cost provider like OVH or hetzner or similar bare metal data centers . It doesn’t even need significant monthly commits to get that pricing if you are running video streaming at scale you are not running on AWS or even tier 2 like OVH for sure
- Aerroon 2y ago
  Are they actually unmetered or are they unmetered*? * up to 2TB per month
  - immibis 2y ago
    It's unmetered* * if our upstream is saturated we're going to look at our biggest users and if the number is really big we'll send them a polite email to please reduce it or pay more. There are reports of people getting emails from Hetzner after sending multiple Gbps continuously for several months. That's the level you have to reach before the * kicks in. Only 1Gbps servers are unmetered, so you'd have to have several. If you want to know a better approximation of their true cost just look at their non-unlimited plans: 20TB/month included for free; 1€/TB (excl VAT) after that. I have one more interesting data point to add: I was quoted 950€/month for a dedicated 10Gbps between Berlin and Amsterdam (about 600km) plus peering at AMS-IX, or 300€ for 1Gbps. (They're not secretive and you can just ask for a quote using their sales contact form). Extrapolating, it seems that 1€ is worth about 2.5 petabyte-kilometers, at least within the dense interconnections of continental Europe. About twice the price of shipping a petabyte of hard drives the same distance.
    - Dylan16807 2y ago
      > If you want to know a better approximation of their true cost just look at their non-unlimited plans: 20TB/month included for free; 1€/TB (excl VAT) after that. I will point out that this is still about 50x-80x cheaper than Amazon. Not far off the claim of 100x.
      - immibis 2y ago
        Mhm. Save literally 98% of your egress bill by avoiding AWS, GCP and Azure.
        manquer 2y ago
        AWS is particularly bad at this for S3. Cloudflare and most other Object Storage Providers either fully free egress for all users or at-least for inter-cloud transit so you can then put a free/cheap CDN like Cloudflare in front and not pay all that much for b/w. AWS refuses to participate. Costs of retrieval of all data plus associated bandwidth is a so high for many people that they stick to S3 including me.
- mtnGoat 2y ago
  But anyone doing things at great scale, isn’t going with OVH. You could use it as an origin I goes but you’d still need CDN for decent content delivery.
  - manquer 2y ago
    Yes, at certain scale you likely have deals with many CDNs. At even higher scale (YouTube, Facebook or even Netflix ) you are going to be putting content caching servers at the local ISP PoPs : it is mutually beneficial to do so .
    - mtnGoat 2y ago
      So you can piggy back on the CDNs that do the same until you can afford to do so, with the same performance. This isn’t a privilege only given to those cloud providers, anyone with a checkbook and a pulse can do it.
      - manquer 2y ago
        The key point is video streaming products are not impaired because of high cost of bandwidth, that only YouTube can only afford because Google subsidizes it from other revenue. YouTube is profitable by itself, the combination of premium and ads is more than enough to pay for it. It is hard to directly compete on long-form video because of audience and content depth advantage Google has with YouTube. There are successful niche players who are fairly large (like Vimeo or Twitch or even OnlyFans) who focus on specific markets that don't require social network advantage like corporate or smaller segments etc. For general purpose media, creators are going to focus on the platform with most audience and vice-versa, very hard to break that.
        mtnGoat 2y ago
        Agreed, I guess my point in my original comment is; yes, bandwidth is cheap but you aren’t going to compete at any scale hosting at OVH as was proposed. And you’ll need a lot more than cheap VMs too.
wolfendin 2y ago
You’re getting scammed with the AWS cost calculator or anything more than $20/mbps

mattmaroon 2y ago

If bandwidth costs were very high a whole lot of the services we love would cost more or not exist.

vundercind 2y ago
I am so confused here. Either I’ve been doing high-bandwidth bit slinging extremely wrong for quite a while or a lot of HN has never done it at all and is opining on it anyway. It’s real money, IME.

Jyaif 2y ago

My home connection uploads at 681mbit/s (just did the test over wifi) for 40 euros/months. At that price, I'd get 13tbit/s for 800k euros. It's a bit surprising that you were not getting significantly better prices than individuals.

vundercind 2y ago
Price business pipes where they won’t cut you off for saturating that 24/7. [edit] and that have good deliverability worldwide, no weird paths to other consumer IPs that intermittently fail to route or inexplicably have dial-up transfer speeds. And have anything like a real SLA.
- swiftcoder 2y ago
  I don't think they legally can in number of European jurisdictions. This isn't like Comcast overselling their copper cables - brand new fiber paid for by the EU comes with certain obligations.
vel0city 2y ago
Try uploading at that rate for a few months non-stop and see how long your residential ISP will continue doing business with you.
callalex 2y ago
That can be taken out by a particularly unfortunate shovel or truck though, and isn’t sufficient to run a business on.
- Jyaif 2y ago
  For 800k I would have 20000 different connections. This would allow me to survive many unfortunate shovels per month.

jononor 2y ago

Assuming 10 Mbit/s per stream that would serve 1 million concurrent streams 24/7. If we assume people watch 2.4 hours per day on average, it could support 10 million active users. Or a cost to serve for bandwidth alone of 1 USD/user per year.

55555 2y ago
Yeah I'm coming to similar numbers. I guess this is why the Cloudflare CTO has said several times on hackernews that at scale bandwidth is free.
- jgrahamc 2y ago
  Pretty sure I didn't say that, but I know who did: Cloudflare's CEO (eastdakota). https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=eastdakota%20bandwidth%20free&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

vundercind 2y ago

> Source: Worked at one of the largest bandwidth consumers in the world. Mindgeek? :-)

Egrodo 2y ago
Just checked his LinkedIn and yes, literally lol
Egrodo 2y ago
Just checked his LinkedIn and yes, literally lol
- vundercind 2y ago
  Oh wow, I didn’t even snoop, just a guess.
E39M5S62 2y ago
They're no longer Mindgeek - they go by Aylo now. I worked for the company that hosted PornHub.

55555 2y ago

Could anyone help me understand what this means in practice, at scale? I keep doing the math and getting numbers either way to small or way too big. If each users streams 200MB over a 10 minute session, and they arrive on the site uniformly distributed over time (false premises, I know), how much would 1,000,000 sessions a month cost? I get confused by terabits and terabytes and how "per second" bandwidth metering actually works. I keep getting to the conclusion that for $1,000,000 you could handle 16,200,000,000 sessions per month, which is more traffic than phub gets and less money than I assume they spend.