8 ms·

Peering agreements between Backblaze and Cloudflare

243 points by decrypt 6y ago

j16sdiz 6y ago
I always thought peer agreements are under NDA. Surprise to see they talk openly.
- brianwski 6y ago
  Disclaimer: I work at Backblaze. > I always thought peer agreements are under NDA. Surprise to see they talk openly. Both Backblaze and CloudFlare are very open about this. It is called the "Bandwidth Alliance", you can read about it at these two links: https://www.backblaze.com/blog/backblaze-and-cloudflare-partner-to-provide-free-data-transfer/ https://www.backblaze.com/blog/backblaze-and-cloudflare-part... https://www.cloudflare.com/bandwidth-alliance/backblaze/ https://www.cloudflare.com/bandwidth-alliance/backblaze/ For Backblaze, it's part of our business strategy to be open and honest with customers. That feels funny to type, but it's true. We have historically disclosed things that other companies have traditionally kept secret, like our internal drive failure rates. I don't know if this is surprising or not, but it has worked well financially for us. Meaning we explain what is going on, more potential customers hear about us, more customers trust us, our sales increase. I'm actually curious why other businesses keep their operations so secret?
erentz 6y ago
My brain is a bit fuzzy today but this comparison seems... off? Yes an interconnect inside a colo is going to be dirt cheap. But you can’t compare it to a huge backbone spanning continents that gives you reach across the world. Everyone should peer with everyone they can locally. Use the tier 1s for reach beyond what you can peer with. (This is not a defense of tier 1s and their peering policies or pricing.) Also this is why it’s always amused me when DC guys quote their raw switchport capacity with pride. Gotta multiply that by distance traveled to get a real comparable metric.
- wmf 6y ago
  Yeah, this is a bad take from Backblaze. If ISPs had 99% profit margin you'd see it in their quarterly reports. Spoiler warning: they don't. Also, I remember when there were only half as many tier 1s, so if they're a cartel keeping new members out they aren't doing a very good job of it.
  - csnover 6y ago
    Frankly, I’ve never seen a good take from Backblaze once I got past their marketing blog posts. Zero-knowledge encryption? You don’t need that[0][1]. Client writes every backup chunk to disk before uploading instead of holding the data in memory, reducing the lifespan of customer SSDs for no reason? It’s fine, don’t worry about it[2][3]. A pattern of violating of basic and well-known software security principles[4][5][6]? Bodge in fixes. Don’t tell customers about it. Cancel the public bug bounty programme, claim it’ll be back in an indeterminate period of time[7] and replace it with a private one instead[8]. [0] https://www.reddit.com/r/backblaze/comments/8oczbl/how_do_i_know_backblaze_can_be_trusted/e095b8x/ https://www.reddit.com/r/backblaze/comments/8oczbl/how_do_i_... [1] https://help.backblaze.com/hc/en-us/articles/217664798-Security-Question-Round-up- https://help.backblaze.com/hc/en-us/articles/217664798-Secur... [2] https://old.reddit.com/r/backblaze/comments/igaqse/please_allow_changing_of_bzdata_directory_heavy/g2ugpwy/ https://old.reddit.com/r/backblaze/comments/igaqse/please_al... [3] https://old.reddit.com/r/backblaze/comments/k764xq/backblaze_temp_drive_on_4gb_ram_drive/geq8x4r/ https://old.reddit.com/r/backblaze/comments/k764xq/backblaze... [4] https://twitter.com/zetafleet/status/1304664097989054464 https://twitter.com/zetafleet/status/1304664097989054464, more discussion at https://news.ycombinator.com/item?id=24842871 https://news.ycombinator.com/item?id=24842871 [5] Hard-coded credentials in deployed apps: https://medium.com/@pig.wig45/from-n-a-to-resolved-for-backblaze-android-app-hackerone-platform-bucket-takeover-f817692a590 https://medium.com/@pig.wig45/from-n-a-to-resolved-for-backb... [6] Relying on client-side validation only: https://www.youtube.com/watch?v=1LC0aEZFVz8 https://www.youtube.com/watch?v=1LC0aEZFVz8 [7] https://news.ycombinator.com/item?id=24849334 https://news.ycombinator.com/item?id=24849334 [8] https://www.backblaze.com/security.html https://www.backblaze.com/security.html
    - 6y ago
Dylan16807 6y ago
> Ok, so here is my proof that the "tier 1" networks are over charging: if Backblaze routes a packet over the "tier 1" network is comes down to about 2/10ths of 1 penny per GByte. That's $0.002/GByte. That roughly matches the information I've been able to find when looking into transit pricing. And then AWS/Azure/GCP gouge you to an absolutely ridiculous extent even beyond that.
- vegardx 6y ago
  You'll likely see prices in the same range with most providers if you have some volume and are willing to commit to it.
  - Dylan16807 6y ago
    By "most" do you mean excluding those three? I know it's not too hard to get a port at a good rate if I have my own server, and it's easy to rent a server with plenty of bandwidth. But if I want to use something like glacier, I'm screwed. Amazon's "contact us" threshold is $28k per month, and anyone under that is paying 30x-40x as much as if it was $.002/GB.
    - balakk 6y ago
      Cant speak for others, but there are some specialized SKUs in Azure you can opt, for non-metered egress data transfer. https://docs.microsoft.com/en-us/azure/expressroute/expressroute-faqs#expressroute-local https://docs.microsoft.com/en-us/azure/expressroute/expressr... It starts about USD 1200 per month, for a 1 Gbps line.
      - welterde 6y ago
        That's still price gouging imho. For the same amount of money you can get 5-10G of transit bandwidth from large carriers. How can it be cheaper for the big carriers to transport my packets halfway around the planet than for Microsoft to transport them inside the local metro-region to their own datacenter?
      - 6y ago
cbg0 6y ago
Public internet exchanges are huge in Europe, you can basically reach every network in Europe by connecting to a few large exchanges for a fraction of what you'd pay for tier 1 transit (granted, you still probably need it to reach some eyeballs). I wish we'd see the same concept pop up in the US, as these exchanges are definitely not as popular there. https://www.de-cix.net/en/locations/germany/frankfurt/statistics https://www.de-cix.net/en/locations/germany/frankfurt/statis... https://www.ams-ix.net/ams/documentation/total-stats https://www.ams-ix.net/ams/documentation/total-stats https://portal.linx.net/ https://portal.linx.net/
- sneak 6y ago
  A big part of it is that simply connecting your own network to these exchanges is a lot more expensive in the US due to increased land size and corrupt legislation. Due to the intense regulations (which are bought and paid for at the legislative level, and mostly exist to protect the large incumbents like Verizon, AT&T, Comcast, CenturyLink, et al from any new competitors) you end up having to buy your "last mile" loop (which, due to the US's size and general car-centric design, is much more than just a mile in most cases) that connects your network (in your office, or data center, or whatever, over to the exchange point) from a small number of monopolists. They're thousands, or tens of thousands of dollars per month. "We don't care. We don't have to. We're the phone company."
- gbil 6y ago
  Internet Exchanges doesn't mean that a peering agreement is also in place automatically. For example: https://www.ams-ix.net/ams/documentation/general-terms-and-conditions https://www.ams-ix.net/ams/documentation/general-terms-and-c... 3.3 Peering arrangements are not covered by these AMS-IX General Terms and Conditions or the AMS-IX Connection Agreement. The Customer is responsible for the negotiation, conclusion and implementation of peering arrangements with other users of the AMS-IX Infrastructure.
  - cbg0 6y ago
    While you are correct, many networks connected have in fact an open peering policy: https://www.ams-ix.net/ams/connected-networks https://www.ams-ix.net/ams/connected-networks
3pt14159 6y ago
Disclaimer: My mum works for AT&T on internet related stuff. This post has large elements of truth to it, especially for dumb hops over tier 1 which should be much cheaper than they're currently billed, but there's a lot more going on at AT&T than most people realize. Especially across the oceans. It's not quite as simplistic as this take makes it out to be, since the 5%ile case for networking a packet over tier 1 really is expensive to handle, maintain, bill, and provision. That said, there should be better regulations on tier 1 inclusion. On the other hand, they did make deep investments into infrastructure that took decades to become profitable. Often times with government assurances. It's one of the reasons telephony had such high penetration in Canada (75%) at a time when it was much lower in France (~25%), Bell was given assurances that we're now paying for in more expensive internet. Not saying it's right, but it's like this for understandable reasons, and it's not strictly raw corruption.
2Gkashmiri 6y ago
so why do vps providers provide a "1-3"TB a month transfer when it costs nothing? talking about cheap plans here
- exikyut 6y ago
  It most definitely does not cost nothing. The whole point here is that the Tier 1s price-gouge everyone else, and from there everything is far more expensive than it needs to be. Prices only go up from there. Cheaper plans sold by smaller outfits are likely being sold to end users through 2-3 layers of resellers. This creates a tug-of-war between multiple resale overheads and customer interest in something at a low price point, so the baseline of what is being offered (CPU, RAM, speed, monthly transfer) can get fairly wobbly. For larger companies, to take Contabo as an example, they basically get to dictate whatever performance metrics they want, because they own the entire stack. So I might have more-or-less-unlimited 100Mbps and 6GB RAM, but CPU performance is... not great, I'm pretty sure my KVM instance winds up almost entirely swapped onto an HDD occasionally, and there are some really harsh QEMU disk I/O quotas.
  - 2Gkashmiri 6y ago
    funny that you bring up contabo. i have been thinking about setting up a selfhosted email instance and that could lead to more stuff later on but bandwidth is like night and day. on contabo cheapest vps is like €4.99 / month, 4 vCPU Cores,8 GB RAM, 200 GB SSD, 200 Mbit/s Port, 1 Snapshot and "32 TB of transfer is already included in each VPS package." while on vultr.com, the ipv4 cheapest is like 25 GB SSD, 1 CPU, 1024 MB RAM, 1 TB transfers for $5/mo. that is such a huge difference that i dont know what is real. i want like 10TB storage vps from contabo but apparently they dont have it. do you know anyone like them? i have looked at scalaway and others but they dont say pricing of bandwidth past the allowance. like only for their s3 they say pricing which is exorbitant like the ones mentioned in the article edit: vultr object storage says "$0.01 per additional GB transferred " https://www.vultr.com/products/object-storage/ https://www.vultr.com/products/object-storage/ and scaleway says Inter-regional* and external outgoing data transfer to other products from a different region and the Internet: 75 GB free every month, then €0.01/GB same pricing for backblaze b2 for downloads
phonon 6y ago
Why doesn't Cloudflare buy GTT? They can easily afford to.
Bkmhfdtyb 6y ago
From what I have read, these networking centric companies have a reputation for noncompetitive pay. Is this true? They otherwise sound like great places to work.
2Gkashmiri 6y ago
so i want to start a peertube server, what storage is best, size and ample bandwidth. $0.01/Gb is too much for transfers.... oh, wasabi is not for video, i tried

243 points by decrypt 6y ago

j16sdiz 6y ago

I always thought peer agreements are under NDA. Surprise to see they talk openly.

brianwski 6y ago
Disclaimer: I work at Backblaze. > I always thought peer agreements are under NDA. Surprise to see they talk openly. Both Backblaze and CloudFlare are very open about this. It is called the "Bandwidth Alliance", you can read about it at these two links: https://www.backblaze.com/blog/backblaze-and-cloudflare-partner-to-provide-free-data-transfer/ https://www.backblaze.com/blog/backblaze-and-cloudflare-part... https://www.cloudflare.com/bandwidth-alliance/backblaze/ https://www.cloudflare.com/bandwidth-alliance/backblaze/ For Backblaze, it's part of our business strategy to be open and honest with customers. That feels funny to type, but it's true. We have historically disclosed things that other companies have traditionally kept secret, like our internal drive failure rates. I don't know if this is surprising or not, but it has worked well financially for us. Meaning we explain what is going on, more potential customers hear about us, more customers trust us, our sales increase. I'm actually curious why other businesses keep their operations so secret?

erentz 6y ago

My brain is a bit fuzzy today but this comparison seems... off? Yes an interconnect inside a colo is going to be dirt cheap. But you can’t compare it to a huge backbone spanning continents that gives you reach across the world. Everyone should peer with everyone they can locally. Use the tier 1s for reach beyond what you can peer with. (This is not a defense of tier 1s and their peering policies or pricing.) Also this is why it’s always amused me when DC guys quote their raw switchport capacity with pride. Gotta multiply that by distance traveled to get a real comparable metric.

wmf 6y ago
Yeah, this is a bad take from Backblaze. If ISPs had 99% profit margin you'd see it in their quarterly reports. Spoiler warning: they don't. Also, I remember when there were only half as many tier 1s, so if they're a cartel keeping new members out they aren't doing a very good job of it.
- csnover 6y ago
  Frankly, I’ve never seen a good take from Backblaze once I got past their marketing blog posts. Zero-knowledge encryption? You don’t need that[0][1]. Client writes every backup chunk to disk before uploading instead of holding the data in memory, reducing the lifespan of customer SSDs for no reason? It’s fine, don’t worry about it[2][3]. A pattern of violating of basic and well-known software security principles[4][5][6]? Bodge in fixes. Don’t tell customers about it. Cancel the public bug bounty programme, claim it’ll be back in an indeterminate period of time[7] and replace it with a private one instead[8]. [0] https://www.reddit.com/r/backblaze/comments/8oczbl/how_do_i_know_backblaze_can_be_trusted/e095b8x/ https://www.reddit.com/r/backblaze/comments/8oczbl/how_do_i_... [1] https://help.backblaze.com/hc/en-us/articles/217664798-Security-Question-Round-up- https://help.backblaze.com/hc/en-us/articles/217664798-Secur... [2] https://old.reddit.com/r/backblaze/comments/igaqse/please_allow_changing_of_bzdata_directory_heavy/g2ugpwy/ https://old.reddit.com/r/backblaze/comments/igaqse/please_al... [3] https://old.reddit.com/r/backblaze/comments/k764xq/backblaze_temp_drive_on_4gb_ram_drive/geq8x4r/ https://old.reddit.com/r/backblaze/comments/k764xq/backblaze... [4] https://twitter.com/zetafleet/status/1304664097989054464 https://twitter.com/zetafleet/status/1304664097989054464, more discussion at https://news.ycombinator.com/item?id=24842871 https://news.ycombinator.com/item?id=24842871 [5] Hard-coded credentials in deployed apps: https://medium.com/@pig.wig45/from-n-a-to-resolved-for-backblaze-android-app-hackerone-platform-bucket-takeover-f817692a590 https://medium.com/@pig.wig45/from-n-a-to-resolved-for-backb... [6] Relying on client-side validation only: https://www.youtube.com/watch?v=1LC0aEZFVz8 https://www.youtube.com/watch?v=1LC0aEZFVz8 [7] https://news.ycombinator.com/item?id=24849334 https://news.ycombinator.com/item?id=24849334 [8] https://www.backblaze.com/security.html https://www.backblaze.com/security.html
  - 6y ago

Dylan16807 6y ago

> Ok, so here is my proof that the "tier 1" networks are over charging: if Backblaze routes a packet over the "tier 1" network is comes down to about 2/10ths of 1 penny per GByte. That's $0.002/GByte. That roughly matches the information I've been able to find when looking into transit pricing. And then AWS/Azure/GCP gouge you to an absolutely ridiculous extent even beyond that.

vegardx 6y ago
You'll likely see prices in the same range with most providers if you have some volume and are willing to commit to it.
- Dylan16807 6y ago
  By "most" do you mean excluding those three? I know it's not too hard to get a port at a good rate if I have my own server, and it's easy to rent a server with plenty of bandwidth. But if I want to use something like glacier, I'm screwed. Amazon's "contact us" threshold is $28k per month, and anyone under that is paying 30x-40x as much as if it was $.002/GB.
  - balakk 6y ago
    Cant speak for others, but there are some specialized SKUs in Azure you can opt, for non-metered egress data transfer. https://docs.microsoft.com/en-us/azure/expressroute/expressroute-faqs#expressroute-local https://docs.microsoft.com/en-us/azure/expressroute/expressr... It starts about USD 1200 per month, for a 1 Gbps line.
    - welterde 6y ago
      That's still price gouging imho. For the same amount of money you can get 5-10G of transit bandwidth from large carriers. How can it be cheaper for the big carriers to transport my packets halfway around the planet than for Microsoft to transport them inside the local metro-region to their own datacenter?
    - 6y ago

cbg0 6y ago

Public internet exchanges are huge in Europe, you can basically reach every network in Europe by connecting to a few large exchanges for a fraction of what you'd pay for tier 1 transit (granted, you still probably need it to reach some eyeballs). I wish we'd see the same concept pop up in the US, as these exchanges are definitely not as popular there. https://www.de-cix.net/en/locations/germany/frankfurt/statistics https://www.de-cix.net/en/locations/germany/frankfurt/statis... https://www.ams-ix.net/ams/documentation/total-stats https://www.ams-ix.net/ams/documentation/total-stats https://portal.linx.net/ https://portal.linx.net/

sneak 6y ago
A big part of it is that simply connecting your own network to these exchanges is a lot more expensive in the US due to increased land size and corrupt legislation. Due to the intense regulations (which are bought and paid for at the legislative level, and mostly exist to protect the large incumbents like Verizon, AT&T, Comcast, CenturyLink, et al from any new competitors) you end up having to buy your "last mile" loop (which, due to the US's size and general car-centric design, is much more than just a mile in most cases) that connects your network (in your office, or data center, or whatever, over to the exchange point) from a small number of monopolists. They're thousands, or tens of thousands of dollars per month. "We don't care. We don't have to. We're the phone company."
gbil 6y ago
Internet Exchanges doesn't mean that a peering agreement is also in place automatically. For example: https://www.ams-ix.net/ams/documentation/general-terms-and-conditions https://www.ams-ix.net/ams/documentation/general-terms-and-c... 3.3 Peering arrangements are not covered by these AMS-IX General Terms and Conditions or the AMS-IX Connection Agreement. The Customer is responsible for the negotiation, conclusion and implementation of peering arrangements with other users of the AMS-IX Infrastructure.
- cbg0 6y ago
  While you are correct, many networks connected have in fact an open peering policy: https://www.ams-ix.net/ams/connected-networks https://www.ams-ix.net/ams/connected-networks

3pt14159 6y ago

Disclaimer: My mum works for AT&T on internet related stuff. This post has large elements of truth to it, especially for dumb hops over tier 1 which should be much cheaper than they're currently billed, but there's a lot more going on at AT&T than most people realize. Especially across the oceans. It's not quite as simplistic as this take makes it out to be, since the 5%ile case for networking a packet over tier 1 really is expensive to handle, maintain, bill, and provision. That said, there should be better regulations on tier 1 inclusion. On the other hand, they did make deep investments into infrastructure that took decades to become profitable. Often times with government assurances. It's one of the reasons telephony had such high penetration in Canada (75%) at a time when it was much lower in France (~25%), Bell was given assurances that we're now paying for in more expensive internet. Not saying it's right, but it's like this for understandable reasons, and it's not strictly raw corruption.

2Gkashmiri 6y ago

so why do vps providers provide a "1-3"TB a month transfer when it costs nothing? talking about cheap plans here

exikyut 6y ago
It most definitely does not cost nothing. The whole point here is that the Tier 1s price-gouge everyone else, and from there everything is far more expensive than it needs to be. Prices only go up from there. Cheaper plans sold by smaller outfits are likely being sold to end users through 2-3 layers of resellers. This creates a tug-of-war between multiple resale overheads and customer interest in something at a low price point, so the baseline of what is being offered (CPU, RAM, speed, monthly transfer) can get fairly wobbly. For larger companies, to take Contabo as an example, they basically get to dictate whatever performance metrics they want, because they own the entire stack. So I might have more-or-less-unlimited 100Mbps and 6GB RAM, but CPU performance is... not great, I'm pretty sure my KVM instance winds up almost entirely swapped onto an HDD occasionally, and there are some really harsh QEMU disk I/O quotas.
- 2Gkashmiri 6y ago
  funny that you bring up contabo. i have been thinking about setting up a selfhosted email instance and that could lead to more stuff later on but bandwidth is like night and day. on contabo cheapest vps is like €4.99 / month, 4 vCPU Cores,8 GB RAM, 200 GB SSD, 200 Mbit/s Port, 1 Snapshot and "32 TB of transfer is already included in each VPS package." while on vultr.com, the ipv4 cheapest is like 25 GB SSD, 1 CPU, 1024 MB RAM, 1 TB transfers for $5/mo. that is such a huge difference that i dont know what is real. i want like 10TB storage vps from contabo but apparently they dont have it. do you know anyone like them? i have looked at scalaway and others but they dont say pricing of bandwidth past the allowance. like only for their s3 they say pricing which is exorbitant like the ones mentioned in the article edit: vultr object storage says "$0.01 per additional GB transferred " https://www.vultr.com/products/object-storage/ https://www.vultr.com/products/object-storage/ and scaleway says Inter-regional* and external outgoing data transfer to other products from a different region and the Internet: 75 GB free every month, then €0.01/GB same pricing for backblaze b2 for downloads

phonon 6y ago

Why doesn't Cloudflare buy GTT? They can easily afford to.

Bkmhfdtyb 6y ago

From what I have read, these networking centric companies have a reputation for noncompetitive pay. Is this true? They otherwise sound like great places to work.

2Gkashmiri 6y ago

so i want to start a peertube server, what storage is best, size and ample bandwidth. $0.01/Gb is too much for transfers.... oh, wasabi is not for video, i tried