7 ms·

My brain is a bit fuzzy today but this comparison seems... off? Yes an interconnect inside a colo is going to be dirt cheap. But you can’t compare it to a huge

by erentz 6y ago

My brain is a bit fuzzy today but this comparison seems... off? Yes an interconnect inside a colo is going to be dirt cheap. But you can’t compare it to a huge backbone spanning continents that gives you reach across the world. Everyone should peer with everyone they can locally. Use the tier 1s for reach beyond what you can peer with. (This is not a defense of tier 1s and their peering policies or pricing.) Also this is why it’s always amused me when DC guys quote their raw switchport capacity with pride. Gotta multiply that by distance traveled to get a real comparable metric.

wmf 6y ago
Yeah, this is a bad take from Backblaze. If ISPs had 99% profit margin you'd see it in their quarterly reports. Spoiler warning: they don't. Also, I remember when there were only half as many tier 1s, so if they're a cartel keeping new members out they aren't doing a very good job of it.
- csnover 6y ago
  Frankly, I’ve never seen a good take from Backblaze once I got past their marketing blog posts. Zero-knowledge encryption? You don’t need that[0][1]. Client writes every backup chunk to disk before uploading instead of holding the data in memory, reducing the lifespan of customer SSDs for no reason? It’s fine, don’t worry about it[2][3]. A pattern of violating of basic and well-known software security principles[4][5][6]? Bodge in fixes. Don’t tell customers about it. Cancel the public bug bounty programme, claim it’ll be back in an indeterminate period of time[7] and replace it with a private one instead[8]. [0] https://www.reddit.com/r/backblaze/comments/8oczbl/how_do_i_know_backblaze_can_be_trusted/e095b8x/ https://www.reddit.com/r/backblaze/comments/8oczbl/how_do_i_... [1] https://help.backblaze.com/hc/en-us/articles/217664798-Security-Question-Round-up- https://help.backblaze.com/hc/en-us/articles/217664798-Secur... [2] https://old.reddit.com/r/backblaze/comments/igaqse/please_allow_changing_of_bzdata_directory_heavy/g2ugpwy/ https://old.reddit.com/r/backblaze/comments/igaqse/please_al... [3] https://old.reddit.com/r/backblaze/comments/k764xq/backblaze_temp_drive_on_4gb_ram_drive/geq8x4r/ https://old.reddit.com/r/backblaze/comments/k764xq/backblaze... [4] https://twitter.com/zetafleet/status/1304664097989054464 https://twitter.com/zetafleet/status/1304664097989054464, more discussion at https://news.ycombinator.com/item?id=24842871 https://news.ycombinator.com/item?id=24842871 [5] Hard-coded credentials in deployed apps: https://medium.com/@pig.wig45/from-n-a-to-resolved-for-backblaze-android-app-hackerone-platform-bucket-takeover-f817692a590 https://medium.com/@pig.wig45/from-n-a-to-resolved-for-backb... [6] Relying on client-side validation only: https://www.youtube.com/watch?v=1LC0aEZFVz8 https://www.youtube.com/watch?v=1LC0aEZFVz8 [7] https://news.ycombinator.com/item?id=24849334 https://news.ycombinator.com/item?id=24849334 [8] https://www.backblaze.com/security.html https://www.backblaze.com/security.html
  - Dylan16807 6y ago
    > Client writes every backup chunk to disk before uploading instead of holding the data in memory, reducing the lifespan of customer SSDs for no reason? It’s fine, don’t worry about it[2][3]. There is basically never going to be a meaningful reduction in SSD life from that amount of wear.
    - deleted 6y ago
      [deleted]
    - csnover 6y ago
      Every file is written once (when you change it) and then a second time (when the client writes the same file into a backup chunk). Halving the SSD lifetime is not something I’d personally call meaningless, especially for QLC drives which have two orders of magnitude fewer P/E cycles than SLC drives[0], especially since there’s no good reason for writing these chunks to disk in the first place. [0] https://www.techradar.com/news/nand-and-cells-slc-qlc-tlc-and-mlc-explained https://www.techradar.com/news/nand-and-cells-slc-qlc-tlc-an...
      - brianwski 6y ago
        > Halving the SSD lifetime is not something I’d personally call meaningless ... No, it doesn't halve the SSD lifetime. It is a little odd of you to make that claim. First of all, you are assuming that 100% of SSDs die from being written, and die so quickly and so often from being written to that this is an enormous concern everybody using an SSD in a laptop should be worried about. In reality, a modern SSD you buy today might last you 10 years of normal use (including running Backblaze), but the rest of your computer simply won't last that long. Personally, I've never had one of my SSDs die. For any reason, including too many writes. Often I throw them in the trash perfectly working because I have upgraded their size for less money. Other times I get a whole new computer which comes with a whole new SSD. Backblaze runs SSDs in all of our "monitoring" computers that write the ever loving bejeebus out of their SSD drives 24 hours a day, 7 days a week FOR YEARS. I think we've had 1 SSD go bad after YEARS of writing at a rate 10x up to 100x higher than any home laptop user could achieve. And we're not sure it died due to too many writes. So even if Backblaze doubled the writes (it doesn't, see below) it wouldn't have any measurable effect on your SSD's life. This is just provably false by watching servers that aren't running Backblaze write 10 or even 100 times as many times for 5 years as any consumer could ever achieve, and they still don't kill the SSD drives. So, is Backblaze doubling the writes on your computer? Heck no. Backblaze only backs up valuable data files, not your operating system and certainly not your log files. The vast majority of writes to an SSD are not writing your one photo that you took today to disk - they are database transactions and log writes and system log write and system registry writes, etc. Backblaze puts in a lot of effort to exclude "chatty" log files (log files that are written all the time, all day long) because it saves our customer's network bandwidth and saves Backblaze space in our datacenter. So IN REALITY Backblaze's behavior is adding maybe a small single digit percentage of additional writes. The exact profile will matter what you use your computer for, but take email -> if you get 200 emails a day saved into an Outlook PST Inbox you might have 600 write transactions per day. But Backblaze won't back that up after every email, it would waste too much customer bandwidth. Backblaze might only back that up once per day. And only the part of the PST file that changed! So in reality, Backblaze only added 3 or 4 writes out of 600. > especially since there’s no good reason for writing these chunks to disk Different customers have different needs and different profiles, and Backblaze has to accommodate them. One reason a full snapshot is taken of the large files is that Backblaze wants a consistent "snapshot" of a file. On slow network connections it takes some customers 4 days to upload their Outlook PST file ONCE. Meanwhile they keep getting mail. If Backblaze didn't take a clean snapshot at one moment of time of this file, you'd get this corrupted file where the first 10 MBytes were from a file on Monday, and the last 10 MBytes were copied on Thursday from the same filename but totally different contents. Surely you see how that might be an issue? And customers may or may not have the RAM required to hold a 10 GByte file in RAM, and none of them have enough RAM required to hold a 500 GByte file in RAM. Backblaze needs to store these "snapshots" someplace, so it stores them on disk. Saying this is "for no reason" is disingenuous.
- Dylan16807 6y ago
  What seems to happen when there's insufficient competition for ISPs is that they under-invest in network upgrades. So you wouldn't expect to see crazy profits, it's more a matter of high prices and inefficient operation.
- rsync 6y ago
  "Also, I remember when there were only half as many tier 1s, so if they're a cartel keeping new members out they aren't doing a very good job of it." As evidence for your point I would point to Hurricane Electric (he.net) who I have found to be very progressive, cost-competitive and easy to work with. For historical reasons we[1] are connected with init7 throughout Europe but in the United States and Asia we work with he.net anywhere that we can. [1] rsync.net has a long-standing (15+ year) relationship with init7 in Zurich.
  - sneak 6y ago
    FWIW, HE has also been pushing for IPv6 to be awesome waaaaay longer than IPv6 has been very popular. I don't know them, but I am supremely glad for their 15+ year efforts along these lines.
zamadatix 6y ago
Cloudflare is providing the huge worldwide backbone and saying it's cheaper for them to peer where you are and transport it on their backbone as needed rather than pay Tier 1's to do it. The difference in cost largely comes from being peering not transit though. I.e. they aren't handling traffic for 3rd parties on their backbone (well, in this case, they do offer that though), just traffic between the mutual peering entities. As a result you're never sending them something they didn't already have to pay to get. Also it's not as simple as distance. A lot of data between 2 world class datacenters is pretty easy and cheap, the fiber is already there. A lot of data to the rest of the world is a PITA and a physical/logistical nightmare.
- erentz 6y ago
  Edit: ok you expanded your initial comment. So yes they aren’t providing transit and your own comment explains why this isn’t a good comparison, it appears cheaper than it really is because Cloudlfare aren’t paying transit to get your data.
  - deleted 6y ago
    [deleted]

by erentz 6y ago

wmf 6y ago

Yeah, this is a bad take from Backblaze. If ISPs had 99% profit margin you'd see it in their quarterly reports. Spoiler warning: they don't. Also, I remember when there were only half as many tier 1s, so if they're a cartel keeping new members out they aren't doing a very good job of it.

csnover 6y ago
Frankly, I’ve never seen a good take from Backblaze once I got past their marketing blog posts. Zero-knowledge encryption? You don’t need that[0][1]. Client writes every backup chunk to disk before uploading instead of holding the data in memory, reducing the lifespan of customer SSDs for no reason? It’s fine, don’t worry about it[2][3]. A pattern of violating of basic and well-known software security principles[4][5][6]? Bodge in fixes. Don’t tell customers about it. Cancel the public bug bounty programme, claim it’ll be back in an indeterminate period of time[7] and replace it with a private one instead[8]. [0] https://www.reddit.com/r/backblaze/comments/8oczbl/how_do_i_know_backblaze_can_be_trusted/e095b8x/ https://www.reddit.com/r/backblaze/comments/8oczbl/how_do_i_... [1] https://help.backblaze.com/hc/en-us/articles/217664798-Security-Question-Round-up- https://help.backblaze.com/hc/en-us/articles/217664798-Secur... [2] https://old.reddit.com/r/backblaze/comments/igaqse/please_allow_changing_of_bzdata_directory_heavy/g2ugpwy/ https://old.reddit.com/r/backblaze/comments/igaqse/please_al... [3] https://old.reddit.com/r/backblaze/comments/k764xq/backblaze_temp_drive_on_4gb_ram_drive/geq8x4r/ https://old.reddit.com/r/backblaze/comments/k764xq/backblaze... [4] https://twitter.com/zetafleet/status/1304664097989054464 https://twitter.com/zetafleet/status/1304664097989054464, more discussion at https://news.ycombinator.com/item?id=24842871 https://news.ycombinator.com/item?id=24842871 [5] Hard-coded credentials in deployed apps: https://medium.com/@pig.wig45/from-n-a-to-resolved-for-backblaze-android-app-hackerone-platform-bucket-takeover-f817692a590 https://medium.com/@pig.wig45/from-n-a-to-resolved-for-backb... [6] Relying on client-side validation only: https://www.youtube.com/watch?v=1LC0aEZFVz8 https://www.youtube.com/watch?v=1LC0aEZFVz8 [7] https://news.ycombinator.com/item?id=24849334 https://news.ycombinator.com/item?id=24849334 [8] https://www.backblaze.com/security.html https://www.backblaze.com/security.html
- Dylan16807 6y ago
  > Client writes every backup chunk to disk before uploading instead of holding the data in memory, reducing the lifespan of customer SSDs for no reason? It’s fine, don’t worry about it[2][3]. There is basically never going to be a meaningful reduction in SSD life from that amount of wear.
  - deleted 6y ago
    [deleted]
  - csnover 6y ago
    Every file is written once (when you change it) and then a second time (when the client writes the same file into a backup chunk). Halving the SSD lifetime is not something I’d personally call meaningless, especially for QLC drives which have two orders of magnitude fewer P/E cycles than SLC drives[0], especially since there’s no good reason for writing these chunks to disk in the first place. [0] https://www.techradar.com/news/nand-and-cells-slc-qlc-tlc-and-mlc-explained https://www.techradar.com/news/nand-and-cells-slc-qlc-tlc-an...
    - brianwski 6y ago
      > Halving the SSD lifetime is not something I’d personally call meaningless ... No, it doesn't halve the SSD lifetime. It is a little odd of you to make that claim. First of all, you are assuming that 100% of SSDs die from being written, and die so quickly and so often from being written to that this is an enormous concern everybody using an SSD in a laptop should be worried about. In reality, a modern SSD you buy today might last you 10 years of normal use (including running Backblaze), but the rest of your computer simply won't last that long. Personally, I've never had one of my SSDs die. For any reason, including too many writes. Often I throw them in the trash perfectly working because I have upgraded their size for less money. Other times I get a whole new computer which comes with a whole new SSD. Backblaze runs SSDs in all of our "monitoring" computers that write the ever loving bejeebus out of their SSD drives 24 hours a day, 7 days a week FOR YEARS. I think we've had 1 SSD go bad after YEARS of writing at a rate 10x up to 100x higher than any home laptop user could achieve. And we're not sure it died due to too many writes. So even if Backblaze doubled the writes (it doesn't, see below) it wouldn't have any measurable effect on your SSD's life. This is just provably false by watching servers that aren't running Backblaze write 10 or even 100 times as many times for 5 years as any consumer could ever achieve, and they still don't kill the SSD drives. So, is Backblaze doubling the writes on your computer? Heck no. Backblaze only backs up valuable data files, not your operating system and certainly not your log files. The vast majority of writes to an SSD are not writing your one photo that you took today to disk - they are database transactions and log writes and system log write and system registry writes, etc. Backblaze puts in a lot of effort to exclude "chatty" log files (log files that are written all the time, all day long) because it saves our customer's network bandwidth and saves Backblaze space in our datacenter. So IN REALITY Backblaze's behavior is adding maybe a small single digit percentage of additional writes. The exact profile will matter what you use your computer for, but take email -> if you get 200 emails a day saved into an Outlook PST Inbox you might have 600 write transactions per day. But Backblaze won't back that up after every email, it would waste too much customer bandwidth. Backblaze might only back that up once per day. And only the part of the PST file that changed! So in reality, Backblaze only added 3 or 4 writes out of 600. > especially since there’s no good reason for writing these chunks to disk Different customers have different needs and different profiles, and Backblaze has to accommodate them. One reason a full snapshot is taken of the large files is that Backblaze wants a consistent "snapshot" of a file. On slow network connections it takes some customers 4 days to upload their Outlook PST file ONCE. Meanwhile they keep getting mail. If Backblaze didn't take a clean snapshot at one moment of time of this file, you'd get this corrupted file where the first 10 MBytes were from a file on Monday, and the last 10 MBytes were copied on Thursday from the same filename but totally different contents. Surely you see how that might be an issue? And customers may or may not have the RAM required to hold a 10 GByte file in RAM, and none of them have enough RAM required to hold a 500 GByte file in RAM. Backblaze needs to store these "snapshots" someplace, so it stores them on disk. Saying this is "for no reason" is disingenuous.
Dylan16807 6y ago
What seems to happen when there's insufficient competition for ISPs is that they under-invest in network upgrades. So you wouldn't expect to see crazy profits, it's more a matter of high prices and inefficient operation.
rsync 6y ago
"Also, I remember when there were only half as many tier 1s, so if they're a cartel keeping new members out they aren't doing a very good job of it." As evidence for your point I would point to Hurricane Electric (he.net) who I have found to be very progressive, cost-competitive and easy to work with. For historical reasons we[1] are connected with init7 throughout Europe but in the United States and Asia we work with he.net anywhere that we can. [1] rsync.net has a long-standing (15+ year) relationship with init7 in Zurich.
- sneak 6y ago
  FWIW, HE has also been pushing for IPv6 to be awesome waaaaay longer than IPv6 has been very popular. I don't know them, but I am supremely glad for their 15+ year efforts along these lines.

zamadatix 6y ago

Cloudflare is providing the huge worldwide backbone and saying it's cheaper for them to peer where you are and transport it on their backbone as needed rather than pay Tier 1's to do it. The difference in cost largely comes from being peering not transit though. I.e. they aren't handling traffic for 3rd parties on their backbone (well, in this case, they do offer that though), just traffic between the mutual peering entities. As a result you're never sending them something they didn't already have to pay to get. Also it's not as simple as distance. A lot of data between 2 world class datacenters is pretty easy and cheap, the fiber is already there. A lot of data to the rest of the world is a PITA and a physical/logistical nightmare.

erentz 6y ago
Edit: ok you expanded your initial comment. So yes they aren’t providing transit and your own comment explains why this isn’t a good comparison, it appears cheaper than it really is because Cloudlfare aren’t paying transit to get your data.
- deleted 6y ago
  [deleted]