3 ms·

Brian from Backblaze here. If we do a full pod swap for a blank, we are disk / CPU limited in the rebuild, not 10 Gbit/sec limited. We have the rebuilds up to

Brian from Backblaze here. If we do a full pod swap for a blank, we are disk / CPU limited in the rebuild, not 10 Gbit/sec limited. We have the rebuilds up to taking about 3/4ths of the 10 Gbit/sec link but we continue to try to change the software and buffering to get this higher. It's MUCH better to lose only one drive, because we distribute the rebuild CPU task across all 20 CPUs in the tome and we can get it synced back up in a few hours. All the chattering back and forth still doesn't come anywhere close to maxing out the 10 Gbit. For uploads, each individual pod has a 10 Gbit/sec connection, so ACTUALLY the vault has an aggregate of 200 Gbit/sec facing towards the internet. However, every byte that a pod accepts it has to ALSO retransmit that internally so that cuts the actual theoretical upload rate to 100 Gbit/sec for a properly parallelized application.

vessenes 10y ago
Thanks for the blogging and open designs, it's inspiring! Spelling error: check for 'utlizing' -> 'utilizing'.
- brianwski 10y ago
  > 'utlizing' -> 'utilizing' Dang it! We will fix, thanks.
nocarrier 10y ago
Thanks for the reply Brian. I'm a big fan of sharding across hosts and doing your own RS. I'm mildly surprised that you're disk and CPU limited, but there's lots of factors that can come into play with that and it sounds like you are happy with the tradeoffs which is all that really matters. I'm a big fan of what Backblaze has done. If you ever want to talk about replication strategies or similar stuff, I've built large scale storage systems in the past--please feel free to reach out to the email in my profile if you'd like to talk shop and trade notes.
scurvy 10y ago
> If we do a full pod swap for a blank, we are disk / CPU limited in the rebuild, not 10 Gbit/sec limited Wow. This is pretty shocking TBH. My basic ceph nodes will crush a 40gbps link doing a rebuild/rebalance. What's your bottleneck?
- mappu 10y ago
  Backblaze use an in-house Java pseudo-RAID layer based on Reed-Solomon codes, with 17 data shards and 3 parity shards. There are some performance numbers here https://www.backblaze.com/blog/reed-solomon/ https://www.backblaze.com/blog/reed-solomon/ (2015).
  - nwmcsween 10y ago
    I wouldn't mind understanding why, I'm assuming at the time ceph wasn't mature enough (and still somewhat is due to btrfs, yes I know it's optional).
    - scurvy 10y ago
      btrfs isn't a requirement nor really a recommendation for production ceph clusters. It was really only there for weirdo benchers. Bluestore and rocks DB really changes the back-end story altogether. No more POSIX filesystem junk with journals and such.
nwmcsween 10y ago
What is the random io like? If it's high why not cache with some ssds, it would help probably run with less power as well as faster and with less wear on the drives due to turning random io into sequential io.

vessenes 10y ago

Thanks for the blogging and open designs, it's inspiring! Spelling error: check for 'utlizing' -> 'utilizing'.

brianwski 10y ago
> 'utlizing' -> 'utilizing' Dang it! We will fix, thanks.

nocarrier 10y ago

Thanks for the reply Brian. I'm a big fan of sharding across hosts and doing your own RS. I'm mildly surprised that you're disk and CPU limited, but there's lots of factors that can come into play with that and it sounds like you are happy with the tradeoffs which is all that really matters. I'm a big fan of what Backblaze has done. If you ever want to talk about replication strategies or similar stuff, I've built large scale storage systems in the past--please feel free to reach out to the email in my profile if you'd like to talk shop and trade notes.

scurvy 10y ago

> If we do a full pod swap for a blank, we are disk / CPU limited in the rebuild, not 10 Gbit/sec limited Wow. This is pretty shocking TBH. My basic ceph nodes will crush a 40gbps link doing a rebuild/rebalance. What's your bottleneck?

mappu 10y ago
Backblaze use an in-house Java pseudo-RAID layer based on Reed-Solomon codes, with 17 data shards and 3 parity shards. There are some performance numbers here https://www.backblaze.com/blog/reed-solomon/ https://www.backblaze.com/blog/reed-solomon/ (2015).
- nwmcsween 10y ago
  I wouldn't mind understanding why, I'm assuming at the time ceph wasn't mature enough (and still somewhat is due to btrfs, yes I know it's optional).
  - scurvy 10y ago
    btrfs isn't a requirement nor really a recommendation for production ceph clusters. It was really only there for weirdo benchers. Bluestore and rocks DB really changes the back-end story altogether. No more POSIX filesystem junk with journals and such.

nwmcsween 10y ago

What is the random io like? If it's high why not cache with some ssds, it would help probably run with less power as well as faster and with less wear on the drives due to turning random io into sequential io.