11 ms·

I am genuinely curious how a ZFS "special" device[1] which absorbs all metadata for a pool would help this kind of a workload. The "special" device is a full-b

by rsync 5y ago

I am genuinely curious how a ZFS "special" device[1] which absorbs all metadata for a pool would help this kind of a workload. The "special" device is a full-blown vdev that you add to a pool (it is not a cache) - typically a 3 or 4-way mirror of SSDs. Now all metadata read and write happen at SSD speeds. We wrote about this in the rsync.net Technical Notes for Q3 of this year[2]. I know what this kind of SSD based vdev does to typical mixed file performance but I'm not sure how metadata-heavy a postgres implementation is ... [1] Yes, they really are called that. [2] https://www.rsync.net/resources/notes/2021-q3-rsync.net_technotes.html https://www.rsync.net/resources/notes/2021-q3-rsync.net_tech...

latk 5y ago
Would separate SSD metadata devices help if the pool, as in Heap.io's case, already consists entirely of SSDs? It's obviously a win for a use case like Rsync.net's where the data is less “hot”and therefore uses much more cost-effective HDDs.
- franga2000 5y ago
  Would be interesting to see if Optane or even just some faster SSDs for the metadata would give any noticeable improvement. I imagine latency would be more important for metadata than throughput, so perhaps SSDs would be more or less equivalent, but I'd be really interested in seeing the numbers for Optane
  - buildbot 5y ago
    This is actually one of the golden use cases for optane! Back in the pre-optane day, the server company I worked for would back SSD/HDD zpools with a device called a ZeusRAM drive: https://www.ebay.com/itm/234256677232 https://www.ebay.com/itm/234256677232 It's all about the latency.
- withinboredom 5y ago
  The benefits would be that metadata type FS queries are OOB from the actual data, theoretically allowing more IOPs on your data disks spent on actual data.
  - smallnamespace 5y ago
    If you just add metadata SSDs you’re also adding IOPS to the pool. The question then becomes whether that improves performance more than if you added those same SSDs without the split (my guess would be not splitting is better, since the IO load will be better balanced across drives).
- nine_k 5y ago
  It could possibly help a little bit if the special device is an extra-fast NVMe SLC, and the bulk are SATA MLCs.
aftbit 5y ago
Thanks for posting about this. TIL about special devices.

by rsync 5y ago

latk 5y ago

Would separate SSD metadata devices help if the pool, as in Heap.io's case, already consists entirely of SSDs? It's obviously a win for a use case like Rsync.net's where the data is less “hot”and therefore uses much more cost-effective HDDs.

franga2000 5y ago
Would be interesting to see if Optane or even just some faster SSDs for the metadata would give any noticeable improvement. I imagine latency would be more important for metadata than throughput, so perhaps SSDs would be more or less equivalent, but I'd be really interested in seeing the numbers for Optane
- buildbot 5y ago
  This is actually one of the golden use cases for optane! Back in the pre-optane day, the server company I worked for would back SSD/HDD zpools with a device called a ZeusRAM drive: https://www.ebay.com/itm/234256677232 https://www.ebay.com/itm/234256677232 It's all about the latency.
withinboredom 5y ago
The benefits would be that metadata type FS queries are OOB from the actual data, theoretically allowing more IOPs on your data disks spent on actual data.
- smallnamespace 5y ago
  If you just add metadata SSDs you’re also adding IOPS to the pool. The question then becomes whether that improves performance more than if you added those same SSDs without the split (my guess would be not splitting is better, since the IO load will be better balanced across drives).
nine_k 5y ago
It could possibly help a little bit if the special device is an extra-fast NVMe SLC, and the bulk are SATA MLCs.

aftbit 5y ago

Thanks for posting about this. TIL about special devices.