4 ms·

How data is split in chunks ? Just curious.

• deleted 4y ago
  [deleted]
• sesm 4y ago
  They mention 'content-defined chunking', but it as far as understand it requires different chunking algorithms for different content types. Does it support plugins for chunking different file formats?
  • ylow 4y ago
    Today we just have a variation of FastCDC in production, but we have alternate experimental chunkers for some file formats (ex: a heuristic chunker for CSV files that will enable almost free subsampling). Hope to have them enter production in the next 6 months.
    • sesm 4y ago
      That's interesting. Can a CSV chunker make adding a column not affect all of the chunks?
      • ylow 4y ago
        The simplest really is to chunk row-wise so adding columns will unfortunately rewrite all the chunks. If you have a parquet file, adding columns will be cheap.
• ylow 4y ago
  CEO/Cofounder here! Content defined chunking. Specifically a variation of FastCDC. We have a paper coming out soon with a lot more technical details.