4 ms·
The trouble with Parquet (and columnar storage) in ML is, 1. You don't really care too-much about accessing subsets of columns 2. You can't easily append stuf
by th24o3j4324234 2y ago
The trouble with Parquet (and columnar storage) in ML is,
1. You don't really care too-much about accessing subsets of columns
2. You can't easily append stuff to closed Parquet files.
3. Batched-row access is presumably slower due to lower cache-hits.
It's okay for map-reduce style stuff where this doesn't matter, but in ML these limitations are an annoyance.
HDF5 (or Zarr, less portably) solves some/many of these issues but it's not quite a settled affair.
- jononor 2y agoRe 2. Parquet can easily be used with chunked/partitioned files. Then appending is just adding another file/chunk. The case of 1. really depends on the workload. For embeddings etc selecting column subsets is rare. In order cases, where one has a a bunch of separate features, doing column subsetting might be rather common. But yes, it is far from every case.
- ismailmaj 2y agoParquet is only a mess if you try to mutate it, usually you consider them as immutable and have the data stored across many files. Also batched-row access is negligible given the compression benefits you get with the columnar format, which is probably why it's still king in ML; I think given what I'm seeing in the industry and recent trends (e.g. Velox).