7 ms·
Parquet + Arrow hopefully seem to be emerging as standards. I would much rather see those standards improved than new formats emerge. Even within those existing
by RyanHamilton 2y ago
Parquet + Arrow hopefully seem to be emerging as standards. I would much rather see those standards improved than new formats emerge. Even within those existing formats there has become enough variation than some platforms only support a subset of functionality. That and the performance and size of the libraries is poor.
e.g. DuckDB / Clickhouse Parquet nanosecond compatibility. https://github.com/duckdb/duckdb/issues/9852 https://github.com/duckdb/duckdb/issues/9852
e.g. The arrow SQL driver is 70+MB in java.
- HackerThemAll 2y agoThe Meta's own ORC is quite popular, too, in addition to Parquet, Arrow, Iceberg, Delta, Velox, Lance or Avro. So I assume the new one will find its way into lake houses/data warehouses as well. Because we need bigger mess and bloat.