5 ms·
The FastLanes File Format [pdf]
- k__ 1y agoPretty cool! WebAssembly planned?
- azimafroozeh 1y agoCUDA and GPU support are next on our list — but we’re definitely interested in WebAssembly as well!
- falcor84 1y agoHow does it compare to Arrow?
- azimafroozeh 1y agoArrow is primarily an in-memory format, while Parquet is commonly used for on-disk storage. Typically, data is stored in Parquet and then read into memory as Arrow. FastLanes is a new on-disk file format, comparable to Parquet, but offers around 40% better compression and faster decoding thanks to its data-parallel encoding design. Disclaimer: I'm the first author of the paper.
- abirch 1y agoWhat about Feather? This is on my to do list, but I thought that Feather was a file format based on Arrow: https://docs.pola.rs/api/python/stable/reference/api/polars.DataFrame.write_ipc.html#polars.DataFrame.write_ipc https://docs.pola.rs/api/python/stable/reference/api/polars.... This is referenced in the link above. https://arrow.apache.org/docs/python/ipc.html https://arrow.apache.org/docs/python/ipc.html Unfortunately I'm stuck with CSV at work for now.
- danking00 1y agoFeather appears to just be block compressed Arrow IPC [1]. Lightweight compression techniques generally achieve two orders of magnitude faster random access compared to block compression. That’s one of the benefits of formats like FastLanes, Vortex, DuckDB native, etc. DuckDB has a good blog post about it here: https://duckdb.org/2022/10/28/lightweight-compression.html https://duckdb.org/2022/10/28/lightweight-compression.html [1]: https://arrow.apache.org/docs/python/feather.html https://arrow.apache.org/docs/python/feather.html
- deleted 1y ago[deleted]
- timhigins 1y agoAny comparisons with Lance/LanceDB?
- azimafroozeh 1y agoWe haven’t benchmarked FastLanes directly against LanceDB yet, but here’s a quick look at the compression side: LanceDB supports: FSST Bit-packing Delta encoding Opaque block codecs: GZIP, LZ4, Snappy, ZLIB So in that regard, it’s quite similar to Parquet — a mix of lightweight codecs and general-purpose block compression. FastLanes, on the other hand, introduces Expression Encoding — a unified compression model that allows combining lightweight encodings to achieve better compression ratios. It also integrates multiple research efforts from CWI into a single file format: The FastLanes Compression Layout: Decoding >100 Billion Integers per Second with Scalar Code (VLDB '23) PDF: https://dl.acm.org/doi/pdf/10.14778/3598581.3598587 https://dl.acm.org/doi/pdf/10.14778/3598581.3598587 ALP (Adaptive Lossless Floating-Point Compression) — SIGMOD '24 https://ir.cwi.nl/pub/33334/33334.pdf https://ir.cwi.nl/pub/33334/33334.pdf G‑ALP (GPU-parallel variant of ALP) — DaMoN '25 https://azimafroozeh.org/assets/papers/g-alp.pdf https://azimafroozeh.org/assets/papers/g-alp.pdf White-box Compression (self-describing, function-based) — CIDR '20 https://www.cidrdb.org/cidr2020/papers/p4-ghita-cidr20.pdf https://www.cidrdb.org/cidr2020/papers/p4-ghita-cidr20.pdf CCC (Exploiting Column Correlations for Compression) — MSc Thesis '23 https://homepages.cwi.nl/~boncz/msc/2023-ThomasGlas.pdf https://homepages.cwi.nl/~boncz/msc/2023-ThomasGlas.pdf
- westonpace 1y agoLance contributor here. This sounds about right. We haven't really innovated too much in the compression space. Most of our efforts have been around getting rid of row groups and the resulting changes in decoding patterns. Our current approach is pretty similar to Parquet for scalar types. We allow a mix of general and lightweight codecs for small types and require lightweight only codecs for larger types (string, binary). Nice work on the paper :)
- timhigins 1y agoHow well does this work with partial downloading of a file based on column statistics via HTTP Range requests? Any plans to integrate FastLanes into DuckDB? You would instantly get more usage from its SDKs in many languages and the CLI.
- azimafroozeh 1y agoWe’d love to bring FastLanes into DuckDB! We're currently working on a DuckDB extension to read and write FastLanes file formats directly from DuckDB. FastLanes is columnar by design, so partial downloading via HTTP Range requests is definitely possible — though not yet implemented. It’s on the roadmap.
- nly 1y agoMy experience with parquet is that the (C++) libraries are pretty awful and resulting performance lackluster
- azimafroozeh 1y agoI totally agree — the C++ implementations of Parquet aren’t the best they could be. To be fair, though, it’s a tough problem: Parquet supports a broad range of encodings and compression schemes, and to do that, it pulls in a lot of dependencies and requires complex dependency management. That’s one of the main reasons we built FastLanes from scratch instead of trying to integrate with Parquet. With FastLanes, we’ve taken a different approach: zero dependencies, no SIMD intrinsics, and a design that’s fully auto-vectorizable. The result is simpler code that still delivers high performance.
- jmakov 1y agoAny comparison with https://github.com/vortex-data/vortex https://github.com/vortex-data/vortex? Also any plans to integrate with polars?
- azimafroozeh 1y agoVortex borrows a few ideas from the FastLanes project, such as bit-packing and ALP. However, it’s unclear how well these are implemented — their performance on ClickBench appears worse than Parquet in both storage size and decompression speed, which is counterintuitive. Technically, Vortex is documented more like a BtrBlocks-style format, which we’ve benchmarked and compared against in depth.
- _willmanning 1y agoI'll chime in (as a Vortex maintainer), that we are greatly indebted to Azim's work on FastLanes & ALP. Vortex heavily utilizes his work to get state-of-the-art performance. I would add that Vortex doesn't have standalone Clickbench results. Azim is presumably referring to the duckdb-vortex results, which were run on an older version of duckdb (1.2) than the duckdb-parquet ones (1.3). We'll get those updated shortly; we just released a new version of Vortex & the duckdb extension. Meanwhile, I believe the DataFusion-Vortex vs DataFusion-Parquet speedups show substantial improvements across the board. The folks over at TUM (who originally authored BtrBlocks) did a reasonable amount of micro-benchmarking of Vortex vs Parquet in their recent "Anyblox" paper for VLDB 2025: https://gienieczko.com/anyblox-paper https://gienieczko.com/anyblox-paper They essentially say in the paper that Vortex is much faster than the original BtrBlocks because it uses better encodings (specifically citing FastLanes & ALP). I'm looking forward to seeing the FastLanes Clickbench results when they're ready, and Azim, we should work together to benchmark FastLanes against Vortex!
- azimafroozeh 1y agoAs we discuss in the FastLanes paper, the way BtrBlocks implements cascaded encodings (Vortex now) is essentially a return to block-based compression such as Zstd — which we're trying to avoid as much as possible. This design doesn't work well with modern vectorized execution engines or GPUs: the decompression granularity is too large to fit in CPU caches or GPU shared memory. So Vortex ends up being yet another Parquet-like file format, repeating the same mistakes. And if it still underperforms compared to Parquet... what’s the point? We just released FastLanes v0.1, and more results — including ClickBench — are coming soon. Please do benchmark FastLanes — and keep us posted!
- thinkharderdev 1y agoHow well does the expression detection scale with number of columns? If I am reading Table 4 correctly FastLanes is ~10x slower at encoding than parquet+snappy (which seems a reasonable tradeoff for the better compression and scan times) but how is that affected for very wide tables (eg 2k columns or something like that)
- azimafroozeh 1y agoThat’s a very valid question. We’ve done zero optimization on the encoding side so far, and improving that is definitely on our roadmap. Technically, once we learn the best expressions, they can be reused — data is often very similar across row groups — which opens the door to caching and amortizing the cost. For very wide tables, expression detection only needs to happen once. Beyond that, we’re also exploring techniques like grouping columns into smaller sets or applying more aggressive heuristics to prune irrelevant columns. These are areas we’re actively investigating, and we plan to support them in future versions of FastLanes.
- Mindjolt 1y agoIs there a place where we can contribute?
- azimafroozeh 1y agoWe’re building a community around FastLanes, and we’d love any contributions or feedback! Please join our Discord so we can discuss things in more detail.