azimafroozeh

5 ms·

1.
by azimafroozeh 1y ago
We’re building a community around FastLanes, and we’d love any contributions or feedback! Please join our Discord so we can discuss things in more detail.
2.
by azimafroozeh 1y ago
As we discuss in the FastLanes paper, the way BtrBlocks implements cascaded encodings (Vortex now) is essentially a return to block-based compression such as Zstd — which we're trying to avoid as much as possible. This design doesn&#x2
3.
by azimafroozeh 1y ago
CUDA and GPU support are next on our list — but we’re definitely interested in WebAssembly as well!
4.
by azimafroozeh 1y ago
That’s a very valid question. We’ve done zero optimization on the encoding side so far, and improving that is definitely on our roadmap. Technically, once we learn the best expressions, they can be reused — data is often very similar across
5.
by azimafroozeh 1y ago
Vortex borrows a few ideas from the FastLanes project, such as bit-packing and ALP. However, it’s unclear how well these are implemented — their performance on ClickBench appears worse than Parquet in both storage size and decompression spe
6.
by azimafroozeh 1y ago
I totally agree — the C++ implementations of Parquet aren’t the best they could be. To be fair, though, it’s a tough problem: Parquet supports a broad range of encodings and compression schemes, and to do that, it pulls in a lot of dependen
7.
by azimafroozeh 1y ago
We’d love to bring FastLanes into DuckDB! We're currently working on a DuckDB extension to read and write FastLanes file formats directly from DuckDB. FastLanes is columnar by design, so partial downloading via HTTP Range requests is d
8.
by azimafroozeh 1y ago
We haven’t benchmarked FastLanes directly against LanceDB yet, but here’s a quick look at the compression side: LanceDB supports: FSST Bit-packing Delta encoding Opaque block codecs: GZIP, LZ4, Snappy, ZLIB So in that regard, it’s quite sim
9.
by azimafroozeh 1y ago
Arrow is primarily an in-memory format, while Parquet is commonly used for on-disk storage. Typically, data is stored in Parquet and then read into memory as Arrow. FastLanes is a new on-disk file format, comparable to Parquet, but offers a