3 ms·
Pgrust v0.2: Now faster than Postgres and Clickhouse Latest
- convolvatron 2mo agothere's some great stuff in here, particularly the direct-to-binary codegen, threads instead of processes and sync batching. you must have some intuitive notion about where the performance gains come from.
- KinetiNode 2mo ago[dead]
- westurner 2mo agoWould there be an advantage to reusing SIMD-optimized libraries for this? Polars Rust is built on Arrow and packed_simd. pola-rs/polars: https://github.com/pola-rs/polars https://github.com/pola-rs/polars polars - Rust API docs: https://docs.pola.rs/api/rust/dev/polars/#simd https://docs.pola.rs/api/rust/dev/polars/#simd lancedb's data format; Lancedb/lance works with [Pandas, DuckDB, Polars, PyArrow,]; https://github.com/lancedb/lance https://github.com/lancedb/lance Narwhals' df interface (Python) https://narwhals-dev.github.io/narwhals/ https://narwhals-dev.github.io/narwhals/ substrait's portable query plans: https://substrait.io/ https://substrait.io/ , https://github.com/ibis-project/ibis-substrait https://github.com/ibis-project/ibis-substrait Arrow RecordBatch, https://news.ycombinator.com/item?id=45495738#45546244 https://news.ycombinator.com/item?id=45495738#45546244 cargo-fuzz, TLA+
- malisper 2mo agoAt least for ClickBench, the remaining bottleneck is memory bandwidth. I think that's mostly going to be solved by better data representations than SIMD. One of the biggest wins was our hash table implementation. Depending on the cardinality of the data, it switches between design that is optimized for L2 cache vs something that is outside of L2.
- westurner 2mo agoTigerBeetle has static allocation for their known workload. SIMD is more useful for heavy parallel analytical workloads. There are probably returns from SIMD even for index updates on transactional inserts. SIMD is faster for: vectorized index comparisons, Vectorized Constraint Validation, Multi-Column SIMD Hashing, Masked Bitmaps for Nullable Fields, gather and scatter, string processing, casting to numeric types. IIUC FWIW Cerebras' does not have L2 cache. TIL Vortex is Zero-Copy compatible with Arrow; > vortex-data/vortex: An extensible, state-of-the-art framework for columnar compression, and the fastest FOSS columnar file format. Formerly at @spiraldb, now an Incubation Stage project at LFAI&Data, part of the Linux Foundation. https://github.com/vortex-data/vortex https://github.com/vortex-data/vortex Vortex also has Segment Profiling and Adaptive Encoding. "What is Vortex? Columnar File Format Explained" https://spice.ai/learn/vortex https://spice.ai/learn/vortex
- jauntywundrkind 2mo agoPort in OrioleDB next please. If our wish-granting really is at such a state. https://github.com/orioledb/orioledb https://github.com/orioledb/orioledb
- jauntywundrkind 2mo agoPort in OrioleDB next please. If our wish-granting really is at such a state. https://github.com/orioledb/orioledb https://github.com/orioledb/orioledb Great submission slipped by yesterday (!) on it's beta15 and beta16 released, on a big stability push. It now passes Postgres's own test suite! https://www.orioledb.com/blog/orioledb-beta15-16-stability https://www.orioledb.com/blog/orioledb-beta15-16-stability https://news.ycombinator.com/item?id=49099108 https://news.ycombinator.com/item?id=49099108
- malisper 2mo agoOnce it's stable, that's definitely something I'm going to look at doing
- kiwicopple 2mo ago(supabase cofounder) reach out if you need any help integrating. contact details are in my profile
- zX41ZdbW 2mo agoYou should submit it to ClickBench for verification.