3 ms·

Idk, but I'd rather see a new iteration of PostgreSQL, similiar to Hydra[0], with a new engine become the upstream instead of a whole new database. There's a lo

by c0balt 3y ago

Idk, but I'd rather see a new iteration of PostgreSQL, similiar to Hydra[0], with a new engine become the upstream instead of a whole new database. There's a lot of experience about db operation and how to approach MVCC encoded in PostgreSQL that shouldn't be underestimated. [0]: https://github.com/hydradatabase/hydra https://github.com/hydradatabase/hydra

refset 3y ago
Another Postgres-based project in this vein that makes use of Apache Arrow: https://heterodb.github.io/pg-strom/ https://heterodb.github.io/pg-strom/ > PG-Strom is an extension module of PostgreSQL designed for version 11 or later. By utilization of GPU (Graphic Processor Unit) device which has thousands cores per chip, it enables to accelerate SQL workloads for data analytics or batch processing to big data set. > PG-Strom has two storage options. The first one is the heap storage system of PostgreSQL. It is not always optimal for aggregation / analysis workloads because of its row data format, on the other hands, it has an advantage to run aggregation workloads without data transfer from the transactional database. The other one is Apache Arrow files, that have structured columnar format. Even though it is not suitable for update per row basis, it enables to import large amount of data efficiently, and efficiently search / aggregate the data through foreign data wrapper (FDW).
cmrdporcupine 3y ago
The MVCC implementation inside Postgres is probably not the greatest, in terms of overall performance. This is a good read https://db.cs.cmu.edu/papers/2017/p781-wu.pdf https://db.cs.cmu.edu/papers/2017/p781-wu.pdf "Postgres ... configurations lead to the worst performance, and the major reason is that the use of append-only storage with O2N ordering severely restricts the scalability of the system.... "

3 ms·

Idk, but I'd rather see a new iteration of PostgreSQL, similiar to Hydra[0], with a new engine become the upstream instead of a whole new database. There's a lo

by c0balt 3y ago

refset 3y ago
Another Postgres-based project in this vein that makes use of Apache Arrow: https://heterodb.github.io/pg-strom/ https://heterodb.github.io/pg-strom/ > PG-Strom is an extension module of PostgreSQL designed for version 11 or later. By utilization of GPU (Graphic Processor Unit) device which has thousands cores per chip, it enables to accelerate SQL workloads for data analytics or batch processing to big data set. > PG-Strom has two storage options. The first one is the heap storage system of PostgreSQL. It is not always optimal for aggregation / analysis workloads because of its row data format, on the other hands, it has an advantage to run aggregation workloads without data transfer from the transactional database. The other one is Apache Arrow files, that have structured columnar format. Even though it is not suitable for update per row basis, it enables to import large amount of data efficiently, and efficiently search / aggregate the data through foreign data wrapper (FDW).
cmrdporcupine 3y ago
The MVCC implementation inside Postgres is probably not the greatest, in terms of overall performance. This is a good read https://db.cs.cmu.edu/papers/2017/p781-wu.pdf https://db.cs.cmu.edu/papers/2017/p781-wu.pdf "Postgres ... configurations lead to the worst performance, and the major reason is that the use of append-only storage with O2N ordering severely restricts the scalability of the system.... "