4 ms·

Supabase Acquires OrioleDB

14 points by gmcabrita 2y ago

kiwicopple 2y ago
hey hn, supabase ceo here we've been fans of Oriole for a while now and have been long-time supporters in case you're jumping straight to the comments: OrioleDB is a table storage extension for Postgres. it acts as a drop-in replacement for the default postgres storage engine using the Table Access Method APIs (pluggable storage). the storage engine changes the representation of table data on disk. its architecture is designed to take advantage of modern hardware like SSDs and NVRAM. it implements MVCC, the feature that allows allows multiple connected users to see different versions of the data depending on when their transaction started, via an UNDO log rather than tuple versioning. one caveat: it requires several patches to the postgres core to expand on the type of features external storage engines extensions can implement. for this reason it could be a while before you see this land as a default engine on supabase. we will probably make it available as an option for customers who want to experiment - no timeline is decided yet. finally, we have been working with the team on decoupled storage and compute [0]. this is experimental but promising, especially with some recent advances in S3 (specifically Express One Zone [1]). we have a demonstration in the blog post. i'll message Alexander in case there are any technical questions [0] https://github.com/orioledb/orioledb/blob/main/doc/usage.md#s3-database-storage-experimental https://github.com/orioledb/orioledb/blob/main/doc/usage.md#... [1] https://aws.amazon.com/s3/storage-classes/express-one-zone/ https://aws.amazon.com/s3/storage-classes/express-one-zone/
- akorotkov 2y ago
  I'm happy to join the Supabase. Working in Supabase gives me a unique opportunity to find use cases for OrioleDB and scale up the development team. I'll be around to answer any technical questions.
Cwizard 2y ago
I didn’t know ‘Table Access Method Interface’ has this been around for a while? Who developed it?
- akorotkov 2y ago
  This was driven by Andres Freund for pg12. Please, check this. https://anarazel.de/talks/2018-10-25-pgconfeu-pluggable-storage/pluggable.pdf https://anarazel.de/talks/2018-10-25-pgconfeu-pluggable-stor... However, the current table AM API in many aspects assumes heap-like storage. This is why OrioleDB comes with a patchset we're intended to upstream to PostgreSQL.
Cwizard 2y ago
> It implements row-level WAL (Write-Ahead Log) and a non-persistent undo log. This significantly reduces IO operations for write transactions. What is the downside of doing this? I assume there is a tradeoff?
- akorotkov 2y ago
  Sure, there are downsides. 1. The replay of WAL records becomes somewhat more CPU-expensive. But at the same time, OrioleDB can do that in parallel. 2. Replace becomes not the file-level equivalent of primary. Thus, you, for instance, can't continue file-level incremental backup started taken from the primary with the replica. Nevertheless, I strongly believe that row-level WAL is a great win on average, and opens brilliant perspectives for further developments.

14 points by gmcabrita 2y ago

hey hn, supabase ceo here we've been fans of Oriole for a while now and have been long-time supporters in case you're jumping straight to the comments: OrioleDB is a table storage extension for Postgres. it acts as a drop-in replacement for the default postgres storage engine using the Table Access Method APIs (pluggable storage). the storage engine changes the representation of table data on disk. its architecture is designed to take advantage of modern hardware like SSDs and NVRAM. it implements MVCC, the feature that allows allows multiple connected users to see different versions of the data depending on when their transaction started, via an UNDO log rather than tuple versioning. one caveat: it requires several patches to the postgres core to expand on the type of features external storage engines extensions can implement. for this reason it could be a while before you see this land as a default engine on supabase. we will probably make it available as an option for customers who want to experiment - no timeline is decided yet. finally, we have been working with the team on decoupled storage and compute [0]. this is experimental but promising, especially with some recent advances in S3 (specifically Express One Zone [1]). we have a demonstration in the blog post. i'll message Alexander in case there are any technical questions [0] https://github.com/orioledb/orioledb/blob/main/doc/usage.md#s3-database-storage-experimental https://github.com/orioledb/orioledb/blob/main/doc/usage.md#... [1] https://aws.amazon.com/s3/storage-classes/express-one-zone/ https://aws.amazon.com/s3/storage-classes/express-one-zone/

akorotkov 2y ago
I'm happy to join the Supabase. Working in Supabase gives me a unique opportunity to find use cases for OrioleDB and scale up the development team. I'll be around to answer any technical questions.

Cwizard 2y ago

I didn’t know ‘Table Access Method Interface’ has this been around for a while? Who developed it?

akorotkov 2y ago
This was driven by Andres Freund for pg12. Please, check this. https://anarazel.de/talks/2018-10-25-pgconfeu-pluggable-storage/pluggable.pdf https://anarazel.de/talks/2018-10-25-pgconfeu-pluggable-stor... However, the current table AM API in many aspects assumes heap-like storage. This is why OrioleDB comes with a patchset we're intended to upstream to PostgreSQL.

Cwizard 2y ago

> It implements row-level WAL (Write-Ahead Log) and a non-persistent undo log. This significantly reduces IO operations for write transactions. What is the downside of doing this? I assume there is a tradeoff?

akorotkov 2y ago
Sure, there are downsides. 1. The replay of WAL records becomes somewhat more CPU-expensive. But at the same time, OrioleDB can do that in parallel. 2. Replace becomes not the file-level equivalent of primary. Thus, you, for instance, can't continue file-level incremental backup started taken from the primary with the replica. Nevertheless, I strongly believe that row-level WAL is a great win on average, and opens brilliant perspectives for further developments.