19 ms·

OrioleDB – solving some PostgreSQL wicked problems

308 points by samokhvalov 5y ago

mmontagna9 5y ago
> Reduced maintenance needs. OrioleDB implements the concepts of undo log and page-mergins, eliminating the need for dedicated garbage collection processes. Additionally, OrioleDB implements default 64-bit transaction identifiers, thus eliminating the well-known and painful wraparound problem. I love this, no more vacuums?
- akorotkov 5y ago
  Right, no more vacuums. And within typical workloads there should be no bloat.
- glogla 5y ago
  On of the major differences between Oracle and Postgres was this - Postgres kept old versions of rows around to be vacuumed, while Oracle moves them to "undo segment". So they are copying the Oracle design. That has its own set of tradeoffs - if you're running a long transaction with higher level of serialization, the database has to look for old data in the undo segment which is not only likely vastly slower than normal table, but might not even have the data anymore (any ETL developer have probably seen "Snapshot too old" error from Oracle). It probably is better tradeoff for some workloads, like high volume of updates with no long-running transactions, but it is more of a different design choice than a "fix" by itself.
  - zigzag312 5y ago
    Isn't the goal to have multiple storage engines, so you could use the one that best fits your needs? Current storage engine probably won't be removed when this is merged.
  - Ginden 5y ago
    Yes, it made me wary - design choice with different tradeoffs is portrayed as "better".
  - akorotkov 5y ago
    > That has its own set of tradeoffs - if you're running a long transaction with higher level of serialization, the database has to look for old data in the undo segment which is not only likely vastly slower than normal table, but might not even have the data anymore (any ETL developer have probably seen "Snapshot too old" error from Oracle). I would say "Snapshot too old" isn't essential drawback of undo log. DBMS can keep the the undo log records until all snapshots, which needs them, are released. And this is how OrioleDB behaves. This is also kind of "bloat", but cleanup is cheaper. You just have to cut unclaimed undo log instead of expensive vacuum.
latch 5y ago
This is exciting: > ...This log architecture is optimized for raft consensus-based replication allowing the implementation of active-active multimaster. I'm a developer but manage 2 PostgreSQL instances each with an async replica. I also manage 1 three-node CockroachDB cluster. It's night and day when it comes to any type of ops (e.g. upgrading a version). There's a lot of reasons to use PostgreSQL (e.g. stored procedures, extensions, ...), but if none of those apply to you, CockroachDB is a better default. I just finished interviewing someone who went into detail about the difficulty they're having doing a large scale migration or a mission critical large PG cluster (from one DC to another)..it's a multi-team, year long initiative that no one is comfortable about. In my experience, PG is always the most out-of-date piece of any system. I still see 9.6 here and there, which is now EOL.
- rad_gruchalski 5y ago
  I'd go for YugabyteDB instead of CockroachDB but your point of distributed SQL being better is spot on.
- mmontagna9 5y ago
  > In my experience, PG is always the most out-of-date piece of any system. Yes
- otterley 5y ago
  Raft doesn’t magically make a database multi master. It’s a consensus algorithm that has a leader election process. There’s still a leader, and therefore a single master to which writes must be directed. The problem it solves is ambiguity about who the active master is at the moment.
  - akorotkov 5y ago
    Right. The idea of active-active OrioleDB multimaster is to apply changes locally and in parallel send it to the leader. Then sync on commit and ensure there is no conflicts. The design document will be published later.
    - 5y ago
jpgvm 5y ago
This looks insanely impressive. Will have to dive in and give this a spin.
themerone 5y ago
Is this a fork or something intended to be upstreamed?
- jpgvm 5y ago
  It's both. There is a PG fork which contains some patches to increase extensibility to certain APIs and then an extension developed against that fork. The idea would be to upstream the fork I would imagine but continue development of the engine out of tree similar to Citus, Timescale, etc.
  - akorotkov 5y ago
    Yes, that's it. 1. The first goal is to become a pure extension. That should be done in 2-3 PostgreSQL release cycles. 2. The long term goal is to become a part of PostgreSQL.
    - InvaderFizz 5y ago
      When you say 2-3 release cycles, do you mean like targeting 14.5(probably 2022), or more like targeting 17.x(probably 2025)?
      - akorotkov 5y ago
        I mean major releases. So, targeting 17.0.
    - abriosi 5y ago
      Awesome. Thank you for helping out the community
CodeWriter23 5y ago
Sounds really exciting. My wife noticed me reading the notes with such intent she asked me what I was reading.
yardstick 5y ago
Awesome concept! Any concerns about the name being very similar to OracleDB? At least when I first saw the name I internally within my head read it as that (so could maybe just be me!).
atdrummond 5y ago
Wow, this looks to potentially save my team hundreds of hours of work. This product and its aims are nearly 1:1 with a massive Postgres extension project we were to start developing next quarter. Who should I get in touch with to see about us contributing instead to Oriole? I would love for our team to support the project with man hours and possibly financial support as well, as it seems this will accelerate our project significantly.
- fuy 5y ago
  this is the guy who leads the project (as far as I know): https://www.linkedin.com/in/alexander-korotkov-41b1b582/ https://www.linkedin.com/in/alexander-korotkov-41b1b582/
- Blahah 5y ago
  https://news.ycombinator.com/user?id=akorotkov https://news.ycombinator.com/user?id=akorotkov
- akorotkov 5y ago
  Hi! My name is Alexander Korotkov, I'm founder of OrioleDB. Thank you for your feedback and support. Please, write me at alexander@orioledb.com Thank you!
- adeptima 5y ago
  Alexandr is a very impressive contributor to Postgres community. Met him few times. He’s on the mission to change and improve community. He has a clear vision what has to be changed, and he’s still 30+ yo and has many many years ahead. Just talking him 15 mins will save you years. Hope he can make a business out of this and find resourceful business partners.
  - sdesol 5y ago
    > he’s still 30+ yo and has many many years ahead I know where you are going with this, but I'm not sure his age is relevant as somebody 50+ yo could have many years ahead of them. Having said all of that, here's his contributions to Postgres which speaks to his credibility: month | commits | files | churn | langs ---------+---------+-------+-------+----------------------------------------------------- 2022-02 | 3 | 7 | 290 | c,sql,text,unknown 2021-12 | 1 | 5 | 30 | c,sql,unknown 2021-11 | 2 | 1 | 38 | c 2021-09 | 2 | 4 | 74 | c,c_header 2021-07 | 3 | 7 | 135 | c,c_header,sql,unknown 2021-06 | 4 | 12 | 992 | c,c_header,sql,unknown 2021-05 | 4 | 5 | 88 | c,sql,unknown 2021-04 | 1 | 4 | 30 | c,c_header,unknown 2021-03 | 1 | 1 | 2 | unknown 2021-02 | 2 | 3 | 20 | c,c_header 2021-01 | 6 | 16 | 1781 | c,c_header,makefile,sql,unknown 2020-12 | 11 | 69 | 12331 | c,c_header,makefile,perl,sql,unknown 2020-11 | 8 | 11 | 288 | c,c_header,makefile,sql,unknown 2020-09 | 4 | 4 | 258 | c,sql,unknown 2020-08 | 1 | 3 | 23 | c 2020-07 | 7 | 12 | 486 | c,c_header,makefile,sql,unknown 2020-06 | 8 | 8 | 766 | c,unknown 2020-05 | 5 | 5 | 96 | c,sql,unknown 2020-04 | 6 | 28 | 1225 | c,c_header,makefile,perl,sql,unknown 2020-03 | 8 | 119 | 5651 | c,c_header,makefile,sql,unknown 2020-01 | 1 | 10 | 627 | c,c_header,sql,unknown 2019-11 | 4 | 5 | 303 | c,text 2019-10 | 2 | 7 | 312 | c,c_header,sql,unknown 2019-09 | 18 | 48 | 3054 | c,c_header,sql,text,unknown 2019-08 | 3 | 4 | 280 | c,sql,unknown 2019-07 | 6 | 18 | 1046 | c,c_header,sql,unknown 2019-06 | 6 | 12 | 134 | c,c_header,sql,unknown 2019-05 | 6 | 9 | 309 | c,unknown 2019-04 | 4 | 17 | 1651 | c,c_header,sql,unknown 2019-03 | 15 | 55 | 11647 | c,c_header,gitignore,makefile,perl,sql,text,unknown 2018-12 | 6 | 24 | 1289 | c,c_header,text,unknown 2018-10 | 3 | 1 | 21 | unknown 2018-09 | 8 | 43 | 2589 | c,c_header,makefile,perl,sql,text,unknown 2018-08 | 5 | 11 | 2474 | c,makefile,sql,unknown 2018-07 | 4 | 17 | 101 | c,c_header 2018-06 | 7 | 9 | 117 | c,unknown Note, the analysis is only for the email akorotkov@postgresql.org, so if there is more, let me know and I'll update the metrics.
eurasiantiger 5y ago
Is it just me or is the database scene the new JavaScript scene, new database every week etc.
- lillecarl 5y ago
  The IT world is still growing, people need scale and existing solutions might not scale like people want.
- kiwicopple 5y ago
  Despite the distinct name, this project aims to upstream all changes to PostgreSQL (see Alexander's comment: https://news.ycombinator.com/reply?id=30464060&goto=item%3Fid%3D30462695%2330464060 https://news.ycombinator.com/reply?id=30464060&goto=item%3Fi...) So rather than a "new database", it's probably better to think of it as an iteration of PostgreSQL
  - thinkx 5y ago
    only the additional “hooks” for Table Access Methods need to get up streamed into the core Postgres. OrioleDB is a separate storage engine and not designed to replace the current engine, and be used as an adjunct storage option
- SigmundA 5y ago
  My only issue is things calling themselves a database when they are either a wrapper layer around an actual DB (most of the new "databases" I see posted) or as in this case a storage engine. Not trying to take away from the great work done here but the product name and link title would be more clear if it was referred to as a storage engine rather than a DB.
  - marco_craveiro 5y ago
    Good point. Especially since the final destination is to merge into Postgres mainline in the future, it would make more sense to call this Oriole Storage Engine...
  - 5y ago
tbrock 5y ago
Finally someone tackling the real problems that everyone glosses over when gushing about postgres. Bravo! This is a really excellent project.
fforflo 5y ago
> PostgreSQL 14 with extensibility patches, can you elaborate on these patches? Have you worked on any parser-level hooks? I'm building an extension on PG, and have been trying to extend the parser without touching PG's tree, but it looks impossible at this point.
- ccleve 5y ago
  @akorotkov I'm interested also. I see the forked and patched version here, https://github.com/orioledb/postgres https://github.com/orioledb/postgres, but I don't see any documentation on what changes you made.
  - akorotkov 5y ago
    There is not exactly a documentation. But there you can see that commits after "Stamp 14.2." are extendability patches. Commit messages briefly describe the changes. https://github.com/orioledb/postgres/commits/patches14 https://github.com/orioledb/postgres/commits/patches14 Patches will be polished and published on pgsql-hackers.
fuy 5y ago
For those who understand russian - there's a great overview of this project on RuPostgres channel https://www.youtube.com/watch?v=1GgyEqLNXiM https://www.youtube.com/watch?v=1GgyEqLNXiM
- samokhvalov 5y ago
  Thanks for mentioning. Alexander also presented OrioleDB in English at an EDB's event: https://vimeo.com/649616139 https://vimeo.com/649616139
stevefan1999 5y ago
Fascinating and outstanding work. I'm looking forward to see how well it integrates with Hasura!
gajus 5y ago
Insanely impressive. If this was available as an extension, I could see us using it in production asap.
CodesInChaos 5y ago
How does this compare to zheap? I believe zheap made some similar design choices (undo based and avoiding the wraparound problems), though the replication features seem unique to oriole.
- akorotkov 5y ago
  Additionally OrioleDB fundamentally changes design of buffer management and improves the vertical scalability a lot. https://www.slideshare.net/AlexanderKorotkov/solving-postgresql-wicked-problems?qid=d3638493-9a21-4143-8aa3-67228e4cd268&v=&b=&from_search=1 https://www.slideshare.net/AlexanderKorotkov/solving-postgre... Also, I have to mention that zheap development is inactive for now. And its undo implementation was never complete. Zheap undo works only if you don't update any indexed column. Otherwise, it works as plain heap with write-amplification and bloat.
hans_castorp 5y ago
How (if at all) is this related to the ZHEAP development? Sounds like this targets more or less the same problems
- akorotkov 5y ago
  Please, see this thread of comments. https://news.ycombinator.com/item?id=30466322 https://news.ycombinator.com/item?id=30466322
jfbaro 5y ago
That's amazing! Thanks for making this open source. Could this be extended to support temporal (and Bitemporal) tables? Some storage level optimizations could benefit performance and easy of use for use cases temporarily is important, like financial services systems. Thanks
- tluyben2 5y ago
  That would be very nice. We ‘solve’ this now with custom ducktape triggers and stored procs. Work in banking.
agacera 5y ago
wow.... this is amazing! I dindn't get this: `OrioleDB implements default 64-bit transaction identifiers, thus eliminating the well-known and painful wraparound problem` I thought that PG's 32-bit transaction identifier and wraparound issue was due to the current MVCC implementation that keeps old versions of rows in page tables, so there is the need to vacuum to clean up the dead rows. So my assumption was that using another approach for MVCC, like the UNDO log, this problem wouldn't exist (since page tables would only have the latest version). What am I missing here?
- akorotkov 5y ago
  PostgreSQL 32-bit transaction identifiers are cycled in a ring. This is why PostgreSQL needs to "freeze" old transaction identifiers before they could start being identified as future. This routine is "vacuum freeze" and it have to scan and re-write pages, which could have no dead rows.
VincentEvans 5y ago
This is a bit of a digression, but since this topic attracted a lot of eyes interested in Postgres and databases… Is there any hope of improvements to database backup (dump) functionality of Postgres? Coming from SQL Server (and it seems to me that there’s not a lot of people who have experience with both) - I was really taken by surprise how badly performant Postgres dumps are in terms of both backup and restore speed, as well as backup size of similarly sized databases.
alberth 5y ago
> OrioleDB ... solving some PostgreSQL wicked problems Why was Postgres forked? I don't want this to come across as negative towards Postgres (or discouraging toward Alexander) but given that these fixes aren't upstream in Postgres - developed by a known/respected contributor, is this indicative to some type of politics / infighting happening within the Postgres community on the architectural direction of their (amazing) database? I see some comments in this thread touch a little on this topic but why does this need to wait 2-3 release cycles before it's upstreamed to Postgres? That's going to be roughly 4 years from now.
- salmo 5y ago
  It’s not a fork. It’s an extension. That’s both in the doc and build instructions. That is a very appropriate way to run ahead and add experimental features that could be folded in later, even just later maintaining the extension as part of the OOB release. It speaks to the architecture of Postgres that this is possible to implement this way. I would love to see this be successful. Much cleaner than stacking software in front of and around Postgres, and not just a wire-compatible DB like Cockroach.
- akorotkov 5y ago
  From my point of view, PostgreSQL development is quite conservative. Every small change may become a subject of heated debates, nothing to say about big changes. This approach to the development have advantages and disadvantages. But if we imagine OrioleDB accepted to the upstream, that would be most dramatical changes since PostgreSQL was released to Open Source. This is why I decided to make a fork and then merge it to upstream step-by-step. It's not that bad to use a fork during 2-3 release cycles give that: 1) It's not highly divergent fork (the patchset is small). 2) They patchset will be smoothly decrease over that period of time. Also, even 2-3 release cycles isn't that bad in comparison with decades while we have small progress in wicked problems.
tbrock 5y ago
How can I contact you to get involved?! This is amazing.
- akorotkov 5y ago
  Feel free to contact me at alexander@orioledb.com

308 points by samokhvalov 5y ago

> Reduced maintenance needs. OrioleDB implements the concepts of undo log and page-mergins, eliminating the need for dedicated garbage collection processes. Additionally, OrioleDB implements default 64-bit transaction identifiers, thus eliminating the well-known and painful wraparound problem. I love this, no more vacuums?

akorotkov 5y ago
Right, no more vacuums. And within typical workloads there should be no bloat.
glogla 5y ago
On of the major differences between Oracle and Postgres was this - Postgres kept old versions of rows around to be vacuumed, while Oracle moves them to "undo segment". So they are copying the Oracle design. That has its own set of tradeoffs - if you're running a long transaction with higher level of serialization, the database has to look for old data in the undo segment which is not only likely vastly slower than normal table, but might not even have the data anymore (any ETL developer have probably seen "Snapshot too old" error from Oracle). It probably is better tradeoff for some workloads, like high volume of updates with no long-running transactions, but it is more of a different design choice than a "fix" by itself.
- zigzag312 5y ago
  Isn't the goal to have multiple storage engines, so you could use the one that best fits your needs? Current storage engine probably won't be removed when this is merged.
- Ginden 5y ago
  Yes, it made me wary - design choice with different tradeoffs is portrayed as "better".
- akorotkov 5y ago
  > That has its own set of tradeoffs - if you're running a long transaction with higher level of serialization, the database has to look for old data in the undo segment which is not only likely vastly slower than normal table, but might not even have the data anymore (any ETL developer have probably seen "Snapshot too old" error from Oracle). I would say "Snapshot too old" isn't essential drawback of undo log. DBMS can keep the the undo log records until all snapshots, which needs them, are released. And this is how OrioleDB behaves. This is also kind of "bloat", but cleanup is cheaper. You just have to cut unclaimed undo log instead of expensive vacuum.

latch 5y ago

This is exciting: > ...This log architecture is optimized for raft consensus-based replication allowing the implementation of active-active multimaster. I'm a developer but manage 2 PostgreSQL instances each with an async replica. I also manage 1 three-node CockroachDB cluster. It's night and day when it comes to any type of ops (e.g. upgrading a version). There's a lot of reasons to use PostgreSQL (e.g. stored procedures, extensions, ...), but if none of those apply to you, CockroachDB is a better default. I just finished interviewing someone who went into detail about the difficulty they're having doing a large scale migration or a mission critical large PG cluster (from one DC to another)..it's a multi-team, year long initiative that no one is comfortable about. In my experience, PG is always the most out-of-date piece of any system. I still see 9.6 here and there, which is now EOL.

rad_gruchalski 5y ago
I'd go for YugabyteDB instead of CockroachDB but your point of distributed SQL being better is spot on.
mmontagna9 5y ago
> In my experience, PG is always the most out-of-date piece of any system. Yes
otterley 5y ago
Raft doesn’t magically make a database multi master. It’s a consensus algorithm that has a leader election process. There’s still a leader, and therefore a single master to which writes must be directed. The problem it solves is ambiguity about who the active master is at the moment.
- akorotkov 5y ago
  Right. The idea of active-active OrioleDB multimaster is to apply changes locally and in parallel send it to the leader. Then sync on commit and ensure there is no conflicts. The design document will be published later.
  - 5y ago

jpgvm 5y ago

This looks insanely impressive. Will have to dive in and give this a spin.

themerone 5y ago

Is this a fork or something intended to be upstreamed?

jpgvm 5y ago
It's both. There is a PG fork which contains some patches to increase extensibility to certain APIs and then an extension developed against that fork. The idea would be to upstream the fork I would imagine but continue development of the engine out of tree similar to Citus, Timescale, etc.
- akorotkov 5y ago
  Yes, that's it. 1. The first goal is to become a pure extension. That should be done in 2-3 PostgreSQL release cycles. 2. The long term goal is to become a part of PostgreSQL.
  - InvaderFizz 5y ago
    When you say 2-3 release cycles, do you mean like targeting 14.5(probably 2022), or more like targeting 17.x(probably 2025)?
    - akorotkov 5y ago
      I mean major releases. So, targeting 17.0.
  - abriosi 5y ago
    Awesome. Thank you for helping out the community

CodeWriter23 5y ago

Sounds really exciting. My wife noticed me reading the notes with such intent she asked me what I was reading.

yardstick 5y ago

Awesome concept! Any concerns about the name being very similar to OracleDB? At least when I first saw the name I internally within my head read it as that (so could maybe just be me!).

atdrummond 5y ago

Wow, this looks to potentially save my team hundreds of hours of work. This product and its aims are nearly 1:1 with a massive Postgres extension project we were to start developing next quarter. Who should I get in touch with to see about us contributing instead to Oriole? I would love for our team to support the project with man hours and possibly financial support as well, as it seems this will accelerate our project significantly.

fuy 5y ago
this is the guy who leads the project (as far as I know): https://www.linkedin.com/in/alexander-korotkov-41b1b582/ https://www.linkedin.com/in/alexander-korotkov-41b1b582/
Blahah 5y ago
https://news.ycombinator.com/user?id=akorotkov https://news.ycombinator.com/user?id=akorotkov
akorotkov 5y ago
Hi! My name is Alexander Korotkov, I'm founder of OrioleDB. Thank you for your feedback and support. Please, write me at alexander@orioledb.com Thank you!
adeptima 5y ago
Alexandr is a very impressive contributor to Postgres community. Met him few times. He’s on the mission to change and improve community. He has a clear vision what has to be changed, and he’s still 30+ yo and has many many years ahead. Just talking him 15 mins will save you years. Hope he can make a business out of this and find resourceful business partners.
- sdesol 5y ago
  > he’s still 30+ yo and has many many years ahead I know where you are going with this, but I'm not sure his age is relevant as somebody 50+ yo could have many years ahead of them. Having said all of that, here's his contributions to Postgres which speaks to his credibility: month | commits | files | churn | langs ---------+---------+-------+-------+----------------------------------------------------- 2022-02 | 3 | 7 | 290 | c,sql,text,unknown 2021-12 | 1 | 5 | 30 | c,sql,unknown 2021-11 | 2 | 1 | 38 | c 2021-09 | 2 | 4 | 74 | c,c_header 2021-07 | 3 | 7 | 135 | c,c_header,sql,unknown 2021-06 | 4 | 12 | 992 | c,c_header,sql,unknown 2021-05 | 4 | 5 | 88 | c,sql,unknown 2021-04 | 1 | 4 | 30 | c,c_header,unknown 2021-03 | 1 | 1 | 2 | unknown 2021-02 | 2 | 3 | 20 | c,c_header 2021-01 | 6 | 16 | 1781 | c,c_header,makefile,sql,unknown 2020-12 | 11 | 69 | 12331 | c,c_header,makefile,perl,sql,unknown 2020-11 | 8 | 11 | 288 | c,c_header,makefile,sql,unknown 2020-09 | 4 | 4 | 258 | c,sql,unknown 2020-08 | 1 | 3 | 23 | c 2020-07 | 7 | 12 | 486 | c,c_header,makefile,sql,unknown 2020-06 | 8 | 8 | 766 | c,unknown 2020-05 | 5 | 5 | 96 | c,sql,unknown 2020-04 | 6 | 28 | 1225 | c,c_header,makefile,perl,sql,unknown 2020-03 | 8 | 119 | 5651 | c,c_header,makefile,sql,unknown 2020-01 | 1 | 10 | 627 | c,c_header,sql,unknown 2019-11 | 4 | 5 | 303 | c,text 2019-10 | 2 | 7 | 312 | c,c_header,sql,unknown 2019-09 | 18 | 48 | 3054 | c,c_header,sql,text,unknown 2019-08 | 3 | 4 | 280 | c,sql,unknown 2019-07 | 6 | 18 | 1046 | c,c_header,sql,unknown 2019-06 | 6 | 12 | 134 | c,c_header,sql,unknown 2019-05 | 6 | 9 | 309 | c,unknown 2019-04 | 4 | 17 | 1651 | c,c_header,sql,unknown 2019-03 | 15 | 55 | 11647 | c,c_header,gitignore,makefile,perl,sql,text,unknown 2018-12 | 6 | 24 | 1289 | c,c_header,text,unknown 2018-10 | 3 | 1 | 21 | unknown 2018-09 | 8 | 43 | 2589 | c,c_header,makefile,perl,sql,text,unknown 2018-08 | 5 | 11 | 2474 | c,makefile,sql,unknown 2018-07 | 4 | 17 | 101 | c,c_header 2018-06 | 7 | 9 | 117 | c,unknown Note, the analysis is only for the email akorotkov@postgresql.org, so if there is more, let me know and I'll update the metrics.

eurasiantiger 5y ago

Is it just me or is the database scene the new JavaScript scene, new database every week etc.

lillecarl 5y ago
The IT world is still growing, people need scale and existing solutions might not scale like people want.
kiwicopple 5y ago
Despite the distinct name, this project aims to upstream all changes to PostgreSQL (see Alexander's comment: https://news.ycombinator.com/reply?id=30464060&goto=item%3Fid%3D30462695%2330464060 https://news.ycombinator.com/reply?id=30464060&goto=item%3Fi...) So rather than a "new database", it's probably better to think of it as an iteration of PostgreSQL
- thinkx 5y ago
  only the additional “hooks” for Table Access Methods need to get up streamed into the core Postgres. OrioleDB is a separate storage engine and not designed to replace the current engine, and be used as an adjunct storage option
SigmundA 5y ago
My only issue is things calling themselves a database when they are either a wrapper layer around an actual DB (most of the new "databases" I see posted) or as in this case a storage engine. Not trying to take away from the great work done here but the product name and link title would be more clear if it was referred to as a storage engine rather than a DB.
- marco_craveiro 5y ago
  Good point. Especially since the final destination is to merge into Postgres mainline in the future, it would make more sense to call this Oriole Storage Engine...
- 5y ago

tbrock 5y ago

Finally someone tackling the real problems that everyone glosses over when gushing about postgres. Bravo! This is a really excellent project.

fforflo 5y ago

> PostgreSQL 14 with extensibility patches, can you elaborate on these patches? Have you worked on any parser-level hooks? I'm building an extension on PG, and have been trying to extend the parser without touching PG's tree, but it looks impossible at this point.

ccleve 5y ago
@akorotkov I'm interested also. I see the forked and patched version here, https://github.com/orioledb/postgres https://github.com/orioledb/postgres, but I don't see any documentation on what changes you made.
- akorotkov 5y ago
  There is not exactly a documentation. But there you can see that commits after "Stamp 14.2." are extendability patches. Commit messages briefly describe the changes. https://github.com/orioledb/postgres/commits/patches14 https://github.com/orioledb/postgres/commits/patches14 Patches will be polished and published on pgsql-hackers.

fuy 5y ago

For those who understand russian - there's a great overview of this project on RuPostgres channel https://www.youtube.com/watch?v=1GgyEqLNXiM https://www.youtube.com/watch?v=1GgyEqLNXiM

samokhvalov 5y ago
Thanks for mentioning. Alexander also presented OrioleDB in English at an EDB's event: https://vimeo.com/649616139 https://vimeo.com/649616139

stevefan1999 5y ago

Fascinating and outstanding work. I'm looking forward to see how well it integrates with Hasura!

gajus 5y ago

Insanely impressive. If this was available as an extension, I could see us using it in production asap.

CodesInChaos 5y ago

How does this compare to zheap? I believe zheap made some similar design choices (undo based and avoiding the wraparound problems), though the replication features seem unique to oriole.

akorotkov 5y ago
Additionally OrioleDB fundamentally changes design of buffer management and improves the vertical scalability a lot. https://www.slideshare.net/AlexanderKorotkov/solving-postgresql-wicked-problems?qid=d3638493-9a21-4143-8aa3-67228e4cd268&v=&b=&from_search=1 https://www.slideshare.net/AlexanderKorotkov/solving-postgre... Also, I have to mention that zheap development is inactive for now. And its undo implementation was never complete. Zheap undo works only if you don't update any indexed column. Otherwise, it works as plain heap with write-amplification and bloat.

hans_castorp 5y ago

How (if at all) is this related to the ZHEAP development? Sounds like this targets more or less the same problems

akorotkov 5y ago
Please, see this thread of comments. https://news.ycombinator.com/item?id=30466322 https://news.ycombinator.com/item?id=30466322

jfbaro 5y ago

That's amazing! Thanks for making this open source. Could this be extended to support temporal (and Bitemporal) tables? Some storage level optimizations could benefit performance and easy of use for use cases temporarily is important, like financial services systems. Thanks

tluyben2 5y ago
That would be very nice. We ‘solve’ this now with custom ducktape triggers and stored procs. Work in banking.

agacera 5y ago

wow.... this is amazing! I dindn't get this: `OrioleDB implements default 64-bit transaction identifiers, thus eliminating the well-known and painful wraparound problem` I thought that PG's 32-bit transaction identifier and wraparound issue was due to the current MVCC implementation that keeps old versions of rows in page tables, so there is the need to vacuum to clean up the dead rows. So my assumption was that using another approach for MVCC, like the UNDO log, this problem wouldn't exist (since page tables would only have the latest version). What am I missing here?

akorotkov 5y ago
PostgreSQL 32-bit transaction identifiers are cycled in a ring. This is why PostgreSQL needs to "freeze" old transaction identifiers before they could start being identified as future. This routine is "vacuum freeze" and it have to scan and re-write pages, which could have no dead rows.

VincentEvans 5y ago

This is a bit of a digression, but since this topic attracted a lot of eyes interested in Postgres and databases… Is there any hope of improvements to database backup (dump) functionality of Postgres? Coming from SQL Server (and it seems to me that there’s not a lot of people who have experience with both) - I was really taken by surprise how badly performant Postgres dumps are in terms of both backup and restore speed, as well as backup size of similarly sized databases.

alberth 5y ago

> OrioleDB ... solving some PostgreSQL wicked problems Why was Postgres forked? I don't want this to come across as negative towards Postgres (or discouraging toward Alexander) but given that these fixes aren't upstream in Postgres - developed by a known/respected contributor, is this indicative to some type of politics / infighting happening within the Postgres community on the architectural direction of their (amazing) database? I see some comments in this thread touch a little on this topic but why does this need to wait 2-3 release cycles before it's upstreamed to Postgres? That's going to be roughly 4 years from now.

salmo 5y ago
It’s not a fork. It’s an extension. That’s both in the doc and build instructions. That is a very appropriate way to run ahead and add experimental features that could be folded in later, even just later maintaining the extension as part of the OOB release. It speaks to the architecture of Postgres that this is possible to implement this way. I would love to see this be successful. Much cleaner than stacking software in front of and around Postgres, and not just a wire-compatible DB like Cockroach.
akorotkov 5y ago
From my point of view, PostgreSQL development is quite conservative. Every small change may become a subject of heated debates, nothing to say about big changes. This approach to the development have advantages and disadvantages. But if we imagine OrioleDB accepted to the upstream, that would be most dramatical changes since PostgreSQL was released to Open Source. This is why I decided to make a fork and then merge it to upstream step-by-step. It's not that bad to use a fork during 2-3 release cycles give that: 1) It's not highly divergent fork (the patchset is small). 2) They patchset will be smoothly decrease over that period of time. Also, even 2-3 release cycles isn't that bad in comparison with decades while we have small progress in wicked problems.

tbrock 5y ago

How can I contact you to get involved?! This is amazing.

akorotkov 5y ago
Feel free to contact me at alexander@orioledb.com