4 ms·

Bridged Indexes in OrioleDB: architecture, internals and everyday use?

76 points by pella 1y ago

apavlo 1y ago
"Bridged Indexes" is a non-standard term. These are just secondary indexes using logical pointers with a mapping index. IIRC, Oracle, Hana, and HyPer do the same thing. Source: https://db.cs.cmu.edu/papers/2017/p781-wu.pdf https://db.cs.cmu.edu/papers/2017/p781-wu.pdf (see Table 1 + Section 6.1)
- _joel 1y ago
  > secondary indexes using logical pointers with a mapping index. rolls off the tongue.
- mattashii 1y ago
  It's more than just one mapping index, though. The CMU paper indicates the logical keys are either TupleID or Primary Key, while the bridged index is actually a TupleID that resolves to a Primary Key, which resolves to the actual tuple - one more level than indicated by 6.1's explanation of logical pointers.
rubenvanwyk 1y ago
Cannot wait for OrioldeDB to reach General Availability. Postgres needs options for open-source separation of storage and compute.
- the_duke 1y ago
  That kind of exists thanks to NeonDB? https://neon.com https://neon.com Althoug with them being recently acquired by Databricks it remains to be seen how the open source version will fare.
  - akorotkov 1y ago
    > That kind of exists thanks to NeonDB? This is unrelated to NeonDB. OrioleDB has been acquired by Supabase. https://supabase.com/blog/supabase-acquires-oriole https://supabase.com/blog/supabase-acquires-oriole
    - mattashii 1y ago
      Neon is indeed unrelated to OreoleDB, but Neon does also provide the separation of storage and compute in Postgres which GP asked about ("Postgres needs options for open-source separation of storage and compute"). A mention of Neon (which is Apache 2 licensed) therefore isn't totally unwarranted.
      - rubenvanwyk 1y ago
        I understand Neon is open source and I think it’s an awesome product, but apart from the risks associated with longevity of open source once a company gets acquired - although the storage engine is open source, the control plane isn’t and is non-trivial to implement oneself. Orioldedb is positioned as a Postgres extension, which is must easier to setup (even on an existing operating database) than migrating to a completely different architecture that Neon provides.
- dkhenry 1y ago
  OrioleDB isn't a separation of storage and compute, its a more efficient storage engine for Postgres to replace the existing HEAP engine. This is like how in MySQL we could swap MyISAM for InnoDB and eventually RocksDB. I did some benchmarks on it previously to show how much of an improvement it gives over the stock HEAP engine EDIT: correct link to the public dashboard below, thanks for the heads up @kiwicopple https://airtable.com/app7jp5t0dEHyDpa8/shr00etqywoDW2N6N https://airtable.com/app7jp5t0dEHyDpa8/shr00etqywoDW2N6N
  - kiwicopple 1y ago
    fwiw I couldn't access your airtable link, but I found this one online: https://airtable.com/app7jp5t0dEHyDpa8/shr00etqywoDW2N6N https://airtable.com/app7jp5t0dEHyDpa8/shr00etqywoDW2N6N thanks for running the benchmarks, it helps to have external parties verifying the progress
  - CathalMullan 1y ago
    There's an experimental feature which separates storage and compute. https://www.orioledb.com/docs/usage/decoupled-storage https://www.orioledb.com/docs/usage/decoupled-storage
  - iamdanieljohns 1y ago
    Is the need for Oriole negated by using a system that separates storage from compute like Neon, Xata?
    - nikita 1y ago
      (Neon CEO) Not really. OrioleDB solve the vacuum problem with the introduction of the undo log. Neon gives you scale out storage which is in a way orthogonal to OrielDB. With some work you can run OrioleDB AND neon storage and get benefits of both.
      - akorotkov 1y ago
        > OrioleDB solve the vacuum problem with the introduction of the undo log. Way more than just this! > With some work you can run OrioleDB AND neon storage and get benefits of both. This would require significant design work, given that significant OrioleDB benefits are derived from row-level WAL.
curtisszmania 1y ago
[dead]

76 points by pella 1y ago

"Bridged Indexes" is a non-standard term. These are just secondary indexes using logical pointers with a mapping index. IIRC, Oracle, Hana, and HyPer do the same thing. Source: https://db.cs.cmu.edu/papers/2017/p781-wu.pdf https://db.cs.cmu.edu/papers/2017/p781-wu.pdf (see Table 1 + Section 6.1)

_joel 1y ago
> secondary indexes using logical pointers with a mapping index. rolls off the tongue.
mattashii 1y ago
It's more than just one mapping index, though. The CMU paper indicates the logical keys are either TupleID or Primary Key, while the bridged index is actually a TupleID that resolves to a Primary Key, which resolves to the actual tuple - one more level than indicated by 6.1's explanation of logical pointers.

rubenvanwyk 1y ago

Cannot wait for OrioldeDB to reach General Availability. Postgres needs options for open-source separation of storage and compute.

the_duke 1y ago
That kind of exists thanks to NeonDB? https://neon.com https://neon.com Althoug with them being recently acquired by Databricks it remains to be seen how the open source version will fare.
- akorotkov 1y ago
  > That kind of exists thanks to NeonDB? This is unrelated to NeonDB. OrioleDB has been acquired by Supabase. https://supabase.com/blog/supabase-acquires-oriole https://supabase.com/blog/supabase-acquires-oriole
  - mattashii 1y ago
    Neon is indeed unrelated to OreoleDB, but Neon does also provide the separation of storage and compute in Postgres which GP asked about ("Postgres needs options for open-source separation of storage and compute"). A mention of Neon (which is Apache 2 licensed) therefore isn't totally unwarranted.
    - rubenvanwyk 1y ago
      I understand Neon is open source and I think it’s an awesome product, but apart from the risks associated with longevity of open source once a company gets acquired - although the storage engine is open source, the control plane isn’t and is non-trivial to implement oneself. Orioldedb is positioned as a Postgres extension, which is must easier to setup (even on an existing operating database) than migrating to a completely different architecture that Neon provides.
dkhenry 1y ago
OrioleDB isn't a separation of storage and compute, its a more efficient storage engine for Postgres to replace the existing HEAP engine. This is like how in MySQL we could swap MyISAM for InnoDB and eventually RocksDB. I did some benchmarks on it previously to show how much of an improvement it gives over the stock HEAP engine EDIT: correct link to the public dashboard below, thanks for the heads up @kiwicopple https://airtable.com/app7jp5t0dEHyDpa8/shr00etqywoDW2N6N https://airtable.com/app7jp5t0dEHyDpa8/shr00etqywoDW2N6N
- kiwicopple 1y ago
  fwiw I couldn't access your airtable link, but I found this one online: https://airtable.com/app7jp5t0dEHyDpa8/shr00etqywoDW2N6N https://airtable.com/app7jp5t0dEHyDpa8/shr00etqywoDW2N6N thanks for running the benchmarks, it helps to have external parties verifying the progress
- CathalMullan 1y ago
  There's an experimental feature which separates storage and compute. https://www.orioledb.com/docs/usage/decoupled-storage https://www.orioledb.com/docs/usage/decoupled-storage
- iamdanieljohns 1y ago
  Is the need for Oriole negated by using a system that separates storage from compute like Neon, Xata?
  - nikita 1y ago
    (Neon CEO) Not really. OrioleDB solve the vacuum problem with the introduction of the undo log. Neon gives you scale out storage which is in a way orthogonal to OrielDB. With some work you can run OrioleDB AND neon storage and get benefits of both.
    - akorotkov 1y ago
      > OrioleDB solve the vacuum problem with the introduction of the undo log. Way more than just this! > With some work you can run OrioleDB AND neon storage and get benefits of both. This would require significant design work, given that significant OrioleDB benefits are derived from row-level WAL.

curtisszmania 1y ago

[dead]