4 ms·

Really cool although quite an undertaking to build an entire SQL engine! I have been working on something pretty similar but using Postgres instead. Basically e

Really cool although quite an undertaking to build an entire SQL engine! I have been working on something pretty similar but using Postgres instead. Basically extending Postgres to run stateless on top of FoundationDB, which would achieve the same thing but with all the Postgres features one would expect (and without some quirks you might not want, like vacuuming). Working with FoundationDB is a real pleasure as many people have noted already, a very intuitive abstraction to build databases on top of.

mike_hearn 1y ago
How do you plan to solve the N+1 query issue and the five second timeout?
- fidotron 1y ago
  This is the interview question you would use to spot people that have actually used FDB vs just talked about it.
- fabianlindfors 1y ago
  The five second timeout remains and trickles through to apply to Postgres transactions instead. This project would very much be for OLTP, so not really fit for using Postgres for OLAP or hybrid workloads. The N+1 issue is a really interesting one which I have a plan for but haven't implemented yet. FoundationDB has what they call mapped ranges [0] to help with this, which works in some cases. More generally, one should make sure to issue read request as soon as possible and not on-demand, given that FoundationDB clients have a futures-based design. This is slightly tricky in Postgres because internally it has a pull-based model where one tuple is pulled from the execution plan at a time, so one needs to implement pre-fetching and not make a read against FDB each time a new tuple is pulled. [0] https://github.com/apple/foundationdb/wiki/Everything-about-GetMappedRange https://github.com/apple/foundationdb/wiki/Everything-about-...
  - fabianlindfors 1y ago
    Neon encounters a similar problem and performs prefetching of pages before Postgres wants to read them: https://neon.tech/docs/extensions/neon#prefetch-option https://neon.tech/docs/extensions/neon#prefetch-option
  - mike_hearn 1y ago
    Right, you'll need to be careful to adjust the read ranges manually to skip pre-fetched but unread tuples. Other aspects that create complexity include SQL's lock based semantics, and that FDB only offers strictly serializable transactions with optimistic concurrency control (requires looping at the originator of the transaction), whereas SQL based apps assume pessimistic non-looping concurrency control.
    - fabianlindfors 1y ago
      Yes, definitely, getting these things right I think are key to making the project actually work. Currently just experimental so we'll see!
fabianlindfors 1y ago
If anybody wants to follow along, I'll be publishing it here once ready: https://github.com/fabianlindfors/pgfdb https://github.com/fabianlindfors/pgfdb (currently just an empty repo!)

mike_hearn 1y ago

How do you plan to solve the N+1 query issue and the five second timeout?

fidotron 1y ago
This is the interview question you would use to spot people that have actually used FDB vs just talked about it.
fabianlindfors 1y ago
The five second timeout remains and trickles through to apply to Postgres transactions instead. This project would very much be for OLTP, so not really fit for using Postgres for OLAP or hybrid workloads. The N+1 issue is a really interesting one which I have a plan for but haven't implemented yet. FoundationDB has what they call mapped ranges [0] to help with this, which works in some cases. More generally, one should make sure to issue read request as soon as possible and not on-demand, given that FoundationDB clients have a futures-based design. This is slightly tricky in Postgres because internally it has a pull-based model where one tuple is pulled from the execution plan at a time, so one needs to implement pre-fetching and not make a read against FDB each time a new tuple is pulled. [0] https://github.com/apple/foundationdb/wiki/Everything-about-GetMappedRange https://github.com/apple/foundationdb/wiki/Everything-about-...
- fabianlindfors 1y ago
  Neon encounters a similar problem and performs prefetching of pages before Postgres wants to read them: https://neon.tech/docs/extensions/neon#prefetch-option https://neon.tech/docs/extensions/neon#prefetch-option
- mike_hearn 1y ago
  Right, you'll need to be careful to adjust the read ranges manually to skip pre-fetched but unread tuples. Other aspects that create complexity include SQL's lock based semantics, and that FDB only offers strictly serializable transactions with optimistic concurrency control (requires looping at the originator of the transaction), whereas SQL based apps assume pessimistic non-looping concurrency control.
  - fabianlindfors 1y ago
    Yes, definitely, getting these things right I think are key to making the project actually work. Currently just experimental so we'll see!

fabianlindfors 1y ago

If anybody wants to follow along, I'll be publishing it here once ready: https://github.com/fabianlindfors/pgfdb https://github.com/fabianlindfors/pgfdb (currently just an empty repo!)