4 ms·

> Neon allows to instantly branch your Postgres database to support a modern development workflow. You can create a branch for your test environments for every

> Neon allows to instantly branch your Postgres database to support a modern development workflow. You can create a branch for your test environments for every code deployment in your CI/CD pipeline. > Branches are virtually free and implemented using the "copy on write" technique. Unless I missed that everyone supports this, this here could be a killer feature and should be advertised higher.

jhgb 4y ago
It sounds like something you might be able to accomplish with a copy-on-write VFS on top of a Firebird database file. (Not sure about PostgreSQL, but with Firebird, you only deal with one file, so with Firebird, this should definitely work.)
- nikita 4y ago
  There is an enterprise company called Delphix that does it on top of Zfs - so the idea was in the air. Instead of duct taping this together with a filesystem we purpose built database storage. The advantage to this is that we can much tighter control execution paths and can profile them end-to-end. Additionally this allows us to integrate with S3 and make it much much cheaper to run.
  - jhgb 4y ago
    Technically Firebird just requires a block device, so you might not even need a filesystem.
zxspectrum1982 4y ago
You can get that feature on any Postgres server by installing Citus
- mattashii 4y ago
  Does Citus provide any such storage-level multi-cluster features? I can't seem to find any documentation on that...
  - zxspectrum1982 4y ago
    I don't understand what you are looking for. Care to explain?
rkwz 4y ago
What are the intended usecases for "branching" a database? Currently, I use separate databases for different environments, are branches better?
- thejosh 4y ago
  It's a great feature on heroku for branches, it shares data between review apps. Quite nice.
- ukd1 4y ago
  Have a staging / qa env, then fork it for a branch for testing. Much faster than reseeding / restoring.
- kelvich 4y ago
  Now the most common setup is to copy the production database to the staging once in a while and test migration against staging. With branching, you can test each PR against its own production database branch -- just put branch creation in your CI config. Hence, it has fewer moving parts, is a bit easier to set up, and reduces the lag between prod and staging.
jvolkman 4y ago
AWS Aurora Postgres supports this to an extent with "clones". You can even clone cross-account. The same copy-on-write stuff applies, so they're relatively cheap and fast. I hope that Google's new AlloyDB will also support it. https://aws.amazon.com/about-aws/whats-new/2019/07/amazon_aurora_supportscloningacrossawsaccounts-/ https://aws.amazon.com/about-aws/whats-new/2019/07/amazon_au... There are some annoying restrictions, though. You can only have a single cross-account clone of a particular db per account.
- samokhvalov 4y ago
  The problem with Aurora's thin clones is extra cost each clone adds For CI/CD, you want multiple clones running on the same compute power, in a shared environment, to keep the budget constant
samokhvalov 4y ago
Agreed, this direction is underestimated and should be developed better -- we (Postgres.ai) do it for any Postgres with our Database Lab Engine [1], and Neon would bring even more power if it's installed on production [1] https://github.com/postgres-ai/database-lab-engine https://github.com/postgres-ai/database-lab-engine

jhgb 4y ago

It sounds like something you might be able to accomplish with a copy-on-write VFS on top of a Firebird database file. (Not sure about PostgreSQL, but with Firebird, you only deal with one file, so with Firebird, this should definitely work.)

nikita 4y ago
There is an enterprise company called Delphix that does it on top of Zfs - so the idea was in the air. Instead of duct taping this together with a filesystem we purpose built database storage. The advantage to this is that we can much tighter control execution paths and can profile them end-to-end. Additionally this allows us to integrate with S3 and make it much much cheaper to run.
- jhgb 4y ago
  Technically Firebird just requires a block device, so you might not even need a filesystem.

zxspectrum1982 4y ago

You can get that feature on any Postgres server by installing Citus

mattashii 4y ago
Does Citus provide any such storage-level multi-cluster features? I can't seem to find any documentation on that...
- zxspectrum1982 4y ago
  I don't understand what you are looking for. Care to explain?

rkwz 4y ago

What are the intended usecases for "branching" a database? Currently, I use separate databases for different environments, are branches better?

thejosh 4y ago
It's a great feature on heroku for branches, it shares data between review apps. Quite nice.
ukd1 4y ago
Have a staging / qa env, then fork it for a branch for testing. Much faster than reseeding / restoring.
kelvich 4y ago
Now the most common setup is to copy the production database to the staging once in a while and test migration against staging. With branching, you can test each PR against its own production database branch -- just put branch creation in your CI config. Hence, it has fewer moving parts, is a bit easier to set up, and reduces the lag between prod and staging.

jvolkman 4y ago

AWS Aurora Postgres supports this to an extent with "clones". You can even clone cross-account. The same copy-on-write stuff applies, so they're relatively cheap and fast. I hope that Google's new AlloyDB will also support it. https://aws.amazon.com/about-aws/whats-new/2019/07/amazon_aurora_supportscloningacrossawsaccounts-/ https://aws.amazon.com/about-aws/whats-new/2019/07/amazon_au... There are some annoying restrictions, though. You can only have a single cross-account clone of a particular db per account.

samokhvalov 4y ago
The problem with Aurora's thin clones is extra cost each clone adds For CI/CD, you want multiple clones running on the same compute power, in a shared environment, to keep the budget constant

samokhvalov 4y ago

Agreed, this direction is underestimated and should be developed better -- we (Postgres.ai) do it for any Postgres with our Database Lab Engine [1], and Neon would bring even more power if it's installed on production [1] https://github.com/postgres-ai/database-lab-engine https://github.com/postgres-ai/database-lab-engine