4 ms·

Writing a Postgres logical replication system in Golang

91 points by ingve 3y ago

dewey 3y ago
> the world's first version-controlled SQL database Isn't that what others like Supabase are already offering or am I missing a distinction here?
- KingOfCoders 3y ago
  Whenever you read "first" just add "with the special meaning of A,B,C" in your head. There was the first person on the moon, or the first person to run 100m below 10s, but for mostly everything else "first" is basically never binary.
  - zachmu 3y ago
    Pretty sure dolt actually is the first SQL db with branch, merge, and diff implemented natively.
- leononame 3y ago
  Supabase offers version controlled database? To be clear, what dolt does is enabling you to branch like git and then merge your branch, rollback old "commits" of your database and similar stuff. It is much more advanced than audit logging
  - boomskats 3y ago
    How does DoltgreSQL compare to Neon's branching?
    - willi59549879 3y ago
      In neon you can create different branches but there is no merge functionality. I think merge functionality is very hard to implement anyway. not sure if it is worth the effort. To have the same state on two branches you would need to run the same queries that you ran on the other branch.
      - andrenotgiant 3y ago
        > not sure if it is worth the effort. Exactly. Teams that are mature enough to use data branching in development likely already have a schema migration tool that handles the merge safely with all the necessary controls.
        elaencam 3y ago
        And how about those teams that are starting out and don't want to build all of that? It's worth the effort given that it's a common problem that teams need to solve over and over.
        zachmu 3y ago
        One of the biggest challenges for us has been getting through people's heads that "version controlled database" doesn't mean "schema migrations with extra steps". Some people get it immediately, but in general there's a real failure of imagination about what kind of use cases this technology enables. Our customers are building workflows that are just impossible to achieve with other databases. There are a few common use cases that come up over and over: * Game development, to version control game configuration data being edited by an entire team of developers at once * CMS-like tools, including online storefronts, so you can have pull request workflows and easy rollback * Collaborative data sourcing, where multiple independent sources get merged together into a single database There's also the distributed use case, where people fork and clone each other's data sets, that we haven't really seen take off yet. Or maybe those people are just unlikely to buy hosting services or file support tickets.
    - zachmu 3y ago
      Neon is basically a copy on write postgres backend. So you can create a branch at near zero cost and start making changes, but there's no way to merge those changes back to the original branch. To do that, you need a storage system that can efficiently compute a diff to do a three-way merge. That's what dolt does. It means you can diff any two commits in the database with cost proportional to the size of the diff, not the size of the data. And it uses a commit graph just like git which enables it to find common ancestors for a three-way merge algorithm, again just like git.
avinassh 3y ago
If anyone from DoltHub team is reading this, please consider fixing [0] the RSS/Atom feed link [1]. I love your blog posts, but I am not able to subscribe to them in my feed reader. Your posts on SwissTable and Prolly Trees are my fav ones. [0] - https://validator.w3.org/feed/check.cgi?url=https%3A%2F%2Fwww.dolthub.com%2Fblog%2Frss.xml https://validator.w3.org/feed/check.cgi?url=https%3A%2F%2Fww... [1] - https://www.dolthub.com/blog/rss.xml https://www.dolthub.com/blog/rss.xml
- begueradj 3y ago
  Also the publication date of this article shows March 8, but we are on March 7 :)
  - zachmu 3y ago
    We used to have a feature on the blog that would hide articles until their publication date even if we submitted them a bit early, but it was kind of buggy so we got rid of it. You guys weren't supposed to find the article until tomorrow, we never know what you're going to decide to send to the front page.
- maxwhoffman 3y ago
  Thank you for drawing our attention to this! Our blog content surpassed the RSS limit awhile ago, we are working on paginating the feed so new content fits.
  - tbantle22 3y ago
    We just updated our RSS feed to only show the most recent 50 blogs so that it fits within the W3C validator size limits. The feed of all blogs is still available at https://www.dolthub.com/blog/rss-all.xml https://www.dolthub.com/blog/rss-all.xml
    - avinassh 3y ago
      I can confirm it works now! Thank you so much :)
cpursley 3y ago
If you're looking for something that's simple to operate (but not quite as robust yet), check out WalEx: https://github.com/cpursley/walex https://github.com/cpursley/walex
linkdd 3y ago
March 08, 2024. News from the future? Who cares about causality?
- radiowave 3y ago
  If it could replicate your database transaction before they've actually happened, that would be really handy.
- deleted 3y ago
  [deleted]

91 points by ingve 3y ago

> the world's first version-controlled SQL database Isn't that what others like Supabase are already offering or am I missing a distinction here?

KingOfCoders 3y ago
Whenever you read "first" just add "with the special meaning of A,B,C" in your head. There was the first person on the moon, or the first person to run 100m below 10s, but for mostly everything else "first" is basically never binary.
- zachmu 3y ago
  Pretty sure dolt actually is the first SQL db with branch, merge, and diff implemented natively.
leononame 3y ago
Supabase offers version controlled database? To be clear, what dolt does is enabling you to branch like git and then merge your branch, rollback old "commits" of your database and similar stuff. It is much more advanced than audit logging
- boomskats 3y ago
  How does DoltgreSQL compare to Neon's branching?
  - willi59549879 3y ago
    In neon you can create different branches but there is no merge functionality. I think merge functionality is very hard to implement anyway. not sure if it is worth the effort. To have the same state on two branches you would need to run the same queries that you ran on the other branch.
    - andrenotgiant 3y ago
      > not sure if it is worth the effort. Exactly. Teams that are mature enough to use data branching in development likely already have a schema migration tool that handles the merge safely with all the necessary controls.
      - elaencam 3y ago
        And how about those teams that are starting out and don't want to build all of that? It's worth the effort given that it's a common problem that teams need to solve over and over.
        zachmu 3y ago
        One of the biggest challenges for us has been getting through people's heads that "version controlled database" doesn't mean "schema migrations with extra steps". Some people get it immediately, but in general there's a real failure of imagination about what kind of use cases this technology enables. Our customers are building workflows that are just impossible to achieve with other databases. There are a few common use cases that come up over and over: * Game development, to version control game configuration data being edited by an entire team of developers at once * CMS-like tools, including online storefronts, so you can have pull request workflows and easy rollback * Collaborative data sourcing, where multiple independent sources get merged together into a single database There's also the distributed use case, where people fork and clone each other's data sets, that we haven't really seen take off yet. Or maybe those people are just unlikely to buy hosting services or file support tickets.
  - zachmu 3y ago
    Neon is basically a copy on write postgres backend. So you can create a branch at near zero cost and start making changes, but there's no way to merge those changes back to the original branch. To do that, you need a storage system that can efficiently compute a diff to do a three-way merge. That's what dolt does. It means you can diff any two commits in the database with cost proportional to the size of the diff, not the size of the data. And it uses a commit graph just like git which enables it to find common ancestors for a three-way merge algorithm, again just like git.

avinassh 3y ago

If anyone from DoltHub team is reading this, please consider fixing [0] the RSS/Atom feed link [1]. I love your blog posts, but I am not able to subscribe to them in my feed reader. Your posts on SwissTable and Prolly Trees are my fav ones. [0] - https://validator.w3.org/feed/check.cgi?url=https%3A%2F%2Fwww.dolthub.com%2Fblog%2Frss.xml https://validator.w3.org/feed/check.cgi?url=https%3A%2F%2Fww... [1] - https://www.dolthub.com/blog/rss.xml https://www.dolthub.com/blog/rss.xml

begueradj 3y ago
Also the publication date of this article shows March 8, but we are on March 7 :)
- zachmu 3y ago
  We used to have a feature on the blog that would hide articles until their publication date even if we submitted them a bit early, but it was kind of buggy so we got rid of it. You guys weren't supposed to find the article until tomorrow, we never know what you're going to decide to send to the front page.
maxwhoffman 3y ago
Thank you for drawing our attention to this! Our blog content surpassed the RSS limit awhile ago, we are working on paginating the feed so new content fits.
- tbantle22 3y ago
  We just updated our RSS feed to only show the most recent 50 blogs so that it fits within the W3C validator size limits. The feed of all blogs is still available at https://www.dolthub.com/blog/rss-all.xml https://www.dolthub.com/blog/rss-all.xml
  - avinassh 3y ago
    I can confirm it works now! Thank you so much :)

cpursley 3y ago

If you're looking for something that's simple to operate (but not quite as robust yet), check out WalEx: https://github.com/cpursley/walex https://github.com/cpursley/walex

linkdd 3y ago

March 08, 2024. News from the future? Who cares about causality?

radiowave 3y ago
If it could replicate your database transaction before they've actually happened, that would be really handy.
deleted 3y ago
[deleted]