3 ms·

Ideas in CRSQL are remarkably similar to Marmot (https://github.com/maxpert/marmot https://github.com/maxpert/marmot) except that it doesn't require an extensio

Ideas in CRSQL are remarkably similar to Marmot (https://github.com/maxpert/marmot https://github.com/maxpert/marmot) except that it doesn't require an extension to be installed, and it can run as sidecar. I can see how having extension makes it simple to intercept writes, and transactions. Since marmot builds on top of NATS, and JetStreams work using Raft, it can easily scale and keep the libsqlite on disk guarantees intact.

lifty 4y ago
Marmot looks interesting, but I am curious how it deals with conflicts. What happens if two writes for the same field happen on two different dbs, and they are conflicting?
- maxpert 4y ago
  Since each write is logged into a JetStream. The last writer will win for conflicting changes.
  - lifty 4y ago
    But since you don't have NATS message ordering (from different sources), it's possible that different DBs in the cluster will end up out of sync, right?
    - maxpert 4y ago
      So per-node there is an order of changes (read internals doc), and all of changes are sent to JS in that commit order. These changes when replayed will always eventually result in same values.
      - lifty 4y ago
        I am talking about writes that come from 2 different nodes. If 2 different nodes, each making a different write at the exact same time, you would need some global ordering or consensus in order to order them. Otherwise you might end up in a different state on different nodes.
tantaman 4y ago
I'd originally done a sidecar route but ultimately went the extension route for portability.
- maxpert 4y ago
  It’s an interesting choice for sure because I heard a lot of ifs and buts for loads of eventually consistent scenarios on why people wanted to not make any changes to their code and don’t really wanna load extensions. Loading extensions is definitely better in terms of transaction control. I did a lot of soul-searching before settling for sidecar approach. In order to not be an interface and stay away from binding complications for various platforms I opted for sidecar. Once done I was already able to scale out of box stuff like PocketBase.

lifty 4y ago

Marmot looks interesting, but I am curious how it deals with conflicts. What happens if two writes for the same field happen on two different dbs, and they are conflicting?

maxpert 4y ago
Since each write is logged into a JetStream. The last writer will win for conflicting changes.
- lifty 4y ago
  But since you don't have NATS message ordering (from different sources), it's possible that different DBs in the cluster will end up out of sync, right?
  - maxpert 4y ago
    So per-node there is an order of changes (read internals doc), and all of changes are sent to JS in that commit order. These changes when replayed will always eventually result in same values.
    - lifty 4y ago
      I am talking about writes that come from 2 different nodes. If 2 different nodes, each making a different write at the exact same time, you would need some global ordering or consensus in order to order them. Otherwise you might end up in a different state on different nodes.

tantaman 4y ago

I'd originally done a sidecar route but ultimately went the extension route for portability.

maxpert 4y ago
It’s an interesting choice for sure because I heard a lot of ifs and buts for loads of eventually consistent scenarios on why people wanted to not make any changes to their code and don’t really wanna load extensions. Loading extensions is definitely better in terms of transaction control. I did a lot of soul-searching before settling for sidecar approach. In order to not be an interface and stay away from binding complications for various platforms I opted for sidecar. Once done I was already able to scale out of box stuff like PocketBase.