4 ms·

I am very interested in any project that implements multi-active SQLite clusters with no cluster size limit, do you have any links for me?

azornathogron 5y ago
rqlite is one of the ones that comes up in discussions. I can't offer a recommendation for or against though, as I've never needed to use it. Not sure why "no cluster size limit" is important given the goals you listed in your comment btw. The rqlite faq mentions a practical cluster size limit of 9 read-write nodes (higher for read-only replicas).
- database_guy 5y ago
  Rqlite is an awesome project but it's not intended for use above a couple GB, like etcd. Sorry, dataset size is what I meant by cluster-size which was a confusing way to put it. You're right, I don't care about the number of nodes in the cluster.
  - azornathogron 5y ago
    Ah, you're right, sorry. I naively assumed rqlite would cope with similar dataset sizes to sqlite itself, which I think is likely to be limited by how much disk you can put on a single node. But it seems rqlite is much more limited on that dimension, so it doesn't fit your criteria.
    - otoolep 5y ago
      Full details on what the limits in rqlite may be, and why, are here: https://github.com/rqlite/rqlite/blob/master/DOC/PERFORMANCE.md#in-memory-database-limits https://github.com/rqlite/rqlite/blob/master/DOC/PERFORMANCE... That said, very large datasets (> 2GB) will work, if you've got the disks and memory to handle it. The link above explains how to do it.
      - database_guy 5y ago
        Perhaps I'll have to test it out at larger data sizes. The performance document makes it sound like this is here-there-be-dragons territory. I'd love to see how the database holds up at 100GB. Rqlite is a great project and I'm glad it exists!
- otoolep 5y ago
  rqlite author here. Happy to answer any questions. Noting the title of this topic, rqlite is specifically designed to be easy to use. I know of no other HA database that is simpler to operate. Installation is trivial, and clustering is easy. You interact with it via a HTTP API. And you get all the power and quality of SQLite as the database engine.
  - otoolep 5y ago
    You can go larger than 9, quite a bit larger. I just suggested 9 as a useful size. But the 7.x series introduced much better support for Kubernetes, so it's much easier now to manage larger clusters. But larger clusters mean a larger quorum, so larger isn't really better. It depends on what you want. If you want a distributed database where more nodes means more write performance, then rqlite may not be a good fit. rqlite is distributed for fault-tolerance and HA, not for performance.
elmerfud 5y ago
Just searching for SQLite replication will get you several options to research to fit your requirements. The concept of multi active and no size limit you would simply solve with a sharding mechanism that you layer on top through a data access method. But if you're looking for some monolith project to provide all of this for you you're going to be in that territory of that fake open source. Because again to build all of this into one tool that has everything from the data access layer all the way down to the metal underneath to ensure that it's replicated and highly available is not a trivial task. Especially if you want things like acid compliance arbitrary joins between data structures and so forth. The file system is effectively a database of things with an API that accesses those things. Simply a file system of properly named files and directories could meet your needs depending on what those are. It's the complexity of your access model and methods that will determine what fits your bill.
- database_guy 5y ago
  Yeah, sounds like most folks agree this doesn't exist simply because it's very challenging to implement.

azornathogron 5y ago

rqlite is one of the ones that comes up in discussions. I can't offer a recommendation for or against though, as I've never needed to use it. Not sure why "no cluster size limit" is important given the goals you listed in your comment btw. The rqlite faq mentions a practical cluster size limit of 9 read-write nodes (higher for read-only replicas).

database_guy 5y ago
Rqlite is an awesome project but it's not intended for use above a couple GB, like etcd. Sorry, dataset size is what I meant by cluster-size which was a confusing way to put it. You're right, I don't care about the number of nodes in the cluster.
- azornathogron 5y ago
  Ah, you're right, sorry. I naively assumed rqlite would cope with similar dataset sizes to sqlite itself, which I think is likely to be limited by how much disk you can put on a single node. But it seems rqlite is much more limited on that dimension, so it doesn't fit your criteria.
  - otoolep 5y ago
    Full details on what the limits in rqlite may be, and why, are here: https://github.com/rqlite/rqlite/blob/master/DOC/PERFORMANCE.md#in-memory-database-limits https://github.com/rqlite/rqlite/blob/master/DOC/PERFORMANCE... That said, very large datasets (> 2GB) will work, if you've got the disks and memory to handle it. The link above explains how to do it.
    - database_guy 5y ago
      Perhaps I'll have to test it out at larger data sizes. The performance document makes it sound like this is here-there-be-dragons territory. I'd love to see how the database holds up at 100GB. Rqlite is a great project and I'm glad it exists!
otoolep 5y ago
rqlite author here. Happy to answer any questions. Noting the title of this topic, rqlite is specifically designed to be easy to use. I know of no other HA database that is simpler to operate. Installation is trivial, and clustering is easy. You interact with it via a HTTP API. And you get all the power and quality of SQLite as the database engine.
- otoolep 5y ago
  You can go larger than 9, quite a bit larger. I just suggested 9 as a useful size. But the 7.x series introduced much better support for Kubernetes, so it's much easier now to manage larger clusters. But larger clusters mean a larger quorum, so larger isn't really better. It depends on what you want. If you want a distributed database where more nodes means more write performance, then rqlite may not be a good fit. rqlite is distributed for fault-tolerance and HA, not for performance.

elmerfud 5y ago

Just searching for SQLite replication will get you several options to research to fit your requirements. The concept of multi active and no size limit you would simply solve with a sharding mechanism that you layer on top through a data access method. But if you're looking for some monolith project to provide all of this for you you're going to be in that territory of that fake open source. Because again to build all of this into one tool that has everything from the data access layer all the way down to the metal underneath to ensure that it's replicated and highly available is not a trivial task. Especially if you want things like acid compliance arbitrary joins between data structures and so forth. The file system is effectively a database of things with an API that accesses those things. Simply a file system of properly named files and directories could meet your needs depending on what those are. It's the complexity of your access model and methods that will determine what fits your bill.

database_guy 5y ago
Yeah, sounds like most folks agree this doesn't exist simply because it's very challenging to implement.