46 ms·

ActorDB takes this about 100 steps further. It replicates the actual WAL pages instead of SQL commands. So the leader executes the SQL, followers just write the

by biokoda 11y ago

ActorDB takes this about 100 steps further. It replicates the actual WAL pages instead of SQL commands. So the leader executes the SQL, followers just write the pages from leader. https://github.com/biokoda/actordb https://github.com/biokoda/actordb It uses LMDB to store everything instead of storing raft log in one place and SQLite in another. It can run thousands of SQLite instances at the same time and it actually compresses the pages when storing to LMDB. You can query it with a MySQL client library (though Thrift is recommended). You can have multiple raft clusters, there is a complementary actordb_console app to manage it easily, it supports distributed transactions and more.

atombender 11y ago
ActorDB is very cool, but from what I gather it's — at least nominally — designed to work with many small databases (one per "actor") as opposed one large one, and that the constraints it puts on the query model means you have to carefully design your data model to be heavily compartmentalized from the beginning. It also poses some problems for database-wide queries, which have to fan out to every separate actor database. Whereas rqlite acts like one single SQLite instance, and can work with existing schemas. Please do correct me if I'm wrong. Would ActorDB be happy and perform well with, say, just one actor?
- biokoda 11y ago
  It would perform well with one actor when compared to a single SQLite instance. Not compared to PostgreSQL/MySQL due to the concurrency model being optimized for many concurrent actors, not concurrent access to a single actor. A single actor can still execute thousands of queries per second however. Compared to a single SQLite instance ActorDB has two major advantages. No write multiplication due to using LMDB and compression which means reads/writes are significantly smaller. SQLite will always write everything twice. First to WAL, then to the SQLite file and it has no compression capabilities. It should completely trounce rqlite in the performance department.
  - lobster_johnson 11y ago
    Thanks, that's useful. I know nothing about SQLite's concurrency model, but it's not surprising that it can't compete with Postgres. Does this mean that ActorDB relies on LMDB to replace the WAL? I don't know enough about LMDB either, but I presume it has some kind of redo log.
    - biokoda 11y ago
      Yes ActorDB takes sqlite3.c, takes out the wal.c code and replaces it with calls to LMDB. There is no redo log, LMDB is designed in such a way that it does not require it. It is actually verified as the safest storage engine design out there.
      - thesz 11y ago
        Do you have a link on the LMDB verification? What I know is that SQLite IS safe to use even on unsafe file systems. There was a discussion here few months ago (link was from danloo.com).
        biokoda 11y ago
        https://www.usenix.org/conference/osdi14/technical-sessions/presentation/pillai https://www.usenix.org/conference/osdi14/technical-sessions/...
- derefr 11y ago
  To me, that just sounds like it forces you to do the Right Thing from the beginning. Eventually, you will need to shard your data in some way†. If you're already locked into a "global queries against tables containing Everyone's Stuff" data model, you end up doing this through Stupid Database Tricks like "see-other" redirection or silent persistent-hash load-balancing (with constant ops-heavy rebalancing as you grow.) If you think about sharding from the beginning, though, you end up just dividing your data into little naturally-atomic "worlds." Like sharding email [headers, not bodies] by user, or sharding StackOverflow posts by community. If you don't think you'll ever scale to the point where you'll be forced to make those calls, just pick a different database. (But then, if you don't think you'll ever reach that scale, then literally any database will do.) --- † Unless you're EVE Online. They could probably write a really good whitepaper about how they've scaled their single-node MSSQL database so far. I presume they're mostly following the same patterns you would when running e.g. Oracle on big iron; but—according to most sources—they're just using some heavily-loaded commodity hardware with no fancy IO offloading et al. No idea how they do it.
  - biokoda 11y ago
    Well put.
  - lobster_johnson 11y ago
    Absolutely. Keep in mind that ActorDB actually requires that you supply the sharding key yourself whenever you do a query. It's not like, say, Elasticsearch, which shards both reads and writes transparently. Many apps do require large queries that span many "partitions": For example, listing all of a StackOverflow member's contributions across all communities, sorted by time. With ActorDB, you would have to plan for this by denormalizing a bit (biokoda might correct me here): Each community would be an actor with a table of QA items, and each user would also have an actor containing a table listing their own QA items for all communities. Since ActorDB apparently has transactions, you can maintain this duplicate data atomically, though you can't maintain foreign-key constraints across actors.
    - derefr 11y ago
continuational 11y ago
This is exactly what I'm looking for. One small database per entity, with the option of distributed transactions across the databases. It sounds wonderful. How mature is it?
- biokoda 11y ago
  We eat our own dog food and use ActorDB for everything we do. Things that are critical for the company. Multiple products are using it without issue.
  - continuational 11y ago
    Thank you. What do you do about secondary indexes? Say, if you want to search for users by name? To elaborate: I imagine sharding my data per user, so that I have one actor per user. But I still want a search bar on my page where I can type in a name and find the user in question. A multi-actor query would probably be too inefficient in this case. Did you have a similar problem, and if so, how did you solve it?
    - biokoda 11y ago
      ActorDB can have multiple actor types, each with a completely different schema. I would create a new actor type for this purpose and have a single actor in it. This would just contain the names of users. It would be easy to query and manage. Another option is to create a new KV type and have the names there. This is much more scalable bit a bit more difficult to query. Needed if you really need many thousands of queries per second. On the other hand if your scale is such that search by name is a big load, it is probably better to complement the DB with elasticsearch and move text search to it. This is how the big boys do it.
      - continuational 11y ago
        That makes sense. Thank you for the explanation.

by biokoda 11y ago

atombender 11y ago

ActorDB is very cool, but from what I gather it's — at least nominally — designed to work with many small databases (one per "actor") as opposed one large one, and that the constraints it puts on the query model means you have to carefully design your data model to be heavily compartmentalized from the beginning. It also poses some problems for database-wide queries, which have to fan out to every separate actor database. Whereas rqlite acts like one single SQLite instance, and can work with existing schemas. Please do correct me if I'm wrong. Would ActorDB be happy and perform well with, say, just one actor?

biokoda 11y ago
It would perform well with one actor when compared to a single SQLite instance. Not compared to PostgreSQL/MySQL due to the concurrency model being optimized for many concurrent actors, not concurrent access to a single actor. A single actor can still execute thousands of queries per second however. Compared to a single SQLite instance ActorDB has two major advantages. No write multiplication due to using LMDB and compression which means reads/writes are significantly smaller. SQLite will always write everything twice. First to WAL, then to the SQLite file and it has no compression capabilities. It should completely trounce rqlite in the performance department.
- lobster_johnson 11y ago
  Thanks, that's useful. I know nothing about SQLite's concurrency model, but it's not surprising that it can't compete with Postgres. Does this mean that ActorDB relies on LMDB to replace the WAL? I don't know enough about LMDB either, but I presume it has some kind of redo log.
  - biokoda 11y ago
    Yes ActorDB takes sqlite3.c, takes out the wal.c code and replaces it with calls to LMDB. There is no redo log, LMDB is designed in such a way that it does not require it. It is actually verified as the safest storage engine design out there.
    - thesz 11y ago
      Do you have a link on the LMDB verification? What I know is that SQLite IS safe to use even on unsafe file systems. There was a discussion here few months ago (link was from danloo.com).
      - biokoda 11y ago
        https://www.usenix.org/conference/osdi14/technical-sessions/presentation/pillai https://www.usenix.org/conference/osdi14/technical-sessions/...
derefr 11y ago
To me, that just sounds like it forces you to do the Right Thing from the beginning. Eventually, you will need to shard your data in some way†. If you're already locked into a "global queries against tables containing Everyone's Stuff" data model, you end up doing this through Stupid Database Tricks like "see-other" redirection or silent persistent-hash load-balancing (with constant ops-heavy rebalancing as you grow.) If you think about sharding from the beginning, though, you end up just dividing your data into little naturally-atomic "worlds." Like sharding email [headers, not bodies] by user, or sharding StackOverflow posts by community. If you don't think you'll ever scale to the point where you'll be forced to make those calls, just pick a different database. (But then, if you don't think you'll ever reach that scale, then literally any database will do.) --- † Unless you're EVE Online. They could probably write a really good whitepaper about how they've scaled their single-node MSSQL database so far. I presume they're mostly following the same patterns you would when running e.g. Oracle on big iron; but—according to most sources—they're just using some heavily-loaded commodity hardware with no fancy IO offloading et al. No idea how they do it.
- biokoda 11y ago
  Well put.
- lobster_johnson 11y ago
  Absolutely. Keep in mind that ActorDB actually requires that you supply the sharding key yourself whenever you do a query. It's not like, say, Elasticsearch, which shards both reads and writes transparently. Many apps do require large queries that span many "partitions": For example, listing all of a StackOverflow member's contributions across all communities, sorted by time. With ActorDB, you would have to plan for this by denormalizing a bit (biokoda might correct me here): Each community would be an actor with a table of QA items, and each user would also have an actor containing a table listing their own QA items for all communities. Since ActorDB apparently has transactions, you can maintain this duplicate data atomically, though you can't maintain foreign-key constraints across actors.
  - derefr 11y ago

continuational 11y ago

This is exactly what I'm looking for. One small database per entity, with the option of distributed transactions across the databases. It sounds wonderful. How mature is it?

biokoda 11y ago
We eat our own dog food and use ActorDB for everything we do. Things that are critical for the company. Multiple products are using it without issue.
- continuational 11y ago
  Thank you. What do you do about secondary indexes? Say, if you want to search for users by name? To elaborate: I imagine sharding my data per user, so that I have one actor per user. But I still want a search bar on my page where I can type in a name and find the user in question. A multi-actor query would probably be too inefficient in this case. Did you have a similar problem, and if so, how did you solve it?
  - biokoda 11y ago
    ActorDB can have multiple actor types, each with a completely different schema. I would create a new actor type for this purpose and have a single actor in it. This would just contain the names of users. It would be easy to query and manage. Another option is to create a new KV type and have the names there. This is much more scalable bit a bit more difficult to query. Needed if you really need many thousands of queries per second. On the other hand if your scale is such that search by name is a big load, it is probably better to complement the DB with elasticsearch and move text search to it. This is how the big boys do it.
    - continuational 11y ago
      That makes sense. Thank you for the explanation.