4 ms·

rqlite author here, happy to answer any questions. Many thanks to Nienke Eijsvogel, Ruben van Baarle and Daan de Graaf for executing this testing. https://gith

rqlite author here, happy to answer any questions. Many thanks to Nienke Eijsvogel, Ruben van Baarle and Daan de Graaf for executing this testing. https://github.com/rqlite/rqlite https://github.com/rqlite/rqlite

muxator 5y ago
Not really related to TFA, but I had a look at the python driver documentation and I was very surprised by this statement [0]: > Limitations > Transactions are not supported Is there something I am badly misreading? How come is a so fundamental feature not supported? Maybe it is just a limitation in the python driver? [0] https://github.com/rqlite/pyrqlite#limitations= https://github.com/rqlite/pyrqlite#limitations=
- otoolep 5y ago
  No, it's due to rqlite. rqlite does support a form of transactions -- you can send a set of SQLite statements, and either all will be successful, or none will be. You can even send a BEGIN statement, and a SQLite transaction will be opened (and you can COMMIT later). However, due to the nature of Raft, restarting the cluster in the middle of an open transaction will not clear the transaction -- and most folks probably don't expect this. That's why I need to think more before advocating use of traditional transactions with rqlite -- distributed transaction functionality is easy to get wrong, so requires careful thought. You can use SQLite transactions, but you must be careful. That's why drivers don't tend to support it yet. https://github.com/rqlite/rqlite/blob/master/DOC/DATA_API.md#transactions https://github.com/rqlite/rqlite/blob/master/DOC/DATA_API.md...
  - ______-_-______ 5y ago
    Taken from the readme: > the behavior of a cluster if it fails while such a manually-controlled transaction is not yet defined Do you envision manually-controlled transactions working seamlessly one day? If not, have you thought about what guarantees are feasible/you hope to provide?
    - otoolep 5y ago
      Yes, I have a design in mind, but it's a fair amount of work. It will require a new API too -- and API that allows clients to explicitly create a new connection, create a transaction on it, and close that transaction when finished.
      - pkaye 5y ago
        I there a go client other than gorqlite which doesn't seem to actively maintained.
  - stavros 5y ago
    Would it be a better compromise to support transactions as long as they were sent as a single block?
    - otoolep 5y ago
      Not sure if I follow you, but that's exactly what rqlite does support. https://github.com/rqlite/rqlite/blob/master/DOC/BULK.md https://github.com/rqlite/rqlite/blob/master/DOC/BULK.md
      - stavros 5y ago
        Sorry, I meant from a marketing perspective, saying "transactions are only supported in batch mode" rather than saying they aren't supported.
- zeckalpha 5y ago
  MySQL didn’t have transactions for years! A transaction layer can be implemented by adding a transaction table, a transaction column to each other table, and joining the two in your queries. where table.transaction_id = transaction.id and transaction.committed = true You would begin a transaction by inserting a new row and then updating it to committed when done. With some metaprogramming you may be able to do this all transparently.
  - srcreigh 5y ago
    How would this provide read / write isolation? Or rollback in case of failure(power outage)?
    - zeckalpha 5y ago
      As an exercise for the reader. :) Both are possible and have been done before.
gleenn 5y ago
Very cool project putting together raft and sqlite. Definitely seems like it's biggest benefit would be super simple operation when you also really need the distributed behaviors. Is that accurate? What do you guys use it for and where do you think the sweet spot is for its application?
- otoolep 5y ago
  Yes, simplicity of operation is a key goal of rqlite. To quote from the FAQ: rqlite is very simple to deploy, run, and manage -- in fact, simplicity-of-operation is a key design goal. It's also lightweight and easy to query. It's a single binary you can drop anywhere on a machine, and just start it, which makes it very convenient. It takes literally seconds to configure and form a cluster, which provides you with fault-tolerance and high-availability. With rqlite you have complete control over your database infrastructure, and the data it stores. https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#why-would-i-use-this-versus-some-other-distributed-database https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#why-...
infogulch 5y ago
Another commenter mentioned automatic sharding to which you replied "no". But short of that what do you think about use cases where the database is "naturally" very sharded, say where every user has a separate database? With such a design the operator might want to scale rqlite down to zero instances when a db is unused for a while and start it back up quickly on demand. Have you thought about how rqlite could fit into this design space?
- zeckalpha 5y ago
  That seems better solved outside of rqlite in the application layer and inetd or k8s.
  - deleted 5y ago
    [deleted]
- otoolep 5y ago
  I have considered it, but it would introduce a very large amount of complexity -- and make rqlite much more complicated to operate. Adding this type of functionality would push mean rqlite would no longer be a worthwhile system to use. It would do something (sharding) probably no better than other systems, but no longer be trivial to operate. rqlite does something somewhat narrow, but does that very well (though I would say that :-) )
  - stavros 5y ago
    I have to say, I like your product thinking. I agree with you.
deleted 5y ago
[deleted]

muxator 5y ago

Not really related to TFA, but I had a look at the python driver documentation and I was very surprised by this statement [0]: > Limitations > Transactions are not supported Is there something I am badly misreading? How come is a so fundamental feature not supported? Maybe it is just a limitation in the python driver? [0] https://github.com/rqlite/pyrqlite#limitations= https://github.com/rqlite/pyrqlite#limitations=

otoolep 5y ago
No, it's due to rqlite. rqlite does support a form of transactions -- you can send a set of SQLite statements, and either all will be successful, or none will be. You can even send a BEGIN statement, and a SQLite transaction will be opened (and you can COMMIT later). However, due to the nature of Raft, restarting the cluster in the middle of an open transaction will not clear the transaction -- and most folks probably don't expect this. That's why I need to think more before advocating use of traditional transactions with rqlite -- distributed transaction functionality is easy to get wrong, so requires careful thought. You can use SQLite transactions, but you must be careful. That's why drivers don't tend to support it yet. https://github.com/rqlite/rqlite/blob/master/DOC/DATA_API.md#transactions https://github.com/rqlite/rqlite/blob/master/DOC/DATA_API.md...
- ______-_-______ 5y ago
  Taken from the readme: > the behavior of a cluster if it fails while such a manually-controlled transaction is not yet defined Do you envision manually-controlled transactions working seamlessly one day? If not, have you thought about what guarantees are feasible/you hope to provide?
  - otoolep 5y ago
    Yes, I have a design in mind, but it's a fair amount of work. It will require a new API too -- and API that allows clients to explicitly create a new connection, create a transaction on it, and close that transaction when finished.
    - pkaye 5y ago
      I there a go client other than gorqlite which doesn't seem to actively maintained.
- stavros 5y ago
  Would it be a better compromise to support transactions as long as they were sent as a single block?
  - otoolep 5y ago
    Not sure if I follow you, but that's exactly what rqlite does support. https://github.com/rqlite/rqlite/blob/master/DOC/BULK.md https://github.com/rqlite/rqlite/blob/master/DOC/BULK.md
    - stavros 5y ago
      Sorry, I meant from a marketing perspective, saying "transactions are only supported in batch mode" rather than saying they aren't supported.
zeckalpha 5y ago
MySQL didn’t have transactions for years! A transaction layer can be implemented by adding a transaction table, a transaction column to each other table, and joining the two in your queries. where table.transaction_id = transaction.id and transaction.committed = true You would begin a transaction by inserting a new row and then updating it to committed when done. With some metaprogramming you may be able to do this all transparently.
- srcreigh 5y ago
  How would this provide read / write isolation? Or rollback in case of failure(power outage)?
  - zeckalpha 5y ago
    As an exercise for the reader. :) Both are possible and have been done before.

gleenn 5y ago

Very cool project putting together raft and sqlite. Definitely seems like it's biggest benefit would be super simple operation when you also really need the distributed behaviors. Is that accurate? What do you guys use it for and where do you think the sweet spot is for its application?

otoolep 5y ago
Yes, simplicity of operation is a key goal of rqlite. To quote from the FAQ: rqlite is very simple to deploy, run, and manage -- in fact, simplicity-of-operation is a key design goal. It's also lightweight and easy to query. It's a single binary you can drop anywhere on a machine, and just start it, which makes it very convenient. It takes literally seconds to configure and form a cluster, which provides you with fault-tolerance and high-availability. With rqlite you have complete control over your database infrastructure, and the data it stores. https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#why-would-i-use-this-versus-some-other-distributed-database https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#why-...

infogulch 5y ago

Another commenter mentioned automatic sharding to which you replied "no". But short of that what do you think about use cases where the database is "naturally" very sharded, say where every user has a separate database? With such a design the operator might want to scale rqlite down to zero instances when a db is unused for a while and start it back up quickly on demand. Have you thought about how rqlite could fit into this design space?

zeckalpha 5y ago
That seems better solved outside of rqlite in the application layer and inetd or k8s.
- deleted 5y ago
  [deleted]
otoolep 5y ago
I have considered it, but it would introduce a very large amount of complexity -- and make rqlite much more complicated to operate. Adding this type of functionality would push mean rqlite would no longer be a worthwhile system to use. It would do something (sharding) probably no better than other systems, but no longer be trivial to operate. rqlite does something somewhat narrow, but does that very well (though I would say that :-) )
- stavros 5y ago
  I have to say, I like your product thinking. I agree with you.

deleted 5y ago

[deleted]