4 ms·

Apparently clustering too https://github.com/rqlite/rqlite https://github.com/rqlite/rqlite

minxomat 7y ago
Hardly, that is more of a hack. There is however a distributed, clustered version of sqlite: Bloomberg's Comdb2. It has the same scaling factors as e.g. percona, can be Geo-replicated (multi-roomed) and has many advanced timing and financial data types. Plus SQL functions can be written in Lua. Oh, and it has super granular serialization guarantees, up to and including full linearizibility. By using Sqlite as the SQL frontend, it retains features other NewSQL engines struggle with (recursive CTE, deep optimization, compatibility with existing queries etc)
- e12e 7y ago
  Wow, thats: https://github.com/bloomberg/comdb2 https://github.com/bloomberg/comdb2 Looks interesting. Any experience using it in real projects?
  - minxomat 7y ago
    Yeah it works well. I also sometimes push code into the db for faster execution (sorry on mobile, can't detail now). Haven't tested out Geo replication though, but it's pretty straightforward. Clock drift maximum and all that.
hardwaresofton 7y ago
Also there is dqlite[0]. I really do think that SQLite can go even farther than people think, but the exploration makes less and less sense the better and easier to run options like postgres become. That said, I'm glad projects like dqlite and rqlite exist. There are also previous threads discussing them but here's one[1]. [0]: https://github.com/CanonicalLtd/dqlite https://github.com/CanonicalLtd/dqlite [1]: https://news.ycombinator.com/item?id=20836331 https://news.ycombinator.com/item?id=20836331
- speedplane 7y ago
  > I really do think that SQLite can go even farther than people think I know different databases support different features (e.g., atomic transactions, ACID compliance, etc.), but it seems like the big differentiator is size, namely: 1. Can your database live within a single computer's RAM? 2. Can your database live within a single computer? 3. Can your database live across a cluster of computers in a single data center? 4. Can your database live across a cluster of computers spread across the planet? Once you pick your category from the above, there are still choices to be made, but the "best" options narrow down significantly.
  - hardwaresofton 7y ago
    I'd argue that feature set (transactions, ACID, etc) are a bigger differentiator because they drive/change application development -- this can have drastic effects on application features and complexity whether immediately or down the road. I agree that the possible second biggest (if not the first) differentiator is size though -- despite the amount of choices out there, once you start narrowing down answering those questions, many options fall away very quickly. I'd also add on that the only real way to scale a database is to shard. There are a bunch of tricks you can do to improve throughput and those are always welcome, and sometimes I'll see articles that proclaim "we just changed to a system with 256GB of RAM to fix our scaling problem!" -- but these improvements don't get past the basic premise that once your data does not fit on one computer/harddrive you need to start figuring out a way for multiple to coordinate. You need to shard, and you need to organize your data in such a way that it can shard, then you need to start worrying about data access patterns. There's a lot of space before something like this is necessary (you really do have to have a lot of data for it to not fit on one machine these days), but I think people don't see this chain coming fast/often enough.
    - speedplane 7y ago
      > I'd argue that feature set (transactions, ACID, etc) are a bigger differentiator because they drive/change application development Handling atomic transactions or ACID on a single computer is a completely different problem from dealing with them across multiple, and still a different problem from dealing with them across datacenters. I'd argue that the size of your data is the first question you need to ask, only then you can figure out if you need/desire these features. > once your data does not fit on one computer/harddrive you need to start figuring out a way for multiple to coordinate Going from one computer to two generally more than doubles the complexity of the system. But when done right, going from two computers to two hundred can be easy. However, going from two hundred (in a single datacenter) to ~two thousand (across multiple datacenters) brings its own unique set of challenges. Picking the database for an app is one of the biggest tech decisions any team can make, certainly more important than the underlying OS, and even potentially more important than choosing a predominant programming language. Even when starting a tiny startup with no initial customers, considering how big about the absolute max you think you your data can grow, and picking a database that can grow with you can save you huge headaches down the road. Case Study: I started a company on Python 2.7 (fine, but not enlightened decision), a fork of Django that supported NoSQL (bad decision), and an underlying NoSQL database that scaled with no configuration (great decision). In the first year, the database had around a gigabyte of data, so NoSQL was definitely overkill. But now, it's over 100 Terabytes, with auto replication making it much larger than that. Changing the Django middleware or even the Python language to 3.x will be difficult, but not impossible. Moving the database is far more terrifying, a non-starter.

Hardly, that is more of a hack. There is however a distributed, clustered version of sqlite: Bloomberg's Comdb2. It has the same scaling factors as e.g. percona, can be Geo-replicated (multi-roomed) and has many advanced timing and financial data types. Plus SQL functions can be written in Lua. Oh, and it has super granular serialization guarantees, up to and including full linearizibility. By using Sqlite as the SQL frontend, it retains features other NewSQL engines struggle with (recursive CTE, deep optimization, compatibility with existing queries etc)

e12e 7y ago
Wow, thats: https://github.com/bloomberg/comdb2 https://github.com/bloomberg/comdb2 Looks interesting. Any experience using it in real projects?
- minxomat 7y ago
  Yeah it works well. I also sometimes push code into the db for faster execution (sorry on mobile, can't detail now). Haven't tested out Geo replication though, but it's pretty straightforward. Clock drift maximum and all that.

hardwaresofton 7y ago

Also there is dqlite[0]. I really do think that SQLite can go even farther than people think, but the exploration makes less and less sense the better and easier to run options like postgres become. That said, I'm glad projects like dqlite and rqlite exist. There are also previous threads discussing them but here's one[1]. [0]: https://github.com/CanonicalLtd/dqlite https://github.com/CanonicalLtd/dqlite [1]: https://news.ycombinator.com/item?id=20836331 https://news.ycombinator.com/item?id=20836331

speedplane 7y ago
> I really do think that SQLite can go even farther than people think I know different databases support different features (e.g., atomic transactions, ACID compliance, etc.), but it seems like the big differentiator is size, namely: 1. Can your database live within a single computer's RAM? 2. Can your database live within a single computer? 3. Can your database live across a cluster of computers in a single data center? 4. Can your database live across a cluster of computers spread across the planet? Once you pick your category from the above, there are still choices to be made, but the "best" options narrow down significantly.
- hardwaresofton 7y ago
  I'd argue that feature set (transactions, ACID, etc) are a bigger differentiator because they drive/change application development -- this can have drastic effects on application features and complexity whether immediately or down the road. I agree that the possible second biggest (if not the first) differentiator is size though -- despite the amount of choices out there, once you start narrowing down answering those questions, many options fall away very quickly. I'd also add on that the only real way to scale a database is to shard. There are a bunch of tricks you can do to improve throughput and those are always welcome, and sometimes I'll see articles that proclaim "we just changed to a system with 256GB of RAM to fix our scaling problem!" -- but these improvements don't get past the basic premise that once your data does not fit on one computer/harddrive you need to start figuring out a way for multiple to coordinate. You need to shard, and you need to organize your data in such a way that it can shard, then you need to start worrying about data access patterns. There's a lot of space before something like this is necessary (you really do have to have a lot of data for it to not fit on one machine these days), but I think people don't see this chain coming fast/often enough.
  - speedplane 7y ago
    > I'd argue that feature set (transactions, ACID, etc) are a bigger differentiator because they drive/change application development Handling atomic transactions or ACID on a single computer is a completely different problem from dealing with them across multiple, and still a different problem from dealing with them across datacenters. I'd argue that the size of your data is the first question you need to ask, only then you can figure out if you need/desire these features. > once your data does not fit on one computer/harddrive you need to start figuring out a way for multiple to coordinate Going from one computer to two generally more than doubles the complexity of the system. But when done right, going from two computers to two hundred can be easy. However, going from two hundred (in a single datacenter) to ~two thousand (across multiple datacenters) brings its own unique set of challenges. Picking the database for an app is one of the biggest tech decisions any team can make, certainly more important than the underlying OS, and even potentially more important than choosing a predominant programming language. Even when starting a tiny startup with no initial customers, considering how big about the absolute max you think you your data can grow, and picking a database that can grow with you can save you huge headaches down the road. Case Study: I started a company on Python 2.7 (fine, but not enlightened decision), a fork of Django that supported NoSQL (bad decision), and an underlying NoSQL database that scaled with no configuration (great decision). In the first year, the database had around a gigabyte of data, so NoSQL was definitely overkill. But now, it's over 100 Terabytes, with auto replication making it much larger than that. Changing the Django middleware or even the Python language to 3.x will be difficult, but not impossible. Moving the database is far more terrifying, a non-starter.