9 ms·

I edit a database newsletter – https://dbweekly.com/ https://dbweekly.com/ – so tend to always have my eyes out for new releases, what's coming along, and what

I edit a database newsletter – https://dbweekly.com/ https://dbweekly.com/ – so tend to always have my eyes out for new releases, what's coming along, and what not. And I thought I'd share a few more things that have jumped out at me recently in case anyone's in the mood for spelunking. 1. QuestDB – https://questdb.io/ https://questdb.io/ – is a performance-focused, open-source time-series database that uses SQL. It makes heavy use of SIMD and vectorization for the performance end of things. 2. GridDB - https://griddb.net/en/ https://griddb.net/en/ - is an in-memory NoSQL time-series database (there's a theme lately with these!) out of Toshiba that was boasting doing 5m writes per second and 60m reads per second on a 20 node cluster recently. 3. MeiliSearch - https://github.com/meilisearch/MeiliSearch https://github.com/meilisearch/MeiliSearch – not exactly a database but basically an Elastic-esque search server written in Rust. Seems to have really taken off. 4. Dolt – https://github.com/liquidata-inc/dolt https://github.com/liquidata-inc/dolt – bills itself as a 'Git for data'. It's relational, speaks SQL, but has version control on everything. TerminusDB, KVRocks, and ImmuDB also get honorable mentions. InfoWorld also had an article recently about 9 'offbeat' databases to check out if you want to go even further: https://www.infoworld.com/article/3533410/9-offbeat-databases-worth-a-look.html https://www.infoworld.com/article/3533410/9-offbeat-database... Exciting times in the database space!

rapnie 6y ago
TerminusDB recently featured on HN. Also a git-for-data approach, time-travel, storing data as RDF and queries return JSON-LD. https://news.ycombinator.com/item?id=22867767 https://news.ycombinator.com/item?id=22867767
- LukeEF 6y ago
  Yes, the latest version of Terminus uses JSON-LD, which is a serialisation of RDF. This enables us to store queries in a database and provides a schema documentation for our datalog query language. Release notes: https://github.com/terminusdb/terminusdb-server/blob/master/RELEASE_NOTES.md https://github.com/terminusdb/terminusdb-server/blob/master/...
heipei 6y ago
Just came to say that I recently discovered your newsletter and am very glad for it. The database space has been seeing so much movement lately that your newsletter is the missing piece for staying on top of it.
mywittyname 6y ago
I'm surprised that it has taken so long for time series databases to take off. I remember in 2015 I wrote an application to wedge time-series features into MySQL because the only option I could find for a native db was paid.
- mulmen 6y ago
  My day job relies heavily on Amazon Redshift. Most of our fact tables are sorted by date and almost always a locale. I wrote a Python script to delete the stale records based on that time key. We have a separate vacuum service that cleans up the mess continuously. Is this considered a time series database? Or is there something a dedicated time series database does differently?
  - manigandham 6y ago
    Time-series databases themselves are usually key/value or relational. The "time-series" features come from the focus around time as a primary component of the data (or some kind of increasing numeric value). Features like automatic partitioning and sharding, better compression, aggregating into time buckets, extracting and managing date values, smoothing missing values in queries, etc.
    - deleted 6y ago
      [deleted]
    - mulmen 6y ago
      Redshift does a lot of that. Is there something unique to a "time series" database that Redshift does not do? Is it the inclusion of some ETL capabilities? I'm not sure what makes a database truly time-series. It sounds like there is some compromise made that requires a time dimension. I'm not sure what that compromise would be beyond the general case of a columnar database.
      - manigandham 6y ago
        Nothing unique. Any database can store "time-series" data. If you look at InfluxDB (key/value) or TimescaleDB (relational), they offer things like high precision time values, higher compression algorithms that work better with timestamps, utilities to make it easier to query for time (buckets, timezones, gap-filling), automatic rollups and aggregations, data deletions based on time, etc. You can do all that yourself on Redshift but they just offer more built-in functionality for it.
manigandham 6y ago
There have been a lot of interesting projects released as alternatives to Elasticsearch and Solr like Sonic, Tantivy, Toshi, Xapiand, Typesense, Blast, etc. List here: https://gist.github.com/manigandham/58320ddb24fed654b57b4ba22aceae25 https://gist.github.com/manigandham/58320ddb24fed654b57b4ba2...
- ekabod 6y ago
  bayard [0]. [0]https://github.com/bayard-search/bayard https://github.com/bayard-search/bayard
  - deleted 6y ago
    [deleted]
  - manigandham 6y ago
    Thanks, already listed but updated the repo link.
  - amelius 6y ago
    > Bayard is a full-text search and indexing server written in Rust built on top of Tantivy that implements Raft Consensus Algorithm and gRPC. Why would Raft be part of a search module when it can be written (and/or used) as a separate library? Curious about the design-choice here.
    - fulmicoton 6y ago
      It is using a separate library from the TiDB project.
gen220 6y ago
Has anybody tried using Dolt? It looks quite young. I think this idea is really valuable, but I usually see it implemented as a time-series extension on top of Postgres or MySQL, like SQLAlchemy-Continuum or TimescaleDB. i.e. you caan get most of the useful git-like time-travel semantics (modulo schema migrations) out of timeseries data with a separate transaction history table. I'm curious what Dolt's performance profile looks like (i.e. how reads and writes scale vs "branch"-count, row-count, and how they handle indices across schema migrations), since the aforementioned solutions on Postgres are building on an already-very-performant core. edit: TerminusDB also looks very cool, although it's approaching this problem from a graph-database angle rather than a relational one. Their architecture (prolog server on top of a rust persistence core) also seems super fascinating, I'd love to read more on how they chose it.
- LukeEF 6y ago
  Thanks! Luke from TerminusDB here. We think a graph is the best way to approach a graphy problem! We like Prolog for Querying, constraint checking and user interaction. Rust is great for low-level data manipulation. Prolog is a superpower - very simple but very powerful, but quite far removed from the hardware and uses abstractions not under our control so not good at nitty-gritty bit manipulation. We like Rust as a low-level memory-safe language (and it has a great community). Much more background on why we made those choices: https://news.ycombinator.com/item?id=22867767 https://news.ycombinator.com/item?id=22867767
- timsehn 6y ago
  CEO of Dolt here. You are indeed correct, we are quite young. We are also focused at this stage on the data distribution use case (ie. use Dolt instead of putting a CSV or JSON file on GitHub). So, we haven't spent much time on query performance or scale. The current model is that the database and all of its history have to fit on the user's hard drive. The format is easily distributed (ie. put content addressed chunks on different servers, optimize for fewer network hops) but that's not how it works today. That being said, we think architecturally we can eventually get pretty close to parity with other RDBMS on the read path. We will be slower on the write path given the need to build the Merkle DAG on writes. Long ways to go though, we only launched open source last August. On a suite of MySQL ~6M correctness benchmarks, we currently execute 3-4X more slowly. These aren't large sets either so we suspect we'll run into some non-linearities in performance. This is just normal SQL. We haven't really tested the limits of how many branches we can handle or how long branch creation or merge takes at scale. Not because we don't want to but because it's not the use case we're focused on. If you are willing to take the journey with us, we'll work with you to get Dolt where it needs to be (if it can, reads should be fine, writes will be more expensive). But we're just at the beginning so there will be bumps along the road. If that's not for you, we completely understand.
Arkanosis 6y ago
I was about to ask if there was a RSS version of your newsletter, and it appears there is: https://cprss.s3.amazonaws.com/dbweekly.com.xml https://cprss.s3.amazonaws.com/dbweekly.com.xml Looks awesome, added to my feeds!
- petercooper 6y ago
  Yep, you got it. We should link these from the front page really :-) The main problem is they're just an "added extra" and we can't always guarantee the formatting is 100% due to our focus on getting the emails just right. It tends to be OK 99% of the time though.
LukeEF 6y ago
Luke from TerminusDB here. Thanks for the honorable mention! Definitely exciting times in database land and great to be in the same list as all these projects. We are focused on the revision control aspects of Terminus - trying to make the lives of data intensive (ML etc.) teams a little easier. We use a delta encoding approach to updates like source control systems such as git and provide the whole suite of revision control features: branch, merge, squash, rollback, blame, and time-travel. Idea is to provide continuous integration for the data layer. Basically squash all the tools and processes mentioned here into a versioned graph: https://martinfowler.com/articles/cd4ml.html https://martinfowler.com/articles/cd4ml.html
lhdj 6y ago
Not really a database - but Parallax[0] is a data warehouse geared at private/sensitive data written in Rust - it allows you to perform data anonymisation at query-time. Furthermore it exposes a privacy-as-code framework allowing you to version control and be very granular about how different entities interact with your datasets. I'm on the core team - if you have any questions I would be more than happy to answer. [0] https://github.com/openquery-io/parallax https://github.com/openquery-io/parallax
murkt 6y ago
MeiliSearch is exactly not Elastic-like search server. It should work out of the box for its intended use case (fast typo-tolerant search), and it doesn’t support basically everything else. No complex queries, no aggregations, no big datasets, no clusters, etc.
arauhala 6y ago
Hi Peter :-) Antti from aito.ai here (https://aito.ai https://aito.ai) What do you think about Aito and the predictive databases? We believe that the predictive databases can be a thing both in the database and the ML space, which is quite exciting. I also feel that the 'smart databases' like Aito or BayesDB form an interesting database category, which will emerge as an everyday softwarw component once the technology matures. According to our internal benchmarks, it is already mature accuracy & performance wise for many applications. - Antti & Aito
willvarfar 6y ago
thx, didn't know about that before, have just subscribed :D
lifty 6y ago
Since you mentioned Dolt, I will add Noms [1], which is the original db that Dolt is based on. The folks that created Noms are working now on Replicache [2]. I started using Noms in my project and the offline first tech is really wonderful; it allows me to do some cool stuff like syncing clients only when they come online. 1. https://github.com/attic-labs/noms https://github.com/attic-labs/noms 2. https://replicache.dev https://replicache.dev
- lessname 6y ago
  Is there also a javascript client for Dolt? I like the idea of Noms but it seems like it is not in active development
santafen 6y ago
Since you mentioned QuestDB ... I'm a former InfluxDB person, now at QuestDB. I'll say this about them: Their performance is off the charts good. Influx performance has been, frankly, headed in the other direction. I was looking at the NYC Taxi data the other day with QuestDB and I ran a sum over 1.6B rows of taxi data in under 150ms. 'select sum(passenger_count) from trips;'

rapnie 6y ago

TerminusDB recently featured on HN. Also a git-for-data approach, time-travel, storing data as RDF and queries return JSON-LD. https://news.ycombinator.com/item?id=22867767 https://news.ycombinator.com/item?id=22867767

LukeEF 6y ago
Yes, the latest version of Terminus uses JSON-LD, which is a serialisation of RDF. This enables us to store queries in a database and provides a schema documentation for our datalog query language. Release notes: https://github.com/terminusdb/terminusdb-server/blob/master/RELEASE_NOTES.md https://github.com/terminusdb/terminusdb-server/blob/master/...

heipei 6y ago

Just came to say that I recently discovered your newsletter and am very glad for it. The database space has been seeing so much movement lately that your newsletter is the missing piece for staying on top of it.

mywittyname 6y ago

I'm surprised that it has taken so long for time series databases to take off. I remember in 2015 I wrote an application to wedge time-series features into MySQL because the only option I could find for a native db was paid.

mulmen 6y ago
My day job relies heavily on Amazon Redshift. Most of our fact tables are sorted by date and almost always a locale. I wrote a Python script to delete the stale records based on that time key. We have a separate vacuum service that cleans up the mess continuously. Is this considered a time series database? Or is there something a dedicated time series database does differently?
- manigandham 6y ago
  Time-series databases themselves are usually key/value or relational. The "time-series" features come from the focus around time as a primary component of the data (or some kind of increasing numeric value). Features like automatic partitioning and sharding, better compression, aggregating into time buckets, extracting and managing date values, smoothing missing values in queries, etc.
  - deleted 6y ago
    [deleted]
  - mulmen 6y ago
    Redshift does a lot of that. Is there something unique to a "time series" database that Redshift does not do? Is it the inclusion of some ETL capabilities? I'm not sure what makes a database truly time-series. It sounds like there is some compromise made that requires a time dimension. I'm not sure what that compromise would be beyond the general case of a columnar database.
    - manigandham 6y ago
      Nothing unique. Any database can store "time-series" data. If you look at InfluxDB (key/value) or TimescaleDB (relational), they offer things like high precision time values, higher compression algorithms that work better with timestamps, utilities to make it easier to query for time (buckets, timezones, gap-filling), automatic rollups and aggregations, data deletions based on time, etc. You can do all that yourself on Redshift but they just offer more built-in functionality for it.

manigandham 6y ago

There have been a lot of interesting projects released as alternatives to Elasticsearch and Solr like Sonic, Tantivy, Toshi, Xapiand, Typesense, Blast, etc. List here: https://gist.github.com/manigandham/58320ddb24fed654b57b4ba22aceae25 https://gist.github.com/manigandham/58320ddb24fed654b57b4ba2...

ekabod 6y ago
bayard [0]. [0]https://github.com/bayard-search/bayard https://github.com/bayard-search/bayard
- deleted 6y ago
  [deleted]
- manigandham 6y ago
  Thanks, already listed but updated the repo link.
- amelius 6y ago
  > Bayard is a full-text search and indexing server written in Rust built on top of Tantivy that implements Raft Consensus Algorithm and gRPC. Why would Raft be part of a search module when it can be written (and/or used) as a separate library? Curious about the design-choice here.
  - fulmicoton 6y ago
    It is using a separate library from the TiDB project.

gen220 6y ago

Has anybody tried using Dolt? It looks quite young. I think this idea is really valuable, but I usually see it implemented as a time-series extension on top of Postgres or MySQL, like SQLAlchemy-Continuum or TimescaleDB. i.e. you caan get most of the useful git-like time-travel semantics (modulo schema migrations) out of timeseries data with a separate transaction history table. I'm curious what Dolt's performance profile looks like (i.e. how reads and writes scale vs "branch"-count, row-count, and how they handle indices across schema migrations), since the aforementioned solutions on Postgres are building on an already-very-performant core. edit: TerminusDB also looks very cool, although it's approaching this problem from a graph-database angle rather than a relational one. Their architecture (prolog server on top of a rust persistence core) also seems super fascinating, I'd love to read more on how they chose it.

LukeEF 6y ago
Thanks! Luke from TerminusDB here. We think a graph is the best way to approach a graphy problem! We like Prolog for Querying, constraint checking and user interaction. Rust is great for low-level data manipulation. Prolog is a superpower - very simple but very powerful, but quite far removed from the hardware and uses abstractions not under our control so not good at nitty-gritty bit manipulation. We like Rust as a low-level memory-safe language (and it has a great community). Much more background on why we made those choices: https://news.ycombinator.com/item?id=22867767 https://news.ycombinator.com/item?id=22867767
timsehn 6y ago
CEO of Dolt here. You are indeed correct, we are quite young. We are also focused at this stage on the data distribution use case (ie. use Dolt instead of putting a CSV or JSON file on GitHub). So, we haven't spent much time on query performance or scale. The current model is that the database and all of its history have to fit on the user's hard drive. The format is easily distributed (ie. put content addressed chunks on different servers, optimize for fewer network hops) but that's not how it works today. That being said, we think architecturally we can eventually get pretty close to parity with other RDBMS on the read path. We will be slower on the write path given the need to build the Merkle DAG on writes. Long ways to go though, we only launched open source last August. On a suite of MySQL ~6M correctness benchmarks, we currently execute 3-4X more slowly. These aren't large sets either so we suspect we'll run into some non-linearities in performance. This is just normal SQL. We haven't really tested the limits of how many branches we can handle or how long branch creation or merge takes at scale. Not because we don't want to but because it's not the use case we're focused on. If you are willing to take the journey with us, we'll work with you to get Dolt where it needs to be (if it can, reads should be fine, writes will be more expensive). But we're just at the beginning so there will be bumps along the road. If that's not for you, we completely understand.

Arkanosis 6y ago

I was about to ask if there was a RSS version of your newsletter, and it appears there is: https://cprss.s3.amazonaws.com/dbweekly.com.xml https://cprss.s3.amazonaws.com/dbweekly.com.xml Looks awesome, added to my feeds!

petercooper 6y ago
Yep, you got it. We should link these from the front page really :-) The main problem is they're just an "added extra" and we can't always guarantee the formatting is 100% due to our focus on getting the emails just right. It tends to be OK 99% of the time though.

LukeEF 6y ago

Luke from TerminusDB here. Thanks for the honorable mention! Definitely exciting times in database land and great to be in the same list as all these projects. We are focused on the revision control aspects of Terminus - trying to make the lives of data intensive (ML etc.) teams a little easier. We use a delta encoding approach to updates like source control systems such as git and provide the whole suite of revision control features: branch, merge, squash, rollback, blame, and time-travel. Idea is to provide continuous integration for the data layer. Basically squash all the tools and processes mentioned here into a versioned graph: https://martinfowler.com/articles/cd4ml.html https://martinfowler.com/articles/cd4ml.html

lhdj 6y ago

Not really a database - but Parallax[0] is a data warehouse geared at private/sensitive data written in Rust - it allows you to perform data anonymisation at query-time. Furthermore it exposes a privacy-as-code framework allowing you to version control and be very granular about how different entities interact with your datasets. I'm on the core team - if you have any questions I would be more than happy to answer. [0] https://github.com/openquery-io/parallax https://github.com/openquery-io/parallax

murkt 6y ago

MeiliSearch is exactly not Elastic-like search server. It should work out of the box for its intended use case (fast typo-tolerant search), and it doesn’t support basically everything else. No complex queries, no aggregations, no big datasets, no clusters, etc.

arauhala 6y ago

Hi Peter :-) Antti from aito.ai here (https://aito.ai https://aito.ai) What do you think about Aito and the predictive databases? We believe that the predictive databases can be a thing both in the database and the ML space, which is quite exciting. I also feel that the 'smart databases' like Aito or BayesDB form an interesting database category, which will emerge as an everyday softwarw component once the technology matures. According to our internal benchmarks, it is already mature accuracy & performance wise for many applications. - Antti & Aito

willvarfar 6y ago

thx, didn't know about that before, have just subscribed :D

lifty 6y ago

Since you mentioned Dolt, I will add Noms [1], which is the original db that Dolt is based on. The folks that created Noms are working now on Replicache [2]. I started using Noms in my project and the offline first tech is really wonderful; it allows me to do some cool stuff like syncing clients only when they come online. 1. https://github.com/attic-labs/noms https://github.com/attic-labs/noms 2. https://replicache.dev https://replicache.dev

lessname 6y ago
Is there also a javascript client for Dolt? I like the idea of Noms but it seems like it is not in active development

santafen 6y ago

Since you mentioned QuestDB ... I'm a former InfluxDB person, now at QuestDB. I'll say this about them: Their performance is off the charts good. Influx performance has been, frankly, headed in the other direction. I was looking at the NYC Taxi data the other day with QuestDB and I ran a sum over 1.6B rows of taxi data in under 150ms. 'select sum(passenger_count) from trips;'