8 ms·

There's been an explosion of new DBs, but I haven't found anything that really beats Postgres or MariaDB for most workloads. The main advantages of these battle

There's been an explosion of new DBs, but I haven't found anything that really beats Postgres or MariaDB for most workloads. The main advantages of these battle tested DBs is that they're easy to operate, well understood, full featured, and can handle most workloads. It does make me wonder what will be the next big leap in DB technology. Most of the NoSQL or distributed DB implementations have a bunch of limitations which make them impractical (or not worth the trade offs) for most applications, IMO. Distributed DBs are great until things go wrong, and then you have a nightmare on your hands. It's a lot easier to optimize simple relational DBs with caching layers, and adding read replicas scales quite effectively too. The only somewhat recent new DB that comes to mind which had a really interesting model was RethinkDB, although it suffered from a variety of issues, including scale problems. Anyway, these days I stick with Postgres for 99% of things, and mix in Redis where needed for key/value stuff.

aloknnikhil 7y ago
The thing is most applications don't really have data across regions that are related. So, there really is no need for distributed databases for most of the use cases which actually can be solved by sharding. Also, most of the applications already gracefully handle DB failures by failing over to stand-by replicas which PgSQL and MariaDB already provide. However, I do think the key innovations are in building control planes around existing relational and NoSQL databses for scaling/sharding them across a set of resources to minimize cost while meeting performance and availability constraints.
ksec 7y ago
Agree. Boring Tech is Good. Postgre + Redis / Memcached is probably good enough for 99% of use case. ( I still wish Postgre made shading easier ) RethinkDB, CockroachDB and FoundationDB are worth keeping an eye on.
- magicalhippo 7y ago
  > I still wish Postgre made shading easier Yeah, MySQL is ahead of the curve there... https://demozoo.org/productions/268459/ https://demozoo.org/productions/268459/
  - ksec 7y ago
    ROFL, I was wondering what is going on there for a moment. Turns out "Sharding"..... is not a word and gets auto corrected to shading.
- bauerd 7y ago
  RethinkDB is pretty much dead at this point I think?
  - Aeolun 7y ago
    They released their new version a few months ago I think. Along with a blog post explaining why it took so long. Still, I can only be carefully optimistic.
heipei 7y ago
The issue that I frequently run into is not that I'm looking for a fancy distributed/sharded database because of reasons of performance, but because I need to store large amounts of data in a way that allows me to grow this datastore by "just adding boxes" while still retaining a few useful database features. I'd love to use Postgres but eventually my single server will run out of disk space. Now, one approach is to just dismiss this use-case by pointing at DynamoDB and similar offerings. But if for some reason you can't use these hosted platforms, what do you use instead? For search, ElasticSearch fortunately fits the bill, the "just keep adding boxes" concept works flawlessly, operating it is a breeze. But you probably don't want to use ElasticSearch as your primary datastore, so what do you use there? I had terrible experiences operating a sharded MongoDB cluster and my next attempt will be using something like ScyllaDB/Cassandra instead since operations seem to require much less work and planning. What other databases would offer that no-advance-planning scaling capability? Somewhat unrelated, by I often wonder what one were to use for a sharded/distributed blob store that offers basic operations like "grep across all blobs" with different query-performance than a real-time search index like ElasticSearch. Would one have to use Hadoop or are there any alternatives which require little operational effort?
- momirlan 7y ago
  yup, if you have infinite time hadoop will fit the bill nicely. but looking at the cost/skills to operate, maybe Elasticsearch is still the better offering
- DelightOne 7y ago
  What is the difference between one server with many TB vs multiple servers with less space?
  - Drdrdrq 7y ago
    With multiple servers you can add space to each of them. With a single one there is a much lower limit to what you can do - that's the idea behind vertical/horizontal scalability. That, and the systems with multiple nodes can be made more reliable than single node servers.
  - 7y ago
deleted 7y ago
[deleted]
hn_throwaway_99 7y ago
Agreed, now that JSON/JSONB support is so good in postgres and MySQL, I see less and less of a reason for the NoSQL databases of yesteryear. There was a really good post from Martin Fowler a while back that the popularity of "NoSQL" was really because it was "NoDBA" - app devs could sidestep the bottleneck of needing to get DBAs involved whenever you needed to persist an extra object field. While it's easy to abuse JSON storage in postgres, for things that are really just "opaque objects", vs. relational properties, appropriate use of JSON columns can save a ton of unnecessary overhead updating schemas.
jialutu 7y ago
If you want distributed DBs built on top boring old SQL databases, there is always https://vitess.io/ https://vitess.io/. Not played with it too much myself, but it's been tried and tested by big companies (it was originally built by youtube), so worth a try.
yingw787 7y ago
I’m excited for the time when databases are built assuming that I/O is no longer the primary bound for distributed deploys, and multi-node by default deploys are a thing :)
- lichtenberger 7y ago
  Yeah, in the advent of byte-addressable NVM... I think we have to rethink a lot of stuff and I'm sure we can get rid of a lot of stuff, which isn't needed anymore or should be replaced with light weight components. I'm trying to achieve some of this with https://sirix.io https://sirix.io. However, I hope more and more people will get involved over time as it's of course completely Open Source.
assface 7y ago
> The only somewhat recent new DB that comes to mind which had a really interesting model was RethinkDB In what way was it interesting? It was a document DBMS that supported MVCC.
StreamBright 7y ago
True. However, the top 1% of users have 1000+ TB data warehouses where Postgres or MariaDB is not an option. These use cases do not require ACID/OLTP, though. This is why projects like Presto thrive. I think the next obvious leap for data management is bridging the OLTP OLAP gap and having the same database providing both, using the same query engine and different storage engines. Moving data from OLTP systems to OLAP always had its challenges; many companies, OSS projects, etc. wanted to solve it, with mixed results.
- sa46 7y ago
  I manage about a petabyte of data using a Citus as the distributed query engine on top of Postgres nodes. It’s nice because you can query with Postgres but there’s a decent number of sharp edges at scale.
bradleyjg 7y ago
> can handle most workloads. They really shine for read heavy workflows that can tolerate a stale read every once in a while. If on top of that you have reasonable shard-ability you get near infinite scalability. While that might cover a large portion of the database usage landscape, I'd hesitate to call it most. There's a reason OLTP was coined as an acronym--it's a pattern that comes up a fair bit.
jnordwick 7y ago
You must not be doing a lot of OLAP or very highly concurrent workloads. Netezza, Informix, KDB and others will all well outperform open source dbs. All open source dbs are absolute trash for time series an other olap type queries.
- kinghajj 7y ago
  We used ScyllaDB at my last employer to store time-series data, it worked quite well, though you do have to add a lot of supporting code, it doesn't come out-of-the-box for free.
  - jnordwick 7y ago
    ScyllaDB isn't really OLAP - it mostly just a glorified logger that throws stuff on disk. It is a terrible TSBD when you really need a TSDB - like basically all open source dbs. we're looking at timescale right now, but basically as the best of the worst. i have the budget no way i'd be using it.
    - PeterCorless 7y ago
      We recommend running KairosDB on top of Scylla for TSDB use cases.
- qaq 7y ago
  ClickHouse is open source it's def not trash for olap and ts
  - jnordwick 7y ago
    clickhouse could barely complete a 15 min moving average; last time I checked it required a very slow correlated subquery. That's pretty much where I stopped evaluating it. edit: after looking it up again, looks like that is still the case, and you have have to be fairly limited with cumulative aggregates if you want the keep performance. maybe someday, but as of now, still not very good.
dustingetz 7y ago
a truly novel competitor to postgresql is Datomic. Datomic takes the ideas of the Clojure programming language (immutability, functional programming, data is namespaced) and extends them through to an ACID database.
- fulafel 7y ago
  Datomic is great but their focus on the paid AWS version limits its potential.
arminiusreturns 7y ago
So for the past few years since about ~2013 I've been on and off keeping an eye on Michael Stonebraker and his work on VoltDB (based on lessons learned from h-store) and Scidb and in general his criticisms of nosql and newsql variants. I think Scidb, being a column-oriented dbms geared for multi-dimensional arrays (datacubes) is very interesting given current trends, and there are only a handful of similar dbs around, the other two that interest me are rasdaman and monetdb. I don't know if OmniScidb counts as a datacube db but it is also really interesting especially due to it's gpu and caching model. As a sysadmin/ops type having to deal with monitoring timeseries db's are also something I like to keep an eye on. It used to be mostly rrdtool in this space, but now I am comparing prometheus and influxdb. Like OmniSci, another case of something that's not quite in the same db model but might even better a better solution for the space (metrics) is Apache Druid. (elasticsearch being another than can be massaged to fit as a quasi tsdb as well) I think there is some room to unify the monitoring/metrics and log storage arena into one space (usually they are separated which adds admin overhead) and right now I really like druid as a potential for this. Another interesting application of timeseriesdb's that I have been keeping an eye on is in the quant/algo trading area. Most people have been using kdb+ there but many are looking for replacements and there are some really good conversations to be found about the kind of limitations they are hitting. I'm just a sysadmin who likes to keep up with whats going on, and my db knowledge is limited, but I do have a process for narrowing my focus to the dbs I reference. It must be open source, bonus points to gpl or apache licenses. The language it is written in is important but not a deal breaker (very tired of so many java based dbs). I don't like it when they are tacked on top of an "older" tech (such as kairos on cassandra, timeseriesdb on top of postgres, opentdsb on top of hbase, kudu on hadoop, etc). Being either filesystem aware or agnostic can be nice features (playing well with ceph, lustre, etc) Not saying this is the sort of selection criteria others should use just giving some info on mine. A few more interesting mentions: clickhouse, gnocchi, marketstore, Atlas (Netflix), opentick (on top of foundationdb).
- tlipcon 7y ago
  Just a small note: Kudu shares no code or provenance with Hadoop. Only commonality is ecosystem (eg you can use spark or Impala to query it)
jeswin 7y ago
This is a database for specialized workloads and not a general purpose DB like Postgres or MariaDB. This offtopic thread could derail discussion on an interesting topic.

aloknnikhil 7y ago

The thing is most applications don't really have data across regions that are related. So, there really is no need for distributed databases for most of the use cases which actually can be solved by sharding. Also, most of the applications already gracefully handle DB failures by failing over to stand-by replicas which PgSQL and MariaDB already provide. However, I do think the key innovations are in building control planes around existing relational and NoSQL databses for scaling/sharding them across a set of resources to minimize cost while meeting performance and availability constraints.

ksec 7y ago

Agree. Boring Tech is Good. Postgre + Redis / Memcached is probably good enough for 99% of use case. ( I still wish Postgre made shading easier ) RethinkDB, CockroachDB and FoundationDB are worth keeping an eye on.

magicalhippo 7y ago
> I still wish Postgre made shading easier Yeah, MySQL is ahead of the curve there... https://demozoo.org/productions/268459/ https://demozoo.org/productions/268459/
- ksec 7y ago
  ROFL, I was wondering what is going on there for a moment. Turns out "Sharding"..... is not a word and gets auto corrected to shading.
bauerd 7y ago
RethinkDB is pretty much dead at this point I think?
- Aeolun 7y ago
  They released their new version a few months ago I think. Along with a blog post explaining why it took so long. Still, I can only be carefully optimistic.

heipei 7y ago

The issue that I frequently run into is not that I'm looking for a fancy distributed/sharded database because of reasons of performance, but because I need to store large amounts of data in a way that allows me to grow this datastore by "just adding boxes" while still retaining a few useful database features. I'd love to use Postgres but eventually my single server will run out of disk space. Now, one approach is to just dismiss this use-case by pointing at DynamoDB and similar offerings. But if for some reason you can't use these hosted platforms, what do you use instead? For search, ElasticSearch fortunately fits the bill, the "just keep adding boxes" concept works flawlessly, operating it is a breeze. But you probably don't want to use ElasticSearch as your primary datastore, so what do you use there? I had terrible experiences operating a sharded MongoDB cluster and my next attempt will be using something like ScyllaDB/Cassandra instead since operations seem to require much less work and planning. What other databases would offer that no-advance-planning scaling capability? Somewhat unrelated, by I often wonder what one were to use for a sharded/distributed blob store that offers basic operations like "grep across all blobs" with different query-performance than a real-time search index like ElasticSearch. Would one have to use Hadoop or are there any alternatives which require little operational effort?

momirlan 7y ago
yup, if you have infinite time hadoop will fit the bill nicely. but looking at the cost/skills to operate, maybe Elasticsearch is still the better offering
DelightOne 7y ago
What is the difference between one server with many TB vs multiple servers with less space?
- Drdrdrq 7y ago
  With multiple servers you can add space to each of them. With a single one there is a much lower limit to what you can do - that's the idea behind vertical/horizontal scalability. That, and the systems with multiple nodes can be made more reliable than single node servers.
- 7y ago

deleted 7y ago

[deleted]

hn_throwaway_99 7y ago

Agreed, now that JSON/JSONB support is so good in postgres and MySQL, I see less and less of a reason for the NoSQL databases of yesteryear. There was a really good post from Martin Fowler a while back that the popularity of "NoSQL" was really because it was "NoDBA" - app devs could sidestep the bottleneck of needing to get DBAs involved whenever you needed to persist an extra object field. While it's easy to abuse JSON storage in postgres, for things that are really just "opaque objects", vs. relational properties, appropriate use of JSON columns can save a ton of unnecessary overhead updating schemas.

jialutu 7y ago

If you want distributed DBs built on top boring old SQL databases, there is always https://vitess.io/ https://vitess.io/. Not played with it too much myself, but it's been tried and tested by big companies (it was originally built by youtube), so worth a try.

yingw787 7y ago

I’m excited for the time when databases are built assuming that I/O is no longer the primary bound for distributed deploys, and multi-node by default deploys are a thing :)

lichtenberger 7y ago
Yeah, in the advent of byte-addressable NVM... I think we have to rethink a lot of stuff and I'm sure we can get rid of a lot of stuff, which isn't needed anymore or should be replaced with light weight components. I'm trying to achieve some of this with https://sirix.io https://sirix.io. However, I hope more and more people will get involved over time as it's of course completely Open Source.

assface 7y ago

> The only somewhat recent new DB that comes to mind which had a really interesting model was RethinkDB In what way was it interesting? It was a document DBMS that supported MVCC.

StreamBright 7y ago

True. However, the top 1% of users have 1000+ TB data warehouses where Postgres or MariaDB is not an option. These use cases do not require ACID/OLTP, though. This is why projects like Presto thrive. I think the next obvious leap for data management is bridging the OLTP OLAP gap and having the same database providing both, using the same query engine and different storage engines. Moving data from OLTP systems to OLAP always had its challenges; many companies, OSS projects, etc. wanted to solve it, with mixed results.

sa46 7y ago
I manage about a petabyte of data using a Citus as the distributed query engine on top of Postgres nodes. It’s nice because you can query with Postgres but there’s a decent number of sharp edges at scale.

bradleyjg 7y ago

> can handle most workloads. They really shine for read heavy workflows that can tolerate a stale read every once in a while. If on top of that you have reasonable shard-ability you get near infinite scalability. While that might cover a large portion of the database usage landscape, I'd hesitate to call it most. There's a reason OLTP was coined as an acronym--it's a pattern that comes up a fair bit.

jnordwick 7y ago

You must not be doing a lot of OLAP or very highly concurrent workloads. Netezza, Informix, KDB and others will all well outperform open source dbs. All open source dbs are absolute trash for time series an other olap type queries.

kinghajj 7y ago
We used ScyllaDB at my last employer to store time-series data, it worked quite well, though you do have to add a lot of supporting code, it doesn't come out-of-the-box for free.
- jnordwick 7y ago
  ScyllaDB isn't really OLAP - it mostly just a glorified logger that throws stuff on disk. It is a terrible TSBD when you really need a TSDB - like basically all open source dbs. we're looking at timescale right now, but basically as the best of the worst. i have the budget no way i'd be using it.
  - PeterCorless 7y ago
    We recommend running KairosDB on top of Scylla for TSDB use cases.
qaq 7y ago
ClickHouse is open source it's def not trash for olap and ts
- jnordwick 7y ago
  clickhouse could barely complete a 15 min moving average; last time I checked it required a very slow correlated subquery. That's pretty much where I stopped evaluating it. edit: after looking it up again, looks like that is still the case, and you have have to be fairly limited with cumulative aggregates if you want the keep performance. maybe someday, but as of now, still not very good.

dustingetz 7y ago

a truly novel competitor to postgresql is Datomic. Datomic takes the ideas of the Clojure programming language (immutability, functional programming, data is namespaced) and extends them through to an ACID database.

fulafel 7y ago
Datomic is great but their focus on the paid AWS version limits its potential.

arminiusreturns 7y ago

So for the past few years since about ~2013 I've been on and off keeping an eye on Michael Stonebraker and his work on VoltDB (based on lessons learned from h-store) and Scidb and in general his criticisms of nosql and newsql variants. I think Scidb, being a column-oriented dbms geared for multi-dimensional arrays (datacubes) is very interesting given current trends, and there are only a handful of similar dbs around, the other two that interest me are rasdaman and monetdb. I don't know if OmniScidb counts as a datacube db but it is also really interesting especially due to it's gpu and caching model. As a sysadmin/ops type having to deal with monitoring timeseries db's are also something I like to keep an eye on. It used to be mostly rrdtool in this space, but now I am comparing prometheus and influxdb. Like OmniSci, another case of something that's not quite in the same db model but might even better a better solution for the space (metrics) is Apache Druid. (elasticsearch being another than can be massaged to fit as a quasi tsdb as well) I think there is some room to unify the monitoring/metrics and log storage arena into one space (usually they are separated which adds admin overhead) and right now I really like druid as a potential for this. Another interesting application of timeseriesdb's that I have been keeping an eye on is in the quant/algo trading area. Most people have been using kdb+ there but many are looking for replacements and there are some really good conversations to be found about the kind of limitations they are hitting. I'm just a sysadmin who likes to keep up with whats going on, and my db knowledge is limited, but I do have a process for narrowing my focus to the dbs I reference. It must be open source, bonus points to gpl or apache licenses. The language it is written in is important but not a deal breaker (very tired of so many java based dbs). I don't like it when they are tacked on top of an "older" tech (such as kairos on cassandra, timeseriesdb on top of postgres, opentdsb on top of hbase, kudu on hadoop, etc). Being either filesystem aware or agnostic can be nice features (playing well with ceph, lustre, etc) Not saying this is the sort of selection criteria others should use just giving some info on mine. A few more interesting mentions: clickhouse, gnocchi, marketstore, Atlas (Netflix), opentick (on top of foundationdb).

tlipcon 7y ago
Just a small note: Kudu shares no code or provenance with Hadoop. Only commonality is ecosystem (eg you can use spark or Impala to query it)

jeswin 7y ago

This is a database for specialized workloads and not a general purpose DB like Postgres or MariaDB. This offtopic thread could derail discussion on an interesting topic.