4 ms·

So, we need a new database engine instead of mere using tables with partitioning in PostgreSQL with a timestamp as a primary key? And it is Java - waste two ti

So, we need a new database engine instead of mere using tables with partitioning in PostgreSQL with a timestamp as a primary key? And it is Java - waste two times more resources than the data you have ™

rodionos 11y ago
Not sure if your Java comment is warranted, a lot of Apache projects are either written in Java or run on JVM. Yes, HBase's memstore can be optimized, in particular for small key values, but that doesn't negate a need for time-series databases with proven scaling. HDFS offers that. How you were planning to design your tables in PostreSQL if you have 1m+ unique series inserting at 100K+/sec. Assuming you use timestamp as a PK does it translate into 1m+ tables?
- dschiptsov 11y ago
  We have already seen how Cassandra clone in C++ perform. I think that adding a new indexing strategy to a decent engine could be a good idea. Informix did this.
  - kev009 11y ago
    Mapr has an HBase compatible stack in C++, I would love to see TSDB numbers on that. TSDB is pretty efficient itself and since metrics collection is such a batchy operation JVM isn't really that much of a problem. You can run some dedicated TSDs for front end/UIs to ameliorate JVM pain there too.
  - rodionos 11y ago
    Do you mean Informix Time-Series Extension? It's been around for a while so maybe there are existing things that came out of IBM recently. Last time I checked its schema it was basically storing serie as a blob not unlike OPC historians like Pi.
krenoten 11y ago
It depends how much time you have to burn, and how much traffic you handle. If you have trivial scaling needs or want to spend 10x the human time managing that setup, then go ahead. You use something like opentsdb when you have hundreds of thousands or millions of ops per second and don't have the time or desire to deal with a database that you have to build your own partitioning + resharding + failover + recovery systems on top of. B+ trees are not optimized for the kinds of write-heavy workloads that you typically stick in HBase (LSM-tree).

rodionos 11y ago

Not sure if your Java comment is warranted, a lot of Apache projects are either written in Java or run on JVM. Yes, HBase's memstore can be optimized, in particular for small key values, but that doesn't negate a need for time-series databases with proven scaling. HDFS offers that. How you were planning to design your tables in PostreSQL if you have 1m+ unique series inserting at 100K+/sec. Assuming you use timestamp as a PK does it translate into 1m+ tables?

dschiptsov 11y ago
We have already seen how Cassandra clone in C++ perform. I think that adding a new indexing strategy to a decent engine could be a good idea. Informix did this.
- kev009 11y ago
  Mapr has an HBase compatible stack in C++, I would love to see TSDB numbers on that. TSDB is pretty efficient itself and since metrics collection is such a batchy operation JVM isn't really that much of a problem. You can run some dedicated TSDs for front end/UIs to ameliorate JVM pain there too.
- rodionos 11y ago
  Do you mean Informix Time-Series Extension? It's been around for a while so maybe there are existing things that came out of IBM recently. Last time I checked its schema it was basically storing serie as a blob not unlike OPC historians like Pi.

krenoten 11y ago

It depends how much time you have to burn, and how much traffic you handle. If you have trivial scaling needs or want to spend 10x the human time managing that setup, then go ahead. You use something like opentsdb when you have hundreds of thousands or millions of ops per second and don't have the time or desire to deal with a database that you have to build your own partitioning + resharding + failover + recovery systems on top of. B+ trees are not optimized for the kinds of write-heavy workloads that you typically stick in HBase (LSM-tree).