3 ms·

What do you plan on doing for benchmarking? I don't expect you to get something like STAC done, but will you try to find general benchmarks that others use? I'

What do you plan on doing for benchmarking? I don't expect you to get something like STAC done, but will you try to find general benchmarks that others use? I've been working with TS databases for a long time now, and it never fails that every database vendor always has benchmarks showing they are the best (no put that more bluntly, when you come out with your own benchmark suite and you are the fastest/smallest/bestest I won't be surprised or believe you). I don't expect you to be the fastest when having a row-oriented architecture, and it would be an unfair comparison against the non-free databases, but I would like realistic numbers. Actually, if you came out 2nd to 3rd against competitors' workloads, I would be far more impresses. (But really? A row-oriented TS database?)

RobAtticus 10y ago
Influx has a framework[1] we are looking at utilizing, while adding other queries that they do not support (e.g., more complex predicates, non-time-based ordering, JOINs). We hope this will give a more accurate set of benchmarks for time-series workloads. Definitely agree that sometimes its hard to take numbers from a company at face value. As you said, we don't necessarily expect to show that we are the best at all types of queries (and will include query patterns where we may not be the best), but for those who need more SQL features and richer queries we expect Timescale to perform well. Additionally, working with people's existing stack and database is another key win. [1] https://github.com/influxdata/influxdb-comparisons https://github.com/influxdata/influxdb-comparisons
- jnordwick 10y ago
  That benchmark looks like its made to hide inefficiencies, if I understand it correctly. Like why is JSON serialization time not included? If you made the interface JSON over HTTP, you should have to live with the performance implications. I notice they only compare against free versions. I wonder if they have tried against any of the commercial vendors? The comercial vendors right now tend to lead the FSS pack by a very large gap.
  - mfreed 10y ago
    We've been using this framework to avoid the appearance of crafting our own that cherrypicks only our strong suits. Don't disagree with comments that you should account for JSON serialization cost in your benchmarks, but that's a criticism of the suite, not Postgres/Timescale/our numbers. To be clear: our tests are all using the standard PostgreSQL interface, not JSON-over-HTTP. And we don't exclude any SQL parsing time in the DB.
    - jnordwick 10y ago
      I tried to read it a little more, that it is incredibly confusing to me. Did you have issues with that too? I have access to a couple commercial products, but just trying to find the basics to run are difficult. STAC was so much easier to understand. Give dataset, X, how fast does query Y return.

RobAtticus 10y ago

Influx has a framework[1] we are looking at utilizing, while adding other queries that they do not support (e.g., more complex predicates, non-time-based ordering, JOINs). We hope this will give a more accurate set of benchmarks for time-series workloads. Definitely agree that sometimes its hard to take numbers from a company at face value. As you said, we don't necessarily expect to show that we are the best at all types of queries (and will include query patterns where we may not be the best), but for those who need more SQL features and richer queries we expect Timescale to perform well. Additionally, working with people's existing stack and database is another key win. [1] https://github.com/influxdata/influxdb-comparisons https://github.com/influxdata/influxdb-comparisons

jnordwick 10y ago
That benchmark looks like its made to hide inefficiencies, if I understand it correctly. Like why is JSON serialization time not included? If you made the interface JSON over HTTP, you should have to live with the performance implications. I notice they only compare against free versions. I wonder if they have tried against any of the commercial vendors? The comercial vendors right now tend to lead the FSS pack by a very large gap.
- mfreed 10y ago
  We've been using this framework to avoid the appearance of crafting our own that cherrypicks only our strong suits. Don't disagree with comments that you should account for JSON serialization cost in your benchmarks, but that's a criticism of the suite, not Postgres/Timescale/our numbers. To be clear: our tests are all using the standard PostgreSQL interface, not JSON-over-HTTP. And we don't exclude any SQL parsing time in the DB.
  - jnordwick 10y ago
    I tried to read it a little more, that it is incredibly confusing to me. Did you have issues with that too? I have access to a couple commercial products, but just trying to find the basics to run are difficult. STAC was so much easier to understand. Give dataset, X, how fast does query Y return.