mfreed

22 ms·

151.
by mfreed 6y ago
Here is a community post on storing logs in TimescaleDB: https://www.komu.engineer/blogs/timescaledb/timescaledb-for-...
152.
by mfreed 6y ago
One thing to recognize that the lowest-level storage guts of TimescaleDB is Postgres, which really provides a super-stable, reliable foundation. This obviously doesn't avoid all distributed bugs, but it's a huge benefit. It'
153.
by mfreed 6y ago
Hi @malisper, we totally appreciate concerns around potential uncertainty what a "Value Added Service" means. In fact, when we were looking at Timescale licensing, we took careful look at what a lot of other like company licenses
154.
by mfreed 6y ago
An important clarification is that Azure, Digital Ocean, Rackspace (Object Rocket), Alibaba Cloud -- which all support managed TimescaleDB today -- only offer the Apache-2 version of TimescaleDB. Many of the more advanced features of Timesc
155.
by mfreed 6y ago
Or that there was a huge development effort going in a private branch that had PRs and code review for each of the 226 commits :) https://github.com/timescale/timescaledb/pull/1923/commits
156.
by mfreed 6y ago
(Timescale person here) Just an observation that the parent post was the unordered "set" nature of tables, and not strictly about ASOF joins. I should point out there is an alternative reason for this in TimescaleDB (and many othe
157.
by mfreed 7y ago
(Timescale person here.) Many of our users store many billions of rows + 10s of millions per day. Native compression will give you significant storage savings, and very flexible options for indexing to support various queries. And if you
158.
by mfreed 7y ago
TimescaleDB mostly does exactly that: Recent data/partitions are maintained in row-based form, and after they reach some age, they are converted into more columnar compressed form.
159.
by mfreed 7y ago
You've probably seen this, but reposting: https://blog.timescale.com/blog/building-columnar-compressio... Our own experience was that most users were getting 3-6x compression running TimescaleDB with zfs. With our
160.
by mfreed 7y ago
Hi @willvarfar: Just to avoid any confusion, TimescaleDB's compression is available in our community edition, which is fully free to use. Once you specify some compression configs, it's automated and transparent. https:/&#
161.
by mfreed 7y ago
Hi @georgewfraser: We understand this point, and you aren't the first to mention it on HN =) But our users, and the users of most so-called "time series databases", do not typically generate the classic scan-oriented workload
162.
by mfreed 7y ago
Continuous aggregates were first introduced in v1.3. Not identical to PipelineDB features, but address some of the very same use cases. We have some cool new features for continuous aggs planned too. Tutorial: https://docs.times
163.
by mfreed 7y ago
PipelineDB was useful primarily if you primarily wanted to maintain a _continuous_ materialization of some aggregate (e.g., an approximation of the distinct items in the database), not necessarily an aggregate per time period. If you are in
164.
by mfreed 7y ago
Gorilla is one of the few simple, efficent, _lossless_ algorithms that compress binary data. Tricks such as renormalizing floats in base-10 (which I believe VictoriaMetrics implements) while great when they work, potentially truncates lower
165.
by mfreed 7y ago
No, but we always love user stories! Shoot me email at mike (at) timescale.com
166.
by mfreed 7y ago
It's also the case that, at least for Postgres, building a new low-level storage format isn't feasible until PG13. And also this approach allows us to leverage alot of the battle-tested stability built into current storage layout
167.
by mfreed 7y ago
It may also be that we're operating in a slightly more delayed fashion (partially based on chunk boundaries), so we can organize across a lot larger range. For example, if you choose to segment by a device_id, it might scan 1M rows to
168.
by mfreed 7y ago
One guess from a super quick scan can be that we use type-specific compression algorithms. So if your table has one column of timestamps, another of floats, another int, another string, the database employs different compression algorithms
169.
by mfreed 7y ago
Hi @pello, article author here. One of the interesting things of our technique is that it doesn't require low-level changes to Postgres, and actually then works with any version of PG that TimescaleDB supports (currently PG10, 11...PG1
170.
by mfreed 7y ago
Not my specialty, but website shows two versions available for download (at least for free non-commercial use): 64-bit "on-demand" or 32-bit. Both have a similar no benchmarking clause. https://kx.com/connect-wit
171.
by mfreed 7y ago
Open-source Time-Series Benchmarking Suite: https://github.com/timescale/tsbs InfluxDB: https://blog.timescale.com/blog/what-is-high-cardinality-how... https://blog.timescale.com&#
172.
by mfreed 7y ago
It's not hierarchical as you describe. You don't first partition on device, then on time. They are done simultaneously -- see this older post for an illustration/comparison: https://blog.timescale.com/blog&
173.
by mfreed 7y ago
To my understanding, KDB+'s license explicitly forbids benchmarking: "1.3 Kdb+ On Demand Software Performance. End User shall not distribute or otherwise make available to any third party any report regarding the performance of th
174.
by mfreed 7y ago
Hi @benwilson-512: We commonly see hypertables (time-series tables) deployed alongside relational tables, often because there exists a relation between them: the relational metadata provides information about the user, sensor, server, secu
175.
by mfreed 7y ago
Hi @dima_vm, we've found that users have really embraced the full SQL and reliability you get from TimescaleDB's approach leveraging PostgreSQL. But we're aware that its standard on-disk format can be more space intensive th
176.
by mfreed 7y ago
Hi @jnordwick: I talk about this in another response and in the parent article. Individual time intervals are also spread across the cluster. So if you are collecting data from a lot of, say, sensors, servers, or financial instruments, th
177.
by mfreed 7y ago
@dominotw: See the parent article =)
178.
by mfreed 7y ago
Hi Peter, as the blog post talks about, our distributed hypertables typically partition by both time _and_ "space" (i.e., some other column like device id, etc.) as a way to better parallelize I/O (reads & writes) for the
179.
by mfreed 7y ago
I'll let the community talk more about their own experiences, but we built some easy tools to enable this migration: Outflux (snapshot migration): https://www.outfluxdata.com/ Telegraph (streaming migration): https:&#
180.
by mfreed 7y ago
Hey @manigandham thanks for the complements on database overall =) I understand conceptually that this is all about splitting data, but I think if you look at most scalable databases that use sharding, it’s really meant as a partitioning of

More ›