valyala

35 ms·

421.
by valyala 6y ago
You can run lightweight vmagent [1] on your laptop. The vmagent collects all the metrics from your laptop and then sends them to remote storage when connection to the remote storage is available. It can send data to any remote storage syste
422.
by valyala 6y ago
Take a look at VictoriaMetrics. It supports both pull and push models. It is inspired by Prometheus and it supports PromQL-inspired query language - MetricsQL [0]. [0] https://victoriametrics.github.io/MetricsQL.html
423.
by valyala 6y ago
Could you give more context on Monarch or Circonus features that are missing in Prometheus? BTW, I'm working on VictoriaMetrics - open source monitoring solution that works out of the box. See https://github.com/Victori
424.
by valyala 6y ago
The following time series databases are popular right now: * ClickHouse (this is a general-purpose OLAP database, but it is easy to adapt it to time series workloads) * InfluxDB * TimescaleDB * M3DB * Cortex * VictoriaMetrics The last three
425.
by valyala 6y ago
If you have cardinality issues in InfluxDB, then just substitute InfluxDB with VictoriaMetrics :) [1] [1] https://medium.com/@valyala/insert-benchmarks-with-inch-infl...
426.
by valyala 6y ago
Prometheus is a monitoring system (it collects metrics and evaluates alerts on them), while TimescaleDB is a database. While it is possible to use TimescaleDB as a remote storage for Prometheus, I'd recommend taking a look at other rem
427.
by valyala 6y ago
100MB data per day looks like a tiny number. Our users ingest 100GB data per day at the ingestion rate of 1M data points per second on a single-node setup [1]. [1] https://victoriametrics.github.io/CaseStudies.html
428.
by valyala 6y ago
PromQL from Prometheus has steep learning curve because of lack of good documentation. When you understand PromQL basics, then it is much easier to write typical queries over time series data in PromQL than in any other query languages (SQL
429.
by valyala 6y ago
Try VictoriaMetrics next time. It supports data push via multiple popular data ingestion protocols [1] and it provides Prometheus-compatible API for Grafana [2]. [1] https://victoriametrics.github.io/#how-to-import-time-seri
430.
by valyala 6y ago
InfluxDB may require big amounts of RAM when working with high cardinality data. It works OK with up to a million of time series. Then it starts eating RAM as crazy [1]. How many time series does your InfluxDB setup contain? [1] https:&#x2
431.
by valyala 6y ago
Absolutely! We are working on this.
432.
by valyala 6y ago
> I really wanted a push rather than pull architecture Then try VictoriaMetrics - it supports both pull and push (including Influx line protocol) [1], it works out of the box and it requires lower amounts of CPU and RAM when working with
433.
by valyala 6y ago
If you want the best of both worlds, then try VictoriaMetrics. It is simple to deploy and operate and it is more resource-efficient comparing to InfluxDB. More on this, it supports data ingestion over Influx line protocol [1]. [1] https:&#
434.
by valyala 6y ago
I believe PromQL [1] and MetricsQL [2] are much better suited for typical queries over time series data than SQL, Flux or InfluxQL. [1] https://medium.com/@valyala/promql-tutorial-for-beginners-9a... [2] https:/&
435.
by valyala 6y ago
90M datapoints per minute means 90M/60=1.5M datapoints per second. Such amounts of data may be easily handled by specialized time series databases even in a single-node setup [1]. > What if a developer wants to add a new dimension t
436.
by valyala 6y ago
InfluxDB may have slightly big RAM requirements when working with high number of time series [1]. [1] https://medium.com/@valyala/insert-benchmarks-with-inch-infl...
437.
by valyala 6y ago
Did you try ClickHouse? [1] We were successfully ingesting hundreds of billions of ad serving events per day to it. It is much faster at query speed than any Postgres-based database (for instance, it may scan tens of billions of rows per se
438.
Vmagent – resource-efficient Prometheus metrics collector(victoriametrics.github.io)
3 points by valyala 6y ago|0 comments
439.
by valyala 6y ago
Great demo! I'm curios why the following query doesn't finish in tens of seconds: select sum(passenger_count) from trips where passenger_count=1 The same query is executed in a hundred of milliseconds on ClickHouse runnin
440.
by valyala 6y ago
BTW, there are algorithms with better compression rate for time series data than Gorilla - see https://medium.com/faun/victoriametrics-achieving-better-com...
441.
by valyala 6y ago
Take a look at ClickHouse [1] and VictoriaMetrics [2]. Both solutions share architecture details and are optimized for high performance and low resource usage. They can handle trillons of rows (i.e. more than 10^12 rows) on a single node an
442.
Sismology: Iguana Solutions’ Monitoring System(medium.com)
1 points by valyala 6y ago|0 comments
443.
VictoriaMetrics TSDB gains replication support(github.com)
5 points by valyala 6y ago|0 comments
444.
by valyala 6y ago
Then it would be great posting updated benchmark results on TimescaleDB blog.
445.
by valyala 6y ago
Multi-node TimescaleDB is a great contribution to open source world! BTW, it would be great comparing multi-node TimescaleDB to VictoriaMetrics cluster [1], which is licensed under vanilla Apache2 open source license [2]. [1] https:/&
446.
by valyala 6y ago
There is more cost effective alternative to BigQuery for storing and analyzing big amounts of logs - LogHouse [1], which is built on ClickHouse. [1] https://github.com/flant/loghouse
447.
by valyala 6y ago
If you use PostgreSQL, then it feels natural to add TimescaleDB extension and start storing time series or analytical data there alongside other relational data. If you need effectively storing trillions of rows and performing real-time OLA
448.
by valyala 7y ago
VictoriaMetrics [1] is written entirely in Go. By default it uses canonical zstd library for compression (the library is written in C), but it supports pure Go mode when built with `make victoria-metrics-pure`. In this mode it uses zstd imp
449.
by valyala 7y ago
Did you try VictoriaMetrics for time series data? It easily handles 2 trillion plus rows (20000 times more than TimescaleDB in your setup) on a single node [1]. [1] https://github.com/VictoriaMetrics/VictoriaMetrics&#x2
450.
by valyala 7y ago
Take a look at VictoriaMetrics. It supports OHLC function (rollup_candlestick) via MetticsQL [1], which is backwards compatible with PromQL - query language used in Prometheus. PromQL is optimized for typical queries over time series data [

More ›