9 ms·
CERN swaps out databases to feed its petabyte-a-day habit
- deleted 3y ago[deleted]
- esafak 3y agotl,dr: Speaking to The Register, Roman Khavronenko, co-founder of VictoriaMetrics, said the previous system had experienced problems with high cardinality, which refers to the level of repeated values – and high churn data – where applications can be redeployed multiple times over new instances. Implementing VictoriaMetrics as backend storage for Prometheus, the CMS monitoring team progressed to using the solution as front-end storage to replace InfluxDB and Prometheus, helping remove cardinality issues, the company said in a statement.
- m3kw9 3y agoOver 24hr period its more then 11 Gigabytes/second or rounding to 100 gbps. Those shards must be pretty crazy
- formerly_proven 3y agoThe headline is about the data processed on their compute, the amount of data in the monitoring system is considerably smaller (but still not small data): > But Brij Kishor Jashal, a scientist in the CMS collaboration, told The Register that his team were currently aggregating 30 terabytes over a 30-day period to monitor their computing infrastructure performance. So 1 TB / day, that's about 10 MB/s.
- Havoc 3y agoThat’s one hell of an endorsement. Marketing team won the jackpot.
- keep_reading 3y agoI also dropped InfluxDB at work due to its terrible performance. VictoriaMetrics is great I was using Promscale (TimescaleDB) but they EOL'd Promscale which forced us to Victoria. But either way both of these are much faster than Influx Don't get fooled into the latest InfluxDB rewrite. I think the latest is cloud hosted only too? So stupid
- pphysch 3y agoI saw the writing on the wall with InfluxDB v2 (doubling down on closed platform / SaaS) and advocated exploring VictoriaMetrics, even though we had some Influx v1 running. No regrets. I also prefer the golang-esque simplicity of the Prometheus ecosystem. Monitoring is the last place I want unnecessary abstraction layers and complicated configuration files.
- contravariant 3y agoHonestly the database isn't half as useful as the tool they wrote to grab the metrics. At least I think telegraf was written by the same people? It seems to have the exact opposite design philosophy.
- heipei 3y agoYup, telegraf is what keeps me with InfluxDB, even if I couldn't care less about the database or the pain in the ass it is to operate.
- contravariant 3y agoVictoriaMetrics does claim to be compatible with the inline format, which makes me wonder if it is possible to keep using telegraf with it.
- keep_reading 3y agoYou can. I also use Telegraf with TimescaleDB. Telegraf actually made interop with their competitors super easy
- somedude82 3y agoInfluxData is killing InfluxDB with their changes. Their v1->v2->v3 changes are beyong insane. They revealed that flux is deprecated in v3, their main selling point for v2. In database domain you would like stability not breaking change in every 2-3 years. See https://news.ycombinator.com/item?id=37206194 https://news.ycombinator.com/item?id=37206194
- amelius 3y agoThis is nothing compared to what dragnet surveillance has to deal with.
- local_crmdgeon 3y agoAnd that's all on MSSQL or RDS, right?
- qwertox 3y agoAt the end of the article it says "InfluxDB said in March this year it had solved the cardinality issue with a new IOx storage engine." Does this mean that in the end it wasn't really necessary to switch to VictoriaMetrics' offering?
- foobiekr 3y agoisn't this like their fourth storage engine? doesn't that get old after awhile?
- heliodor 3y agoOne half-baked thing after another!
- ericb 3y agoThat should read: InfluxDB said in March this year it had solved the cardinality issue with a new IOx storage engine for it's hosting customers only There isn't an influxdb that you can download with cardinality solved.
- coder543 3y agoAs of today, they announced a plan for open sourcing a version of v3, but they make it clear in the article that it won't be a great fit for every existing use case. https://www.influxdata.com/blog/the-plan-for-influxdb-3-0-open-source/ https://www.influxdata.com/blog/the-plan-for-influxdb-3-0-op...
- ilyt 3y agoI really like VictoriaMetrics's architecture vmagent takes care of all the pesky edge things like emulating prometheus config parsing and various scraping bits. It also does buffering in case you lose network connection for a while, and accept vast spread of different protocols vminsert/vmselect scale separately from eachother and your queries don't bother your ingest all that much. vmstorage does just that, storage. Only thing that bothers me (compared to say, Elasticsearch), is that data can't migrate between nodes so you can't "just" start a new one and drain an old one, but a tiny bit ops work in rare cases is IMO price worth paying for straightforwardness of the stack.. PromQL compatibility is also great, tools like Grafana "just work" without anyone having to write support for it. We started migrating from InfluxDB at work, and on my private stuff I already did. Soo much less memory usage too.
- theossuary 3y agoWhat version of Influx were you running? I'm interested if v3 will be more competitive than v2.
- ilyt 3y ago1.8, migration path to 2.0 was a no-no. Don't remember exact reasons back then but we decided to have wait-and-see approach and see how alternatives grow up as our data generally grows in predictable rate Also frankly Prometheus support is a massive positive. For better or worse industry standarized on apps using Prometheus as ingest for metrics, and also most of the materials related to that will of course give examples in PromQL Flux is frankly hieroglyphs for people using it 20 minutes a month like our developers This is given example on how to raise value in Flux to power of two |> map(fn: (r) => ({ r with _value: r._value * r._value })) This is example of that in prometheus value ^ 2 This is example of calculating percentage in Flux (from their webpage) data |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") |> map( fn: (r) => ({ _time: r._time, _field: "used_percent", _value: float(v: r.used) / float(v: r.total) * 100.0, }), ) This is how you do it in PromQL space_used / space_total * 100 Flux is atrocious for "normal users".
- bouvin 3y agoOne of my fondest memories as a summer student at CERN in 1993 (in the Electronics and Computing for Physics department) was the visit to the basement beneath the main computing facility, where a colossal tape robot was in operation. Even at that time, CERN was grappling with exceedingly vast amounts of data.
- jjtheblunt 3y agoI saw the same at FermiLab as a high school kid in 1985. Those things were neat.
- lmihaig 3y agoI was a summer student this summer, working right next to that exact building. Unfortunately we did not get to see the tape robot, we did get visits to the experiments though. I can say from what I've seen that the amount of data they have to deal with is still the #1 problem. Before the multi layered filters they generate a petabyte of data per second.
- ComputerGuru 3y agoMissing from the title: leaving InfluxDB and Prometheus for VictoriaMetrics.
- hintymad 3y agoThis is puzzling. I'm not sure how VictoriaMetrics solved the cardinality problem? When running an aggregate query that sums up some counters for a single metric over the dimension of instances in a time window of larger than a few hours, VictoriaMetrics would barf with error for the querying having too many time series (or data points? I forgot the exact wording). This clearly shows that 1/ Victoria Metrics does not treat a time series with multiple dimensions as a single time series; 2/ VictoriaMetrics does not perform hierarchical aggregation. That is, VictoriaMetrics has not really built a true time series DB that handles reasonable cardinalities.
- deleted 3y ago[deleted]
- narism 3y agoThere are a number of circuit breakers in VictoriaMetrics that limit the number of time series/datapoints in queries to limit CPU/RAM usage. These can be tweaked with the -search.max* command line flags. I think the default query limits for data points per series are 30M and number of series is 300K. What do you consider reasonable cardinalities or a true TSDB?
- hintymad 3y agoIn my particular example, the cardinality of the instances (i.e., the number of of unit instance count in a query's time range) should not even matter. I was summing a counter over all the instances, and VictoriaMetrics should just add up the counters for every time unit while scanning all the data points -- this is implemented by pretty much all the OLAP engines. Or put it another way, logically I was query over a single time series. It's just that each value in the time series was associated with multiple values. Given such logical model, a good time series database should not not even bother me with any concern of cardinality.
- foota 3y agoWeird that the title talks about the petabyte a day, while the article is actually about their monitoring tooling, not the thing ingesting the data from experiments, iiuc.
- iFire 3y agoOPENSOURCE, APACHE2 LICENSE https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/LICENSE https://github.com/VictoriaMetrics/VictoriaMetrics/blob/mast...
- deleted 3y ago[deleted]
- sgt101 3y agoI can do this on my laptop /tumbleweed...
- jmakov 3y agoHow come they don't support wire protocols for analytical workloads like arrow streaming or others like Clickhouse. Looks like they don't want to compete with CK.
- zaps 3y agoJust use sqlite amirite
- faeriechangling 3y agoI’m amazed years after I heard about it how the tiny VictoriaMetrics team is thumping what far bigger organizations manage to do. The biggest reason I’ve heard people don’t want to adopt it is if the lead maintainer gets bussed the project is liable to fall apart. I looked at some of the alternatives to victoriametrics for Prometheus and they all seem… much much worse…
- fijiaarone 3y agoAmazing how much data you can generate with a small cluster piping out /dev/urandom continuously over every possible socket.
- eclark 3y agoWe've been using and building with VictoriaMetrics for a while at Batteries Included. I have probably created and torn down 100+ clusters now. It's a remarkably easy to use piece of software for something with the capabilities.
- inv2004 3y agoDo not have very positive experience with influxdb, but the strange for me that clickhouse was not even mentioned in the article
- __turbobrew__ 3y agoI’m surprised nobody has mentioned grafana mimir yet. You get all the niceties of the prometheus ecosystem with a backend which can scale into the billions of metric streams.
- valyala 3y agoGrafana Mimir is a great product, but it also has some rough edges - https://victoriametrics.com/blog/mimir-benchmark/ https://victoriametrics.com/blog/mimir-benchmark/