4 ms·
You'd be surprised how many serious tech shops have close to zero performance metrics collected and utilised. I've done this in fintech a few times already and
by bitcharmer 6y ago
You'd be surprised how many serious tech shops have close to zero performance metrics collected and utilised.
I've done this in fintech a few times already and the best stack that worked from my experience was telegraf + influxdb + grafana.
There are many things you can get wrong with metrics collection (what you collect, how, units, how the data is stored, aggregated and eventually presented) and I learned about most of them the hard way.
However, when done right and covering all layers of your software execution stack this can be a game changer, both in terms of capacity planing/picking low hanging perf fruit and day to day operations.
Highly recommend giving it a try as the tools are free, mature and cover a wide spectrum of platforms and services.
- chrisweekly 6y agoAgreed; same exp (w many years of web perf optimization as ui arch and/or perf strategy consulting gigs. Even after all these years, simply using WebPageTest to capture test runs and share the results is often a transformational experience.
- benraskin92 6y agoMight also want to give Prometheus a try as it's extremely simple to setup, and it is very well supported in the open source community.
- bitcharmer 6y agoPrometheus was another option but millisecond-precise timestamps are a deal breaker in my field.
- amenod 6y agoCurious, would microseconds suffice? Or are we talking about higher precision still?
- simcop2387 6y agoVery likely they need nanosecond precision, https://www.thetradenews.com/fintech-firms-reduce-trading-time-to-120-nanoseconds/ https://www.thetradenews.com/fintech-firms-reduce-trading-ti...
- bitcharmer 6y agoJust like the sibling comment indicated, we (HFT, algo trading) need nanosecond precision for some metrics.
- benraskin92 6y agoThat makes sense -- have you taken a look at M3DB (https://www.m3db.io/ https://www.m3db.io/)?
- contravariant 6y agoNote that this isn't an either/or. You can use telegraf to push data to prometheus, or pull data from prometheus compliant http endpoints. You can also use grafana to show data from both InfluxDB and Prometheus. This might be obvious to some but I thought it worth mentioning.
- duckmysick 6y ago> There are many things you can get wrong with metrics collection Can you share the important things that are overlooked?
- chucky_z 6y agoI'd like to followup on this and say InfluxDB is fantastic -- with the caveat that you know exactly what you want, and use it with that in mind. It's great at storing some metrics with high volume, but I've found the OSS version falls over hard in some circumstances (high cardinality in tags absolutely murders performance, making it nearly unusable). I'm in the middle of doing a test-run of vmagent + victoria-metrics as a mostly-replacement (see: not replacing the cases where it's known exactly what's wanted, and used explicitly for that), and victoria-metrics fully support telegraf pushes which is a big bonus. Also they're apparently going to be dropping InfluxQL entirely, for Flux, which is weird to me? The docs don't outright state this, but InfluxDB 2.0 has only Flux examples in the docs I can find. It's a better language, however the learning curve is not trivial.
- sciurus 6y agoThankfully they still support influxql via transpiling it to flux. They've talked about supporting promql eventually as well. Tickscript is gone in 2.0, though. I'm not in love with Flux, but I certainly won't miss tickscript.
- chucky_z 6y agoOh, interesting. What's the core difference between Tickscript and Flux? Was Tickscript what Kapacitor used for InfluxDB 1.7?
- sciurus 6y ago> Was Tickscript what Kapacitor used for InfluxDB 1.7? Yes. The learning curve was steep, and debugging was near-impossible. Flux is rough too, but it's already getting better documentation and tooling support than Tickscript had.