15 ms·
Grafana Tempo, a scalable distributed tracing system
- sna1l 6y agoIngestion/storage of the volume is one thing, but what about the additional overhead for every request?
- strofcon 6y agoGood question! Depends on the instrumentation libraries for sure, but in our Go services we've been using Jaeger protocol for our traces, and "sampling" at 100%, with negligible impact to request / response times. We spit out some 30k+ spans per second, FWIW. :-) Edit: Disclaimer, we're not using Tempo.
- sna1l 6y agoThanks for the reply! I think I must be missing something, but it seems like the big difference between Tempo and a traditional tracing system is the storage indexing & database (ES/C* vs object store and index all fields vs key/value lookup by ID). I vaguely remember reading something that latency even from EC2 -> S3 can be around 200-300ms. Wouldn't this cause the overhead to rise? Feel free to point me to any documentation that clear this up!
- netingle 6y agoWrites are batches up and committed asynchronously to s3 - this should add much if any latency to your services.
- number101010 6y agoWe are ingesting 170k+ with Tempo. It is 100% of our read/query path. Disclaimer: I am using Tempo :) (and from Grafana)
- RhodesianHunter 6y agoEven with batching from Tempo, wouldn't that cost many thousands per month in S3 PUT costs alone?
- number101010 6y agoWe batch up traces in a block and write a block at a time. Internally we are currently configured to write 100k traces in one batch.
- atombender 6y agoDoesn't this cause explosive memory usage? What happens if there's some congestion? Is there a circuit breaker to start dumping (discarding) log entries past a certain limit? I was testing Google Pub/Sub's Go client for publishing internal API event data for later ingest to BigQuery, and it turns out Pub/Sub publishing is not that much faster than writing directly to BigQuery. The buffer sizes we'd need to avoid adding latency to our APIs would have to be ridiculously high; the Pub/Sub client buffers and submits batches in the background (its default buffer size is 100MB!). I don't like the idea of having huge buffers that increase with the request rate. Conversely, pushing the data to NATS in recent time without any buffering or batching turned out to be fast enough to not add any latency. You have to be able to receive messages very fast on the consumer side (as NATS will start dropping messages if consumers can't keep up), but you can simply run a few big horizontally autoscaled ingest processes that can sit there ingesting as fast as they can, which never impacts API latency at all.
- marcinzm 6y agoS3 PUTs are $0.005 per 1000. If you're writing twice a second that comes out to $25/month.
- RhodesianHunter 6y agoYeah, but the person I'm responding to is suggesting 170k+ spans per second so how is twice per second relevant?
- SEJeff 6y agoI'd love to see how this compares to Jaeger.
- strofcon 6y agoI'm hopeful that with it effectively being a KV store pointed at a "simple" object store (S3, GCS, etc), it will be dramatically simpler to manage than Jaeger, and much more performant. Backing Jaeger with Elasticsearch or Cassandra was a nightmare. :-|
- gouthamve 6y agoSee some more details around the motivations and history here: https://gouthamve.dev/tempo-a-game-of-trade-offs/ https://gouthamve.dev/tempo-a-game-of-trade-offs/ It will be easier and cheaper to manage compared to Jaeger but doesn't yet have any ability to search inbuilt (which is one of the reasons Jaeger is expensive).
- netingle 6y agoThat’s our motivation and experience over the last 6 months running it internally at Grafana Labs. Much cheaper and easier to operate. Not to bash Jaeger though, its more powerful than tempo in that it allows you to search for traces. Tempo is about integrating with Grafana, Loki and Prometheus for finding traces.
- SEJeff 6y agoSo if you're not running Loki, you're not going to be able to find traces using tempo? We use grafana and prom, but not loki.
- annanay 6y agoSince Tempo is a k/v store that can retrieve traces given a traceID, we need either a metric system that can store traceIDs in exemplars OR any logging framework to log traceIDs that can be copied over to the Tempo Query UI.
- Axsuul 6y agoIs this a good fit for storing every network request we make (so that we can trace all requests made to an API on behalf of a specific customer for debugging) or is Loki better for that?
- number101010 6y agoit depends on your needs. if a single log line is sufficient to log all the information you need to do your debugging then Loki is a great choice. if you need to see the full request as it passed through your system then distributed tracing/Tempo is a great fit!
- RedShift1 6y agoI really wish Grafana would spend more time on fixing the 2.7k bug reports that are open for their core product...
- netingle 6y agoPRs welcome!
- corford 6y agoYou just raised $50 million....
- buro9 6y agoWe also have open roles https://grafana.com/about/careers/#jobs https://grafana.com/about/careers/#jobs
- corford 6y agoNot sure why this is being downvoted... Hiring people to fix things is a lot more reasonable (and a good idea!) than asking for free labour when you have $50 million in the bank.
- nullsense 6y agoBug fixes appreciated
- Sodman 6y agoThere are certainly some real bugs in there, but saying there are 2.7k "bug reports" is a little disingenuous. These are "GitHub Issues", many of which are feature requests, and many more are new users trying out this free open source software for the first time. Many of these "bugs" are from users asking why their setup isn't working, and then failing to produce any kind of technical debug info when asked by the maintainers.
- 6y ago
- cbsmith 6y agoOh dear... this seems like a move in the exact opposite direction of what one would hope for. It's basically shoving in structured data into what is at best semi-structured log data, storing all of it, and then using that for things that a tracing system would be better suited to. I can't help but feel they've learned the wrong lessons from their challenges with tracing.
- annanay 6y agoThe semi structured nature of logs works to the advantage of Tempo, because as developers we have the flexibility to log _anything_, high cardinality values like cust-id, request latency, gobble-de-gook .. the equivalent of span tags. Instead of indexing these as tags, we get advance search features through a powerful query language landing in Loki (LogQLv2).
- cbsmith 6y agoBut the data starts out structured. It becomes semi-structured when you log it. I'm telling you, from first hand experience, this does not end well. There's no reason that your tracing system should not be indexing your tags in an engine that provides advanced search features through a powerful query language.
- pushrax 6y agoI agree, if anything the eventual goal should be to invert it. In applications I work on right now, trace tags contain the richest and best-described request metadata. Tags are indexed differently depending on their cardinality, and there is no cardinality limit. Tempo's implementation seems pragmatic as a short to medium term solution though. Log engines still have a lot more investment and maturity than trace engines. In my work, even though the trace tags contain the best data quality, the tracing system is currently worse at answering a good deal of my questions. It's simply that Splunk has many tools that work well, and the tracing system is behind.
- cbsmith 6y ago
- tikkabhuna 6y agoSlightly off-topic, but how are users of Grafana and other monitoring tools justifying the investment? And I'm not necessarily talking about monetary amount, its also people cost. What features are you looking for and how do you rank them?
- syoc 6y agoI use a telegraf, influxdb, grafana stack. I find the cost in maintainance and initial setup quite low. Telegraf is super easy, just uncomment the things you want it to collect from the config file. Influxdb is just adding the correct users and a database, never touched it since. Grafana can be a time sink if you want to bikeshed your dashboards, but there are a lot of pre-made ones that handle common usecases. It's really nice to have some metrics when for instance a service goes down. It's super easy to spot a OOM situation or other vertical scaling issues.
- tikkabhuna 6y agoThanks. I completely agree with the bikeshedding. We're playing with Prometheus/Grafana + ELK, and being able to visualise all this data, its hard to work out what is useful and what is just fun to play with. I'm now wondering where the line is between necessary monitoring and bikeshedding? I could look at introducing distributed tracing, but will it actually add any meaningful value?
- lawrjone 6y agoWe run Prometheus with 60d retention (2TB), and an Elasticsearch cluster for logs (also 60d, about 30TB cluster) with Kibana on top. Prometheus is by far the cheapest, both for infrastructure spend and for human cost. The only time we spend really working on Prometheus is configuring our service discovery, which is the shipping component you’d have with any monitoring tool. I estimate we spend about 1 developer day a month un Prometheus upkeep. Logging is much more fiddly, and required a large up front investment. Now it’s running though, we get a lot of value from our setup. It costs much more though, think about $10k/month in infrastructure spend and about 3 developer days a month to maintain. Grafana is effortless to run, really dead simple. You can consider this to be ‘free’, both for infra and maintenance. Hope that gives you a sense of this? I think we come out equivalent to managed services for cost when you account for infra and human time, but have far more flexibility in how we use the tools and develop the skills to properly leverage each product along the way.
- sciurus 6y agoHoneycomb also stores traces in S3 and supports searching them via AWS lambda. I wonder if a model like that is useful for tempo, which doesn't seem to support search at all now. https://www.honeycomb.io/blog/secondary-storage-to-just-storage/ https://www.honeycomb.io/blog/secondary-storage-to-just-stor...