5 ms·

Is there yet any way of having a front end for this which doesn't significantly dent your revenue stream either in staffing, infrastructure or license fees? We

• number101010 3y ago
  Most observability vendors support OTEL at this point. To plug the OSS project I work on that supports OTEL ingestion: https://github.com/grafana/tempo/ https://github.com/grafana/tempo/
  • baz00 3y ago
    Thanks I had no idea Tempo even existed despite using Grafana. Will read into it.
    • candiddevmike 3y ago
      IMO, Jaeger is easier to setup/manage and has a better interface than Grafana/Tempo. It's easy to add Jaeger to your local dev stack so you can have tracing while developing. FWIW I now use Tempo because I have everything else in Grafana (Prometheus, Loki), but I do miss using Jaeger.
      • number101010 3y ago
        > It's easy to add Jaeger to your local dev stack so you can have tracing while developing. Tempo can be spun up with docker compose using a local disk for ephemeral storage/querying: https://github.com/grafana/tempo/blob/main/example/docker-compose/local/docker-compose.yaml https://github.com/grafana/tempo/blob/main/example/docker-co... Maybe this meets your needs? > Jaeger is easier to setup/manage and has a better interface than Grafana/Tempo What do you enjoy about the Jaeger interface? Perhaps it's a gap in Tempo we can improve.
      • sofixa 3y ago
        Jaeger can use multiple backends for storage, including Tempo, so it's not an either/or situation. I'm fairly sure there was an official Grafana-provided Jaeger gRPC plugin for Tempo, but can't easily find it, only this one: https://github.com/flitnetics/jaeger-tempo https://github.com/flitnetics/jaeger-tempo
• secondcoming 3y ago
  Why is this downvoted? This is a pain point for us too, except we're at 500k requests/second. We're currently Datadog but everyone knows they're too expensive.
  • phillipcarter 3y ago
    Cost management via sampling is still largely a vendor concern. Each vendor has a different solution, and while some are better than others, all can be effective at bringing costs down.
  • lukaszwojtow 3y ago
    Try coralogix.
  • spockz 3y ago
    We tracé everything and first collects all spans in their trace. Then we sample the successful traces and we retain all traces ending in a failed state. This greatly reduces the need for storage.
• candiddevmike 3y ago
  I would ask this a different way--is there a single OSS project that handles collecting all of the OTEL metrics/logs/traces? Folks keep saying you can't do this in data store/format, Elastic seems to, and having to manage separate storage/infrastructure for all of these tools is intense. Could Prometheus be augmented to store metrics, logs, and traces somehow? I don't really mind if it doesn't scale well on a single instance or is highly available, I'll just add more instances and aggregate them.
  • kortex 3y ago
    It's really not that intense. I basically set up my last co's telemetry infrastructure all by myself, using terraform, otel-python, jaeger, and AWS elasticsearch. This TF project does most of the heavy lift. https://github.com/telia-oss/terraform-aws-jaeger https://github.com/telia-oss/terraform-aws-jaeger
    • candiddevmike 3y ago
      Jaeger for tracing, Elasticsearch for logs? What are you using for metrics?
      • mmclean 3y ago
        Likely Prometheus - Jaeger for tracing, Elastic for logs, Prometheus for metrics is a pretty common and effective OSS observability stack
  • CubsFan1060 3y ago
    I've been watching this with a lot of interest: https://signoz.io/ https://signoz.io/
    • candiddevmike 3y ago
      That looks really promising, thank you for sharing it
• rdoherty 3y ago
  Grafana seems to be an option? Handles metrics, logs and traces. I don't know what storage costs look like though if you are self hosting.. https://grafana.com/ https://grafana.com/
• Thaxll 3y ago
  Sampling is the answer. Sample 1% of success and all the errors. Cost is one thing but you would be surprised how heavy observability can be on a service, it's uses a lot of %cpu.
  • deleted 3y ago
    [deleted]
  • kiitos 3y ago
    Sampling is fine at the query layer, but if you sample at the ingest layer -- and therefore drop a majority of your telemetry data outright -- that's a total hack, and I can't see how the resulting system is anything but unsound.
    • BHSPitMonkey 3y ago
      You use sampling rates that are statically significant enough for the system you're observing. You can always make exceptions to the default rate based on other heuristics. What's wrong with that, for the kinds of insights it provides?
      • kiitos 3y ago
        General-purpose observability systems serve two use cases: presenting a high-level summary of system behavior, and allowing operators to inspect telemetry data associated with a specific e.g. request. The former use case is often solved by a specific and narrow kind of observability data, which is metrics. A common tool for that purpose is Prometheus. You certainly can't query Prometheus for individual requests, which is fine, and accepting that invariant allows Prometheus to treat input data as statistical, in the sense that you mean in your comment. But if we're talking about general-purpose telemetry, we're talking about more than just high-level summaries of system behavior, we also need to be able to inspect individual log events, trace spans, etc. If a user made a request an hour ago with reqid 123, I expect to be able to query my telemetry system for reqid 123 and see all of the metadata related to that request. A telemetry system that samples prior to ingest certainly delivers value, but it can only ever solve the first use case, and never the second.
• kortex 3y ago
  Define "significant". At $lastco, we routed traces to Cassandra and stored them in an AWS elasticsearch domain. Jaeger was used to visualize traces. We also wrote some elasticsearch queries to generate some basic reports, eg finding the most sluggish queries. Pretty standard stuff if you follow the OTEL/jaeger tutorials. Traces came on the order of hundreds/second, but we didn't have downsampling turned on, just collected all. Traces were saved 7 days (configurable). Very little actual optimization at the point where I left. I think it cost on the order of dozens to hundreds of dollars a month. There's an environment variable which you can set on your containers which defines how the tracing sampler behaves. It's in the docs. See OTEL_TRACES_SAMPLER https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/ https://opentelemetry.io/docs/specs/otel/configuration/sdk-e...
  • conor- 3y ago
    I think the issue described is when you're storing in a vendor backend, e.g. Data Dog or New Relic. If you're not careful with how much tracing data you ingest you can be looking at significant being 7 figures worth of bills.
• memset 3y ago
  I am building https://scratchdb.com/ https://scratchdb.com/ to address this. It's an HTTP wrapper around Clickhouse, and uses pay-as-you-go pricing. It's also open source and easy to host yourself (single go binary.) Currently have users sending in the 1000 requests per second so definitely have capacity. Typically seeing a 0.1 compression ratio on data before other optimizations. I have it connected to Fly.io here: https://scratchdb.com/blog/fly-logs-to-clickhouse/ https://scratchdb.com/blog/fly-logs-to-clickhouse/ I'd be really grateful to learn more about what you're looking for (how are you even managing logs today?) Even if you end up not using scratchdb it'll help me figure out the next thing to build!
• singron 3y ago
  I recommend sampling traces if you aren't. I've been unimpressed with datadog apm, which has no affordable configuration. We've been running our own Jaeger stack with 0.1% sampling, and it's negligible to run compared to datadog apm. For metrics and logs, sampling isn't so useful, so I don't have a good answer. Datadog has 80% gross margin, so at most 20% of what you pay them is the infra, so you stand to save a lot of money running your own open source stacks if your labor costs would be less than that 80%. With datadog, we have a project every 3 months to reduce usage, so it's not like we aren't constantly babysitting it anyway.
  • phamilton 3y ago
    > I've been unimpressed with datadog apm, which has no affordable configuration FWIW we sample in datadog APM and it works fine to control costs, I'm not sure what issues you hit.
  • pranay01 3y ago
    If you’re looking for open source APM stack which is OpenTelemetry native and you can self host - you can check out SigNoz ( https://github.com/SigNoz/signoz https://github.com/SigNoz/signoz)
  • paulddraper 3y ago
    > datadog apm, which has no affordable configuration In my experience, Datadog's ingestion sampling works pretty well. And there's retention filters you can use to override.
• pondidum 3y ago
  Honeycomb's pricing is pretty reasonable - but looking at your volume, some sampling might help also
  • yourapostasy 3y ago
    If you’re using Honeycomb, how did you find their solution’s developer/DevOps engineer ergonomics, their support, and their overall experience?
    • pondidum 3y ago
      The only thing I can recall being a bit of a stumbling point at the beginning was finding "my" trace from a local environment. I just didn't know where to find it. Once I figured that out I've not had problems. The high-level view and being able to draw a box around something that looks weird on my graph, and honeycomb tells me what's different inside and outside the box is amazing (called bubble up, if you're searching). It's faster than any other tool I've used; usually data is available by the time I've switched to their ui from the curl command to our API. It's mind blowing actually. Other saas and self hosted options I've tried have all been awful in some way or other; honeycomb is a breath of fresh air, and going back to other tools after using honeycomb is painful. I'm not sponsored or working for them btw, they just make one of the few products that I genuinely love using.
• AndreasHae 3y ago
  Super nitpick, but you meant profit, not revenue, right?
• jdwyah 3y ago
  I'm working on giving you more options for saving money on logging with dynamic log levels: https://prefab.cloud/features/log-levels/ https://prefab.cloud/features/log-levels/ The hypothesis being that you can save money by turning things down, but easily turn them back up when you're actively investigating. Or turn the volume up for a targeted sub-segment of your traffic. We've done some exploration into providing the same for APM and the rest of OTEL and I think it's pretty doable. hmu if you want to talk.
• adra 3y ago
  Whats the validity of the logs when they can be boiled down to metrics instead way more concisely (at least with olap-like storage). For traces, as mentioned elsewhere sampling is key. Some systems use % of requests, and other systems cap traces to traces/sec. Logs are just expensive but easy to use, so the best strategy is to lean on filtering to avoid a bunch of pointless repetition which don't add value. TLDR, metrics require more planning but can end up being much cheaper than logs.
  • adra 3y ago
    Oh and the operational overhead of performing full trace recording of 2k/requests a sec are non-trivial. You'll start to feel the overhead of adding extra telemetry.