7 ms·

At KubeCon Europe a very good chunk of booths were observability stacks. Everyone was claiming they're better than the competitors (with some of the just justif

At KubeCon Europe a very good chunk of booths were observability stacks. Everyone was claiming they're better than the competitors (with some of the just justifying themselves by saying "it's written in Rust). Having dealt with Prometheus (+Thanos) / Grafana / OTEL and other stacks (e.g: custom solution on ClickHouse, Victoria{Metrics,Logs}, Jaeger/Tempo, Loki, ...) and even cloud ones (Google's Monarch rebranded as Prometheus)... what's your selling point? This to me seems like yet another way to re-invent the wheel. If it's just for running locally, okay, fine, but when it comes to production (where the stack really matters) at scale, you end up with lots of tradeoffs and approaches. Why is this one a winning one compared to the overwhelming "competition"? Seems like we're re-inventing the wheel for the 100th time instead of focusing on unifying the efforts in making the existing solutions better. Thankfully we now have OTEL, so at least the interoperability part is somewhat solved (or mitigated)

yard2010 5mo ago
I have tried to self host grafana (loki prom and alloy) as o11y stack for prepbook.app. This is hard. I have a bsc in cs not that it says something. I managed to do it eventually, after some research. It was not plug and play in any way. The docs kept saying this solution is not production ready even. I couldn't find the production guide, only the "forget about self hosting and simply pay for us hosting this". After I deployed it the UX was so abrasive my partner won't even try to go into it to figure out a problem. It was a few months ago. Since then new solutions have arrived and I'm waiting to have the time to migrate. I saw PostHog have a solution but I prefer something I could self host and completely own. I thought how come no one is trying to solve this problem. It looks like it's just a matter of time. With that being said, my experience can be very skewed since prepbook is a passion project running on a VPS with essentially 0 scale. All I care about is the UX of the stack, not scale. Just for context.
- embedding-shape 5mo ago
  FWIW, I have no CS degree and barely attended school at all, and found Grafana + Prometheus + Loki fairly easy to setup, at least compared to what we used to use before those tools were available. Maybe it's because I used NixOS for the setup, but besides learning some new domain-specific things I didn't know since before, I don't recall hitting any particular bumps or roadblocks, I also went the 100% self-hosted route (spread across two hosts at home). What exactly were you struggling with when it came to the setup? Just a ton of new concepts to learn which took time, or something specific to Grafana/Prometheus/Loki?
  - dijit 5mo ago
    "Getting it running" is the easy part. "Getting it ready for production" is a different game. I've fallen on my sword many times by trying to explain that prometheus fails every metric of production ready; in fact Google themselves replaced borgmon (prometheus) for Monarch because the "tiny unreliable time series databases everywhere" was in fact, not the successful and reliable deployment strategy that they had claimed. But, it is very easy to set up. Just don't go looking for failure modes, because they're everywhere and every single one of them is catastrophic.
    - denysvitali 5mo ago
      There are ways to scale Prometheus (look at Thanos), but none of the solutions is really bug free. See this PR for example (https://github.com/prometheus/prometheus/pull/18364 https://github.com/prometheus/prometheus/pull/18364) - this used to impact a production deployment I worked on. Prometheus, Thanos and even OpenTelemetry are full of those kind of problems - but at the same time it's the best we have and we should be grateful they're free and open source. I'd still choose an open source stack (and contribute to it) rather than go for a proprietary solution - we've all seen what happens with DataDog & co. Please don't take my words lightly, I worked with the rest of my team in a large scale observability platform and scalability should not be underestimated - at the same time DataDog / Splunk prices are simply unjustified. It's ironically cheaper to build a team of engineers that will maintain a sane observability stack instead of feeding the monster(s).
      - otterley 5mo ago
        > It's ironically cheaper to build a team of engineers that will maintain a sane observability stack instead of feeding the monster(s). Can you show the math here? This is a very bold claim, and I’m super curious. A shared Google Sheet would work well.
        smuchow1962 5mo ago
        [flagged]
yuppiepuppie 5mo ago
I was thinking this might be a result of the Cheap-money (post covid) era ending and everyone scrambling to reduce their Datadog/Cloud costs. Thinking back on 2023/2024, lots of companies were leaking large amounts of capital to those vendors and I imagine lots of people saw an opportunity for creating leaner and cheaper stacks.
- robertlagrant 5mo ago
  This is my instinct too. I've had the pleasure of using DataDog and the pain of negotiating with their salespeople!
  - gnyadav 5mo ago
    [dead]
  - Ocha 5mo ago
    Yes. Their sales people don’t even negotiate - they just tell you this is price and done. Dunno why they need sales person if prices are non-negotiable
    - SOLAR_FIELDS 5mo ago
      It’s because you’re in a leveraged position. Why would they negotiate when they don’t need to. Tell them thanks and that you’re churning tomorrow and watch the “OH WAIT”’s come flying through the door. The insidious thing about datadog is that it snakes its way into your entire business line and so it’s really hard to extricate yourself from it down the road.
- dusanstanojevic 5mo ago
  No need to guess, I'll tell you the exact story of why I made Traceway! Last Dec I had a customer complaint, took me 2 days to find the issue. I had to pay $800 for Sentry and a bit more for New Relic. The issue was a locking problem that happened only in very very specific cases, erroring in diff places and timing out in others, unfortunately power users were running into it often. I had two systems, no SLO to catch this and they were completely disconnected. Super annoying. Anyhow, I spent a day looking at those and eventually went, screw this, I'm gonna just make this actually work. So I spent a few hours, hooked it up, no auth or anything nice, pulled the traces and found the issue. Turns out it was locking due to a long transaction existing in a scheduled task, it existed for years. The big things for me is it automatically flagging issues, prioritizing them and taking into account: errors, response codes, timing. That's why I'm making it, no venture capital, funded by actual revenue from the start (not paying for Sentry or New Relic anymore). It's really a dev focused tool to help smallish teams find and fix issues before customers even have time to complain. Anyhow, hope that explains it, kinda related to cloud costs, mostly just my personal frustration with existing tools. Also I did NOT want to host a 5 service stack (grafana, otel collector, prometheus, mimir, loki, k8s) for something that can be done in a 60mb go binary that runs on a 3$ server...
ting0 5mo ago
Do you think Prometheus + Grafana is the way to go?
- denysvitali 5mo ago
  Really depends on the use case. Home lab? Probably. Production? As soon as you scale you need a proper solution. Prometheus (by itself) doesn't scale - you need Mimir or Thanos (or similar). Clickhouse (the "clickstack") seems to be the new kid on the block. Looks very promising.
  - parliament32 5mo ago
    Note Clickhouse is quite old (2010ish?) but they've always been a "web server access log analytics" solution. The pivot to "we do observability too" is new, we'll see how that plays out. Not terribly optimistic given how badly a similar pivot went for Elastic, but who knows.
    - NortySpock 5mo ago
      I thought observability was shoved on Clickhouse by other stacks deciding to use Clickhouse as their recommended database for observability (SigNoz springs to mind but they were not the only one)
      - valyala 5mo ago
        VictoriaMetrics CTO here. We at VictoriaMetrics took another approach. We tried using ClickHouse as a database for metrics in 2017, but then decided implementing a specialized database for metrics. This database uses ClickHouse architecture ideas for achieving the best performance and the lowest resource usage. The main difference between ClickHouse and VictoriaMetrics is that VictoriaMetrics is optimized solely for typical observability tasks. It supports all the popular data ingestion protocols, it provides promql-compatible querying API, it provides Graphite-compatible querying API, it provides Prometheus-compatible service discovery and relabeling, it provides Prometheus-compatible alerting and recording rules. It provides built-in web UI for quick exploration and analysis of the ingested metrics, with the ability to investigate the source of high cardinality. It consists of a single small executable (~20MB) without external dependencies with minimum configs and minimum maintenance. See https://altinity.com/wp-content/uploads/2021/11/How-ClickHouse-Inspired-Us-to-Build-a-High-Performance-Time-Series-Database.pdf https://altinity.com/wp-content/uploads/2021/11/How-ClickHou... for more details. We used the same approach for building VictoriaLogs - a specialized database for logs. It uses the most appropriate architecture ideas from ClickHouse for achieving high performance and low resource usage. It is schemaless and zero-config. It contains of a single small executable without external dependencies. It accepts logs via popular data ingestion protocols. It provides a specialized query language for typical queries over production logs - LogsQL. This language is much simpler to use than SQL for querying typical logs. It provides a built-in web UI for quick exploration of the ingested logs. It provides a Grafana plugin for building arbitrary complex dashboards from the stored logs. It provides the ability to build alerts and metrics from the stored logs. See https://docs.victoriametrics.com/victorialogs/faq/#what-is-the-difference-between-victorialogs-and-clickhouse https://docs.victoriametrics.com/victorialogs/faq/#what-is-t...
CyberDildonics 5mo ago
Is "observability stack" the new term for logs and stats?
- denysvitali 5mo ago
  You have more than that nowadays. Tracing and profiling are part of O11y too
NortySpock 5mo ago
If I can ask a separate question: what scalability problems did you run into with Victoria{Metrics|Logs|Traces}, and at what scale did you hit them? VictoriaMetrics and Logs have worked fine in my quiet homelab, and VictoriaMetrics appeared to work great for the infrastructure team of an open source online video game I contribute to (say about 10 physical nodes and 20 applications/services ) ... I was going to suggest VictoriaLogs to them next but wanted to ask what roadblocks could come up.
- dusanstanojevic 5mo ago
  I honestly think you are a bot. When ever I see Victoria mentioned it is always the same, always asking about hitting a scaling problem + promoting it, never responding to any comments. Hope I'm wrong, but it's been one too many. I refuse to use a product that is this dishonest.
  - hagen1778 5mo ago
    I work at VictoriaMetrics. Just to clarify: VictoriaMetrics doesn't use bots for HN or for any other media for promotion. I don't know the person who you responded to. Most of the activity you see is coming from community members who genuinely use the project or from the core engineering team trying to answer user's questions or address misunderstandings. > never responding to any comments Could you please share examples like this? I can't say for community members, but our internal policy for engineers is very much focused on great support. You can check our slack/github to see that every question is answered and well explained.
    - dusanstanojevic 5mo ago
      Hi, first of all, thank you for your response, I really appreciate actual Victoria team commenting. This is probably the 5th comment (almost identical) I have seen about VictoriaMetrics, mostly on Reddit. I engaged with a few trying to learn more about your product and eventually just gave up. If you really want you can comb through my reddit comments, but be warned, I have commented on a lot of things... a lot... You should be proud of what you have built, I've looked a bit more and your product looks incredible. I personally think that a sales person might have been testing an automation tool, but if it was actual customers that just shows how good the product is! Traceway is not working on addressing the problems y'all are solving, it is more focused on having an out of the box experience with preconfigured dashboards, SLOs, integrations, automatic endpoint ranking, frontend session replays/RUM, symbolication etc. Again, thank you for your comment.
parliament32 5mo ago
FWIW we've also tried all sorts of different things, and honestly the very vanilla (prometheus -> central thanos, fluentbit -> central loki, grafana) ends up on top. The resource consumption is surprisingly minimal (for a sense of scale, we run about 200k eps for metrics and 1k eps for logs). For all these solutions, I find myself asking the same question as you.. what problem are you trying to solve? Is there anything actually different about your product other than less stability than the battle-tested stack?
- dusanstanojevic 5mo ago
  Hi, sorry for not responding sooner this one slipped through the cracks. I've tried to explain my reasons for starting traceway and what I've been building with it. It's not aimed to be the fastest ingestion tool out there, but it's backed by clickhouse and does minimal processing of the data, you can expect the perf to be as close to clickhouse as possible. I'm working on a comprehensive benchmark of Traceway performance on different hardware configurations. The most I've tested with was the smallest managed ch instance with 250k traces per sec, handled it without a hiccup (but that's empirical). You can checkout the traceway git, there is an issue I've opened for benchmarking and you can subscribe/comment on it if you're interested. I'm benchmarking across sqlite, self hosted clickhouse and managed clickhouse. I am a huge fan of systematic, realistic and most of all reproducible benchmarks, so I am really excited about the progress on that. Anyhow, you can checkout traceway and see what it offers, it's aimed at providing SLOs out of the box, session replays, alerting, configurable dashboards and great exception tracking (automatic symbolication) etc...
dusanstanojevic 5mo ago
Hi, creator of Traceway here. Sorry for the late response, I didn't know this got posted and then my account was rate limiting on comments. A lot of tools in this space, most pretty good. The goals when I started Traceway were: - simple to host and reason about - cheap to host - comes pre configured for sub 15 dev teams - completely open source, no paid ad-ons It's not aimed at teams that can afford SREs (yet), the idea was to provide a good tool for smaller teams and startups in the sub 15 dev range. The base of Traceway is Clickhouse, nothing special there, if you want you can run it with sqlite for self hosting. Sessions are also stored in S3 so the costs are minimal. It is opinionated, it comes with preconfigured SLOs for flagging issues with endpoints and it will never try to sell you an AI SRE, you can file your exceptions/slo issues with the git integration and run what ever AI you want on it (I was sick of observability tools trying to sell me an AI). The goal is to have a one line setup, for OpenTelemetry, that gets you everything you need in Traceway without anything needing to be additionally configured. It's Datadog/Sentry but combined and fully open sourced. I'm a huge fan of open source, here is what we've done so far for making existing solutions better: 1 - Session Replays/RUM Session replays are usually a premium/expensive feature. With Traceway you can self host them and add them to your app in minutes. I am working on making this a standalone feature that ties into the otel sdks for mobile/js so that you can get your spans/logs/metrics/exceptions from any platform connected to your session replays in Traceway. At one point I got nerd snipped into making it work with Flutter, so we are the only solution I know of that has affordable usable session replays for Flutter. 2 - Symfony Otel Symfony, the php framework, had no library that offered a few line setup and worked out of the box with open telemetry. We wrote one, you can use it with any tool out there. 3 - Symbolicator We're working on a symbolicator that will be Open Telemetry Collector compatible, so that you can get your stack traces for Js/Flutter/Android/iOS resolved back. From what I can tell no good solution exists for this currently. I will make a proper HN post at some point with more info on the project, right now I am focusing on building. If you have any ideas or things you'd like to see feel free to comment, join our discord community or open the issue in our git, we're always happy to accept PRs.

yard2010 5mo ago

I have tried to self host grafana (loki prom and alloy) as o11y stack for prepbook.app. This is hard. I have a bsc in cs not that it says something. I managed to do it eventually, after some research. It was not plug and play in any way. The docs kept saying this solution is not production ready even. I couldn't find the production guide, only the "forget about self hosting and simply pay for us hosting this". After I deployed it the UX was so abrasive my partner won't even try to go into it to figure out a problem. It was a few months ago. Since then new solutions have arrived and I'm waiting to have the time to migrate. I saw PostHog have a solution but I prefer something I could self host and completely own. I thought how come no one is trying to solve this problem. It looks like it's just a matter of time. With that being said, my experience can be very skewed since prepbook is a passion project running on a VPS with essentially 0 scale. All I care about is the UX of the stack, not scale. Just for context.

embedding-shape 5mo ago
FWIW, I have no CS degree and barely attended school at all, and found Grafana + Prometheus + Loki fairly easy to setup, at least compared to what we used to use before those tools were available. Maybe it's because I used NixOS for the setup, but besides learning some new domain-specific things I didn't know since before, I don't recall hitting any particular bumps or roadblocks, I also went the 100% self-hosted route (spread across two hosts at home). What exactly were you struggling with when it came to the setup? Just a ton of new concepts to learn which took time, or something specific to Grafana/Prometheus/Loki?
- dijit 5mo ago
  "Getting it running" is the easy part. "Getting it ready for production" is a different game. I've fallen on my sword many times by trying to explain that prometheus fails every metric of production ready; in fact Google themselves replaced borgmon (prometheus) for Monarch because the "tiny unreliable time series databases everywhere" was in fact, not the successful and reliable deployment strategy that they had claimed. But, it is very easy to set up. Just don't go looking for failure modes, because they're everywhere and every single one of them is catastrophic.
  - denysvitali 5mo ago
    There are ways to scale Prometheus (look at Thanos), but none of the solutions is really bug free. See this PR for example (https://github.com/prometheus/prometheus/pull/18364 https://github.com/prometheus/prometheus/pull/18364) - this used to impact a production deployment I worked on. Prometheus, Thanos and even OpenTelemetry are full of those kind of problems - but at the same time it's the best we have and we should be grateful they're free and open source. I'd still choose an open source stack (and contribute to it) rather than go for a proprietary solution - we've all seen what happens with DataDog & co. Please don't take my words lightly, I worked with the rest of my team in a large scale observability platform and scalability should not be underestimated - at the same time DataDog / Splunk prices are simply unjustified. It's ironically cheaper to build a team of engineers that will maintain a sane observability stack instead of feeding the monster(s).
    - otterley 5mo ago
      > It's ironically cheaper to build a team of engineers that will maintain a sane observability stack instead of feeding the monster(s). Can you show the math here? This is a very bold claim, and I’m super curious. A shared Google Sheet would work well.
      - smuchow1962 5mo ago
        [flagged]

yuppiepuppie 5mo ago

I was thinking this might be a result of the Cheap-money (post covid) era ending and everyone scrambling to reduce their Datadog/Cloud costs. Thinking back on 2023/2024, lots of companies were leaking large amounts of capital to those vendors and I imagine lots of people saw an opportunity for creating leaner and cheaper stacks.

robertlagrant 5mo ago
This is my instinct too. I've had the pleasure of using DataDog and the pain of negotiating with their salespeople!
- gnyadav 5mo ago
  [dead]
- Ocha 5mo ago
  Yes. Their sales people don’t even negotiate - they just tell you this is price and done. Dunno why they need sales person if prices are non-negotiable
  - SOLAR_FIELDS 5mo ago
    It’s because you’re in a leveraged position. Why would they negotiate when they don’t need to. Tell them thanks and that you’re churning tomorrow and watch the “OH WAIT”’s come flying through the door. The insidious thing about datadog is that it snakes its way into your entire business line and so it’s really hard to extricate yourself from it down the road.
dusanstanojevic 5mo ago
No need to guess, I'll tell you the exact story of why I made Traceway! Last Dec I had a customer complaint, took me 2 days to find the issue. I had to pay $800 for Sentry and a bit more for New Relic. The issue was a locking problem that happened only in very very specific cases, erroring in diff places and timing out in others, unfortunately power users were running into it often. I had two systems, no SLO to catch this and they were completely disconnected. Super annoying. Anyhow, I spent a day looking at those and eventually went, screw this, I'm gonna just make this actually work. So I spent a few hours, hooked it up, no auth or anything nice, pulled the traces and found the issue. Turns out it was locking due to a long transaction existing in a scheduled task, it existed for years. The big things for me is it automatically flagging issues, prioritizing them and taking into account: errors, response codes, timing. That's why I'm making it, no venture capital, funded by actual revenue from the start (not paying for Sentry or New Relic anymore). It's really a dev focused tool to help smallish teams find and fix issues before customers even have time to complain. Anyhow, hope that explains it, kinda related to cloud costs, mostly just my personal frustration with existing tools. Also I did NOT want to host a 5 service stack (grafana, otel collector, prometheus, mimir, loki, k8s) for something that can be done in a 60mb go binary that runs on a 3$ server...

ting0 5mo ago

Do you think Prometheus + Grafana is the way to go?

denysvitali 5mo ago
Really depends on the use case. Home lab? Probably. Production? As soon as you scale you need a proper solution. Prometheus (by itself) doesn't scale - you need Mimir or Thanos (or similar). Clickhouse (the "clickstack") seems to be the new kid on the block. Looks very promising.
- parliament32 5mo ago
  Note Clickhouse is quite old (2010ish?) but they've always been a "web server access log analytics" solution. The pivot to "we do observability too" is new, we'll see how that plays out. Not terribly optimistic given how badly a similar pivot went for Elastic, but who knows.
  - NortySpock 5mo ago
    I thought observability was shoved on Clickhouse by other stacks deciding to use Clickhouse as their recommended database for observability (SigNoz springs to mind but they were not the only one)
    - valyala 5mo ago
      VictoriaMetrics CTO here. We at VictoriaMetrics took another approach. We tried using ClickHouse as a database for metrics in 2017, but then decided implementing a specialized database for metrics. This database uses ClickHouse architecture ideas for achieving the best performance and the lowest resource usage. The main difference between ClickHouse and VictoriaMetrics is that VictoriaMetrics is optimized solely for typical observability tasks. It supports all the popular data ingestion protocols, it provides promql-compatible querying API, it provides Graphite-compatible querying API, it provides Prometheus-compatible service discovery and relabeling, it provides Prometheus-compatible alerting and recording rules. It provides built-in web UI for quick exploration and analysis of the ingested metrics, with the ability to investigate the source of high cardinality. It consists of a single small executable (~20MB) without external dependencies with minimum configs and minimum maintenance. See https://altinity.com/wp-content/uploads/2021/11/How-ClickHouse-Inspired-Us-to-Build-a-High-Performance-Time-Series-Database.pdf https://altinity.com/wp-content/uploads/2021/11/How-ClickHou... for more details. We used the same approach for building VictoriaLogs - a specialized database for logs. It uses the most appropriate architecture ideas from ClickHouse for achieving high performance and low resource usage. It is schemaless and zero-config. It contains of a single small executable without external dependencies. It accepts logs via popular data ingestion protocols. It provides a specialized query language for typical queries over production logs - LogsQL. This language is much simpler to use than SQL for querying typical logs. It provides a built-in web UI for quick exploration of the ingested logs. It provides a Grafana plugin for building arbitrary complex dashboards from the stored logs. It provides the ability to build alerts and metrics from the stored logs. See https://docs.victoriametrics.com/victorialogs/faq/#what-is-the-difference-between-victorialogs-and-clickhouse https://docs.victoriametrics.com/victorialogs/faq/#what-is-t...

CyberDildonics 5mo ago

Is "observability stack" the new term for logs and stats?

denysvitali 5mo ago
You have more than that nowadays. Tracing and profiling are part of O11y too

NortySpock 5mo ago

If I can ask a separate question: what scalability problems did you run into with Victoria{Metrics|Logs|Traces}, and at what scale did you hit them? VictoriaMetrics and Logs have worked fine in my quiet homelab, and VictoriaMetrics appeared to work great for the infrastructure team of an open source online video game I contribute to (say about 10 physical nodes and 20 applications/services ) ... I was going to suggest VictoriaLogs to them next but wanted to ask what roadblocks could come up.

dusanstanojevic 5mo ago
I honestly think you are a bot. When ever I see Victoria mentioned it is always the same, always asking about hitting a scaling problem + promoting it, never responding to any comments. Hope I'm wrong, but it's been one too many. I refuse to use a product that is this dishonest.
- hagen1778 5mo ago
  I work at VictoriaMetrics. Just to clarify: VictoriaMetrics doesn't use bots for HN or for any other media for promotion. I don't know the person who you responded to. Most of the activity you see is coming from community members who genuinely use the project or from the core engineering team trying to answer user's questions or address misunderstandings. > never responding to any comments Could you please share examples like this? I can't say for community members, but our internal policy for engineers is very much focused on great support. You can check our slack/github to see that every question is answered and well explained.
  - dusanstanojevic 5mo ago
    Hi, first of all, thank you for your response, I really appreciate actual Victoria team commenting. This is probably the 5th comment (almost identical) I have seen about VictoriaMetrics, mostly on Reddit. I engaged with a few trying to learn more about your product and eventually just gave up. If you really want you can comb through my reddit comments, but be warned, I have commented on a lot of things... a lot... You should be proud of what you have built, I've looked a bit more and your product looks incredible. I personally think that a sales person might have been testing an automation tool, but if it was actual customers that just shows how good the product is! Traceway is not working on addressing the problems y'all are solving, it is more focused on having an out of the box experience with preconfigured dashboards, SLOs, integrations, automatic endpoint ranking, frontend session replays/RUM, symbolication etc. Again, thank you for your comment.

parliament32 5mo ago

FWIW we've also tried all sorts of different things, and honestly the very vanilla (prometheus -> central thanos, fluentbit -> central loki, grafana) ends up on top. The resource consumption is surprisingly minimal (for a sense of scale, we run about 200k eps for metrics and 1k eps for logs). For all these solutions, I find myself asking the same question as you.. what problem are you trying to solve? Is there anything actually different about your product other than less stability than the battle-tested stack?

dusanstanojevic 5mo ago
Hi, sorry for not responding sooner this one slipped through the cracks. I've tried to explain my reasons for starting traceway and what I've been building with it. It's not aimed to be the fastest ingestion tool out there, but it's backed by clickhouse and does minimal processing of the data, you can expect the perf to be as close to clickhouse as possible. I'm working on a comprehensive benchmark of Traceway performance on different hardware configurations. The most I've tested with was the smallest managed ch instance with 250k traces per sec, handled it without a hiccup (but that's empirical). You can checkout the traceway git, there is an issue I've opened for benchmarking and you can subscribe/comment on it if you're interested. I'm benchmarking across sqlite, self hosted clickhouse and managed clickhouse. I am a huge fan of systematic, realistic and most of all reproducible benchmarks, so I am really excited about the progress on that. Anyhow, you can checkout traceway and see what it offers, it's aimed at providing SLOs out of the box, session replays, alerting, configurable dashboards and great exception tracking (automatic symbolication) etc...

dusanstanojevic 5mo ago

Hi, creator of Traceway here. Sorry for the late response, I didn't know this got posted and then my account was rate limiting on comments. A lot of tools in this space, most pretty good. The goals when I started Traceway were: - simple to host and reason about - cheap to host - comes pre configured for sub 15 dev teams - completely open source, no paid ad-ons It's not aimed at teams that can afford SREs (yet), the idea was to provide a good tool for smaller teams and startups in the sub 15 dev range. The base of Traceway is Clickhouse, nothing special there, if you want you can run it with sqlite for self hosting. Sessions are also stored in S3 so the costs are minimal. It is opinionated, it comes with preconfigured SLOs for flagging issues with endpoints and it will never try to sell you an AI SRE, you can file your exceptions/slo issues with the git integration and run what ever AI you want on it (I was sick of observability tools trying to sell me an AI). The goal is to have a one line setup, for OpenTelemetry, that gets you everything you need in Traceway without anything needing to be additionally configured. It's Datadog/Sentry but combined and fully open sourced. I'm a huge fan of open source, here is what we've done so far for making existing solutions better: 1 - Session Replays/RUM Session replays are usually a premium/expensive feature. With Traceway you can self host them and add them to your app in minutes. I am working on making this a standalone feature that ties into the otel sdks for mobile/js so that you can get your spans/logs/metrics/exceptions from any platform connected to your session replays in Traceway. At one point I got nerd snipped into making it work with Flutter, so we are the only solution I know of that has affordable usable session replays for Flutter. 2 - Symfony Otel Symfony, the php framework, had no library that offered a few line setup and worked out of the box with open telemetry. We wrote one, you can use it with any tool out there. 3 - Symbolicator We're working on a symbolicator that will be Open Telemetry Collector compatible, so that you can get your stack traces for Js/Flutter/Android/iOS resolved back. From what I can tell no good solution exists for this currently. I will make a proper HN post at some point with more info on the project, right now I am focusing on building. If you have any ideas or things you'd like to see feel free to comment, join our discord community or open the issue in our git, we're always happy to accept PRs.