11 ms·

Show HN: SigNoz – open-source alternative to DataDog, NewRelic

510 points by pranay01 4y ago

We just released a major upgrade to SigNoz with support for Logs management based on ClickHouse. Would love to get any feedback from the community here on what you think and any questions you may have for us Many big companies like Uber and Cloudflare have been shifting to ClickHouse as their main workhorse for Logs management seeing much better performance. for e.g Cloudflare recently shifted from Elastic to ClickHouse and are seeing 8x improvement in memory/cpu resource requirement in ingestion. This is our first release with Logs support and we have added support for: - Filtering logs based on fields - Full text search in logs - Live mode to see logs coming in realtime - Detailed view of logs in table and json format with ability to add filters quickly - Ability to specify interesting fields which will be indexed by default

pranay01 4y ago
We just released a major upgrade to SigNoz with support for Logs management based on ClickHouse. Would love to get any feedback from the community here on what you think and any questions you may have for us Many big companies like Uber and Cloudflare have been shifting to ClickHouse as their main workhorse for Logs management seeing much better performance. for e.g Cloudflare recently shifted from Elastic to ClickHouse and are seeing 8x improvement in memory/cpu resource requirement in ingestion. This is our first release with Logs support and we have added support for: - Filtering logs based on fields - Full text search in logs - Live mode to see logs coming in realtime - Detailed view of logs in table and json format with ability to add filters quickly - Ability to specify interesting fields which will be indexed by default
ketzu 4y ago
Is there any introduction to all the services that are included in the docker-compose file [1]? I wanted to give signoz a try, but the sheer amount of services attached discouraged me, especially as I have to reconfigure them all to work with my setup. (Don't run them directly on the host, instead in a separated network, put the network interface behind traefik, figure out which access they need, provide all the configuration in a nice way without having to clone the full repo just to have the configuration files.) [1] https://github.com/SigNoz/signoz/blob/develop/deploy/docker/clickhouse-setup/docker-compose.yaml https://github.com/SigNoz/signoz/blob/develop/deploy/docker/...
- rad_gruchalski 4y ago
  Not associated with the project but a quick look suggests: a database (clickhouse), alertmanager, query-service, frontend are signoz components, otel* and hotrod are for distributed tracing. Otel stands for open telemetry (https://opentelemetry.io/ https://opentelemetry.io/), hotrod is a tracing demo app from jaeger: https://github.com/jaegertracing/jaeger/tree/main/examples/hotrod https://github.com/jaegertracing/jaeger/tree/main/examples/h.... Without thinking too much about it, I assume that: hotrod is a demo data source pushing traces to the otel collector, which stores data in clickhouse. Frontend fetches data from clickhouse using the query service. Alert manager probably looks at traces coming in and detects anomalies, so that you can get real-time alerts when things don’t look normal.
  - ketzu 4y ago
    Thank you for the quick explanation, that helps a lot and gives me at least a starting point. Although sample services surprise me in the suggested docker-compose from the "how to install" section.
    - pranay01 4y ago
      yeah, it's not ideal - but we kept it to give users who are installing for the first time to get a hang of the product You can easily remove the sample services if you want - https://signoz.io/docs/operate/docker-standalone/#remove-the-sample-application https://signoz.io/docs/operate/docker-standalone/#remove-the...
- pranay01 4y ago
  The comment by rad_gruchalski is mostly accurate SigNoz has the following components - Frontend - ClickHouse (datastore) - Alert Manager (this is to monitor metrics and create alerts which you configure in SigNoz) - Query Service (which is the backend service which talks to datastore & frontend) - Otel collector ( which the collector provided by opentelemetry to collect telemetry data) The other two components for sample app which can be commented out - Hotrod ( which is a sample app) - load generator More details here - https://signoz.io/docs/architecture/#architecture-components https://signoz.io/docs/architecture/#architecture-components
reilly3000 4y ago
I just learned about SigNoz. I spend a lot of time flipping between New Relic and Splunk, and my company spends a ton on both. I’m interested to hear from users and learn if it’s prod ready as a New Relic alternative. I’m especially interested in something that could run locally so I can profile using the same tools that run in prod. Any feedback?
- makeavish 4y ago
  Here are few case studies of production users published by SigNoz: https://signoz.io/case-study/ https://signoz.io/case-study/
jakswa 4y ago
Goodness I hope this is good because I will make my company gobble it upppp if so.
- pranay01 4y ago
  feel free to test it out. We have an active slack community as well if you have any questions on how to setup etc. https://signoz.io/slack https://signoz.io/slack
- saintfiends 4y ago
  We thought so too until we found out it doesn't support any kind of SSO https://github.com/SigNoz/signoz/issues/1188 https://github.com/SigNoz/signoz/issues/1188
  - bogota 4y ago
    I mean if that is the single thing stopping your company from using it then just add support for it. The cost of DD monthly bill more than supports spending a week or less it would take to add
    - rad_gruchalski 4y ago
      That or just put the frontend behind a reverse proxy with sso.
      - hunter2_ 4y ago
        SSO is about making a user account inside a service provider (e.g. TFA) which mirrors that same user account in the identity provider (e.g. Okta). A reverse proxy isn't able to write to the upstream application's user store or otherwise assert the identity of the current user to the upstream application, as far as I'm aware. It could do some kind of binary proxy-or-don't-proxy based on a valid assertion from the IdP, but the application would just attribute all traffic to a single user. Or is there some kind of gateway standard that I'm unaware of?
toastercat 4y ago
I used New Relic for the first time at my current job, and while service it provides to our team is invaluable, the UI is slow and very confusing; I can never remember what button or menu to click to get to where I need to go. The fact that it loads so slowly just makes it worse. So I am optimistic for alternative solutions like this!
- pranay01 4y ago
  thanks for the kind words. To give it a try and let us know if you have any feedback/questions. We also have an active slack community if you have any questions on how to set up or have an feedback for us - https://signoz.io/slack https://signoz.io/slack
- jiggawatts 4y ago
  At $dayjob the devs use both Azure Application Insights and New Relic. Comparing the two, I had the same comments: New Relic is appears useful but is too slow in practice. The developers generally avoid it in their day-to-day work, which defeats the purpose entirely. We're not in the US, so the cloud-hosted version of New Relic is especially slow because of the added latency of the trans-oceanic network hop. For comparison, Azure Application Insights can be deployed in our own region. It's not massively faster, but for some UX design reason it feels faster and more pleasant to use. It might be literally just the network latency, and nothing more, but the end result is that it's used more often. Application Insights isn't perfect by any means: - Deployment is a PitA and breaks regularly. The documentation related to installing it is confusing, out-of-date, and will guide you down dead ends. For example, I got the profiler component working once on virtual machines, then it broke, and I can't get it working again for the life of me. - The underlying Log Analytics workspaces are crazy, crazy expensive. They're far more expensive than the competition, which then makes high-level services like Sentinel and App Insights built on top also too expensive for most orgs. For comparison, Log Analytics is about 5x as expensive per GB as AWS CloudWatch logs, and up to 30x as expensive as some other similar services. If Microsoft just fixed the installer and used reasonable pricing for Log Analytics, the App Insights would be very hard to beat, especially for .NET shops. I'm hoping open-source tools like SigNoz force down the pricing from "highway robbery" to merely "greedy".
gclawes 4y ago
How's this compare to the grafana/prometheus/tempo/loki set of tools?
- pranay01 4y ago
  You can find answers to some similar questions here - https://signoz.io/docs/faqs/product/#how-is-signoz-different-from-prometheus https://signoz.io/docs/faqs/product/#how-is-signoz-different... Let me know if you have a specific question which is not addressed in the above link and will try to answer
deleted 4y ago
[deleted]
dominotw 4y ago
good to see solid oss projects coming out of india. I will def keep an eye on this one.
- pranay01 4y ago
  thanks. do give it a try and let us know if you have any feedback
codetrotter 4y ago
They say: > if you want to have a seamless experience between metrics and traces, then current experience of stitching together Prometheus & Jaeger is not great. But I wonder if using Promscale https://github.com/timescale/promscale https://github.com/timescale/promscale would make Prometheus & Jaeger not such a big problem as SigNoz imply. Promscale readme: > Promscale is a unified metric and trace observability backend for Prometheus, Jaeger and OpenTelemetry built on PostgreSQL and TimescaleDB. Either way, SigNoz seems interesting indeed. And am glad to see that SigNoz supports OpenTelemetry.
- pranay01 4y ago
  yeah, promscale is an interesting project with similar goal - trying to bring together different signals ( only metrics & traces for promscale) under one hood. I have not explored the project in detail, but as far as I understand it uses Grafana and Jaeger UI, so I am not sure how seamless is the UI interaction, while SigNoz has a UI built ground up for observability use cases.
- 0x457 4y ago
  You don't even need Promscale for this. Grafana has support for displaying information from both plus logs. As long as log messages include TraceID is straightforward use Grafana for debugging.
- ankitnayan 4y ago
  Basically, the choice of DB is different, relational vs analytical DB. IMO clickhouse should be better at ingestion rate and aggregation queries.
FridgeSeal 4y ago
This looks really good! Might give this a spin next week, currently using NR, but it’s slow, expensive and the in-cluster collection services are frustratingly fragile.
- pranay01 4y ago
  thanks, if you have any questions while setting things up, feel free to drop by on our slack community - https://signoz.io/slack https://signoz.io/slack Also, can you explain a bit more on in-cluster collection services are frustratingly fragile do you mean the agent they use for sending application metrics from clusters breaks down?
  - FridgeSeal 4y ago
    > do you mean the agent they use for sending application metrics from clusters breaks down? Yeah basically, a number of the kubernetes deployments are configured with too-low memory and resource limits, so they're constantly crashing (and triggering all sorts of alerting false-positives). The limits are semi-hardcoded, so we can't override them, and at the moment, the effort required to fix the deeply-interlinked helm chart they provide isn't worth it. The NRI-Bundle helm chart also installs a lot into your cluster - it brings its own NATS instance, there's multiple daemon sets, etc. I'd likely get quite a lot of spare compute back by swapping to something lighter.
gregwebs 4y ago
Click house is a great DB but still not the best at storing timeseries metrics. Do you have plans to incorporate a db optimized for time series storage?
- datalopers 4y ago
  I see this viewpoint occasionally but have never seen what features a TSDB offers that Clickhouse can’t do just as effectively if not better?
  - hiyer 4y ago
    I would have thought better compression was one, but it looks like Clickhouse now supports all the time series-optimized encodings like delta and gorilla - https://altinity.com/blog/2019/7/new-encodings-to-improve-clickhouse https://altinity.com/blog/2019/7/new-encodings-to-improve-cl....
- RhodesianHunter 4y ago
  Clickhouse is absolutely incredible at storing time series metrics.
  - ankitnayan 4y ago
    What does your schema for metrics look like? Using materialised views? I am particularly interested in storing metrics with labels(key/val pairs) Eg, Prometheus metrics. You can't flatten them out into columns due to high dimensionality(can need millions of columns). Do you store the labels in an array?
    - datalopers 4y ago
      Not the person you're replying to, but you can see how SigNoz did it here [1] Essentially you add: keys Array(String), values Array(String) and write the pairs accordingly. https://github.com/SigNoz/signoz/blob/develop/deploy/docker-swarm/clickhouse-setup/docker-entrypoint-initdb.d/init-db.sql https://github.com/SigNoz/signoz/blob/develop/deploy/docker-...
andrewmcwatters 4y ago
I don't know what other people's experience has been with DataDog, but the user interface experience I had with it was definitely reminiscent of something produced from a dog's rear end. Definitely a product where no one in the org said "no" to an idea. Felt very Atlassian.
- pranay01 4y ago
  Yeah, we have had some of our users mentioning that DataDog has become too bulky and too many bells and whistles. Curious though, what specific features/UX did you not like in DataDog?
- waynesonfire 4y ago
  gatta keep polishing that turd! there is a promo packet that needs filling.
- aloknnikhil 4y ago
  Second this. It’s pretty terrible when you use it at scale particularly. Eg: If you have a monitor with some scopes that you use for exceptions, it gets gnarly in the UI when you have like 50 of them. And the inconsistent UI. Notebooks are cool but so half-assed in their features that it kind becomes useless for anything other than basic charts. Oh and feature discovery is a pain too. It took me so long to figure out a way to add a second Y-axis to my charts.
  - pranay01 4y ago
    Interesting. do you mean this by scopes - https://docs.datadoghq.com/api/latest/scopes/ https://docs.datadoghq.com/api/latest/scopes/ ?
    - aloknnikhil 4y ago
      Not exactly. I’m talking about this workflow. https://www.datadoghq.com/blog/mute-datadog-alerts-planned-downtime/ https://www.datadoghq.com/blog/mute-datadog-alerts-planned-d... You use specific tags to mute a monitor by adding a scope. This does not scale well with the UI.
cebert 4y ago
I like DataDog but it is prohibitively expensive for monitoring serverless applications, even with negotiated rates. I don’t think their sales team has identified a reasonable way to bill applications that entirety use serverless AWS services. We’re looking for alternatives. Paying for DataDog is many orders of magnitude higher than our AWS bill.
- pranay01 4y ago
  Paying for DataDog is many orders of magnitude higher than our AWS bill. Wow! This is blowing my mind. Do you this this case for most companies monitoring serverless applications with DataDog, or there is something specific about your infra which cause this
  - cebert 4y ago
    The application I am currently working on is all lambdas ~ 45, DynamoDB, S3, CloudFront, Cognito, and SQS and SNS. My employer has several serverless applications with relatively moderate use, and I work on one of them. Our total cloud costs for the product I work on for our DEV/STAGE/PROD/SANDBOX environments is currently less than $1,000/mo. Our estimated cost of DataDog monitoring for the next year just on my application is at least 23k/yr using negotiated rates. We don’t have crazy traffic, but do have global users invoking all of our lambdas at least once an hour. DataDog charges a fixed monthly cost for each lambda invoked at least once an hour on average. Then, you also need to pay for ingestion, storage, and custom metrics. Just on my product alone with multiple isolated environments, this gets expensive. Many other product teams at my work have lightly used serverless apps. The DataDog costs simply aren’t feasible for serverless apps. We’re actively looking into alternatives such as just using CloudWatch, Elastic, etc as it’s a huge cost for us.
    - pranay01 4y ago
      thanks for the detailed note. I was just checking Datadog pricing for serverless, there is says - $7.20 per active function per month. If you are using 45 lambdas, is the number of functions much higher? I am guessing ~200 or so? Though I can see, how charging based on functions can quickly shoot up the bill
InTheArena 4y ago
Is there any outlier analysis on this? That's a key advantage of DataDog and NewRelic.
- pranay01 4y ago
  By outlier analysis, do you mean anomaly detection in metrics to send alerts? As of now, we have fixed threshold based alerting capabilities - but more advanced ML/seasonality based anomaly detection is on the roadmap. we are tracking this here - https://github.com/SigNoz/signoz/issues/295 https://github.com/SigNoz/signoz/issues/295 What type of outlier analysis do you generally do in DataDog/NewRelic?
  - InTheArena 4y ago
    Week over week, or day over day values that are more then x deviations from norm.
zrosenbauer 4y ago
Logs be SUPER expensive... our biggest freaking expense
- pranay01 4y ago
  what do you use currently for logs?
daigoba66 4y ago
Curious how it compares to https://github.com/uptrace/uptrace https://github.com/uptrace/uptrace And are there any other similar projects.
skanga 4y ago
I tried installation on Windows 10 via Rancher-Desktop using "other platform" docs but ran into issues with dependencies on sh/bash. Are there any windows specific instructions?
- pranay01 4y ago
  hey, windows is not officially supported. If you have a Ubuntu machine/Mac, you can try in that https://signoz.io/docs/install/docker/#prerequisites https://signoz.io/docs/install/docker/#prerequisites
  - skanga 4y ago
    Gotcha, thanks. Do the linux instructions work on any WSL distro?
    - thegagne 4y ago
      I got it running on WSL w/ docker using docker compose. I didn’t do much more than get it running though.
      - pranay01 4y ago
        Awesome! Let us know if you have nay feedback when you get a chance to dive deeper. Here's our slack community- https://signoz.io/slack https://signoz.io/slack
tomschwiha 4y ago
Looks interesting, will have a look at it. Thank you!
- pranay01 4y ago
  Awesome! Do give it a try and let us know what do you think. We also have an active slack community if you have any questions on how to set up or have an feedback for us - https://signoz.io/slack https://signoz.io/slack
yawniek 4y ago
what drove you to move away from MIT licencing?
- pranay01 4y ago
  Since we follow an open core business model, we wanted the flexibility to introduce proprietary features in the future. Hence, we added a folder ee/ which will have proprietary code. The rest of the code is still MIT licensed. And if you remove the ee/ folder, the project would work without any issue This licensing model is very similar to what folks like Gitlab and PostHog do today
  - lhoff 4y ago
    Don't do that. Open Core is going to hurt you. Especially if things like SSO are paid only. I understand you want to make money and you deserve that but open core will hurt you more then it will help you. - Open core means no direct path from testing to using (and paying). If I want to make a case for a software to be included in our stack I don't want to constantly run into paywalls while trying out the product but I also don't want to go through the hoops of getting a licence (because of internal bureaucracy). - You steer away users that cost you nothing but help you spread to word and report issues and might even provide a PR. I'm talking about OpenSource Projects, student organisations and companies with limited resources (NGOs, early stage startup's). I used to be head admin in a student organisation. We developed our own internal tools and only relied on open-source components. I know of at least two cases where other students got to know the tools and after they finished university joined companies and introduced the very tools we used to their employees resulting in paid support contracts. - It creates tempting opportunities for investors to force you into ruining the open source tier. In the beginning you only have features behind the paywall that are useful to big enterprises but if your business is not growing fast enough (from the perspective of investors) they might force you to push more users to be paying customers. That might help in the short term but will ruin your reputation in the long run. - Paying for services (aka insurance to get help if something goes haywire) is easier to justify with execs then paying an unreasonable amount for that one feature that is behind the paywall. ("Can't you just make it work without it"). Its a purely psychological argument but decision processes in companies are not always rational and your allies are the devs and you should be helping them make a case to buy your product.
chrisandchris 4y ago
Question: Is it licensed under MIT? The license reads as it could be MIT but there is no reference to it.
- ankitnayan 4y ago
  https://github.com/SigNoz/signoz/blob/main/LICENSE https://github.com/SigNoz/signoz/blob/main/LICENSE
  - chrisandchris 4y ago
    Does not match the file I see when clicking "View License", e.g. will not match the LICENSE file the next release may have? https://github.com/SigNoz/signoz/blob/develop/LICENSE https://github.com/SigNoz/signoz/blob/develop/LICENSE
    - pranay01 4y ago
      yes, upcoming release will have license which is currently in `develop` branch
- pranay01 4y ago
  The ee/ folder is under proprietary license. Rest of the code is under MIT license
  - chrisandchris 4y ago
    Thanks for the response. I've realized that the license between the branch develop and main is not the same at the moment.
mritchie712 4y ago
How are you dealing with join's in clickhouse? Do you just avoid them all together? We use clickhouse at Luabase and join performance is the only weak point we've hit.
- ankitnayan 4y ago
  Yes...we are avoiding join altogether. Currently we used join in timeseries but we are probably moving away from that due to perf. Single table is amazingly fast
spaceman10 4y ago
Hi, I want to love this product. This will be the 3rd or 4th time I have looked at the docs to figure out basic setup. Each time I look into this. I look to see how it can report CPU/Memory/Disk of the systems at large. When I read, all I find on open telemetry and SigNoz is how to integrate with application stacks. Am I missing something fundamental in that SygNoz/OpenTelemetry do not integrate with host level metrics? I really, really want to use this product. But this extremely BASIC implementation to use the service is 100% missing for documentation as far as I can tell. Even the example page has nothing listed https://signoz.io/docs/tutorials/ https://signoz.io/docs/tutorials/ ... So I am either trying to find out how to make SygNoz do something it was not built to do... or the documentation has a huge hole in it. Let me know how to proceed, if you have time. It's appreciated so that I dont keep walking down the wrong road and hoping to find something useful. Thanks!
- pranay01 4y ago
  Hey, thanks for writing. Our initial focus was on application monitoring - that is why docs are more oriented towards it. But I can understand you may find it tough to understand for infra metrics (CPU/Memory/Disk) As of now, there are couple of ways to do this: 1. You can enable hostmetrics receiver in opentelemetry collector 2. You can use something like prometheus node exporter and enable prometheus receiver in opentelemetry collector If you follow this for a VM setup, you should be able to get your host metrics - https://signoz.io/docs/userguide/send-metrics/#enable-a-specific-metric-receiver https://signoz.io/docs/userguide/send-metrics/#enable-a-spec... If you are on k8s, check out this - https://signoz.io/docs/tutorial/kubernetes-infra-metrics/ https://signoz.io/docs/tutorial/kubernetes-infra-metrics/ If it is still confusing, do drop by in our slack community ( https://signoz.io/slack https://signoz.io/slack) We are quite active there and should be able to help you get started
  - spaceman10 4y ago
    Many thanks. I took notice of your prometheus mention and had also seen that peppered in the docs for both services (openT and SigN) as a way to gather data. As time investment is a dangerous thing I didn't plop myself down for some R&D... But... thanks to your post I'll have another go at it and determine how to get it to hum. Thanks again!
jaxn 4y ago
I am hesitant to run my APM on the same infra as our application. I love the idea of reducing an external dependency/cost, and it looks easy enough to add to our Kubernetes. It just seems a little like host our own status page on the same servers as our SaaS. Is this something you have an answer for?
- pranay01 4y ago
  We recommend users to run SigNoz in a separate k8s cluster/VM. In this way, even if your applications servers/clusters get overloaded, your observability stack (SigNoz) will still be running seamlessly. Many of our users use SigNoz in a similar fashion.

510 points by pranay01 4y ago

pranay01 4y ago

ketzu 4y ago

Is there any introduction to all the services that are included in the docker-compose file [1]? I wanted to give signoz a try, but the sheer amount of services attached discouraged me, especially as I have to reconfigure them all to work with my setup. (Don't run them directly on the host, instead in a separated network, put the network interface behind traefik, figure out which access they need, provide all the configuration in a nice way without having to clone the full repo just to have the configuration files.) [1] https://github.com/SigNoz/signoz/blob/develop/deploy/docker/clickhouse-setup/docker-compose.yaml https://github.com/SigNoz/signoz/blob/develop/deploy/docker/...

rad_gruchalski 4y ago
Not associated with the project but a quick look suggests: a database (clickhouse), alertmanager, query-service, frontend are signoz components, otel* and hotrod are for distributed tracing. Otel stands for open telemetry (https://opentelemetry.io/ https://opentelemetry.io/), hotrod is a tracing demo app from jaeger: https://github.com/jaegertracing/jaeger/tree/main/examples/hotrod https://github.com/jaegertracing/jaeger/tree/main/examples/h.... Without thinking too much about it, I assume that: hotrod is a demo data source pushing traces to the otel collector, which stores data in clickhouse. Frontend fetches data from clickhouse using the query service. Alert manager probably looks at traces coming in and detects anomalies, so that you can get real-time alerts when things don’t look normal.
- ketzu 4y ago
  Thank you for the quick explanation, that helps a lot and gives me at least a starting point. Although sample services surprise me in the suggested docker-compose from the "how to install" section.
  - pranay01 4y ago
    yeah, it's not ideal - but we kept it to give users who are installing for the first time to get a hang of the product You can easily remove the sample services if you want - https://signoz.io/docs/operate/docker-standalone/#remove-the-sample-application https://signoz.io/docs/operate/docker-standalone/#remove-the...
pranay01 4y ago
The comment by rad_gruchalski is mostly accurate SigNoz has the following components - Frontend - ClickHouse (datastore) - Alert Manager (this is to monitor metrics and create alerts which you configure in SigNoz) - Query Service (which is the backend service which talks to datastore & frontend) - Otel collector ( which the collector provided by opentelemetry to collect telemetry data) The other two components for sample app which can be commented out - Hotrod ( which is a sample app) - load generator More details here - https://signoz.io/docs/architecture/#architecture-components https://signoz.io/docs/architecture/#architecture-components

reilly3000 4y ago

I just learned about SigNoz. I spend a lot of time flipping between New Relic and Splunk, and my company spends a ton on both. I’m interested to hear from users and learn if it’s prod ready as a New Relic alternative. I’m especially interested in something that could run locally so I can profile using the same tools that run in prod. Any feedback?

makeavish 4y ago
Here are few case studies of production users published by SigNoz: https://signoz.io/case-study/ https://signoz.io/case-study/

jakswa 4y ago

Goodness I hope this is good because I will make my company gobble it upppp if so.

pranay01 4y ago
feel free to test it out. We have an active slack community as well if you have any questions on how to setup etc. https://signoz.io/slack https://signoz.io/slack
saintfiends 4y ago
We thought so too until we found out it doesn't support any kind of SSO https://github.com/SigNoz/signoz/issues/1188 https://github.com/SigNoz/signoz/issues/1188
- bogota 4y ago
  I mean if that is the single thing stopping your company from using it then just add support for it. The cost of DD monthly bill more than supports spending a week or less it would take to add
  - rad_gruchalski 4y ago
    That or just put the frontend behind a reverse proxy with sso.
    - hunter2_ 4y ago
      SSO is about making a user account inside a service provider (e.g. TFA) which mirrors that same user account in the identity provider (e.g. Okta). A reverse proxy isn't able to write to the upstream application's user store or otherwise assert the identity of the current user to the upstream application, as far as I'm aware. It could do some kind of binary proxy-or-don't-proxy based on a valid assertion from the IdP, but the application would just attribute all traffic to a single user. Or is there some kind of gateway standard that I'm unaware of?

toastercat 4y ago

I used New Relic for the first time at my current job, and while service it provides to our team is invaluable, the UI is slow and very confusing; I can never remember what button or menu to click to get to where I need to go. The fact that it loads so slowly just makes it worse. So I am optimistic for alternative solutions like this!

pranay01 4y ago
thanks for the kind words. To give it a try and let us know if you have any feedback/questions. We also have an active slack community if you have any questions on how to set up or have an feedback for us - https://signoz.io/slack https://signoz.io/slack
jiggawatts 4y ago
At $dayjob the devs use both Azure Application Insights and New Relic. Comparing the two, I had the same comments: New Relic is appears useful but is too slow in practice. The developers generally avoid it in their day-to-day work, which defeats the purpose entirely. We're not in the US, so the cloud-hosted version of New Relic is especially slow because of the added latency of the trans-oceanic network hop. For comparison, Azure Application Insights can be deployed in our own region. It's not massively faster, but for some UX design reason it feels faster and more pleasant to use. It might be literally just the network latency, and nothing more, but the end result is that it's used more often. Application Insights isn't perfect by any means: - Deployment is a PitA and breaks regularly. The documentation related to installing it is confusing, out-of-date, and will guide you down dead ends. For example, I got the profiler component working once on virtual machines, then it broke, and I can't get it working again for the life of me. - The underlying Log Analytics workspaces are crazy, crazy expensive. They're far more expensive than the competition, which then makes high-level services like Sentinel and App Insights built on top also too expensive for most orgs. For comparison, Log Analytics is about 5x as expensive per GB as AWS CloudWatch logs, and up to 30x as expensive as some other similar services. If Microsoft just fixed the installer and used reasonable pricing for Log Analytics, the App Insights would be very hard to beat, especially for .NET shops. I'm hoping open-source tools like SigNoz force down the pricing from "highway robbery" to merely "greedy".

gclawes 4y ago

How's this compare to the grafana/prometheus/tempo/loki set of tools?

pranay01 4y ago
You can find answers to some similar questions here - https://signoz.io/docs/faqs/product/#how-is-signoz-different-from-prometheus https://signoz.io/docs/faqs/product/#how-is-signoz-different... Let me know if you have a specific question which is not addressed in the above link and will try to answer

deleted 4y ago

[deleted]

dominotw 4y ago

good to see solid oss projects coming out of india. I will def keep an eye on this one.

pranay01 4y ago
thanks. do give it a try and let us know if you have any feedback

codetrotter 4y ago

They say: > if you want to have a seamless experience between metrics and traces, then current experience of stitching together Prometheus & Jaeger is not great. But I wonder if using Promscale https://github.com/timescale/promscale https://github.com/timescale/promscale would make Prometheus & Jaeger not such a big problem as SigNoz imply. Promscale readme: > Promscale is a unified metric and trace observability backend for Prometheus, Jaeger and OpenTelemetry built on PostgreSQL and TimescaleDB. Either way, SigNoz seems interesting indeed. And am glad to see that SigNoz supports OpenTelemetry.

pranay01 4y ago
yeah, promscale is an interesting project with similar goal - trying to bring together different signals ( only metrics & traces for promscale) under one hood. I have not explored the project in detail, but as far as I understand it uses Grafana and Jaeger UI, so I am not sure how seamless is the UI interaction, while SigNoz has a UI built ground up for observability use cases.
0x457 4y ago
You don't even need Promscale for this. Grafana has support for displaying information from both plus logs. As long as log messages include TraceID is straightforward use Grafana for debugging.
ankitnayan 4y ago
Basically, the choice of DB is different, relational vs analytical DB. IMO clickhouse should be better at ingestion rate and aggregation queries.

FridgeSeal 4y ago

This looks really good! Might give this a spin next week, currently using NR, but it’s slow, expensive and the in-cluster collection services are frustratingly fragile.

pranay01 4y ago
thanks, if you have any questions while setting things up, feel free to drop by on our slack community - https://signoz.io/slack https://signoz.io/slack Also, can you explain a bit more on in-cluster collection services are frustratingly fragile do you mean the agent they use for sending application metrics from clusters breaks down?
- FridgeSeal 4y ago
  > do you mean the agent they use for sending application metrics from clusters breaks down? Yeah basically, a number of the kubernetes deployments are configured with too-low memory and resource limits, so they're constantly crashing (and triggering all sorts of alerting false-positives). The limits are semi-hardcoded, so we can't override them, and at the moment, the effort required to fix the deeply-interlinked helm chart they provide isn't worth it. The NRI-Bundle helm chart also installs a lot into your cluster - it brings its own NATS instance, there's multiple daemon sets, etc. I'd likely get quite a lot of spare compute back by swapping to something lighter.

gregwebs 4y ago

Click house is a great DB but still not the best at storing timeseries metrics. Do you have plans to incorporate a db optimized for time series storage?

datalopers 4y ago
I see this viewpoint occasionally but have never seen what features a TSDB offers that Clickhouse can’t do just as effectively if not better?
- hiyer 4y ago
  I would have thought better compression was one, but it looks like Clickhouse now supports all the time series-optimized encodings like delta and gorilla - https://altinity.com/blog/2019/7/new-encodings-to-improve-clickhouse https://altinity.com/blog/2019/7/new-encodings-to-improve-cl....
RhodesianHunter 4y ago
Clickhouse is absolutely incredible at storing time series metrics.
- ankitnayan 4y ago
  What does your schema for metrics look like? Using materialised views? I am particularly interested in storing metrics with labels(key/val pairs) Eg, Prometheus metrics. You can't flatten them out into columns due to high dimensionality(can need millions of columns). Do you store the labels in an array?
  - datalopers 4y ago
    Not the person you're replying to, but you can see how SigNoz did it here [1] Essentially you add: keys Array(String), values Array(String) and write the pairs accordingly. https://github.com/SigNoz/signoz/blob/develop/deploy/docker-swarm/clickhouse-setup/docker-entrypoint-initdb.d/init-db.sql https://github.com/SigNoz/signoz/blob/develop/deploy/docker-...

andrewmcwatters 4y ago

I don't know what other people's experience has been with DataDog, but the user interface experience I had with it was definitely reminiscent of something produced from a dog's rear end. Definitely a product where no one in the org said "no" to an idea. Felt very Atlassian.

pranay01 4y ago
Yeah, we have had some of our users mentioning that DataDog has become too bulky and too many bells and whistles. Curious though, what specific features/UX did you not like in DataDog?
waynesonfire 4y ago
gatta keep polishing that turd! there is a promo packet that needs filling.
aloknnikhil 4y ago
Second this. It’s pretty terrible when you use it at scale particularly. Eg: If you have a monitor with some scopes that you use for exceptions, it gets gnarly in the UI when you have like 50 of them. And the inconsistent UI. Notebooks are cool but so half-assed in their features that it kind becomes useless for anything other than basic charts. Oh and feature discovery is a pain too. It took me so long to figure out a way to add a second Y-axis to my charts.
- pranay01 4y ago
  Interesting. do you mean this by scopes - https://docs.datadoghq.com/api/latest/scopes/ https://docs.datadoghq.com/api/latest/scopes/ ?
  - aloknnikhil 4y ago
    Not exactly. I’m talking about this workflow. https://www.datadoghq.com/blog/mute-datadog-alerts-planned-downtime/ https://www.datadoghq.com/blog/mute-datadog-alerts-planned-d... You use specific tags to mute a monitor by adding a scope. This does not scale well with the UI.

cebert 4y ago

I like DataDog but it is prohibitively expensive for monitoring serverless applications, even with negotiated rates. I don’t think their sales team has identified a reasonable way to bill applications that entirety use serverless AWS services. We’re looking for alternatives. Paying for DataDog is many orders of magnitude higher than our AWS bill.

pranay01 4y ago
Paying for DataDog is many orders of magnitude higher than our AWS bill. Wow! This is blowing my mind. Do you this this case for most companies monitoring serverless applications with DataDog, or there is something specific about your infra which cause this
- cebert 4y ago
  The application I am currently working on is all lambdas ~ 45, DynamoDB, S3, CloudFront, Cognito, and SQS and SNS. My employer has several serverless applications with relatively moderate use, and I work on one of them. Our total cloud costs for the product I work on for our DEV/STAGE/PROD/SANDBOX environments is currently less than $1,000/mo. Our estimated cost of DataDog monitoring for the next year just on my application is at least 23k/yr using negotiated rates. We don’t have crazy traffic, but do have global users invoking all of our lambdas at least once an hour. DataDog charges a fixed monthly cost for each lambda invoked at least once an hour on average. Then, you also need to pay for ingestion, storage, and custom metrics. Just on my product alone with multiple isolated environments, this gets expensive. Many other product teams at my work have lightly used serverless apps. The DataDog costs simply aren’t feasible for serverless apps. We’re actively looking into alternatives such as just using CloudWatch, Elastic, etc as it’s a huge cost for us.
  - pranay01 4y ago
    thanks for the detailed note. I was just checking Datadog pricing for serverless, there is says - $7.20 per active function per month. If you are using 45 lambdas, is the number of functions much higher? I am guessing ~200 or so? Though I can see, how charging based on functions can quickly shoot up the bill

InTheArena 4y ago

Is there any outlier analysis on this? That's a key advantage of DataDog and NewRelic.

pranay01 4y ago
By outlier analysis, do you mean anomaly detection in metrics to send alerts? As of now, we have fixed threshold based alerting capabilities - but more advanced ML/seasonality based anomaly detection is on the roadmap. we are tracking this here - https://github.com/SigNoz/signoz/issues/295 https://github.com/SigNoz/signoz/issues/295 What type of outlier analysis do you generally do in DataDog/NewRelic?
- InTheArena 4y ago
  Week over week, or day over day values that are more then x deviations from norm.

zrosenbauer 4y ago

Logs be SUPER expensive... our biggest freaking expense

pranay01 4y ago
what do you use currently for logs?

daigoba66 4y ago

Curious how it compares to https://github.com/uptrace/uptrace https://github.com/uptrace/uptrace And are there any other similar projects.

skanga 4y ago

I tried installation on Windows 10 via Rancher-Desktop using "other platform" docs but ran into issues with dependencies on sh/bash. Are there any windows specific instructions?

pranay01 4y ago
hey, windows is not officially supported. If you have a Ubuntu machine/Mac, you can try in that https://signoz.io/docs/install/docker/#prerequisites https://signoz.io/docs/install/docker/#prerequisites
- skanga 4y ago
  Gotcha, thanks. Do the linux instructions work on any WSL distro?
  - thegagne 4y ago
    I got it running on WSL w/ docker using docker compose. I didn’t do much more than get it running though.
    - pranay01 4y ago
      Awesome! Let us know if you have nay feedback when you get a chance to dive deeper. Here's our slack community- https://signoz.io/slack https://signoz.io/slack

tomschwiha 4y ago

Looks interesting, will have a look at it. Thank you!

pranay01 4y ago
Awesome! Do give it a try and let us know what do you think. We also have an active slack community if you have any questions on how to set up or have an feedback for us - https://signoz.io/slack https://signoz.io/slack

yawniek 4y ago

what drove you to move away from MIT licencing?

pranay01 4y ago
Since we follow an open core business model, we wanted the flexibility to introduce proprietary features in the future. Hence, we added a folder ee/ which will have proprietary code. The rest of the code is still MIT licensed. And if you remove the ee/ folder, the project would work without any issue This licensing model is very similar to what folks like Gitlab and PostHog do today
- lhoff 4y ago
  Don't do that. Open Core is going to hurt you. Especially if things like SSO are paid only. I understand you want to make money and you deserve that but open core will hurt you more then it will help you. - Open core means no direct path from testing to using (and paying). If I want to make a case for a software to be included in our stack I don't want to constantly run into paywalls while trying out the product but I also don't want to go through the hoops of getting a licence (because of internal bureaucracy). - You steer away users that cost you nothing but help you spread to word and report issues and might even provide a PR. I'm talking about OpenSource Projects, student organisations and companies with limited resources (NGOs, early stage startup's). I used to be head admin in a student organisation. We developed our own internal tools and only relied on open-source components. I know of at least two cases where other students got to know the tools and after they finished university joined companies and introduced the very tools we used to their employees resulting in paid support contracts. - It creates tempting opportunities for investors to force you into ruining the open source tier. In the beginning you only have features behind the paywall that are useful to big enterprises but if your business is not growing fast enough (from the perspective of investors) they might force you to push more users to be paying customers. That might help in the short term but will ruin your reputation in the long run. - Paying for services (aka insurance to get help if something goes haywire) is easier to justify with execs then paying an unreasonable amount for that one feature that is behind the paywall. ("Can't you just make it work without it"). Its a purely psychological argument but decision processes in companies are not always rational and your allies are the devs and you should be helping them make a case to buy your product.

chrisandchris 4y ago

Question: Is it licensed under MIT? The license reads as it could be MIT but there is no reference to it.

ankitnayan 4y ago
https://github.com/SigNoz/signoz/blob/main/LICENSE https://github.com/SigNoz/signoz/blob/main/LICENSE
- chrisandchris 4y ago
  Does not match the file I see when clicking "View License", e.g. will not match the LICENSE file the next release may have? https://github.com/SigNoz/signoz/blob/develop/LICENSE https://github.com/SigNoz/signoz/blob/develop/LICENSE
  - pranay01 4y ago
    yes, upcoming release will have license which is currently in `develop` branch
pranay01 4y ago
The ee/ folder is under proprietary license. Rest of the code is under MIT license
- chrisandchris 4y ago
  Thanks for the response. I've realized that the license between the branch develop and main is not the same at the moment.

mritchie712 4y ago

How are you dealing with join's in clickhouse? Do you just avoid them all together? We use clickhouse at Luabase and join performance is the only weak point we've hit.

ankitnayan 4y ago
Yes...we are avoiding join altogether. Currently we used join in timeseries but we are probably moving away from that due to perf. Single table is amazingly fast

spaceman10 4y ago

Hi, I want to love this product. This will be the 3rd or 4th time I have looked at the docs to figure out basic setup. Each time I look into this. I look to see how it can report CPU/Memory/Disk of the systems at large. When I read, all I find on open telemetry and SigNoz is how to integrate with application stacks. Am I missing something fundamental in that SygNoz/OpenTelemetry do not integrate with host level metrics? I really, really want to use this product. But this extremely BASIC implementation to use the service is 100% missing for documentation as far as I can tell. Even the example page has nothing listed https://signoz.io/docs/tutorials/ https://signoz.io/docs/tutorials/ ... So I am either trying to find out how to make SygNoz do something it was not built to do... or the documentation has a huge hole in it. Let me know how to proceed, if you have time. It's appreciated so that I dont keep walking down the wrong road and hoping to find something useful. Thanks!

pranay01 4y ago
Hey, thanks for writing. Our initial focus was on application monitoring - that is why docs are more oriented towards it. But I can understand you may find it tough to understand for infra metrics (CPU/Memory/Disk) As of now, there are couple of ways to do this: 1. You can enable hostmetrics receiver in opentelemetry collector 2. You can use something like prometheus node exporter and enable prometheus receiver in opentelemetry collector If you follow this for a VM setup, you should be able to get your host metrics - https://signoz.io/docs/userguide/send-metrics/#enable-a-specific-metric-receiver https://signoz.io/docs/userguide/send-metrics/#enable-a-spec... If you are on k8s, check out this - https://signoz.io/docs/tutorial/kubernetes-infra-metrics/ https://signoz.io/docs/tutorial/kubernetes-infra-metrics/ If it is still confusing, do drop by in our slack community ( https://signoz.io/slack https://signoz.io/slack) We are quite active there and should be able to help you get started
- spaceman10 4y ago
  Many thanks. I took notice of your prometheus mention and had also seen that peppered in the docs for both services (openT and SigN) as a way to gather data. As time investment is a dangerous thing I didn't plop myself down for some R&D... But... thanks to your post I'll have another go at it and determine how to get it to hum. Thanks again!

jaxn 4y ago

I am hesitant to run my APM on the same infra as our application. I love the idea of reducing an external dependency/cost, and it looks easy enough to add to our Kubernetes. It just seems a little like host our own status page on the same servers as our SaaS. Is this something you have an answer for?

pranay01 4y ago
We recommend users to run SigNoz in a separate k8s cluster/VM. In this way, even if your applications servers/clusters get overloaded, your observability stack (SigNoz) will still be running seamlessly. Many of our users use SigNoz in a similar fashion.