15 ms·

Stack Exchange's monitoring system is now open source

147 points by waffle_ss 13y ago

pyrox420 13y ago
There are a few issues with the GIT repo right now that I'm helping Nick Craver work through. I see this project maturing over time to be quite awesome.
rsync 13y ago
Is it GUI based ? If so, are there screenshots ?
- GeorgeBeech 13y ago
  Yep. Here are the screenshots from out Velocity Presentation: http://imgur.com/a/dawwf http://imgur.com/a/dawwf
  - qixiang 13y ago
    thanks
jamesRaybould 13y ago
Having a quick poke around and I notice that they are storing all exceptions in an SQL database. I've been looking at storing all the errors we get in our various applications in a central repository and was wondering what the general consensus was? Currently I'm going a centralised logstash server and using a logstash shipper on each of my servers to push the exceptions, from a standard logfile to it. I was toying with the idea of pushing all my errors at source to an SQL database but figured if I was having database problems I'd be missing all the exceptions that I could be using to trigger the alerts that I'm having database problems!
- rhizome 13y ago
  There is no general consensus, but there's no rule that say you can only have one central watchdog. Collect errors one place, watch over you db somewhere else, watch over your watchdogs somewhere else still. This is basic sysadmin CYA.
- js2 13y ago
  Take a look at http://squash.io http://squash.io before you build your own.
- SEJeff 13y ago
  Seriously take a look at Sentry[1]. It supports just about every major language out there, is open source and used on some very large web properties (disqus, which powers comments for cnn.com amongst others), and is just generally awesome software. If you don't want to set it up yourself, use their hosted version[2]. If you are a .net / C# guy check out their csharp raven client[3]. Raven is the client to sentry which automatically sends all exceptions. [1] https://github.com/getsentry/sentry https://github.com/getsentry/sentry [2] https://getsentry.com/welcome https://getsentry.com/welcome [3] https://github.com/getsentry/raven-csharp https://github.com/getsentry/raven-csharp
- sk5t 13y ago
  We built our monitoring package at Zetetic to be extremely flexible about what events go where, with multiple levels of non-blocking filtering and routing, so that everything remotely interesting could go to SQL (with or without a durable local queue in front of it), and just very critical stuff could also go to a local embedded database / lpr / MQ / ZeroMQ publisher, etc. My preference is for decoupled ZMQ subscribers to handle anything very gnarly. Internal errors in the monitoring software itself first surface in NLog, which could--but probably oughtn't--be configured to feed even more errors into the monitoring system; obviously this could create a terrible feedback loop if left unchecked.
- Nick-Craver 13y ago
  We indeed use SQL for errors (though Exceptional has a couple of stores...and you can add a new one). The reason a SQL outage isn't an issue is a) we have other alarms for that, and b) Exceptional will fall back to an in-memory exception queue and flush to the DB when it's available again. In the event of a store loss (file share, SQL server...whatever your store is) then it queues exceptions in memory with rollups to reduce memory usage, and will flush to the store when it's available again. It does a connectivity test every 2 seconds in the event of failure. Exceptional is open source and is the basis for what's used in Opserver...the UI is even very, very similar it's just that Opserver has more features for a multi-application view. You can see the source here: https://github.com/NickCraver/StackExchange.Exceptional https://github.com/NickCraver/StackExchange.Exceptional
grundprinzip 13y ago
Easy to deploy as it's written in C# :)
- zackbloom 13y ago
  Just because you're already on a .NET stack, or is there another reason?
  - grundprinzip 13y ago
    Actually, this was irony, as all the systems that I have currently deployed run a classical Linux stack and I would have to think a long time to start deploying a .NET / Mono app if it's worth the integration effort.
pygy_ 13y ago
Wow... Nice one. Does anybody know if it works with Mono?
- michaelfdeberry 13y ago
  I haven't pulled the code yet, but it looks like it targeting MVC4. The documentation[1] states that mono currently partially supports it, everything but the async stuff, but that doesn't mean there isn't anything else in there that will cause it to not be compatible. [1]http://www.mono-project.com/Compatibility http://www.mono-project.com/Compatibility
AsymetricCom 13y ago
Looks like a tangled mess of a decision engine. Why would anyone want this? Surely, there are other solutions out there that are more mature? (nagios comes to mind but is a poor example)
- rch 13y ago
  Any and all answers to this question would be appreciated, even those that just qualify as 'promising' and 'active' (as opposed to mature). All I know of is nagios really.
  - ojilles 13y ago
    http://sensuapp.org/ http://sensuapp.org/ and http://riemann.io/ http://riemann.io/
- hamburglar 13y ago
  The fact that you're labeling the only alternative you can think of "a poor example" seems significant. Edit: and I agree that nagios is a poor example, and welcome newcomers.
beck5 13y ago
Could someone please list the equivalent systems this is could replace? (linux or windows)
- viraptor 13y ago
  I see parts of cacti, nagios/icinga + graphing, munin, zabbix, sentry (exceptions/errors), shinken, etc. There are many similar projects, but nothing so much integrated as far as I know.
emillon 13y ago
I understand that it's not worth the cost to cleanup the history, but it's always frustrating to see such a project come with a 100kloc "initial" commit.
- crazygringo 13y ago
  Interesting. Why? What would be valuable about the commit history for you?
  - kingkilr 13y ago
    The same things that are valuable about any commit history?
    - meowface 13y ago
      I would imagine that if you're a random person only just finding the project now and aren't already involved in it (by working for or with Stack Exchange), then you probably wouldn't get all that much value out of the commit history. Feel free to dispute that, though. There may be some scenarios I'm not considering.
      - tedunangst 13y ago
        if (x < 42 && y > 1776) useConfabulator = false; Why? Who knows? People don't comment their code for many reasons, not least of which is that it's not technically required and you can easily put it off and forget. Every checkin generally requires at least some comment. You can still enter meaningless messages like "fix", but you're not liable to do so by accident.
  - emillon 13y ago
    I think that the commit history says a lot about the development process and the company culture. First, is that it's interesting to see how these companies work internally. Second, it's easier to dive into the code by reading the latest commits. You can see what's going on and what pieces are changed together. Finally, there is a substantial difference between: - a repository that's handled as a first class "source code ledger" (pick one or several of: proper branching/merging, atomic commits, issues referenced in commit messages, etc). - a repository full of "git commit -a -m 'ill fix this bug later, TGIF'" whose history makes no sense. I don't want to open the debate on which one is preferable or not (most people agree that git bisect is handy, but it is open to debate whether it's worth the cost). What I'm wondering is how it looked and why it was rewritten. The most probable reason is that they wanted to make sure that nothing sensitive was exposed (non-redistributable embedded dependencies, information on their infrastructure, ...). But it's also possible that this was started as a type-2 repository and they don't want to expose this image. I tend to think that every code should be written (and developed) as it was open sourced one day, but as always there's a cost trade-off in here. Anyway, thanks for open sourcing this piece of software!
toyg 13y ago
Looks nice, sadly it's just overkill for what I need. Has anyone built a very simple solution to start/stop an arbitrary set of Windows services across several boxes, in a specific order? It'd be nice to have a simple GUI for this sort of thing. I've started working on it, but I suck at desktop programming (well, at programming in general, probably)...
- gecko 13y ago
  Unless you really need a GUI, it's insanely easy to do exactly this with PowerShell workflows (http://technet.microsoft.com/en-us/library/jj134242.aspx http://technet.microsoft.com/en-us/library/jj134242.aspx).
elwell 13y ago
Wow, they weren't kidding when they said .NET
rdtsc 13y ago
It is so strange to look at that C# code. I know they use it, it is just looks kind of odd compared to the rest of young (I mean less than 5 year old) web companies. No Ubuntu, nginx, node, jvm, but instead C#. I don't know, it just stands out.
saneshark 13y ago
Stackoverflow runs Microsoft?!? I feel like I was just punched in the gut by a best friend. Throwing up...
stevepotter 13y ago
The anti .net kneejerk reactions on HN really disturbs me. You spend all day on stackoverflow then blindly bash their tech stack. Developing, deploying, and hosting .net apps is just fine. Many brilliant people choose .net and are plenty happy with it. Maybe rather than jumping to conclusions, you could give it a shot.
- rurounijones 13y ago
  What anti .net kneejerk? At time of this comment there are 28 comments, only 2 of which are anti .net / Microsoft (with no replies) and they are already well on their way to being downvoted into oblivion.
- Styck 13y ago
  Many of us have .NET experience, really like C# but have sensible reasons for not recommending it. A Microsoft only stack is nowhere near as flexible as Linux and if you were to reach Stack Overflow's levels of traffic you'll not be able to avoid the need to run a part of your infrastructure on Linux. I seriously doubt Stack Overflow is running Redis, ElasticSearch & HAProxy* on Windows. Unless you already have a team proficient in .NET, there are few good reasons to use it instead of a JVM language. *https://github.com/opserver/Opserver/tree/master/Opserver.Core/Data https://github.com/opserver/Opserver/tree/master/Opserver.Co...
  - garethadams 13y ago
    I imagine the Stackoverflow guys are familiar with Linux architectures, they also run the open source forum Discourse[1] which is a Ruby on Rails app that comes with a recommendation to set up and develop in a *NIX environment[2]. Certainly the point is to use the right tool for the job, and I wouldn't dismiss your final comment: having a team proficient in .NET is a great reason to build your tools with/for that platform. But no, I don't think it would be fair to reply to this article saying "Why didn't they build this for [my preferred language]?" [1]: http://www.discourse.org/ http://www.discourse.org/ [2]: http://blog.discourse.org/2013/04/discourse-as-your-first-rails-app/ http://blog.discourse.org/2013/04/discourse-as-your-first-ra...
  - 13y ago
rajbala 13y ago
They lost me at IIS.

147 points by waffle_ss 13y ago

There are a few issues with the GIT repo right now that I'm helping Nick Craver work through. I see this project maturing over time to be quite awesome.

rsync 13y ago

Is it GUI based ? If so, are there screenshots ?

GeorgeBeech 13y ago
Yep. Here are the screenshots from out Velocity Presentation: http://imgur.com/a/dawwf http://imgur.com/a/dawwf
- qixiang 13y ago
  thanks

jamesRaybould 13y ago

Having a quick poke around and I notice that they are storing all exceptions in an SQL database. I've been looking at storing all the errors we get in our various applications in a central repository and was wondering what the general consensus was? Currently I'm going a centralised logstash server and using a logstash shipper on each of my servers to push the exceptions, from a standard logfile to it. I was toying with the idea of pushing all my errors at source to an SQL database but figured if I was having database problems I'd be missing all the exceptions that I could be using to trigger the alerts that I'm having database problems!

rhizome 13y ago
There is no general consensus, but there's no rule that say you can only have one central watchdog. Collect errors one place, watch over you db somewhere else, watch over your watchdogs somewhere else still. This is basic sysadmin CYA.
js2 13y ago
Take a look at http://squash.io http://squash.io before you build your own.
SEJeff 13y ago
Seriously take a look at Sentry[1]. It supports just about every major language out there, is open source and used on some very large web properties (disqus, which powers comments for cnn.com amongst others), and is just generally awesome software. If you don't want to set it up yourself, use their hosted version[2]. If you are a .net / C# guy check out their csharp raven client[3]. Raven is the client to sentry which automatically sends all exceptions. [1] https://github.com/getsentry/sentry https://github.com/getsentry/sentry [2] https://getsentry.com/welcome https://getsentry.com/welcome [3] https://github.com/getsentry/raven-csharp https://github.com/getsentry/raven-csharp
sk5t 13y ago
We built our monitoring package at Zetetic to be extremely flexible about what events go where, with multiple levels of non-blocking filtering and routing, so that everything remotely interesting could go to SQL (with or without a durable local queue in front of it), and just very critical stuff could also go to a local embedded database / lpr / MQ / ZeroMQ publisher, etc. My preference is for decoupled ZMQ subscribers to handle anything very gnarly. Internal errors in the monitoring software itself first surface in NLog, which could--but probably oughtn't--be configured to feed even more errors into the monitoring system; obviously this could create a terrible feedback loop if left unchecked.
Nick-Craver 13y ago
We indeed use SQL for errors (though Exceptional has a couple of stores...and you can add a new one). The reason a SQL outage isn't an issue is a) we have other alarms for that, and b) Exceptional will fall back to an in-memory exception queue and flush to the DB when it's available again. In the event of a store loss (file share, SQL server...whatever your store is) then it queues exceptions in memory with rollups to reduce memory usage, and will flush to the store when it's available again. It does a connectivity test every 2 seconds in the event of failure. Exceptional is open source and is the basis for what's used in Opserver...the UI is even very, very similar it's just that Opserver has more features for a multi-application view. You can see the source here: https://github.com/NickCraver/StackExchange.Exceptional https://github.com/NickCraver/StackExchange.Exceptional

grundprinzip 13y ago

Easy to deploy as it's written in C# :)

zackbloom 13y ago
Just because you're already on a .NET stack, or is there another reason?
- grundprinzip 13y ago
  Actually, this was irony, as all the systems that I have currently deployed run a classical Linux stack and I would have to think a long time to start deploying a .NET / Mono app if it's worth the integration effort.

pygy_ 13y ago

Wow... Nice one. Does anybody know if it works with Mono?

michaelfdeberry 13y ago
I haven't pulled the code yet, but it looks like it targeting MVC4. The documentation[1] states that mono currently partially supports it, everything but the async stuff, but that doesn't mean there isn't anything else in there that will cause it to not be compatible. [1]http://www.mono-project.com/Compatibility http://www.mono-project.com/Compatibility

AsymetricCom 13y ago

Looks like a tangled mess of a decision engine. Why would anyone want this? Surely, there are other solutions out there that are more mature? (nagios comes to mind but is a poor example)

rch 13y ago
Any and all answers to this question would be appreciated, even those that just qualify as 'promising' and 'active' (as opposed to mature). All I know of is nagios really.
- ojilles 13y ago
  http://sensuapp.org/ http://sensuapp.org/ and http://riemann.io/ http://riemann.io/
hamburglar 13y ago
The fact that you're labeling the only alternative you can think of "a poor example" seems significant. Edit: and I agree that nagios is a poor example, and welcome newcomers.

beck5 13y ago

Could someone please list the equivalent systems this is could replace? (linux or windows)

viraptor 13y ago
I see parts of cacti, nagios/icinga + graphing, munin, zabbix, sentry (exceptions/errors), shinken, etc. There are many similar projects, but nothing so much integrated as far as I know.

emillon 13y ago

I understand that it's not worth the cost to cleanup the history, but it's always frustrating to see such a project come with a 100kloc "initial" commit.

crazygringo 13y ago
Interesting. Why? What would be valuable about the commit history for you?
- kingkilr 13y ago
  The same things that are valuable about any commit history?
  - meowface 13y ago
    I would imagine that if you're a random person only just finding the project now and aren't already involved in it (by working for or with Stack Exchange), then you probably wouldn't get all that much value out of the commit history. Feel free to dispute that, though. There may be some scenarios I'm not considering.
    - tedunangst 13y ago
      if (x < 42 && y > 1776) useConfabulator = false; Why? Who knows? People don't comment their code for many reasons, not least of which is that it's not technically required and you can easily put it off and forget. Every checkin generally requires at least some comment. You can still enter meaningless messages like "fix", but you're not liable to do so by accident.
- emillon 13y ago
  I think that the commit history says a lot about the development process and the company culture. First, is that it's interesting to see how these companies work internally. Second, it's easier to dive into the code by reading the latest commits. You can see what's going on and what pieces are changed together. Finally, there is a substantial difference between: - a repository that's handled as a first class "source code ledger" (pick one or several of: proper branching/merging, atomic commits, issues referenced in commit messages, etc). - a repository full of "git commit -a -m 'ill fix this bug later, TGIF'" whose history makes no sense. I don't want to open the debate on which one is preferable or not (most people agree that git bisect is handy, but it is open to debate whether it's worth the cost). What I'm wondering is how it looked and why it was rewritten. The most probable reason is that they wanted to make sure that nothing sensitive was exposed (non-redistributable embedded dependencies, information on their infrastructure, ...). But it's also possible that this was started as a type-2 repository and they don't want to expose this image. I tend to think that every code should be written (and developed) as it was open sourced one day, but as always there's a cost trade-off in here. Anyway, thanks for open sourcing this piece of software!

toyg 13y ago

Looks nice, sadly it's just overkill for what I need. Has anyone built a very simple solution to start/stop an arbitrary set of Windows services across several boxes, in a specific order? It'd be nice to have a simple GUI for this sort of thing. I've started working on it, but I suck at desktop programming (well, at programming in general, probably)...

gecko 13y ago
Unless you really need a GUI, it's insanely easy to do exactly this with PowerShell workflows (http://technet.microsoft.com/en-us/library/jj134242.aspx http://technet.microsoft.com/en-us/library/jj134242.aspx).

elwell 13y ago

Wow, they weren't kidding when they said .NET

rdtsc 13y ago

It is so strange to look at that C# code. I know they use it, it is just looks kind of odd compared to the rest of young (I mean less than 5 year old) web companies. No Ubuntu, nginx, node, jvm, but instead C#. I don't know, it just stands out.

saneshark 13y ago

Stackoverflow runs Microsoft?!? I feel like I was just punched in the gut by a best friend. Throwing up...

stevepotter 13y ago

The anti .net kneejerk reactions on HN really disturbs me. You spend all day on stackoverflow then blindly bash their tech stack. Developing, deploying, and hosting .net apps is just fine. Many brilliant people choose .net and are plenty happy with it. Maybe rather than jumping to conclusions, you could give it a shot.

rurounijones 13y ago
What anti .net kneejerk? At time of this comment there are 28 comments, only 2 of which are anti .net / Microsoft (with no replies) and they are already well on their way to being downvoted into oblivion.
Styck 13y ago
Many of us have .NET experience, really like C# but have sensible reasons for not recommending it. A Microsoft only stack is nowhere near as flexible as Linux and if you were to reach Stack Overflow's levels of traffic you'll not be able to avoid the need to run a part of your infrastructure on Linux. I seriously doubt Stack Overflow is running Redis, ElasticSearch & HAProxy* on Windows. Unless you already have a team proficient in .NET, there are few good reasons to use it instead of a JVM language. *https://github.com/opserver/Opserver/tree/master/Opserver.Core/Data https://github.com/opserver/Opserver/tree/master/Opserver.Co...
- garethadams 13y ago
  I imagine the Stackoverflow guys are familiar with Linux architectures, they also run the open source forum Discourse[1] which is a Ruby on Rails app that comes with a recommendation to set up and develop in a *NIX environment[2]. Certainly the point is to use the right tool for the job, and I wouldn't dismiss your final comment: having a team proficient in .NET is a great reason to build your tools with/for that platform. But no, I don't think it would be fair to reply to this article saying "Why didn't they build this for [my preferred language]?" [1]: http://www.discourse.org/ http://www.discourse.org/ [2]: http://blog.discourse.org/2013/04/discourse-as-your-first-rails-app/ http://blog.discourse.org/2013/04/discourse-as-your-first-ra...
- 13y ago

rajbala 13y ago

They lost me at IIS.