7 ms·

Homegrown DevOps Tools at Stack Exchange

144 points by KyleBrandt 13y ago

pionar 13y ago
I wonder how much investment they have in this vs. going with a pre-existing monitoring system like ExtraHop?
- KyleBrandt 13y ago
  I wasn't aware of extrahop so I will have to look into that. We currently use Solarwinds Orion which is where status gets some of its data. We have definitely outgrown Orion, and a lot of stuff in Orion is very rough, sloppy, and not well integrated. We don't really like the idea of cloud hosted monitoring (which is a lot of what more modern monitoring systems are). Alternatives also seem very expensive. So if we are going to make a investment (cash or labor) I would rather we get a system that fulfills all of our needs (fit) and share it with everyone.
  - dvanduzer 13y ago
    Had you tried Nagios and Puppet/Chef when you started developing these? The logging tool is the only one I'm not sure about, if you're parsing 1200 events a second. I'm not familiar enough with Orion to understand why it was insufficient. Would you say these in-house tools are primarily about integrating / presenting the data, or are there custom parts doing more heavy lifting too?
    - KyleBrandt 13y ago
      We use Puppet a lot and are generally quite happy. We have recently started leveraging Hiera which is addressing our main issue we had with Puppet: Reusability problems in the code since Modules (code) and our data were not easily separated. A lot of these tools are about integrating / presenting but that isn't entirely the case. Status does its own polling of things like Redis and SQL. Realog parses the data, structures it in Redis etc, and the patching dashboard can kick of updates.
      - dvanduzer 13y ago
        Status handles the scheduling of the polling, or the actual polling, or both? One thing Nagios handles well, although it isn't exactly well polished, is distribution, scheduling, and aggregation of the polling. Also, anecdotally, "nobody" seems to know how easy it is to do simple-intermediate monitoring of MS-SQL in Nagios. I happen to be using this, at the moment: https://github.com/scot0357/check_mssql_collection https://github.com/scot0357/check_mssql_collection edit: And thank you for the concise explanation of Hiera. I had entirely ignored it as just another add-on as Puppet "goes enterprise"
    - stevenjmurawski 13y ago
      We tried Nagios a while back, before going to Orion. Nagios has some great structural ideas, but fell down in other areas (don't know all of them, that was before I came on board). We use Puppet to manage our Linux infrastructure and are testing Desired State Configuration for our Windows systems. Puppet and DSC solve a different problem than monitoring. Our tools span the gamut of aggregating and presenting data, to doing "heavy lifting" of things like installing patches, managing our load balancers, and removing bad query plans from our database servers. Our in-house tooling fills in gaps where existing tooling wasn't responsive enough or made it difficult to deal with certain edge cases. General purpose tools like Orion satisfy 80% of our use cases, but we are fanatical about performance and functionality so we want to fill in that additional 20%. In fact, the majority of our Orion monitors are custom script monitors, which we have to create ourselves as it is. If what we do can work for others (I've worked in several environments with Orion and had the same issues in each environment), that's a bonus. We've found we are building a significant number of these projects and that's why we are looking for a dedicated developer for our team.
      - dvanduzer 13y ago
        I'll have to look at the overlap between Puppet's Windows tools and DSC. These are definitely distinct from monitoring, but monitoring configuration should be populated from the same configuration store as Puppet. (Sometimes Puppet is the configuration store.) I've never evaluated Orion in particular, but it's slightly puzzling if you're creating a lot of custom monitoring scripts. In the low-touch Nagios deployments I've seen, this is often because someone didn't understand good places to use macros, and centralize more of the parameters.
        jacques_chester 13y ago
        The problem is due to history. Configuration systems have historically looked at config as bits-on-a-disk. Supervisor systems look at bits-in-memory. And they emerged and evolved independently. So there's pain points and impedance mismatches regardless of which one you start with. What's needed is a system that sees that configuration and supervision are the same problem: you have a directed graph of what a system can look like, plus a compare-and-repair mechanism to drag the system to that state frequently. I've personally looked at Chef, Puppet and Cfengine, none of which really does all of them the way I would like. http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reigning-in-chaos/ http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reign...
  - berkay 13y ago
    "We don't really like the idea of cloud hosted monitoring" Can you elaborate on the reasons?
    - DougWebb 13y ago
      I'll bet one of the reasons is the amount of data. At a previous employer I built a system similar to this which produced nightly, weekly, monthly, and annual reports of weblog analysis for an app that had 6 million+ http requests per day. It was tough enough to consolidate those logs across the load-balanced servers in a single data-center. We never consolidated across data-centers (separate reports for each instead) and I doubt we could have shipped all of that data to a cloud service in a reasonable amount of time. BTW, we made heavy use of setting and logging http headers too. One trick I liked was capturing performance timing metrics as a request was processed and stuffing it into a response header as the response went out. We then logged the response headers, which gave us the ability to report on the performance metrics. We also had a debug mode in the app on the browser side so we could see the performance metrics from the headers there too.
      - dvanduzer 13y ago
        When was this? ~70 events per second doesn't sound like much to capture and aggregate. How much of this parsing did you need to perform in real time? Creating a unique token to pair requests/responses shouldn't add much overhead at all.
        DougWebb 13y ago
        Development started around 2000, and stabilized around 2008. As far as I know the reporting scripts are still being run every day. During this period we had purchased a 1TB storage rack from EMC for a million dollars, to give you some perspective on the differences between then and now. - No real-time parsing; it's all nightly batch processing after devops rotates the Apache server logs to a storage volume. The logs sit there for a while then get compressed and moved to offline tape archives. - No DB storage of the logs; space was too expensive and the Oracle database we had couldn't have kept up. It was already heavily burdened with a completely separate usage statistics system that fed into user-facing reporting and billing, which had a much higher event rate, about 100x higher, than the http logs. - We had unique tokens, but they identified a particular user session that tied together all of the user's http requests from login to logoff/abandonment, and which also tied into the Oracle-based statistics for that user, that user's organization, and the customer responsible for the user (often multi-organization). My reports had breakdowns for individual user experiences, session-level metrics, and user type/organization/customer/region/etc metrics. - I don't recall how long the analysis took; it was between half an hour to two hours I think. A lot of that time was spent on disk I/O reading the logs. I had optimized the parsing, analysis, and results recording about as much as I could. - This stuff was written in Perl, and ran on Solaris servers from that time era... probably not a lot more powerful than a handful of smartphones today, though they did have lots of cpus. I don't think traffic has grown much since I left the company (we had pretty full market penetration already) so it's likely those servers haven't been upgraded.
stephengillie 13y ago
That dashboard looks really neat. I've been searching for a good Windows dashboard, and I like the patching views. Where's the download link?
- RossM 13y ago
  As it says at the bottom of the article, it's unreleased until they can make it more generic.
- KyleBrandt 13y ago
  None of this is open sourced yet. Nick is investing a lot of time to get what we currently call "status" ready to open source. Part of the reason for the open position linked to in the post ( http://careers.stackoverflow.com/jobs/39983/developer-site-reliability-team-stack-exchange http://careers.stackoverflow.com/jobs/39983/developer-site-r... ) is to make it so we have more manpower to get this stuff open sourced.
lifeisstillgood 13y ago
This oddly sounds like a death knell for Windows. I am not seeing anything here that is not in the LAMP / OSS stack as standard (graphite, nagios / Munin) If we rephrase the blog post as "we could not find any good tools in the Windows devops space so we wrote them" and add it to the departure of the only CEO willing to dance on stage chanting developers developers developers and Windows is not an ecosystem but a hub with a few brave outlying satellites. I am impressed by the stacke change folks and their story and skills but it feels like amazing stories of software skill written for one company and never released in the open - it just leaves no legacy
- jc4p 13y ago
  We (can I say we? I don't work on Windows stack stuff at here but whatever) are actually one of the biggest open source contributors in the .NET world, granted that's just my opinion and I don't have data behind it, but here's a list of things we've open sourced: http://blog.stackoverflow.com/2012/02/stack-exchange-open-source-projects/ http://blog.stackoverflow.com/2012/02/stack-exchange-open-so... MiniProfiler is my favorite one, and now exists in .NET, Ruby and Node.js due to my insanely smart co-workers.
  - lifeisstillgood 13y ago
    Please don't take it as a criticism of your work - but the fact that one small company (<100?) that does OSS as a side effect (plus recruitment retention effort) can be one of the largest OSS contributors in the dotnet ecosystem says its not a vibrant and thriving ecosystem. SE has a very high reputation as far as I can tell so its not a quality issue - it is in this case a problem of quantity. As someone commented the idea is to focus on Azure and mobile - but that is no substitute for lots and lots of developers in a culture of releasing stuff outside their immediate company and so driving each other to new heights
    - CodeCube 13y ago
      The snowball has been growing here for the last few years. The .NET community was historically not very "open" ... but ever since nuget and github, attitudes have definitely been improving.
cllns 13y ago
An interesting consulting niche would be to help companies open source software they want to release. Basically: - clean up code, - make sure the infrastructure is sufficient, - help with marketing and adoption, - write documentation
- coolsunglasses 13y ago
  Very few companies want to pay for that, fewer still when in-house and third party devs are often begging to do that work if they'd just let them license the code as OSS.
- VikingCoder 13y ago
  I see it more likely as getting someone like the Stack Exchange guys agree to open source it, if we pay them enough. A Kickstarter project, for instance.
jmelloy 13y ago
Where does the patching dashboard pull data from? Is it tracked by hand or is there a scanner? We use Orion at work, and it's got a decent amount of data in it, but is kind of kludgy and slow.
- stevenjmurawski 13y ago
  The patching dashboard has scheduled jobs on the clients (PowerShell scripts on the Windows boxes and Ruby scripts on the Linux boxes). We use Puppet to deploy the client to our Linux boxes and for Windows we deploy the task and scripts with Group Policy (soon to be replaced by Desired State Configuration). This information isn't something we need to poll often for (and on Windows, there are difficulties interacting with the Windows Update apis remotely). We add and replace servers often, so the client adds themselves to the dashboard, as well as updating their status. We have integrated some reporting into Orion, but that was a side effect of not having the dashboard before.
benjaminwootton 13y ago
It's always nice to see new products in the DevOps space, but be careful not to re-invent the wheel if you do this kind of stuff as the open source world is coming on leaps and bounds. LogStash, ElasticSearch and Kibana are a great open source stack for log management. StatsD and Graphite are nice tools for metric tracking and visualization. There are lots of open source dashboard offerings which combined with a bit of scripting can get you far. You are also spoiled for choice with SAAS monitoring stuff such as NewRelic and Server Density, even if the OP isn't a fan of cloud based tools.
- KyleBrandt 13y ago
  I did an experiment with logstash, elasticsearch, and Kibana for our HAProxy logs. The default with logstash was to store each field, and then the whole text so stuff go quite large for our web logs. Also the Kibana interface was pretty buggy. Parsing our logs (~2k entries) a second doesn't work well with Regex so in our version we do a bunch of substring stuff. I'm excited about Kibana/ES for the rest of our logs over with their recent hire. When I looked at StatsD and Graphite last time I didn't really see an API. I really like the model of data being queryable and returning nice serialized format like json (like OpenTSDB does). I'm also not that fond of the "many files" model and the automatic data summarization as it ages (it does save space, but makes forecasting difficult as it can skew data).
  - zwily 13y ago
    We're parsing 2k lines/sec with logstash using regexes. We scaled it out to 6 logstash processes across 2 nodes. They pull off a shared redis queue and then insert the results directly into elasticsearch. (That said, I'd like to configure our apache logs to just output the json logstash expects.) Graphite has a very simple and powerful JSON API [1]. Any graph URL can include &format=json and you'll get back the raw JSON values for the datapoints. I haven't used OpenTSDB yet - I'm curious how its API is better. You get to choose the levels of summarization. If you want to keep 1 second intervals for a year, you're welcome to do so. [1] http://graphite.readthedocs.org/en/0.9.12/render_api.html http://graphite.readthedocs.org/en/0.9.12/render_api.html
WestCoastJustin 13y ago
I think Stack Exchange has a secret weapon which will likely greatly improve their backend systems. Tom Limoncelli, who used to work at Google as a Site reliability engineer (SRE), now works at Stack Exchange [1]. He pretty much wrote the bible for sysadmins entitle "The Practice of System and Network Administration" [2]. I wouldn't be surprised if we start seeing more posts like this! [1] http://everythingsysadmin.com/2013/09/the-team-im-on-at-stack-exchan.html http://everythingsysadmin.com/2013/09/the-team-im-on-at-stac... [2] http://www.amazon.com/dp/0321492668/tomontime-20 http://www.amazon.com/dp/0321492668/tomontime-20
- Fuxy 13y ago
  Wow that's cool I would love to work with those guys wouldn't be much help though only have basic Linux administration skills.
  - gsands 13y ago
    Keep learning and soon you will be beyond basic. If you have passion, you will have plenty to offer.
jamra 13y ago
I am curious to how they deploy database schema and stored procedure updates. That is a much harder problem.
- JasonPunyon 13y ago
  We have a forwards only migration runner that takes care of the deployment for us. We don't use many stored procedures, but they're taken care of the same way (in a migration). Our policy is that pushed code be backwards compatible by however many migrations are being deployed in a production build.
stevenjmurawski 13y ago
jmelloy (https://news.ycombinator.com/item?id=6334778 https://news.ycombinator.com/item?id=6334778) had a really good point that wasn't covered in the post, since it was a post about what we are doing not necessarily why we are doing it. One of our major problems with existing monitoring and management systems is the lack of good APIs. We are a shop of developers and sysadmins who all understand the real management systems need to be composable. The system needs to understand that it won't solve every case out of the box and expose hooks into management and functionality, allowing us to tie disparate systems together and enhance their coverage. I'd rather take a bunch of existing products and put some cool dashboards on top, but most enterprise solutions (and some of the open source ones) don't offer a decent API to work with.
- dvanduzer 13y ago
  It's quite likely that I'm confusing "decent API" and "ease of extending and integrating" and all that gets wrapped up with my long term familiarity with Nagios. Nagios check plugins don't have an API per se, but they have a very simple standard for exit codes to be interpreted by the Orchestration component. Nagios reporting/alerting plugins primarily use RRD, so you always have that API to do interesting things like trend analysis. (One theory I have is that there is a such a large and growing culture of technologists that are thinking "integration tool" but only know how to say "HTTP API")
dredmorbius 13y ago
OK, I'll be that guy. This makes that page much easier to read. Especially the headings, which are all mashed together for some reason... #content { margin-left: auto; margin-right: auto; float: none; width: 40em; font-size: 15pt; line-height: 1.4em; } h1 { font-size: 180%; line-height: 1.2em; margin-top: 2em; margin-bottom: 0.5em; } h2 { font-size: 160%; line-height: 1.2em; margin-top: 2em; margin-bottom: 0.5em; } #wrap { width: 100%; }

144 points by KyleBrandt 13y ago

pionar 13y ago

I wonder how much investment they have in this vs. going with a pre-existing monitoring system like ExtraHop?

KyleBrandt 13y ago
I wasn't aware of extrahop so I will have to look into that. We currently use Solarwinds Orion which is where status gets some of its data. We have definitely outgrown Orion, and a lot of stuff in Orion is very rough, sloppy, and not well integrated. We don't really like the idea of cloud hosted monitoring (which is a lot of what more modern monitoring systems are). Alternatives also seem very expensive. So if we are going to make a investment (cash or labor) I would rather we get a system that fulfills all of our needs (fit) and share it with everyone.
- dvanduzer 13y ago
  Had you tried Nagios and Puppet/Chef when you started developing these? The logging tool is the only one I'm not sure about, if you're parsing 1200 events a second. I'm not familiar enough with Orion to understand why it was insufficient. Would you say these in-house tools are primarily about integrating / presenting the data, or are there custom parts doing more heavy lifting too?
  - KyleBrandt 13y ago
    We use Puppet a lot and are generally quite happy. We have recently started leveraging Hiera which is addressing our main issue we had with Puppet: Reusability problems in the code since Modules (code) and our data were not easily separated. A lot of these tools are about integrating / presenting but that isn't entirely the case. Status does its own polling of things like Redis and SQL. Realog parses the data, structures it in Redis etc, and the patching dashboard can kick of updates.
    - dvanduzer 13y ago
      Status handles the scheduling of the polling, or the actual polling, or both? One thing Nagios handles well, although it isn't exactly well polished, is distribution, scheduling, and aggregation of the polling. Also, anecdotally, "nobody" seems to know how easy it is to do simple-intermediate monitoring of MS-SQL in Nagios. I happen to be using this, at the moment: https://github.com/scot0357/check_mssql_collection https://github.com/scot0357/check_mssql_collection edit: And thank you for the concise explanation of Hiera. I had entirely ignored it as just another add-on as Puppet "goes enterprise"
  - stevenjmurawski 13y ago
    We tried Nagios a while back, before going to Orion. Nagios has some great structural ideas, but fell down in other areas (don't know all of them, that was before I came on board). We use Puppet to manage our Linux infrastructure and are testing Desired State Configuration for our Windows systems. Puppet and DSC solve a different problem than monitoring. Our tools span the gamut of aggregating and presenting data, to doing "heavy lifting" of things like installing patches, managing our load balancers, and removing bad query plans from our database servers. Our in-house tooling fills in gaps where existing tooling wasn't responsive enough or made it difficult to deal with certain edge cases. General purpose tools like Orion satisfy 80% of our use cases, but we are fanatical about performance and functionality so we want to fill in that additional 20%. In fact, the majority of our Orion monitors are custom script monitors, which we have to create ourselves as it is. If what we do can work for others (I've worked in several environments with Orion and had the same issues in each environment), that's a bonus. We've found we are building a significant number of these projects and that's why we are looking for a dedicated developer for our team.
    - dvanduzer 13y ago
      I'll have to look at the overlap between Puppet's Windows tools and DSC. These are definitely distinct from monitoring, but monitoring configuration should be populated from the same configuration store as Puppet. (Sometimes Puppet is the configuration store.) I've never evaluated Orion in particular, but it's slightly puzzling if you're creating a lot of custom monitoring scripts. In the low-touch Nagios deployments I've seen, this is often because someone didn't understand good places to use macros, and centralize more of the parameters.
      - jacques_chester 13y ago
        The problem is due to history. Configuration systems have historically looked at config as bits-on-a-disk. Supervisor systems look at bits-in-memory. And they emerged and evolved independently. So there's pain points and impedance mismatches regardless of which one you start with. What's needed is a system that sees that configuration and supervision are the same problem: you have a directed graph of what a system can look like, plus a compare-and-repair mechanism to drag the system to that state frequently. I've personally looked at Chef, Puppet and Cfengine, none of which really does all of them the way I would like. http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reigning-in-chaos/ http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reign...
- berkay 13y ago
  "We don't really like the idea of cloud hosted monitoring" Can you elaborate on the reasons?
  - DougWebb 13y ago
    I'll bet one of the reasons is the amount of data. At a previous employer I built a system similar to this which produced nightly, weekly, monthly, and annual reports of weblog analysis for an app that had 6 million+ http requests per day. It was tough enough to consolidate those logs across the load-balanced servers in a single data-center. We never consolidated across data-centers (separate reports for each instead) and I doubt we could have shipped all of that data to a cloud service in a reasonable amount of time. BTW, we made heavy use of setting and logging http headers too. One trick I liked was capturing performance timing metrics as a request was processed and stuffing it into a response header as the response went out. We then logged the response headers, which gave us the ability to report on the performance metrics. We also had a debug mode in the app on the browser side so we could see the performance metrics from the headers there too.
    - dvanduzer 13y ago
      When was this? ~70 events per second doesn't sound like much to capture and aggregate. How much of this parsing did you need to perform in real time? Creating a unique token to pair requests/responses shouldn't add much overhead at all.
      - DougWebb 13y ago
        Development started around 2000, and stabilized around 2008. As far as I know the reporting scripts are still being run every day. During this period we had purchased a 1TB storage rack from EMC for a million dollars, to give you some perspective on the differences between then and now. - No real-time parsing; it's all nightly batch processing after devops rotates the Apache server logs to a storage volume. The logs sit there for a while then get compressed and moved to offline tape archives. - No DB storage of the logs; space was too expensive and the Oracle database we had couldn't have kept up. It was already heavily burdened with a completely separate usage statistics system that fed into user-facing reporting and billing, which had a much higher event rate, about 100x higher, than the http logs. - We had unique tokens, but they identified a particular user session that tied together all of the user's http requests from login to logoff/abandonment, and which also tied into the Oracle-based statistics for that user, that user's organization, and the customer responsible for the user (often multi-organization). My reports had breakdowns for individual user experiences, session-level metrics, and user type/organization/customer/region/etc metrics. - I don't recall how long the analysis took; it was between half an hour to two hours I think. A lot of that time was spent on disk I/O reading the logs. I had optimized the parsing, analysis, and results recording about as much as I could. - This stuff was written in Perl, and ran on Solaris servers from that time era... probably not a lot more powerful than a handful of smartphones today, though they did have lots of cpus. I don't think traffic has grown much since I left the company (we had pretty full market penetration already) so it's likely those servers haven't been upgraded.

stephengillie 13y ago

That dashboard looks really neat. I've been searching for a good Windows dashboard, and I like the patching views. Where's the download link?

RossM 13y ago
As it says at the bottom of the article, it's unreleased until they can make it more generic.
KyleBrandt 13y ago
None of this is open sourced yet. Nick is investing a lot of time to get what we currently call "status" ready to open source. Part of the reason for the open position linked to in the post ( http://careers.stackoverflow.com/jobs/39983/developer-site-reliability-team-stack-exchange http://careers.stackoverflow.com/jobs/39983/developer-site-r... ) is to make it so we have more manpower to get this stuff open sourced.

lifeisstillgood 13y ago

This oddly sounds like a death knell for Windows. I am not seeing anything here that is not in the LAMP / OSS stack as standard (graphite, nagios / Munin) If we rephrase the blog post as "we could not find any good tools in the Windows devops space so we wrote them" and add it to the departure of the only CEO willing to dance on stage chanting developers developers developers and Windows is not an ecosystem but a hub with a few brave outlying satellites. I am impressed by the stacke change folks and their story and skills but it feels like amazing stories of software skill written for one company and never released in the open - it just leaves no legacy

jc4p 13y ago
We (can I say we? I don't work on Windows stack stuff at here but whatever) are actually one of the biggest open source contributors in the .NET world, granted that's just my opinion and I don't have data behind it, but here's a list of things we've open sourced: http://blog.stackoverflow.com/2012/02/stack-exchange-open-source-projects/ http://blog.stackoverflow.com/2012/02/stack-exchange-open-so... MiniProfiler is my favorite one, and now exists in .NET, Ruby and Node.js due to my insanely smart co-workers.
- lifeisstillgood 13y ago
  Please don't take it as a criticism of your work - but the fact that one small company (<100?) that does OSS as a side effect (plus recruitment retention effort) can be one of the largest OSS contributors in the dotnet ecosystem says its not a vibrant and thriving ecosystem. SE has a very high reputation as far as I can tell so its not a quality issue - it is in this case a problem of quantity. As someone commented the idea is to focus on Azure and mobile - but that is no substitute for lots and lots of developers in a culture of releasing stuff outside their immediate company and so driving each other to new heights
  - CodeCube 13y ago
    The snowball has been growing here for the last few years. The .NET community was historically not very "open" ... but ever since nuget and github, attitudes have definitely been improving.

cllns 13y ago

An interesting consulting niche would be to help companies open source software they want to release. Basically: - clean up code, - make sure the infrastructure is sufficient, - help with marketing and adoption, - write documentation

coolsunglasses 13y ago
Very few companies want to pay for that, fewer still when in-house and third party devs are often begging to do that work if they'd just let them license the code as OSS.
VikingCoder 13y ago
I see it more likely as getting someone like the Stack Exchange guys agree to open source it, if we pay them enough. A Kickstarter project, for instance.

jmelloy 13y ago

Where does the patching dashboard pull data from? Is it tracked by hand or is there a scanner? We use Orion at work, and it's got a decent amount of data in it, but is kind of kludgy and slow.

stevenjmurawski 13y ago
The patching dashboard has scheduled jobs on the clients (PowerShell scripts on the Windows boxes and Ruby scripts on the Linux boxes). We use Puppet to deploy the client to our Linux boxes and for Windows we deploy the task and scripts with Group Policy (soon to be replaced by Desired State Configuration). This information isn't something we need to poll often for (and on Windows, there are difficulties interacting with the Windows Update apis remotely). We add and replace servers often, so the client adds themselves to the dashboard, as well as updating their status. We have integrated some reporting into Orion, but that was a side effect of not having the dashboard before.

benjaminwootton 13y ago

It's always nice to see new products in the DevOps space, but be careful not to re-invent the wheel if you do this kind of stuff as the open source world is coming on leaps and bounds. LogStash, ElasticSearch and Kibana are a great open source stack for log management. StatsD and Graphite are nice tools for metric tracking and visualization. There are lots of open source dashboard offerings which combined with a bit of scripting can get you far. You are also spoiled for choice with SAAS monitoring stuff such as NewRelic and Server Density, even if the OP isn't a fan of cloud based tools.

KyleBrandt 13y ago
I did an experiment with logstash, elasticsearch, and Kibana for our HAProxy logs. The default with logstash was to store each field, and then the whole text so stuff go quite large for our web logs. Also the Kibana interface was pretty buggy. Parsing our logs (~2k entries) a second doesn't work well with Regex so in our version we do a bunch of substring stuff. I'm excited about Kibana/ES for the rest of our logs over with their recent hire. When I looked at StatsD and Graphite last time I didn't really see an API. I really like the model of data being queryable and returning nice serialized format like json (like OpenTSDB does). I'm also not that fond of the "many files" model and the automatic data summarization as it ages (it does save space, but makes forecasting difficult as it can skew data).
- zwily 13y ago
  We're parsing 2k lines/sec with logstash using regexes. We scaled it out to 6 logstash processes across 2 nodes. They pull off a shared redis queue and then insert the results directly into elasticsearch. (That said, I'd like to configure our apache logs to just output the json logstash expects.) Graphite has a very simple and powerful JSON API [1]. Any graph URL can include &format=json and you'll get back the raw JSON values for the datapoints. I haven't used OpenTSDB yet - I'm curious how its API is better. You get to choose the levels of summarization. If you want to keep 1 second intervals for a year, you're welcome to do so. [1] http://graphite.readthedocs.org/en/0.9.12/render_api.html http://graphite.readthedocs.org/en/0.9.12/render_api.html

WestCoastJustin 13y ago

I think Stack Exchange has a secret weapon which will likely greatly improve their backend systems. Tom Limoncelli, who used to work at Google as a Site reliability engineer (SRE), now works at Stack Exchange [1]. He pretty much wrote the bible for sysadmins entitle "The Practice of System and Network Administration" [2]. I wouldn't be surprised if we start seeing more posts like this! [1] http://everythingsysadmin.com/2013/09/the-team-im-on-at-stack-exchan.html http://everythingsysadmin.com/2013/09/the-team-im-on-at-stac... [2] http://www.amazon.com/dp/0321492668/tomontime-20 http://www.amazon.com/dp/0321492668/tomontime-20

Fuxy 13y ago
Wow that's cool I would love to work with those guys wouldn't be much help though only have basic Linux administration skills.
- gsands 13y ago
  Keep learning and soon you will be beyond basic. If you have passion, you will have plenty to offer.

jamra 13y ago

I am curious to how they deploy database schema and stored procedure updates. That is a much harder problem.

JasonPunyon 13y ago
We have a forwards only migration runner that takes care of the deployment for us. We don't use many stored procedures, but they're taken care of the same way (in a migration). Our policy is that pushed code be backwards compatible by however many migrations are being deployed in a production build.

stevenjmurawski 13y ago

jmelloy (https://news.ycombinator.com/item?id=6334778 https://news.ycombinator.com/item?id=6334778) had a really good point that wasn't covered in the post, since it was a post about what we are doing not necessarily why we are doing it. One of our major problems with existing monitoring and management systems is the lack of good APIs. We are a shop of developers and sysadmins who all understand the real management systems need to be composable. The system needs to understand that it won't solve every case out of the box and expose hooks into management and functionality, allowing us to tie disparate systems together and enhance their coverage. I'd rather take a bunch of existing products and put some cool dashboards on top, but most enterprise solutions (and some of the open source ones) don't offer a decent API to work with.

dvanduzer 13y ago
It's quite likely that I'm confusing "decent API" and "ease of extending and integrating" and all that gets wrapped up with my long term familiarity with Nagios. Nagios check plugins don't have an API per se, but they have a very simple standard for exit codes to be interpreted by the Orchestration component. Nagios reporting/alerting plugins primarily use RRD, so you always have that API to do interesting things like trend analysis. (One theory I have is that there is a such a large and growing culture of technologists that are thinking "integration tool" but only know how to say "HTTP API")

dredmorbius 13y ago

OK, I'll be that guy. This makes that page much easier to read. Especially the headings, which are all mashed together for some reason... #content { margin-left: auto; margin-right: auto; float: none; width: 40em; font-size: 15pt; line-height: 1.4em; } h1 { font-size: 180%; line-height: 1.2em; margin-top: 2em; margin-bottom: 0.5em; } h2 { font-size: 160%; line-height: 1.2em; margin-top: 2em; margin-bottom: 0.5em; } #wrap { width: 100%; }