8 ms·

One thing this glances over is that you should have something monitoring your production systems to make sure that they are running correctly. To start with ge

One thing this glances over is that you should have something monitoring your production systems to make sure that they are running correctly. To start with get something to monitor errors/exceptions and email you. To name a few services: https://airbrake.io/ https://airbrake.io/ https://rollbar.com/ https://rollbar.com/ https://github.com/errbit/errbit https://github.com/errbit/errbit (can be hosted on Heroku for free) Also make sure that you have accessible logs that log useful information (timestamps, the user making the request, unique request ID). Then use syslog or a SaaS service to aggregate logs from all servers in one place, and keep them for as long as you can.

mwarkentin 11y ago
https://getsentry.com/ https://getsentry.com/ is another option (my favourite) for exception monitoring.
- js2 11y ago
  It's also open-source https://github.com/getsentry/sentry https://github.com/getsentry/sentry
  - sytse 11y ago
    We use it for GitLab.com and love Sentry.
nicko84 11y ago
I'd also add https://raygun.io https://raygun.io as great option here. Those services are more for the DIYer / startup market but if you need a professional error monitoring tool I recommend you check out Raygun.
- detaro 11y ago
  You are a raygun employee? If yes, please disclose that if you "recommend" a product.
  - pc86 11y ago
    Agreed. It's pretty spammy otherwise, especially with one comment in nearly two years and just a few Raygun-specific posts.
    - kuschku 11y ago
      You should see what the Nylas team does on here, or the ABP guys. Lots of spam, never announcing they're promoting their own product (until you call them out on it, then they admit it, at least), and both like to lie.
  - michaelmior 11y ago
    Same username, Technical Marketer at Raygun https://growthhackers.com/members/nicko84 https://growthhackers.com/members/nicko84
    - ChristianBundy 11y ago
      Ouch. Definitely staying the hell away from those people.
- deleted 11y ago
  [deleted]
ultramancool 11y ago
Great point - a small note though, for those of us who don't like using other people's hosted services and have some of our own infrastructure to host - use Zabbix, Nagios, Cacti, Spiceworks, Munin, PRTG, etc. Lots of nice options. I know everyone loves cloud services these days, but I feel absolutely no temptation to put any of my business infrastructure in other people's hands when I don't absolutely have to. I use a combinations of Zabbix and a very agressive Smokeping mesh deployed with docker (10 pings per minute from all separate DCs all other DCs) to monitor our worldwide nodes and be able to account for routing problems on the backbone rather than our nodes themselves. Very handy tool, particularly if you work with latency sensitive applications. I've been surprised some of the datacenters we use appear not to have access to similar...
- vacri 11y ago
  I wouldn't describe Nagios a 'nice' option. Every time I have to install or reconfigure it, it opens the wounds afresh (and I only have a relatively small deployment) :)
  - tachion 11y ago
    Go for Sensu, you wont look back and it has terapeutic powers, when it comes to monitoring ;)
  - scurvy 11y ago
    Hard to have a discussion about open source monitoring without mentioning OpenNMS. Best monitoring back end by far.
Mandatum 11y ago
The ELK stack is also good but will require some config - you'll need to pay for a subscription for alerts however there are free alternatives like Yelp's ElastAlert on Github.
- orthecreedence 11y ago
  Personally I found the EHK (ES, Heka, Kibana) stack much nicer than ELK. I especially like that each node can run Heka locally and ship directly to ES without having some intermediary server we have to maintain. On top of that, Heka is great for general event streams and can grow quite a bit more into your infrastructure as a general tool than logstash (at least last time I used logstash, which was a few years ago). Heka can also monitor and send you alerts via email ("Hey this server got 3 exceptions in the past 5 minutes!") on certain conditions. We do use ES as a core part of our infrastructure, so our only real barrier was setting up Heka.
brightball 11y ago
I'll vouch for Airbrake. It does an excellent job of identifying patterns and grouping similar errors, tracking resolution across deploys, recording request context to help you figure out exactly what was going on while avoiding spamming you with error messages. I haven't had much of a need to look at logs since I started using Airbrake three years ago. For the price, it's really hard to beat the value when you compare the price to various log aggregation systems.
sync 11y ago
Another error tracking system for your list: https://bugsnag.com/ https://bugsnag.com/ It has been rock solid for us with no event limits and cheap pricing. (not an employee, just a fan)
- maslam 11y ago
  Same here, we just started using it and are happy with it!
bbrazil 11y ago
One of my clients got Graylog up and running pretty quickly. Logs are only one part of monitoring though, and it's easy to miss the wood for the trees if you're drinking from the logs firehose. Some metrics monitoring is also vital, naturally I recommend http://prometheus.io http://prometheus.io as I'm one of the core developers :) https://blog.raintank.io/logs-and-metrics-and-graphs-oh-my/ https://blog.raintank.io/logs-and-metrics-and-graphs-oh-my/ discusses this a bit more.
jly 11y ago
A great list of resource - glad I ran into this post! I have rolled my own log scraping and exception monitoring and notification, but some of services do a lot more. I have been using https://www.pingdom.com https://www.pingdom.com for uptime monitoring and some basic perf/latency tracking. It works well and doesn't cost a lot, plus it can hit my services from multiple regions.
kuschku 11y ago
> timestamps, the user making the request, unique request ID […] keep them for as long as you can That's a sure way to go to jail. Don't store which IP requested what for more than necessary — in many countries, there are limits if a few months on how long you may store metadata.
- teraflop 11y ago
  Can you elaborate? I find this incredibly hard to believe. If it's true, you'd think it would be widely-known and publicized in those countries (whichever ones they are) but I can't find any reference to such a policy existing anywhere.
  - detaro 11y ago
    Germany (and probably other countries n the EU as well) considers IP addresses PII, and you have to have consent to store it. Short-term logs might be justifiable as technical necessity (but the lawyers I've heard talk about it didn't want to put a "safe" time frame on that), but certainly not "keep as long as possible". That's the reason Google Analytics, Piwik and others have an option to blank out parts of the IP address. Jail is a bit dramatic though (at least here), unless you do something bad with them fines + maybe damages would be the maximum, and it is quite unlikely you'll actually get hit. But if you are ever in a situation where this would come to light (= already in legal trouble), it could make things a lot more complicated, especially if you process data for other companies.
  - angelbob 11y ago
    It's commonly known in business. I've heard of it repeatedly, usually in terms of "here's what our retention policy has to look like for European customers."
  - rmc 11y ago
    It falls under personal data about someone. Many (but not all) EU countries consider IP addresses to be personal information, and hence "Store it forever for no real reason" is illegal. You need to have a legitimate, legal, reason to store PI, you need the person's unambiguos freely given consent, etc. There is currently a court case as to whether dynamic IP addresses from ISPs counts as personal information. But static IP address probably count. And all the other information (user X logged in at time Y and send message to other user Z) would be PI too.
blister 11y ago
Shameless plug, but my co-founder and I are about to launch our powerful logging service this month. One of our features is session-specific logs and the ability to send alerts at user-defined thresholds. We'd love to get some feedback on our product when we launch to beta, so if you're interested, please sign up for our email list at http://logdebug.com http://logdebug.com
- joshmanders 11y ago
  One suggestion is remove the "Most Popular" banner above Small Business on your pricing. Being as your site is plastered with under construction, it's a turnoff seeing such false advertisement.
  - blister 11y ago
    Sure, good tip. Though the pedantic butt-head in me wants to point out that out of all of our paying customers, they're all using that level of service. :) So technically...
    - johns 11y ago
      So keep it. Ditch the under construction stuff instead if you have customers.
- AznHisoka 11y ago
  Let us know when it's actually out. This is a big pain of mine right now but frustrating I have to sign up to be notified. That pain might cease to exist when you're actually ready
- mholt 11y ago
  Your website is still new, I see, so I imagine it will be more explanatory about how your service works. ;) One thing about the plans that made me cringe was that "SSL" encryption was not available to the Startup plan. I feel like we're entering an era where HTTPS isn't really something that should be offered as a perk, ya know?
akurtzhs 11y ago
I've used https://opbeat.com/ https://opbeat.com/ for a small side project and been fairly happy - they track exceptions, releases and performance.
rmc 11y ago
> keep them for as long as you can. Careful, you can run into Data Protection legal issues with that approach.
- JupiterMoon 11y ago
  Data is a liability as well as an asset.
  - rmc 11y ago
    The operator of Pinboard says that companies should treat personal data like radioactive waste. You should not keep it around, or keep as little as is required/possible, and keep it secure and away from everything else. It's a great way to think of it.,
fizerkhan 11y ago
Let me start with a disclaimer that I work for Atatus. Front-end monitoring is the other end of the circle, that complements production monitoring. With Atatus that is what we have done. We monitoring the front-end for both errors and performance issues - https://www.atatus.com/ https://www.atatus.com/
scurvy 11y ago
syslog and SEC is a good place to start before deciding to ship all your logdata to a 3rd party.

mwarkentin 11y ago

https://getsentry.com/ https://getsentry.com/ is another option (my favourite) for exception monitoring.

js2 11y ago
It's also open-source https://github.com/getsentry/sentry https://github.com/getsentry/sentry
- sytse 11y ago
  We use it for GitLab.com and love Sentry.

nicko84 11y ago

I'd also add https://raygun.io https://raygun.io as great option here. Those services are more for the DIYer / startup market but if you need a professional error monitoring tool I recommend you check out Raygun.

detaro 11y ago
You are a raygun employee? If yes, please disclose that if you "recommend" a product.
- pc86 11y ago
  Agreed. It's pretty spammy otherwise, especially with one comment in nearly two years and just a few Raygun-specific posts.
  - kuschku 11y ago
    You should see what the Nylas team does on here, or the ABP guys. Lots of spam, never announcing they're promoting their own product (until you call them out on it, then they admit it, at least), and both like to lie.
- michaelmior 11y ago
  Same username, Technical Marketer at Raygun https://growthhackers.com/members/nicko84 https://growthhackers.com/members/nicko84
  - ChristianBundy 11y ago
    Ouch. Definitely staying the hell away from those people.
deleted 11y ago
[deleted]

ultramancool 11y ago

Great point - a small note though, for those of us who don't like using other people's hosted services and have some of our own infrastructure to host - use Zabbix, Nagios, Cacti, Spiceworks, Munin, PRTG, etc. Lots of nice options. I know everyone loves cloud services these days, but I feel absolutely no temptation to put any of my business infrastructure in other people's hands when I don't absolutely have to. I use a combinations of Zabbix and a very agressive Smokeping mesh deployed with docker (10 pings per minute from all separate DCs all other DCs) to monitor our worldwide nodes and be able to account for routing problems on the backbone rather than our nodes themselves. Very handy tool, particularly if you work with latency sensitive applications. I've been surprised some of the datacenters we use appear not to have access to similar...

vacri 11y ago
I wouldn't describe Nagios a 'nice' option. Every time I have to install or reconfigure it, it opens the wounds afresh (and I only have a relatively small deployment) :)
- tachion 11y ago
  Go for Sensu, you wont look back and it has terapeutic powers, when it comes to monitoring ;)
- scurvy 11y ago
  Hard to have a discussion about open source monitoring without mentioning OpenNMS. Best monitoring back end by far.

Mandatum 11y ago

The ELK stack is also good but will require some config - you'll need to pay for a subscription for alerts however there are free alternatives like Yelp's ElastAlert on Github.

orthecreedence 11y ago
Personally I found the EHK (ES, Heka, Kibana) stack much nicer than ELK. I especially like that each node can run Heka locally and ship directly to ES without having some intermediary server we have to maintain. On top of that, Heka is great for general event streams and can grow quite a bit more into your infrastructure as a general tool than logstash (at least last time I used logstash, which was a few years ago). Heka can also monitor and send you alerts via email ("Hey this server got 3 exceptions in the past 5 minutes!") on certain conditions. We do use ES as a core part of our infrastructure, so our only real barrier was setting up Heka.

brightball 11y ago

I'll vouch for Airbrake. It does an excellent job of identifying patterns and grouping similar errors, tracking resolution across deploys, recording request context to help you figure out exactly what was going on while avoiding spamming you with error messages. I haven't had much of a need to look at logs since I started using Airbrake three years ago. For the price, it's really hard to beat the value when you compare the price to various log aggregation systems.

sync 11y ago

Another error tracking system for your list: https://bugsnag.com/ https://bugsnag.com/ It has been rock solid for us with no event limits and cheap pricing. (not an employee, just a fan)

maslam 11y ago
Same here, we just started using it and are happy with it!

bbrazil 11y ago

One of my clients got Graylog up and running pretty quickly. Logs are only one part of monitoring though, and it's easy to miss the wood for the trees if you're drinking from the logs firehose. Some metrics monitoring is also vital, naturally I recommend http://prometheus.io http://prometheus.io as I'm one of the core developers :) https://blog.raintank.io/logs-and-metrics-and-graphs-oh-my/ https://blog.raintank.io/logs-and-metrics-and-graphs-oh-my/ discusses this a bit more.

jly 11y ago

A great list of resource - glad I ran into this post! I have rolled my own log scraping and exception monitoring and notification, but some of services do a lot more. I have been using https://www.pingdom.com https://www.pingdom.com for uptime monitoring and some basic perf/latency tracking. It works well and doesn't cost a lot, plus it can hit my services from multiple regions.

kuschku 11y ago

> timestamps, the user making the request, unique request ID […] keep them for as long as you can That's a sure way to go to jail. Don't store which IP requested what for more than necessary — in many countries, there are limits if a few months on how long you may store metadata.

teraflop 11y ago
Can you elaborate? I find this incredibly hard to believe. If it's true, you'd think it would be widely-known and publicized in those countries (whichever ones they are) but I can't find any reference to such a policy existing anywhere.
- detaro 11y ago
  Germany (and probably other countries n the EU as well) considers IP addresses PII, and you have to have consent to store it. Short-term logs might be justifiable as technical necessity (but the lawyers I've heard talk about it didn't want to put a "safe" time frame on that), but certainly not "keep as long as possible". That's the reason Google Analytics, Piwik and others have an option to blank out parts of the IP address. Jail is a bit dramatic though (at least here), unless you do something bad with them fines + maybe damages would be the maximum, and it is quite unlikely you'll actually get hit. But if you are ever in a situation where this would come to light (= already in legal trouble), it could make things a lot more complicated, especially if you process data for other companies.
- angelbob 11y ago
  It's commonly known in business. I've heard of it repeatedly, usually in terms of "here's what our retention policy has to look like for European customers."
- rmc 11y ago
  It falls under personal data about someone. Many (but not all) EU countries consider IP addresses to be personal information, and hence "Store it forever for no real reason" is illegal. You need to have a legitimate, legal, reason to store PI, you need the person's unambiguos freely given consent, etc. There is currently a court case as to whether dynamic IP addresses from ISPs counts as personal information. But static IP address probably count. And all the other information (user X logged in at time Y and send message to other user Z) would be PI too.

blister 11y ago

Shameless plug, but my co-founder and I are about to launch our powerful logging service this month. One of our features is session-specific logs and the ability to send alerts at user-defined thresholds. We'd love to get some feedback on our product when we launch to beta, so if you're interested, please sign up for our email list at http://logdebug.com http://logdebug.com

joshmanders 11y ago
One suggestion is remove the "Most Popular" banner above Small Business on your pricing. Being as your site is plastered with under construction, it's a turnoff seeing such false advertisement.
- blister 11y ago
  Sure, good tip. Though the pedantic butt-head in me wants to point out that out of all of our paying customers, they're all using that level of service. :) So technically...
  - johns 11y ago
    So keep it. Ditch the under construction stuff instead if you have customers.
AznHisoka 11y ago
Let us know when it's actually out. This is a big pain of mine right now but frustrating I have to sign up to be notified. That pain might cease to exist when you're actually ready
mholt 11y ago
Your website is still new, I see, so I imagine it will be more explanatory about how your service works. ;) One thing about the plans that made me cringe was that "SSL" encryption was not available to the Startup plan. I feel like we're entering an era where HTTPS isn't really something that should be offered as a perk, ya know?

akurtzhs 11y ago

I've used https://opbeat.com/ https://opbeat.com/ for a small side project and been fairly happy - they track exceptions, releases and performance.

rmc 11y ago

> keep them for as long as you can. Careful, you can run into Data Protection legal issues with that approach.

JupiterMoon 11y ago
Data is a liability as well as an asset.
- rmc 11y ago
  The operator of Pinboard says that companies should treat personal data like radioactive waste. You should not keep it around, or keep as little as is required/possible, and keep it secure and away from everything else. It's a great way to think of it.,

fizerkhan 11y ago

Let me start with a disclaimer that I work for Atatus. Front-end monitoring is the other end of the circle, that complements production monitoring. With Atatus that is what we have done. We monitoring the front-end for both errors and performance issues - https://www.atatus.com/ https://www.atatus.com/

scurvy 11y ago

syslog and SEC is a good place to start before deciding to ship all your logdata to a 3rd party.