4 ms·
1) Log to local disk (most people will tell you this is bad practice and that you should directly log to socket or whatever, but it's more likely for your netwo
by wenc 7y ago
1) Log to local disk (most people will tell you this is bad practice and that you should directly log to socket or whatever, but it's more likely for your network to be down than for your disk to fail).
In Python, use the RotatingFileHandler to avoid running out of space.
2) Incrementally forward your log files to a server using something like fluentd that can pre-aggregate and/or filter messages.
Big advantage of logging to disk: if logging server is unreachable, forwarder can resume once it's up again. If you log directly over network, if things fail the very log messages you need to troubleshoot the failure are potentially gone.
3) Visualize. Create alerts.
I've evaluated a bunch of logging solutions. Splunk is the best, and affordable at low data volumes (they have a pricing calculator, you can check for yourself). It's medium hard to setup.
Sumo Logic is the easiest to set up, and at low data volumes, prices are similar to Splunk. You can get something working within an hour or less.
ELK stack is free only in bits but not in engineering time.
I've not actually tried Sentry.io but I saw it at PyCon and it looks pretty impressive. If you only care about tracking errors/events and not about general-purpose logging functionality per se, I would take a serious look at it.
- linsomniac 7y agoI set up Sentry a few years ago, based on a PyCon hallway-track talk I saw. We had a few false starts with it, but have integrated it with a couple of our newer platforms and have liked it. It takes a kind of "ticket" approach to messages, it'll deduplicate and combine similar errors, and you go into a dashboard and see "We got ten thousand of this error, let me track it down, fix it, ack it and see if we keep getting it."
- caseyf7 7y agoOne should keep in mind that ELK is quite expensive once you reach the point where you need to pay for it.
- viraptor 7y agoSentry is not for logging really. It's designed for errors/exceptions. I believe you should use rollbar/sentry/airbrake regardless of whether you use centralised logging. Or even before it.
- penagwin 7y ago+1 for sentry. You can actually self host it for free too!
- LifeIsBio 7y agoAnother +1 for sentry. I used it (basically accidentally) for a year because it's part of the cookiecutter-django framework[1]. I don't think there was a single issue with it the whole time we used it. [1] https://github.com/pydanny/cookiecutter-django https://github.com/pydanny/cookiecutter-django
- tuco86 7y ago+1. Sentry is amazing. We self host although they stopped shipping updates for over a year until recently. I'd pay for saas if it was up to me, tho.
- codemac 7y ago> it's more likely for your network to be down than for your disk to fail For most people, the network being down means they can't reach the disk. Buffering unsent logs via local disk or RAM is critical due to network flakiness for sure, but not logging over the network as well is a bad idea 100% of the time.
- scarface74 7y agoHe specifically mentioned mobile apps. When developing mobile apps, you almost always operate in a “semi connected” state where ideally you can function without network access and rely on syncing.
- aflag 7y agoHe's actually working in the backend, not in the mobile app itself.
- madhadron 7y ago> For most people, the network being down means they can't reach the disk. If you're talking about cloud solutions, that's what instance stores are for. Logging to local disk and then forwarding is still the best answer. And there's still a lot of world out there that's running on actual hardware.
- mandeepj 7y ago> network to be down than for your disk to fail If network is down then how your users are going to reach your app?
- GauntletWizard 7y agoNetsplits happen. User actions can be queued on a server that has just lost connectivity. Your logging server can be down or broken itself. A file on a unix system isn't going anywhere fast.
- MuffinFlavored 7y agoIt blows my mind that there isn't an easily Googable example of how to pipe JSON logs on disk to logstash. It's 2019. The software should be able to come online, check what logs it has already indexed, and then `tail -f` files matching a pattern. I really thought this was a more common use case but it's obvious things went the "directly log to socket" route you mentioned.
- jasonrojas 7y ago“ELK stack is free only in bits but not in engineering time.” — best thing I’ve read all week. Thank you.
- ameyv 7y agoThis is so true!
- envolt 7y agoAfter 3-4 attempts of 7 days over the last 1 year. I finally had it up and running happliy.
- AlchemistCamp 7y agoHow long would it take you to do it again on another project?
- jetru 7y agoWe do a similar thing. Additionally, we use logrotate to move these files into S3, which later get ETLed into Parquet so we can scoop it for long term analytics.
- devonkim 7y agoWith cloud options available for both Elasticsearch and Splunk these days difficulty of setup / ease of use may be better evaluated from the client perspective. I’ve setup both ES and Splunk in the past few years and it’s not terribly different at lower end scale (< 100 GB logs / mo). But currently Splunk is not as good as ELK for metrics and vice verse and with the recent SignalFX acquisition that may change in a couple years but it definitely isn’t now. Also, there’s tons of options for streaming logs to ES besides Logstash including Filebeat which is at least written in Go (Splunk’s forwarders are probably in C given I swear they’ve been mostly the same since the early 2000s).
- hrktb 7y agoElastic cloud is not a great option for beginners, at least not at this point. For instance there is no purge of old indexes out of the box, a lot of configuration options are not available, performance issues are inscrutable. I think it needs at least of few more months or years to be a no-brainer to choose.
- TrickyRick 7y agohttps://sematext.com/cloud/ https://sematext.com/cloud/ Sematext Cloud has been working well for me in a relatively small startup (Couple of thousands active users). Cheap solution and easy to get started.