5 ms·

Ask HN: Best Tools for Log Management?

What is your go to tool for viewing and understanding manually created application logs? I know there are a few tools for high volume of automated logs (Papertrail, CloudWatch, Kinesis), but I am looking for something that can ingest small amount of debugging logs, and then make it searchable, ideally by time, or order of events. For example, I would like to log from the server side: 1- Log("Event 1 happened") 2- Log("Event 2 happened") 3 -Log("Event 3 happened") Then I want to get this information in a list/graph, that shows these flows in much the same way I would see user flow in modern analytics tools. I guess I could use analytics tools, like Mixpanel, but that seems an overkill. What would you use?

02020202 6y ago
tech is irrelevant. all you need is to keep the correlation id, and possibly causation id, in the events and you generate the entire graph of user's movement(or whatever is the subject of the event stream).
- kirillzubovsky 6y ago
  Perfect. Excel FTW. I agree!
backslash_16 6y ago
Really easy and good homegrown logging. Use a structured logging library, this is key because it lets you easily filter on the properties attached to a log event. Then log those to a database. If you don't have a ton of volume and want to use really reliable tech and be safe(ish) send your logs to a relational DB used only for logging. I have never tried sending to just a separate table within the production database but I know that can be done too. Create read-only users on that logging DB or table. Download a GUI client for that DB and you're up and running. You can use SQL to filter, order by timestamp, and create views. If you still want to stay homegrown it's straightforward to put an API in front of it that powers some dashboards or easy investigation tools. For example, submit correlation/request ID and get back every log event in-order for that, color error logs red and boom - 1st level data visualization for investigations done too.
- mattmanser 6y ago
  Yeah, at moderate volumes (or low depending on your perspective) there's no problem putting it straight into a table in the DB. If it ever causes a problem, you can move it. You shouldn't need to order by timestamp, it should already be ordered by the very nature of logs.
  - backslash_16 6y ago
    I completely agree. Because I think this area is really interesting I'm going to add a few more thoughts. The first web service I built used a table inside the production database for structured logging and I had one kind of log event that I just included more and more in over time, using empty fields when needed, which ends up being an anti-pattern like a god class. That wasn't terrible, I could filter for the logs I wanted by including a statement like RequestId IS NULL. The biggest problem I have seen with using a table in the DB and not a completely separate DB, is if it's a traditional relational DB (I'm sort of making up that term to refer to systems like MS SQL Server, PostgreSQL, MySQL) the schema is static, and each table can only contain one type of log or event unless you stuff it all into a dynamic-ish column type like JSON or XML (ewww using XML as a logging format). If you have multiple log formats or types (with the type system meaning of types) you need to use multiple tables to keep the filtering easy. An unintended bonus is that joining across tables in a relational DB is far faster than joining across events types in the other logging systems I have worked with. I guess I'm trying to say you should consider using a separate DB once you have more than a few log types (which are different tables in the prod DB), which still 100% backs up what you said. Start off in the prod DB and move it if and when you need to. In the worst case scenario you can do unstructured logging with like 4 common fields (LogCreationTime, LogIngestionTime, RequestID, HostName) and a message field and use that until it doesn't work anymore, and then you have a service that is complex enough to warrant spending more time on the logging.
tstack 6y ago
Try the Log File Navigator (http://lnav.org http://lnav.org). It works directly with log files. Can understand many log formats by default and be customized. The log messages from separate files are collated into a single view where highlighting is also applied. You can even do analyses on the messages by executing SQLite queries (without having to insert the messages into a database).
aleixventa 6y ago
If you plan to send logs from a mobile app or a javascript frontend app, we created a tool called Bugfender (https://bugfender.com https://bugfender.com) that will help you. You can use it for free up to 100K logs and then we have paid plans to fit almost any budget. For server logs, there are a lot of tools that will help you. I wouldn't recommend you to create your own service, might look easy at the beginning but it ends up being very complex (this is how we started Bugfender, creating a tool for ourselves but it get more and more complex).

02020202 6y ago

tech is irrelevant. all you need is to keep the correlation id, and possibly causation id, in the events and you generate the entire graph of user's movement(or whatever is the subject of the event stream).

kirillzubovsky 6y ago
Perfect. Excel FTW. I agree!

backslash_16 6y ago

Really easy and good homegrown logging. Use a structured logging library, this is key because it lets you easily filter on the properties attached to a log event. Then log those to a database. If you don't have a ton of volume and want to use really reliable tech and be safe(ish) send your logs to a relational DB used only for logging. I have never tried sending to just a separate table within the production database but I know that can be done too. Create read-only users on that logging DB or table. Download a GUI client for that DB and you're up and running. You can use SQL to filter, order by timestamp, and create views. If you still want to stay homegrown it's straightforward to put an API in front of it that powers some dashboards or easy investigation tools. For example, submit correlation/request ID and get back every log event in-order for that, color error logs red and boom - 1st level data visualization for investigations done too.

mattmanser 6y ago
Yeah, at moderate volumes (or low depending on your perspective) there's no problem putting it straight into a table in the DB. If it ever causes a problem, you can move it. You shouldn't need to order by timestamp, it should already be ordered by the very nature of logs.
- backslash_16 6y ago
  I completely agree. Because I think this area is really interesting I'm going to add a few more thoughts. The first web service I built used a table inside the production database for structured logging and I had one kind of log event that I just included more and more in over time, using empty fields when needed, which ends up being an anti-pattern like a god class. That wasn't terrible, I could filter for the logs I wanted by including a statement like RequestId IS NULL. The biggest problem I have seen with using a table in the DB and not a completely separate DB, is if it's a traditional relational DB (I'm sort of making up that term to refer to systems like MS SQL Server, PostgreSQL, MySQL) the schema is static, and each table can only contain one type of log or event unless you stuff it all into a dynamic-ish column type like JSON or XML (ewww using XML as a logging format). If you have multiple log formats or types (with the type system meaning of types) you need to use multiple tables to keep the filtering easy. An unintended bonus is that joining across tables in a relational DB is far faster than joining across events types in the other logging systems I have worked with. I guess I'm trying to say you should consider using a separate DB once you have more than a few log types (which are different tables in the prod DB), which still 100% backs up what you said. Start off in the prod DB and move it if and when you need to. In the worst case scenario you can do unstructured logging with like 4 common fields (LogCreationTime, LogIngestionTime, RequestID, HostName) and a message field and use that until it doesn't work anymore, and then you have a service that is complex enough to warrant spending more time on the logging.

tstack 6y ago

Try the Log File Navigator (http://lnav.org http://lnav.org). It works directly with log files. Can understand many log formats by default and be customized. The log messages from separate files are collated into a single view where highlighting is also applied. You can even do analyses on the messages by executing SQLite queries (without having to insert the messages into a database).

aleixventa 6y ago

If you plan to send logs from a mobile app or a javascript frontend app, we created a tool called Bugfender (https://bugfender.com https://bugfender.com) that will help you. You can use it for free up to 100K logs and then we have paid plans to fit almost any budget. For server logs, there are a lot of tools that will help you. I wouldn't recommend you to create your own service, might look easy at the beginning but it ends up being very complex (this is how we started Bugfender, creating a tool for ourselves but it get more and more complex).