3 ms·

What happens when a program logs messages faster than the log daemon can process them? Since it's non-blocking, the program no longer pauses to wait for it to c

by oofabz 13y ago

What happens when a program logs messages faster than the log daemon can process them? Since it's non-blocking, the program no longer pauses to wait for it to catch up. Does it drop messages in this situation? Is that why you warn that "liblogfaf should not be used in an environment where reliable log message delivery is required."? That is probably the most elegant solution.

pilif 13y ago
Because the log messages are being sent as UDP packets, what's going to happen is that the receiving daemon's socket buffer fills up and the kernel drops packets. Thus you lose log messages. It does not, however, affect the application that is sending the log messages because sending the UDP packet is always possible (minus other kernel issues). Depending on your use-case you might want not start dropping log messages when the load is high enough for a log processor to not be able to process them. My reasoning: At that point, something is seriously wrong and according to Murphy, the message that's going to be dropped is the one that contains the actual reason for the issue amidst all the other log spam caused by the real issue.
- IgorPartola 13y ago
  What's more, the (receiving) kernel in that case will drop the newest packets, not the oldest. For logging that might be fine, but I worked on at least one project where we had updates coming in every second to the same piece of information (telemetry). There, you want to drop the oldest packets. In fact, you want to drop all packets by the absolute newest at the time you are processing it. The solution of course was to make accepting datagrams and processing them two asynchronous tasks (well, really three: receive, process, write to the data store).
Eiwatah4 13y ago
When there are too many messages, rsyslog will drop them in the default configuration, too. http://www.rsyslog.com/tag/systemlogratelimitinterval/ http://www.rsyslog.com/tag/systemlogratelimitinterval/

by oofabz 13y ago

pilif 13y ago

Because the log messages are being sent as UDP packets, what's going to happen is that the receiving daemon's socket buffer fills up and the kernel drops packets. Thus you lose log messages. It does not, however, affect the application that is sending the log messages because sending the UDP packet is always possible (minus other kernel issues). Depending on your use-case you might want not start dropping log messages when the load is high enough for a log processor to not be able to process them. My reasoning: At that point, something is seriously wrong and according to Murphy, the message that's going to be dropped is the one that contains the actual reason for the issue amidst all the other log spam caused by the real issue.

IgorPartola 13y ago
What's more, the (receiving) kernel in that case will drop the newest packets, not the oldest. For logging that might be fine, but I worked on at least one project where we had updates coming in every second to the same piece of information (telemetry). There, you want to drop the oldest packets. In fact, you want to drop all packets by the absolute newest at the time you are processing it. The solution of course was to make accepting datagrams and processing them two asynchronous tasks (well, really three: receive, process, write to the data store).

Eiwatah4 13y ago

When there are too many messages, rsyslog will drop them in the default configuration, too. http://www.rsyslog.com/tag/systemlogratelimitinterval/ http://www.rsyslog.com/tag/systemlogratelimitinterval/