5 ms·
The better way to do this is to write the logs to a file or an in-memory ring buffer and have a separate thread/process push logs from the file/ring-buffer to t
by supriyo-biswas 3y ago
The better way to do this is to write the logs to a file or an in-memory ring buffer and have a separate thread/process push logs from the file/ring-buffer to the logging service, allowing for retries if the logging service is down or slow (for moderately short values of down/slow).
Promtail[1] can do this if you're using Loki for logging.
[1] https://grafana.com/docs/loki/latest/send-data/promtail/ https://grafana.com/docs/loki/latest/send-data/promtail/
- insanitybit 3y agoBut that's still not guaranteed delivery. You're doing what the OP presented - choosing to drop logs under some circumstances when the system is down. a) If your service crashes and it's in-memory, you lose logs b) If your service can't push logs off (upstream service is down or slow) you either drop logs, run out of memory, or block
- hgfghui7 3y agoYou are thinking too much in terms of the stated requirements instead of what people actually want: good uptime and good debugability. Falling back to local logging means a blip in logging availability doesn't turn into all hands on deck everything is on fire. And it means that logs will very likely be available for any failures. In other words it's good enough.
- mort96 3y ago"Good uptime and good reliability but no guarantees" is just a good best effort system.
- insanitybit 3y agoGood enough is literally "best effort delivery", you're just agreeing with them that this is ultimately a distributed systems problem and you either choose CP or AP.
- kbenson 3y agoYeah, what the "best effort" actually means in practice is usually a result of how much resources you want to throw at the problem. Those give you runway on how much of a problem you can withstand and perhaps recover from without any loss of data (logs), but in the end you're usually still just buying time. That's usually enough though.
- o11c 3y agoLogging to `mmap`ed files is resilient to service crashes, just not hardware crashes.
- sroussey 3y agoWe did something like this at Weebly for stats. The app sent the stats to a local service via UDP, so shoot and forget. That service aggregated for 1s and then sent off server.
- laurencerowe 3y agoWhy UDP for a local service rather than a unix socket?
- sroussey 3y agoSend and forget. Did not want to wait on an ack from a broken process.
- laurencerowe 3y agoAck is a TCP thing to acknowledge that the packet has been received over the unreliable network, it’s not relevant to Unix sockets which are for communicating between processes on the same host.