3 ms·

From the article: > a better conceptual model is to treat logs as time-ordered streams At scale it's probably better still to re-think logs as weakly-ordered

From the article: > a better conceptual model is to treat logs as time-ordered streams At scale it's probably better still to re-think logs as weakly-ordered lossy streams. One form of weak-ordering is the inevitable jitter that comes with having multiple processes, threads or machines; without some kind of global lock (which would be impactful to performance) it stops being possible to have a true before/after relationship between individual log entries. Another form of weak ordering is that it's very common for log entries to be recorded only at the end of an operation, irrespective of its duration; so a single instantaneous entry really represents a time-span of activity with all sorts of fuzzy before/after/concurrent-to relationships to other entries. But maybe the most overlooked kind of weak ordering is one that is rarely found in logging systems, but is highly desirable: log streams should ideally be processed in LIFO order. If you're building some kind of analytical or visualisation system or near real-time processor for log data, you care most about "now". Inevitably there are processing queues and batches and so on to deal with; but practically every logging system just orders the entries by "time" and handles those queues as FIFO. If a backlog arises; you must wait for the old data to process before seeing the new. Change these queues and batching systems to LIFOs and you get really powerful behavior; recent data always takes priority but you can still backfill historical gaps. Unix files are particularly poorly suited to this pattern though - even though a stack is a simple data-structure, it's not something that you can easily emulate with a file-system and command line tools.

justincormack 12y ago
You could write some simple code that seeks if there is sufficient data between the file position and the end and puts the gap on a queue for later processing. You need to be able to find end of record markers but that's it.
jimmaswell 12y ago
>it stops being possible to have a true before/after relationship between individual log entries. Timestamps solve this easily. Synchronizing the time between machines isn't hard
- jasode 12y ago
  > Synchronizing the time between machines isn't hard Well, maybe "hard" can be relative to scope and scale. It took google engineers some trial and error before settling on the concept of "smearing a leap second" across the entire day. Relying on plain vanilla NTP sync and GPS satellites was not enough. http://googleblog.blogspot.com/2011/09/time-technology-and-leaping-seconds.html http://googleblog.blogspot.com/2011/09/time-technology-and-l...
- ColinCera 12y ago
  > Timestamps solve this easily. Synchronizing the time between machines isn't hard I can't tell if this is snark or not. Either way, it made me giggle.
peterwwillis 12y ago
You can have strongly-ordered logs too. The fact that nobody actually builds robust logging into their application doesn't mean it has to be that way :) As for the order, it's really a matter of how you are getting the data [right now] and what tools you have to work with. If you're just receiving an unending network stream, you have no choice but FIFO. If you have log files you can start from the bottom and work up, but it's usually not as efficient as top-down. If you have clustered infrastructure you can process the same log(s) faster using parallel jobs. If you build log processing tools into each system from the start you can distribute the simple parts of data processing to each node. You have a lot of freedom in finding new ways to make logging more efficient. LIFO is nice for old data you want to see now, but isn't as useful for new data being processed constantly.

justincormack 12y ago

You could write some simple code that seeks if there is sufficient data between the file position and the end and puts the gap on a queue for later processing. You need to be able to find end of record markers but that's it.

jimmaswell 12y ago

>it stops being possible to have a true before/after relationship between individual log entries. Timestamps solve this easily. Synchronizing the time between machines isn't hard

jasode 12y ago
> Synchronizing the time between machines isn't hard Well, maybe "hard" can be relative to scope and scale. It took google engineers some trial and error before settling on the concept of "smearing a leap second" across the entire day. Relying on plain vanilla NTP sync and GPS satellites was not enough. http://googleblog.blogspot.com/2011/09/time-technology-and-leaping-seconds.html http://googleblog.blogspot.com/2011/09/time-technology-and-l...
ColinCera 12y ago
> Timestamps solve this easily. Synchronizing the time between machines isn't hard I can't tell if this is snark or not. Either way, it made me giggle.

peterwwillis 12y ago

You can have strongly-ordered logs too. The fact that nobody actually builds robust logging into their application doesn't mean it has to be that way :) As for the order, it's really a matter of how you are getting the data [right now] and what tools you have to work with. If you're just receiving an unending network stream, you have no choice but FIFO. If you have log files you can start from the bottom and work up, but it's usually not as efficient as top-down. If you have clustered infrastructure you can process the same log(s) faster using parallel jobs. If you build log processing tools into each system from the start you can distribute the simple parts of data processing to each node. You have a lot of freedom in finding new ways to make logging more efficient. LIFO is nice for old data you want to see now, but isn't as useful for new data being processed constantly.