4 ms·

I don't think "distributed event processing framework" describes this system at all. It's exactly what it says on the tin: a replicated log (i.e. event log) sto

I don't think "distributed event processing framework" describes this system at all. It's exactly what it says on the tin: a replicated log (i.e. event log) storage system that one can build other systems on top of. It's not processing events as far as I can see?

tcoppi 10y ago
You put events in one end, and it distributes them, reliably and consistently, to distributed end-points that can consume them. That is far more powerful than simply being used to log events for later retrieval, though it can be used for that. I have seen systems using Kafka that process tens of thousands of time-sensitive events a second and distribute them all over the world for independent processing as part of a larger system.
- toomuchtodo 10y ago
  "real-time event journaling and dissemination system"?
  - tcoppi 10y ago
    Yes I like that, much more accurate descriptor than "log", though it is more verbose. In the end it is really just nitpicking I guess, but the discoverability of calling these systems "log services" is very low compared to what they are really capable of.
    - deleted 10y ago
      [deleted]
    - Terr_ 10y ago
      Somehow I associate "journal" with temporary/transitional data, but that's probably just me thinking of journaling file-systems.
      - toomuchtodo 10y ago
        When I think journal, I think a permanent record of rdbms transactions or perhaps even a blockchain. I agree with you though that the persistence of data needs to be defined when discussing a journal of any sorts.
      - grogers 10y ago
        You might prefer 'ledger' to emphasize the immutability of the journal.
jcrites 10y ago
"Log-structured file system" or "journaling file system" is often how such things are described. The notion being that the primary operation of interest is appending new data records, rather than modifying existing data. https://en.wikipedia.org/wiki/Log-structured_file_system https://en.wikipedia.org/wiki/Log-structured_file_system The use of the term "log" in some contexts implies concepts like lossy, unimportant, non-durable, etc. However, in the context of log-structured file systems or journaling file systems, the sense that's intended is append-only as a primitive, atomic, consistent, durable (replicated) operation.

tcoppi 10y ago

You put events in one end, and it distributes them, reliably and consistently, to distributed end-points that can consume them. That is far more powerful than simply being used to log events for later retrieval, though it can be used for that. I have seen systems using Kafka that process tens of thousands of time-sensitive events a second and distribute them all over the world for independent processing as part of a larger system.

toomuchtodo 10y ago
"real-time event journaling and dissemination system"?
- tcoppi 10y ago
  Yes I like that, much more accurate descriptor than "log", though it is more verbose. In the end it is really just nitpicking I guess, but the discoverability of calling these systems "log services" is very low compared to what they are really capable of.
  - deleted 10y ago
    [deleted]
  - Terr_ 10y ago
    Somehow I associate "journal" with temporary/transitional data, but that's probably just me thinking of journaling file-systems.
    - toomuchtodo 10y ago
      When I think journal, I think a permanent record of rdbms transactions or perhaps even a blockchain. I agree with you though that the persistence of data needs to be defined when discussing a journal of any sorts.
    - grogers 10y ago
      You might prefer 'ledger' to emphasize the immutability of the journal.

jcrites 10y ago

"Log-structured file system" or "journaling file system" is often how such things are described. The notion being that the primary operation of interest is appending new data records, rather than modifying existing data. https://en.wikipedia.org/wiki/Log-structured_file_system https://en.wikipedia.org/wiki/Log-structured_file_system The use of the term "log" in some contexts implies concepts like lossy, unimportant, non-durable, etc. However, in the context of log-structured file systems or journaling file systems, the sense that's intended is append-only as a primitive, atomic, consistent, durable (replicated) operation.