5 ms·

This could be an interesting competitor to Apache Kafka, which is singularly unique in this space as far as I'm aware. On another note, I find it somewhat funn

by tcoppi 10y ago

This could be an interesting competitor to Apache Kafka, which is singularly unique in this space as far as I'm aware. On another note, I find it somewhat funny that these are called "log" services, logging is probably the least interesting use case for these things I can think of. A better description in my mind would be as a distributed event processing framework, since what they are really doing is distributing discrete events in a reliable manner.

allengeorge 10y ago
I don't think "distributed event processing framework" describes this system at all. It's exactly what it says on the tin: a replicated log (i.e. event log) storage system that one can build other systems on top of. It's not processing events as far as I can see?
- tcoppi 10y ago
  You put events in one end, and it distributes them, reliably and consistently, to distributed end-points that can consume them. That is far more powerful than simply being used to log events for later retrieval, though it can be used for that. I have seen systems using Kafka that process tens of thousands of time-sensitive events a second and distribute them all over the world for independent processing as part of a larger system.
  - toomuchtodo 10y ago
    "real-time event journaling and dissemination system"?
    - tcoppi 10y ago
      Yes I like that, much more accurate descriptor than "log", though it is more verbose. In the end it is really just nitpicking I guess, but the discoverability of calling these systems "log services" is very low compared to what they are really capable of.
      - deleted 10y ago
        [deleted]
      - Terr_ 10y ago
        Somehow I associate "journal" with temporary/transitional data, but that's probably just me thinking of journaling file-systems.
        toomuchtodo 10y ago
        When I think journal, I think a permanent record of rdbms transactions or perhaps even a blockchain. I agree with you though that the persistence of data needs to be defined when discussing a journal of any sorts.
        grogers 10y ago
        You might prefer 'ledger' to emphasize the immutability of the journal.
- jcrites 10y ago
  "Log-structured file system" or "journaling file system" is often how such things are described. The notion being that the primary operation of interest is appending new data records, rather than modifying existing data. https://en.wikipedia.org/wiki/Log-structured_file_system https://en.wikipedia.org/wiki/Log-structured_file_system The use of the term "log" in some contexts implies concepts like lossy, unimportant, non-durable, etc. However, in the context of log-structured file systems or journaling file systems, the sense that's intended is append-only as a primitive, atomic, consistent, durable (replicated) operation.
goejoegg 10y ago
If you read the CS papers surrounding distributed systems, you will often see the notion of a 'journal' or a 'log', meaning an append-only structure, which typically contains numerous agreed-upon facts.
- tcoppi 10y ago
  I agree that term describes the foundational data structure these services provide, but in common parlance that is not really what it means at all and it is confounded by a common(and boring) use case being to synchronize processing of actual, text-based, logs.
  - sheriff 10y ago
    Jay Kreps, architect of Kafka, calls it a log. https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...
  - manigandham 10y ago
    What does "boring" have to do with any use-case?
- vdm 10y ago
  aka atomic broadcast
SEJeff 10y ago
I'm pretty sure that this post inspired DL. It was written by Jay, one of the 3 founders of kafka (Jay, Jun, and Neha) and should be recommended reading for every software engineer if you've not read it: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... An ordered append only datastructure is rightfully called a log. The fact that text based files are called logs is just an annoying feature in common english usage.
- i0exception 10y ago
  IIRC, we started working on Distributed Log at least a year before that post.
  - mioelnir 10y ago
    The Twitter blog post in other comments says that "At design time we had concerns about Kafka’s I/O model...", which means that at the least Kafka was extensively studied during the design phase of DistributedLog, which may have started earlier than Kafka's release of course.
manigandham 10y ago
Kafka isn't that unique. There are several hosted options as well as commercial software that can do what Kafka does. Kafka has the open-source/java community and ecosystem that fits in well with the rest of the current big data processing stuff though.

by tcoppi 10y ago

allengeorge 10y ago

I don't think "distributed event processing framework" describes this system at all. It's exactly what it says on the tin: a replicated log (i.e. event log) storage system that one can build other systems on top of. It's not processing events as far as I can see?

tcoppi 10y ago
You put events in one end, and it distributes them, reliably and consistently, to distributed end-points that can consume them. That is far more powerful than simply being used to log events for later retrieval, though it can be used for that. I have seen systems using Kafka that process tens of thousands of time-sensitive events a second and distribute them all over the world for independent processing as part of a larger system.
- toomuchtodo 10y ago
  "real-time event journaling and dissemination system"?
  - tcoppi 10y ago
    Yes I like that, much more accurate descriptor than "log", though it is more verbose. In the end it is really just nitpicking I guess, but the discoverability of calling these systems "log services" is very low compared to what they are really capable of.
    - deleted 10y ago
      [deleted]
    - Terr_ 10y ago
      Somehow I associate "journal" with temporary/transitional data, but that's probably just me thinking of journaling file-systems.
      - toomuchtodo 10y ago
        When I think journal, I think a permanent record of rdbms transactions or perhaps even a blockchain. I agree with you though that the persistence of data needs to be defined when discussing a journal of any sorts.
      - grogers 10y ago
        You might prefer 'ledger' to emphasize the immutability of the journal.
jcrites 10y ago
"Log-structured file system" or "journaling file system" is often how such things are described. The notion being that the primary operation of interest is appending new data records, rather than modifying existing data. https://en.wikipedia.org/wiki/Log-structured_file_system https://en.wikipedia.org/wiki/Log-structured_file_system The use of the term "log" in some contexts implies concepts like lossy, unimportant, non-durable, etc. However, in the context of log-structured file systems or journaling file systems, the sense that's intended is append-only as a primitive, atomic, consistent, durable (replicated) operation.

goejoegg 10y ago

If you read the CS papers surrounding distributed systems, you will often see the notion of a 'journal' or a 'log', meaning an append-only structure, which typically contains numerous agreed-upon facts.

tcoppi 10y ago
I agree that term describes the foundational data structure these services provide, but in common parlance that is not really what it means at all and it is confounded by a common(and boring) use case being to synchronize processing of actual, text-based, logs.
- sheriff 10y ago
  Jay Kreps, architect of Kafka, calls it a log. https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...
- manigandham 10y ago
  What does "boring" have to do with any use-case?
vdm 10y ago
aka atomic broadcast

SEJeff 10y ago

I'm pretty sure that this post inspired DL. It was written by Jay, one of the 3 founders of kafka (Jay, Jun, and Neha) and should be recommended reading for every software engineer if you've not read it: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... An ordered append only datastructure is rightfully called a log. The fact that text based files are called logs is just an annoying feature in common english usage.

i0exception 10y ago
IIRC, we started working on Distributed Log at least a year before that post.
- mioelnir 10y ago
  The Twitter blog post in other comments says that "At design time we had concerns about Kafka’s I/O model...", which means that at the least Kafka was extensively studied during the design phase of DistributedLog, which may have started earlier than Kafka's release of course.

manigandham 10y ago

Kafka isn't that unique. There are several hosted options as well as commercial software that can do what Kafka does. Kafka has the open-source/java community and ecosystem that fits in well with the rest of the current big data processing stuff though.