14 ms·
Twitter open-sources a high-performance replicated log service
- tlrobinson 11y agoHow does this compare to Kafka?
- scaleout1 11y agoHere is a pdf going briefly into kafka comparison and design motivation http://goo.gl/J9XdsG http://goo.gl/J9XdsG . During my time at twitter I remember when Twitter switch from Kafka to distributedlog. They have an internal layer that adapts Kafka API to distributedlog, I am not sure if they have open sourced that
- gshx 11y agoVery nice, now that we have some good competition in this space - kafka, ddl and wormhole. If only Fb can open src wormhole, there will be some good related systems software to choose from and contribute to.
- tcoppi 11y agoThis could be an interesting competitor to Apache Kafka, which is singularly unique in this space as far as I'm aware. On another note, I find it somewhat funny that these are called "log" services, logging is probably the least interesting use case for these things I can think of. A better description in my mind would be as a distributed event processing framework, since what they are really doing is distributing discrete events in a reliable manner.
- allengeorge 11y agoI don't think "distributed event processing framework" describes this system at all. It's exactly what it says on the tin: a replicated log (i.e. event log) storage system that one can build other systems on top of. It's not processing events as far as I can see?
- tcoppi 11y agoYou put events in one end, and it distributes them, reliably and consistently, to distributed end-points that can consume them. That is far more powerful than simply being used to log events for later retrieval, though it can be used for that. I have seen systems using Kafka that process tens of thousands of time-sensitive events a second and distribute them all over the world for independent processing as part of a larger system.
- toomuchtodo 11y ago"real-time event journaling and dissemination system"?
- tcoppi 11y agoYes I like that, much more accurate descriptor than "log", though it is more verbose. In the end it is really just nitpicking I guess, but the discoverability of calling these systems "log services" is very low compared to what they are really capable of.
- deleted 11y ago[deleted]
- Terr_ 11y agoSomehow I associate "journal" with temporary/transitional data, but that's probably just me thinking of journaling file-systems.
- toomuchtodo 11y agoWhen I think journal, I think a permanent record of rdbms transactions or perhaps even a blockchain. I agree with you though that the persistence of data needs to be defined when discussing a journal of any sorts.
- grogers 11y ago
- goejoegg 11y agoIf you read the CS papers surrounding distributed systems, you will often see the notion of a 'journal' or a 'log', meaning an append-only structure, which typically contains numerous agreed-upon facts.
- tcoppi 11y agoI agree that term describes the foundational data structure these services provide, but in common parlance that is not really what it means at all and it is confounded by a common(and boring) use case being to synchronize processing of actual, text-based, logs.
- sheriff 11y agoJay Kreps, architect of Kafka, calls it a log. https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...
- manigandham 11y agoWhat does "boring" have to do with any use-case?
- vdm 11y agoaka atomic broadcast
- SEJeff 11y agoI'm pretty sure that this post inspired DL. It was written by Jay, one of the 3 founders of kafka (Jay, Jun, and Neha) and should be recommended reading for every software engineer if you've not read it: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... An ordered append only datastructure is rightfully called a log. The fact that text based files are called logs is just an annoying feature in common english usage.
- i0exception 11y agoIIRC, we started working on Distributed Log at least a year before that post.
- mioelnir 11y agoThe Twitter blog post in other comments says that "At design time we had concerns about Kafka’s I/O model...", which means that at the least Kafka was extensively studied during the design phase of DistributedLog, which may have started earlier than Kafka's release of course.
- manigandham 11y agoKafka isn't that unique. There are several hosted options as well as commercial software that can do what Kafka does. Kafka has the open-source/java community and ecosystem that fits in well with the rest of the current big data processing stuff though.
- crgwbr 11y agoAnyone know how this similar and different from Kafka? Or why Twittered decide to build their own instead of using and contributing to Kafka?
- murph 11y agoThere's a great blog post on this: https://blog.twitter.com/2015/building-distributedlog-twitter-s-high-performance-replicated-log-service https://blog.twitter.com/2015/building-distributedlog-twitte...
- DannyBee 11y agoWhich says " At design time we had concerns about Kafka’s I/O model and its lack of strong durability guarantees‐a non-starter for an application like a distributed transaction log[3]" Seems reasonable, right? Except "[3] Kafka addressed these durability concerns in version 0.8" So they built a whole thing, because they didn't bother to ask or say "hey, if we help fix the durability, would that we welcome?" or even "do you guys have a plan and timeline to fix it?" That's .... not great. Now, maybe there are other reasons, but they aren't elucidated in this blog post :P. Even then, my general view would be "did you approach the community and discuss your concerns or just dismiss them out of hand as infeasible", mainly because my experience is that if you do this, you often find they have exactly the same set of concerns/goals, and just need more resources to make it happen. The desire to build shiny objects is very large. Outside of the paper plans of engineering teams, these things rarely end up up more shiny than what already exists or will be built by the time you are done.
- sametmax 11y agoOr even "we are ready to pay 3 people for a year to build that, if we don't and give you the money instead what can you do ?". The software industry has a real problem contributing to open source, the stuff, you know, allowing them to make money in the first place.
- placeybordeaux 11y ago
- alphadevx 11y agoWonder how it compares to Facebook Scribe?
- manojlds 11y ago> This is an archived project and is no longer supported or updated by Facebook
- sandspit 11y agoScribe is a log aggregation system, it doesn't really do topics/streams or have a good durability/streaming performance story. Its mostly used for aggregating data into a log storage system like DL, or Kafka, or into HDFS.
- merb 11y ago100% Java. I'm wondering. Why doesn't this has any Scala?
- fosk 11y agoBecause Scala adds overhead.
- poooogles 11y agoWhere? In operational cost? In development time? In performance?
- jbooth 11y agoYes. Depending on use-case, of course, there's lot of places where scala is preferable to Java. A byte-shuttling service where performance is important, mutability reigns and expressiveness is irrelevant.. not one of those places.
- balamaci 11y agoYes. Joke aside, at least in hiring people it should.
- niccaluim 11y agoRumors of Scala's dominance at Twitter are slightly exaggerated. While it's true that the non-revenue-related backend is almost all Scala, on the ads eng side, it's almost 100% Java. Which language a new project is written in has more to do with who's writing it than anything else.
- deleted 11y ago[deleted]
- merb 11y agoActually I just wondered cause of Finangle. I thought that their projects uses the RPC service heavily.
- moondowner 11y agoBy checking out the code quickly it looks like it's built on top of Apache Bookkeeper http://bookkeeper.apache.org/ http://bookkeeper.apache.org/ ?
- Xylakant 11y agoThis blogpost https://blog.twitter.com/2015/building-distributedlog-twitter-s-high-performance-replicated-log-service https://blog.twitter.com/2015/building-distributedlog-twitte... mentioned somewhere else in the discussion confirms it's built on bookkeeper.
- tlrobinson 11y agoMore info on the stack: https://twitter.github.io/distributedlog/html/architecture/main.html#software-stack https://twitter.github.io/distributedlog/html/architecture/m...
- moondowner 11y agoNice docs, thanks!
- heavenlyhash 11y agoThis looks potentially fantastic. If I could beg one wish from the developers of this (and almost every other project anywhere near this space), though, it would be one tiny piece of documentation: What's your unique ID scheme? Let's say I'm willing to believe[1] that you've got Durable and Consistent down, once messages make it committed in to the system. What's the story for messages on their way in? My application logs are buffered to the local disk, now I'm streaming them into central storage, and halfway through a TCP connection that's shuffled 2mb of thousands of messages into storage, the connection terminates -- unexpectedly, midmessage. Could the service have committed more messages than it acknowledged? Or many less than I've sent? Both could be true from the network standpoint.[2] So, what I need to know, and what should be very easy to answer, front-and-center in your docs, pretty please: 1) Where should my log uploader resume? 2) Is there any danger of repeatedly entering some lines? 3) If I have log lines that are legitimately duplicates, will they be stored at the correct count? These are questions that may have a different answer than the durability after data makes it fully into the system. It also may provide useful information about how complexity the code in a submitting client is, because good answers tend to require some kind of ID sequence being assigned on submitting clients, afaict. And it's really just plain critical to sanity. ---- [1] well, no, I'm not, "trust by verify" in all things etc etc; but let's suppose that's more believable and something I have to mechanically verify anyway, and doesn't have an obviously observable boolean at the protocol level as to whether it's going to work well or not, and system internals simply don't have such a sordid history of being over-simplified until they're broken like client interfaces so often are, so...! We'll handwave that to a later and more involved step of quality investigation. [2] https://en.wikipedia.org/wiki/Two_Generals'_Problem https://en.wikipedia.org/wiki/Two_Generals'_Problem
- chillacy 11y agoI stumbled upon this page while reading the docs: https://twitter.github.io/distributedlog/html/design/main.html#consistency https://twitter.github.io/distributedlog/html/design/main.ht... It looks like they use fencing and a two-phase commit to prevent duplicate writes. Whether that covers all failure scenarios I'm not sure.
- sandspit 11y ago
- vruiz 11y agoFor those who like to watch talks, I found one on youtube: https://www.youtube.com/watch?v=QW1OEQxcjZc https://www.youtube.com/watch?v=QW1OEQxcjZc
- ryanobjc 11y agoI was interested until I saw the Zookeeper dependency. I have had too many deployment nightmares with Zookeeper. I would prefer to avoid it as much as possible, plus systems software in Java, sigh.
- nicobn 11y agoFor better or for worse, Java runs a big chunk of this world. Not running a Java application because "ewww Java is gross" will prevent you from enjoying some of the most exciting database technologies out there.
- asimuvPR 11y agoMind sharing a bit more?
- ryanobjc 11y agoThe classic problem of ZooKeeper is the client and the server upon initial startup will DNS resolve and permanently cache the IP addresses of the ZK cluster. Thus any cluster migrations require you to restart _EVERYTHING_. So running zookeeper in a dynamic environment becomes very risky. Aka AWS. Perhaps on GCE it's less dangerous, because the host migration is pretty good. There are other operational issues people have noted above as well.
- eropple 11y agoYou're right in that Zookeeper can be a hassle in a dynamic environment, but in my experience Exhibitor largely removes that concern.
- rad_gruchalski 11y agoThen configure it using IP addresses. I've been using Zookeeper like this for 4 years in public EC2, swapping machines in and out and it always worked. What's the problem.
- 11y ago
- hvmonk 11y agoI wish they had kept some of the commit history - would be tough to parse the project without it.
- swang 11y agoWhat? No bird related name for this project?!
- sulam 11y agoBird projects got outlawed during the interregnum.
- trungonnews 11y agoDoes DL has a limit on the number of partitions like Kafka?
- mgodave 11y agoA DL stream is a single partition if you are thinking in terms of kafka. We stitch them together as a partitioned stream a la Kafka using a different system. So, to answer your question, no, there is no limit to the number of DL streams you can put together into a bigger logical stream, we have some very large ones.
- alexatkeplar 11y agoAh! That actually explains the strange syntax I was seeing in the tutorials: // Create Stream `basic-stream-{3-7}` // dlog tool create -u ${distributedlog-uri} -r ${stream-prefix} -e ${stream-regex} ./distributedlog-core/bin/dlog tool create -u distributedlog://127.0.0.1:7000/messaging/distributedlog -r basic-stream- -e 3-7 You use a regex to progammatically create N streams, which would be N shards in Kinesis or N partitions in Kafka.
- throwaway_xx9 11y agoNo, actually Java is a bane to the database world. Cassandra doesn't work, and Hadoop is a complete waste of hosts for most companies (hence the move to Spark.)
- slantedview 11y ago> hence the move to Spark ...which also runs on the Java Virtual Machine and is subject to the same pros and cons.
- sgt101 11y agoheh - you beat me too it!
- sgt101 11y agoDisagree on two fronts : - By hadoop I assume you mean Map Reduce ? There are other Engines like SAMSA and FLINK and Kafka makes a great event store. Anyway, MR is super for massive throughput batch jobs, for example huge HIVE queries or Pig jobs and if you are reading, breaking the heap size, doing one thing and then writing there is no bonus from doing it in SPARK. - SPARK is written in Scala, which runs on the JVM. And it has a nice Java API as well !
- growse 11y agoI'd be interested to know how you can assert that Hadoop is a 'complete waste of hosts for most companies'. Also, don't underestimate the many, many people successfully running Spark on YARN at scale. Hadoop is actually quite helpful to some workloads.
- dkersten 11y agoMost companies simply don't have the data volume to make Hadoop worthwhile. You can process tens of TB in an RDBMS on a beefy machine cheaper than a Hadoop cluster. Hadoop is slow, but on huge data volume the overheads are dwarfed by the parallelism gained. Most companies don't have huge volume though. For example recently I saw someone propose using Hadoop for a sub-TB dataset...
- boredandroid 11y agoSeveral people asked how this compares to Kafka (I'm one of the people who created Kafka at LinkedIn). Here's my take: I think the motivations they list are: 1. Different I/O model 2. Was started before Kafka had replication (the first release of Kafka with replication was in late 2013 I think) The I/O model I'm less sure about, we looked at similar things for Kafka and they didn't seem worth it (basically you're doing a ton of stuff at the app level that the OS does pretty well--namely caching and buffering linear I/O), we'd have to look at actual benchmarks to know. Here is my take on the pros and cons of the core tech. Pros: - Seems to have better built in support for fencing/idempotence - Better geo placement? Cons: - Lots more moving pieces. Already people are irritated that there are both Kafka nodes and ZK to set up. This system seems to split this over separate physical tiers for serving, core, storage, and zookeeper. My experience has been lot's of tiers is generally a big headache. Neutral: - There seems to be a built in achival to HDFS. I think if the consumer is fast and efficient then you don't need to reach around your consumer api which will be high latency (since you have to wait for files to be closed out). There is also a bunch of stuff Kafka does that I'm just not sure about how complete it is in DistributedLog: - Clients in a bunch of languages - Integration with all the major stream processing frameworks - Log compaction http://kafka.apache.org/documentation.html#compaction http://kafka.apache.org/documentation.html#compaction - Connector management http://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines http://www.confluent.io/blog/announcing-kafka-connect-buildi... - Quotas/throttling - Security/ACLs
- duggan 11y agoPeople talk about Zookeeper as a negative, but in a quorum it's been one of the most stable and reliable pieces of software I've deployed, despite being a bit frustrating to set up / configure. Netflix's Exhibitor[1] is an indispensable addition to it. Also, once you're on the Big Data™ train, a lot of things like to plug into Zookeeper, so it becomes more of a convenience. Kafka, and presumably DL, are at their most useful when you're pushing the limits of NIC and/or HDD performance for throughput. Zookeeper's configuration is a footnote in the complexity of managing one of these systems, and lets them avoid implementing their own byzantine coordination system. Also, folks seem to appreciate Aphyr's opinion, and he states it pretty plainly: Use Zookeeper. It’s mature, well-designed, and battle-tested. [2] [1] https://github.com/Netflix/exhibitor https://github.com/Netflix/exhibitor [2] https://aphyr.com/posts/291-jepsen-zookeeper https://aphyr.com/posts/291-jepsen-zookeeper
- sambe 11y agoI found Zookeeper to be pretty easy to learn about and deploy. I also found it got a semi-bad reputation with others because they did not want to learn anything. Two major examples being: * attempting thousands (tens of thousands?) of simultaneous writes across data centres/continents at the same instant and then saying it was slow. Subscribing all clients to updates on all parts of the tree. The architecture had been grown by people who didn't understand the guarantees and constraints. * calling it unreliable after arbitrarily moving nodes around without changing connect strings and generally mis-configuring it. Essentially, pointing clients at machines that no longer contained nodes and blaming ZK for this not working. It could be easier to setup. People often don't want to think about such things at all. But it's also not the hardest, and I found it to be very resilient to node and network failures when deployed correctly.
- duggan 11y agoAgreed, some of the problems I encountered were because I did not carefully read the documentation. Some are because a lot of tooling has become easier to deploy in the years since Zookeeper was released, so the bar has been raised. Which is just to say there's room for improvement. A dependency on Zookeeper is fine if you've already got a configured cluster, and a cognitive speed bump if not.
- stemuk 11y agoI am not that into this topic, but could you compare DL to Soundclouds Roshi[0], and if not, whats different? Thanks! [0] https://github.com/soundcloud/roshi https://github.com/soundcloud/roshi
- DenisM 11y agoSpeaking of logs, I want to put some logging in place for my web server. I log every single request with extensive details, so I can debug things later if needed. It's several gigabytes per day now, so I can no longer just dump it on disk as I did for the last couple of years. Since I'm on AWS EC2, I want to try this: - Write the logs to local SSD, asynchronously so as not hold back the http request. - Have a separate cron job that loops through the log directory and scoops up all the files. - The job will then stuff those files into a Kinesis Firehose. AFAIK, Kinesis Firehose does not require any capacity provisioning, unlike the Kinesis Streams, so I'm set "for life" (up to 5MB/second) - The firehose will accumulate the logs and put them into S3. Hurray unlimited storage! - S3 will trigger a Lambda. - Lambda will parse through the log from S3, pull out interesting properties (IP address, user id, session id, etc) and stuff them into a DynamoDb table. - If I need to see data from one user/ip/session I will use DynamoDb to find the right S3 blobs. - If I need to reprocess the logs to extract a new piece of data that I did not foresee earlier, I can run a map-reduce task Except the last piece, this looks like something I can half-ass in a couple of days and forget about it for another couple of years. Any opinions? I don't really want to use a SaaS log service because gigabytes per day.
- nathwill 11y agoWe just went through an extensive ecosystem survey and reached an identical design, which we're currently implementing. So far, so good! AWS has aws-kinesis-agent, which you can deploy for the log aggregation bit, which is very easy to use.
- DenisM 11y agoI just found out that an s3-triggered lambda is only retried 3 times and then it gives up.
- d23 11y agoI've heard some bad things about Kinesis in general. Why not have your cron job just put the logs onto S3 directly? I've used Lambda a bit. The debugging process can be a pain, since you're forced the upload a ZIP file, and if your code times out Lambda doesn't give you any traceback to indicate what happened. There's also a maximum run time of each Lambda invocation, which I believe is 5 minutes. Is there a chance your parsing may run longer than that? Also, what will you do if you upload some bad code and the parsing fails? Will it be the end of the world if you lose data while you fix the parsing? Oh, I see you plan on doing map-reduce to re-parse the logs, so maybe that part isn't as big a deal. You could also consider doing something like rsyslog -> db-of-choice while also rotating the files off to S3 for long-term map-reduces. This is all to ignore the obvious ELK cluster solution, which will give you good data visualization and investigation options, but may be more of a headache to set up and maintain than you are looking for. Anyway, those are my thoughts. Hope they help.