Hello! I work at YT and would like to answer a question that was asked in a flagged thread about the comparison between YT and Hive and Zookeeper. Both Cypress

Hello! I work at YT and would like to answer a question that was asked in a flagged thread about the comparison between YT and Hive and Zookeeper. Both Cypress and Zookeeper are fault-tolerant distributed hierarchical filesystems that can be used for distributed coordination, but Cypress has much richer functionality. Recall that Zookeeper's data model is just a tree consisting of homogeneous nodes that can be either ephemeral or persistent, along with a set of sessions that control the lifetime of ephemeral nodes. This simple model allows to implement multiple primitives of distributed synchronization, such as leader election, exactly-once queue processing, or two-phase commits. However, it is not always easy to integrate Zookeeper with third-party systems. For example, if you want to elect a leader via Zookeeper and use it to insert data into a database, it is mandatory that the instance remains the leader during the commit into the database, which is not easy to implement without races or some additional assumptions. In YTsaurus, transactions permeate our entire system. You can start a transaction and acquire an exclusive lock at some Cypress node (which is a way to make a leader election), and after that, the transaction becomes the leader lease. You can then modify Cypress, run MapReduce or YQL operations using the transaction as a prerequisite, lock some files and tables in the same transaction, and do many other things. Currently, we are working on the ability to use Cypress locks as prerequisites for dynamic table commits. There are many other features in Cypress that are not implemented in Zookeeper, such as symlinks, automatic expiration of unused nodes, and many others. Moreover, Cypress can be sharded using Portals about which I wrote in a previous comment, so this filesystem is scalable unlike Zookeeper. Even without sharding a single primary master of YTsaurus can hold tens of gigabytes of metadata of Cypress while Zookeeper state size is limited with hundreds of megabytes accoring to etcd vs Zookeeper comparision [1]. One major disadvantage of Cypress compared to Zookeeper is the lack of watches, so all changes tracking should be done via short polling. The good news is that Cypress is well-optimized for read queries with the possibility to read from followers and from multiple threads, so this is not a big problem. In the meantime, we are considering the possibility of adding some kind of watches to Cypress. The big difference between Cypress and Zookeeper is the replicated state machine implementation. With all due respect to Zookeeper developers, Zookeeper was implemented over 15 years ago when the world of distributed algorithms was different. Today we see that ZAB (the consensus algorithm used in Zookeeper) has some shortcomings in failover speed and stability. There are multiple reports of Zookeeper being unstable under heavy load. In YTsaurus, we use an in-house library called Hydra for RSM implementation. This is our consensus algorithm very similar to RAFT that has proven itself to be both efficient and fault-tolerant. We use Hydra for master servers, clock servers, and tablet cells (RSMs that store data in dynamic tables). I even had an idea to implement a Zookeeper API using Hydra both to simplify migration to YTsaurus and check Hydra performance and correctness via multiple tests implemented for Zookeeper (Jepsen, for instance), but did not have enough time to finish this project. This comment is already quite long, so I will write about the YT vs Hive comparison in another comment later on. [1] -- https://etcd.io/docs/v3.3/learning/why/ https://etcd.io/docs/v3.3/learning/why/