9 ms·

Yandex open-sources its exabyte-scale big data platform

275 points by xpl 4y ago

cynicalsecurity 4y ago
[flagged]
- DanTheManPR 4y ago
  As opposed to the IT department of the NSA?
- omgtehlion 4y ago
  Oh come on already... Of course FSB has a backdoor to *data* collected by Yandex, but the code itself (as well as coders) have nothing to do with any three-letter-agency within Russia.
  - screamingninja 4y ago
    > the code itself (as well as coders) have nothing to do with any three-letter-agency within Russia. citation needed
    - dna_polymerase 4y ago
      Let's keep in mind, the only companies that were ever found to collude with three-letter agencies were the big players in the US. But for some reason, Yandex needs to prove something here…
    - eddsh1994 4y ago
      Why don't you read the code? The amount of people commenting on this, if there was something to hide they'd have probably caught it already and if not surely by next week.
  - deleted 4y ago
    [deleted]
- ivan_gammel 4y ago
  Maybe this is exactly the reason why it’s worth checking? They open-sourced it.
  - Aldipower 4y ago
    Yeah, like Apache Mesos is running the NSA data center in Utah. One of the biggest in the world.
    - mdaniel 4y ago
      Do you happen to have any link where I can read more about this?
- bertil 4y ago
  I’m all for holding companies that have supported dangerous regimes to account. However, when it comes to data management, totalitarian regimes rarely indicate inadequate implementation. IBM’s role in Germany in the 40s was horrific, but it proved their ideas of tabulations and files were promising. Just like with rocketry, there were many valuable things to learn that defined the rest of the XXth century. The FSB likely has a lot of crimes to atone for. Still, suppose one of their specialists publishes something on data management or how to manage hundreds of sock-puppet social media accounts. In that case, I’d be tempted to listen and learn from a likely expert—unless you suspect that they think this article is not sincere and meant as a distraction from actual good practices. Similarly, the CIA has done very problematic things, but the people who worked in the disguise department have a creative take on changing your appearance. I’m unsure when I would have to do that, but I’m always curious about how data is stored efficiently. And yes, like the FSB, the NSA has opinions about that, and those are typically well-informed. Was their practice constitutional? Seemingly not, IANAL. But do they have good insights into caching video files at scale? Definitely.
  - medo-bear 4y ago
    > IBM’s role in Germany in the 40s was horrific, but it proved their ideas of tabulations and files were promising source: https://en.m.wikipedia.org/wiki/IBM_and_the_Holocaust https://en.m.wikipedia.org/wiki/IBM_and_the_Holocaust
- throwaw12 4y ago
  brainwashing at its peak. When USA does things, it is for the good of society, democracy. When Russia does things, it is hurting people, bad for society. Come on buddy, time to wake up and understand every country does things for its own good and whatever your media is telling about Russia is bad, it is because they're applying 3 letter agency brainwashing methods on to you. Code is open source, if you read code you will not get under Russian propoganda.
  - garbagecoder 4y ago
    [flagged]
    - medo-bear 4y ago
      nice head in the sand
      - garbagecoder 4y ago
        That's a hilarious response from a Russia apologist. You think telling other people they're naive about the CIA means you can ignore all the bad stuff about Russia but that's not how the world works and you are either paid to have say this or are doing it for free and I'm not sure which is worse. I try not to put myself in the hands of any spy agency. "Other team bad so my team good" is reasoning barely worthy of the human mind.
        medo-bear 4y ago
        there is no need to slander me. im definitely not paid by anyone to give my opinions imperialism is imperialism. it should be denounced if you dont think there is a US imperialist element involved in Ukraine you need to inform yourself a bit more. you can start with reading about North Stream 2 before the war in Ukraine i used to think that, in comparison the US, at least Russia bombs countries without calling it freedom [1]. When Russia started its "special operation" against Ukraine i now think that at best Putin is trolling the US, and at worst he is trying to be the US [1] https://en.m.wikipedia.org/wiki/Operation_Enduring_Freedom https://en.m.wikipedia.org/wiki/Operation_Enduring_Freedom
- jones6ofMont 4y ago
  [flagged]
  - stef25 4y ago
    > remember FSB is NOT KGB because Russia now is a democratic country and not a communist USSR Secret services in a democratic country being better / worse than their communist counterparts aside, "democracy" is a bit of stretch when describing Russia.
  - Idiot_in_Vain 4y ago
    Russia now is a democratic country??? Only a complete idiot would think that.
    - newaccount2023 4y ago
      [flagged]
garbagecoder 4y ago
Didn't this just leak?
- xpl 4y ago
  AFAIK it wasn't in the leak (that "YT" platform specifically). Also, open-sourcing proprietary projects of this scale is a pretty hard job and it can't be done quickly — seems they had been doing it for a long time, starting long before the leak.
  - arivkin 4y ago
    That's true. We have been preparing for opening YT for a long time when we found out about the leak. We even joked inside that they couldn't even do it normally and we still have to open our part of the code ourselves :) It's truly a hard work, because we were very tightly tied to the Yandex infrastructure and we had to learn how to deploy in k8s from scratch. Also you need a new brand and clean your documentation from irrelevant things... All this takes months.
matrix_overload 4y ago
Looks like a reasonable response to a recent leak [0] [0] https://news.ycombinator.com/item?id=34525936 https://news.ycombinator.com/item?id=34525936
panki27 4y ago
Wasn't this "open sourced" in their recent sourcecode leak already?
- omgtehlion 4y ago
  No, it was not, see https://arseniyshestakov.com/2023/01/26/yandex-services-source-code-leak/ https://arseniyshestakov.com/2023/01/26/yandex-services-sour...
galkk 4y ago
There is a conspiracy theory that recent open sourcing of Yandex tech and to some extent even a leak is a preparation for global Yandex exodus from Russia.
- slt2021 4y ago
  exodus has already happened
- speed_spread 4y ago
  You can take Yandex out of Russia... but you can not take Russia out of Yandex. Admittedly, the same could probably be said about any other $big_search:$nuclear_power pair. It's just that the other companies either can't or have no reason for exile. And so I suspect Yandex leaving it's home turf would essentially be a covert invasion of wherever they'd swarm to. Somehow, money tells me UK would be a likely target. Source: worked briefly for an exiled Russian company. Would not repeat.
  - ClumsyPilot 4y ago
    > a covert invasion of wherever they'd swarm to. > an exiled Russian company. Either you are exhiled or you are invading, how can you be both?
    - helge9210 4y ago
      For example, you wholeheartedly support the policy, but are not comfortable living under the sanctions.
      - ClumsyPilot 4y ago
        My understanding of Exile is that you are rejected by the society you are exiled from, due to your actions or beliefs. Like profound disagreement with the policy. I have certainly seen the kind of people you talk about, they support (or at least used to) Putin, but don't want their kids to live in Russia. I would call them more like emigrants of convenience.
reisse 4y ago
There is an interesting take that Russian part of the Yandex group opensources as much as possible, in order for the overseas companies of the group to leverage technologies without legal or financial ties with Russia. For me this seems very plausible, as for the last year they first did everything to distance from anything related to politics (e. g. they sold their news and their blogging platform to the basically state-owned VK), and then to separate Russian and overseas businesses as much as possible.
- the_mitsuhiko 4y ago
  Would be a clever move but I have doubts that no longer Yandex has much to say in Russian Yandex. The latter seems very close to the Kremlin now.
- 0xDEF 4y ago
  Yandex is incorporated in the Netherlands and the founders live in Israel. I think they will create several spin-off open source companies (like ClickHouse Inc.) outside Russia to continue doing B2B business with the outside world.
  - deepsun 4y ago
    It doesn't matter where the company is incorporated -- when Kremlin wants the access to the data Yandex cannot say no. One cannot operate in Russia and not play by Kremlin rules, especially media companies.
  - konart 4y ago
    They did some time ago actually. Like https://nebius.com/about https://nebius.com/about for example. Not to mention that Yandex been operation in countries outside of Russia under different name from the beginning (like https://en.wikipedia.org/wiki/Yango_(ride_sharing) https://en.wikipedia.org/wiki/Yango_(ride_sharing))
  - malaya_zemlya 4y ago
    Unforytunately, only one founder, Arkady Volozh, is still alive. Ilya Segalovich has died 10 years ago.
gre 4y ago
github link https://github.com/YTsaurus/YTsaurus https://github.com/YTsaurus/YTsaurus
perryh2 4y ago
[flagged]
- maxdo 4y ago
  It's a problem of entire Russia. My Asian girlfriend from china was denied to enter a restaurant in Moscow because of her origins... It's absolutely legal to put restrict your rental apartments nationality. In yandex you can ads like " will lease my house only to russians". It's everywhere. If you will ride a taxi in russia, an offensive words to other nationalities/ethnicities is everywhere. The problem is so big, that even non russian/slavic people applied to offend each other. I've been in a taxi when two middle east decent people start screaming to each other "churka" even they are both could apply to this definition. I personally trying not to invest into anything Russian, simply because that society is very, very sick and they are very far away even from recognition of this problem. They think it's their strength...
  - TechBro8615 4y ago
    > I've been in a taxi when two middle east decent people start screaming to each other "churka" Another way of interpreting this sort of culture is that they know not to take things too personally and that intent behind words is more important than the words themselves. It's a kind of liberating way of interacting with each other, and not uncommon to see in environments like fraternities or sports teams. Some people will call that culture exclusionary, but I might call those people neurotic. To put it another way, who are you to be offended on behalf of the people calling each other churka? Clearly they're not offended by it, so shouldn't you let them have their fun?
    - 2-718-281-828 4y ago
      [dead]
  - konart 4y ago
    > My Asian girlfriend from china was denied to enter a restaurant in Moscow because of her origins... While I'm not questioning your statement - this sound very questionable, considering the number of Chinese tourists in Moscow every year. >It's absolutely legal to put restrict your rental apartments nationality >I personally trying not to invest into anything Russian, simply because that society is very, very sick Do you as a gaijin invest in anything japanese in this case? It is not uncommon to see a "gaijins are not allowed" sign in Japan either. While it is true that Russia has (and will probably have for a long time) some racism problems (just like superiority complex) - I wouldn't say it's as bad as you think it is (especially compared to 90s for example)
sciencesama 4y ago
Who cares ?
- konart 4y ago
  Same people who use Clickhouse or YDB at least.
IYasha 4y ago
Funny to see 0 of 50 comments on a HN tech article being tech-related...
- booi 4y ago
  I think what you're seeing is a resurgence of tech ethics where the source and support of a project is as important as the technology itself.
- hotstickyballs 4y ago
  It’s not technically interesting since a lot of big data solutions already solve this problem so I guess only the geopolitics are left.
  - xpl 4y ago
    I doubt about "a lot". Also, "already solve" does not mean "solve better" or even "good enough". It would be very interesting to see some in-depth comparisons with already-existing open source technology (like Hadoop, Hive, Iceberg, ZooKeeper) to get a sense of when and where YT could be more effective.
  - hamilyon2 4y ago
    I am obviously biased, but, yes, technically it is very, very interesting. Distributed transactions, kiparis, YQL are interesting. Another aspect, the only other open alternative is Hadoop, hbase and hive. They don't compete with yt on usability and developer experience aspects. Yt is much more polished, despite historical quirks.
- l0b0 4y ago
  Agreed, but this is presumably also an absolutely massive project hardly anyone here has even used before. So it's not surprising that there are no big tech insights on the day of the release. An `scc` printout might be interesting, but any in-depth analysis is going to take a long time.
  - arivkin 4y ago
g9yuayon 4y ago
The github repo has 5 million lines of C++ code (headers included), 1.6 million lines of C code, and even nearly 1 million lines of Scala + Java code. We'd need some serious docs to adopt this technology. The most interesting part of YT is Cypress. I'm particularly interested in how they make their master cluster horizontally scalable.
- gritukan 4y ago
  Historically, the master server of YTsaurus was a single RSM (replicated state machine) that contained all the meta-information about the cluster. This included the tree of the distributed filesystem, transactions, information about users and tables, placement of chunks, and much more. However, this approach proved to be non-scalable as the memory amount and throughput of the master server soon became insufficient. To address this issue, we implemented Multicell technology. With Multicell, there are multiple RSMs called secondary masters that store information about chunks of the tables and their placement. The primary master still stores information about the distributed filesystem and transactions but is now single and non-sharded. After a few years, the masters became overloaded again, and we implemented Portals. With Portals, one can select a subtree of Cypress and place it in one of the secondary masters. This technology is used nowadays, and home directories of some active users are hosted on secondary masters. However, we anticipate that this approach will also become insufficient in a few years. Therefore, we are currently working on a new technology called Sequoia, which stores information about the Cypress tree shape in horizontally scalable dynamic tables. It is hard to describe all aspects of master server internals in one comment. Therefore, feel free to join our chat at t.me/ytsaurus for further discussion!
  - ddorian43 4y ago
    > Therefore, we are currently working on a new technology called Sequoia, which stores information about the Cypress tree shape in horizontally scalable dynamic tables. Why not just use a database for the metadata? Something that can be sharded and has transactions like YugabyteDB/Yandex-ydb/etc?
    - 4y ago
magundu 4y ago
Is it an alternative to Snowflake?
- rch 4y ago
  I work at Cloudera and I see it as roughly analogous to CDP, but I didn't see anything about hybrid on-prem/cloud deployments.
  - arivkin 4y ago
    It's true. We are very close to CDP and Apache Hadoop. We are only about on-prem right now. It's not a secret that we didn't need a cloud deployment at Yandex. But we see and understand the demand for it. And we have a plan:)
    - rch 4y ago
      Interesting! I'm looking forward to seeing what you have lined up.
  - deleted 4y ago
    [deleted]
- alephone1 4y ago
  We provide YQL for running large-scale OLAP SQL queries. In this regard YT can be compared to Snowflake. However, we target on prem deployment in the first place, while Snowflake runs in aws/gcp/azure, and queries are performed over data that sits in S3.
ushakov 4y ago
How many companies exist that will utilise its full capabilities? My bet would be 50-100
- arivkin 4y ago
  Why do you think so? How many companies use Apache Hadoop?
- RobotToaster 4y ago
  Most people don't utilise the full capabilities of the tools they use. Sure it would be overkill for a lot of applications, but so is redis, react.js, etc.
- xpl 4y ago
  My guess it could be successfully used for relatively small (terabyte-scale) workloads. It's the same as when people use k8s not utilizing its full capabilities, only to be able to massively scale up when needed.
VWWHFSfQ 4y ago
It's my understanding that the original Yandex founders and tech left Russia several years ago. Is this new technology? Or is it stuff being extricated from the (now) Kremlin-controlled codebase?
- xpl 4y ago
  It isn't new, in the article they state that it originated back in 2006. It's been in development since then.
gritukan 4y ago
Hello! I work at YT and would like to answer a question that was asked in a flagged thread about the comparison between YT and Hive and Zookeeper. Both Cypress and Zookeeper are fault-tolerant distributed hierarchical filesystems that can be used for distributed coordination, but Cypress has much richer functionality. Recall that Zookeeper's data model is just a tree consisting of homogeneous nodes that can be either ephemeral or persistent, along with a set of sessions that control the lifetime of ephemeral nodes. This simple model allows to implement multiple primitives of distributed synchronization, such as leader election, exactly-once queue processing, or two-phase commits. However, it is not always easy to integrate Zookeeper with third-party systems. For example, if you want to elect a leader via Zookeeper and use it to insert data into a database, it is mandatory that the instance remains the leader during the commit into the database, which is not easy to implement without races or some additional assumptions. In YTsaurus, transactions permeate our entire system. You can start a transaction and acquire an exclusive lock at some Cypress node (which is a way to make a leader election), and after that, the transaction becomes the leader lease. You can then modify Cypress, run MapReduce or YQL operations using the transaction as a prerequisite, lock some files and tables in the same transaction, and do many other things. Currently, we are working on the ability to use Cypress locks as prerequisites for dynamic table commits. There are many other features in Cypress that are not implemented in Zookeeper, such as symlinks, automatic expiration of unused nodes, and many others. Moreover, Cypress can be sharded using Portals about which I wrote in a previous comment, so this filesystem is scalable unlike Zookeeper. Even without sharding a single primary master of YTsaurus can hold tens of gigabytes of metadata of Cypress while Zookeeper state size is limited with hundreds of megabytes accoring to etcd vs Zookeeper comparision [1]. One major disadvantage of Cypress compared to Zookeeper is the lack of watches, so all changes tracking should be done via short polling. The good news is that Cypress is well-optimized for read queries with the possibility to read from followers and from multiple threads, so this is not a big problem. In the meantime, we are considering the possibility of adding some kind of watches to Cypress. The big difference between Cypress and Zookeeper is the replicated state machine implementation. With all due respect to Zookeeper developers, Zookeeper was implemented over 15 years ago when the world of distributed algorithms was different. Today we see that ZAB (the consensus algorithm used in Zookeeper) has some shortcomings in failover speed and stability. There are multiple reports of Zookeeper being unstable under heavy load. In YTsaurus, we use an in-house library called Hydra for RSM implementation. This is our consensus algorithm very similar to RAFT that has proven itself to be both efficient and fault-tolerant. We use Hydra for master servers, clock servers, and tablet cells (RSMs that store data in dynamic tables). I even had an idea to implement a Zookeeper API using Hydra both to simplify migration to YTsaurus and check Hydra performance and correctness via multiple tests implemented for Zookeeper (Jepsen, for instance), but did not have enough time to finish this project. This comment is already quite long, so I will write about the YT vs Hive comparison in another comment later on. [1] -- https://etcd.io/docs/v3.3/learning/why/ https://etcd.io/docs/v3.3/learning/why/
karsinkk 4y ago
This looks very impressive! As another commenter echoed, the code base is ~5million lines of C++ code, but almost no comments at all. Unless the documentation is excellent, maintenance/open source work is going to be difficult.
- xpl 4y ago
  The docs, for the reference: https://ytsaurus.tech/docs/en/ https://ytsaurus.tech/docs/en/ P.S. I wonder if LLMs could be used to generate docs and comments for big hairy codebases. Seems that the current generation of LLMs lack context to do it, but maybe it's "just one or two more papers down the line"®...
  - klysm 4y ago
    The cost of wrong docs is pretty high. You’d need someone knowledgeable to make corrections
ALLTaken 4y ago
I've read the whole article, it's very long, but it was written in such a refreshingly clean way. Hats-up! I believe the engineers really spent a lot of their time building this from the ground up. I'm thankful that those large companies open-source their software and algorithms.

275 points by xpl 4y ago

cynicalsecurity 4y ago

[flagged]

DanTheManPR 4y ago
As opposed to the IT department of the NSA?
omgtehlion 4y ago
Oh come on already... Of course FSB has a backdoor to *data* collected by Yandex, but the code itself (as well as coders) have nothing to do with any three-letter-agency within Russia.
- screamingninja 4y ago
  > the code itself (as well as coders) have nothing to do with any three-letter-agency within Russia. citation needed
  - dna_polymerase 4y ago
    Let's keep in mind, the only companies that were ever found to collude with three-letter agencies were the big players in the US. But for some reason, Yandex needs to prove something here…
  - eddsh1994 4y ago
    Why don't you read the code? The amount of people commenting on this, if there was something to hide they'd have probably caught it already and if not surely by next week.
- deleted 4y ago
  [deleted]
ivan_gammel 4y ago
Maybe this is exactly the reason why it’s worth checking? They open-sourced it.
- Aldipower 4y ago
  Yeah, like Apache Mesos is running the NSA data center in Utah. One of the biggest in the world.
  - mdaniel 4y ago
    Do you happen to have any link where I can read more about this?
bertil 4y ago
I’m all for holding companies that have supported dangerous regimes to account. However, when it comes to data management, totalitarian regimes rarely indicate inadequate implementation. IBM’s role in Germany in the 40s was horrific, but it proved their ideas of tabulations and files were promising. Just like with rocketry, there were many valuable things to learn that defined the rest of the XXth century. The FSB likely has a lot of crimes to atone for. Still, suppose one of their specialists publishes something on data management or how to manage hundreds of sock-puppet social media accounts. In that case, I’d be tempted to listen and learn from a likely expert—unless you suspect that they think this article is not sincere and meant as a distraction from actual good practices. Similarly, the CIA has done very problematic things, but the people who worked in the disguise department have a creative take on changing your appearance. I’m unsure when I would have to do that, but I’m always curious about how data is stored efficiently. And yes, like the FSB, the NSA has opinions about that, and those are typically well-informed. Was their practice constitutional? Seemingly not, IANAL. But do they have good insights into caching video files at scale? Definitely.
- medo-bear 4y ago
  > IBM’s role in Germany in the 40s was horrific, but it proved their ideas of tabulations and files were promising source: https://en.m.wikipedia.org/wiki/IBM_and_the_Holocaust https://en.m.wikipedia.org/wiki/IBM_and_the_Holocaust
throwaw12 4y ago
brainwashing at its peak. When USA does things, it is for the good of society, democracy. When Russia does things, it is hurting people, bad for society. Come on buddy, time to wake up and understand every country does things for its own good and whatever your media is telling about Russia is bad, it is because they're applying 3 letter agency brainwashing methods on to you. Code is open source, if you read code you will not get under Russian propoganda.
- garbagecoder 4y ago
  [flagged]
  - medo-bear 4y ago
    nice head in the sand
    - garbagecoder 4y ago
      That's a hilarious response from a Russia apologist. You think telling other people they're naive about the CIA means you can ignore all the bad stuff about Russia but that's not how the world works and you are either paid to have say this or are doing it for free and I'm not sure which is worse. I try not to put myself in the hands of any spy agency. "Other team bad so my team good" is reasoning barely worthy of the human mind.
      - medo-bear 4y ago
        there is no need to slander me. im definitely not paid by anyone to give my opinions imperialism is imperialism. it should be denounced if you dont think there is a US imperialist element involved in Ukraine you need to inform yourself a bit more. you can start with reading about North Stream 2 before the war in Ukraine i used to think that, in comparison the US, at least Russia bombs countries without calling it freedom [1]. When Russia started its "special operation" against Ukraine i now think that at best Putin is trolling the US, and at worst he is trying to be the US [1] https://en.m.wikipedia.org/wiki/Operation_Enduring_Freedom https://en.m.wikipedia.org/wiki/Operation_Enduring_Freedom
jones6ofMont 4y ago
[flagged]
- stef25 4y ago
  > remember FSB is NOT KGB because Russia now is a democratic country and not a communist USSR Secret services in a democratic country being better / worse than their communist counterparts aside, "democracy" is a bit of stretch when describing Russia.
- Idiot_in_Vain 4y ago
  Russia now is a democratic country??? Only a complete idiot would think that.
  - newaccount2023 4y ago
    [flagged]

garbagecoder 4y ago

Didn't this just leak?

xpl 4y ago
AFAIK it wasn't in the leak (that "YT" platform specifically). Also, open-sourcing proprietary projects of this scale is a pretty hard job and it can't be done quickly — seems they had been doing it for a long time, starting long before the leak.
- arivkin 4y ago
  That's true. We have been preparing for opening YT for a long time when we found out about the leak. We even joked inside that they couldn't even do it normally and we still have to open our part of the code ourselves :) It's truly a hard work, because we were very tightly tied to the Yandex infrastructure and we had to learn how to deploy in k8s from scratch. Also you need a new brand and clean your documentation from irrelevant things... All this takes months.

matrix_overload 4y ago

Looks like a reasonable response to a recent leak [0] [0] https://news.ycombinator.com/item?id=34525936 https://news.ycombinator.com/item?id=34525936

panki27 4y ago

Wasn't this "open sourced" in their recent sourcecode leak already?

omgtehlion 4y ago
No, it was not, see https://arseniyshestakov.com/2023/01/26/yandex-services-source-code-leak/ https://arseniyshestakov.com/2023/01/26/yandex-services-sour...

galkk 4y ago

There is a conspiracy theory that recent open sourcing of Yandex tech and to some extent even a leak is a preparation for global Yandex exodus from Russia.

slt2021 4y ago
exodus has already happened
speed_spread 4y ago
You can take Yandex out of Russia... but you can not take Russia out of Yandex. Admittedly, the same could probably be said about any other $big_search:$nuclear_power pair. It's just that the other companies either can't or have no reason for exile. And so I suspect Yandex leaving it's home turf would essentially be a covert invasion of wherever they'd swarm to. Somehow, money tells me UK would be a likely target. Source: worked briefly for an exiled Russian company. Would not repeat.
- ClumsyPilot 4y ago
  > a covert invasion of wherever they'd swarm to. > an exiled Russian company. Either you are exhiled or you are invading, how can you be both?
  - helge9210 4y ago
    For example, you wholeheartedly support the policy, but are not comfortable living under the sanctions.
    - ClumsyPilot 4y ago
      My understanding of Exile is that you are rejected by the society you are exiled from, due to your actions or beliefs. Like profound disagreement with the policy. I have certainly seen the kind of people you talk about, they support (or at least used to) Putin, but don't want their kids to live in Russia. I would call them more like emigrants of convenience.

reisse 4y ago

There is an interesting take that Russian part of the Yandex group opensources as much as possible, in order for the overseas companies of the group to leverage technologies without legal or financial ties with Russia. For me this seems very plausible, as for the last year they first did everything to distance from anything related to politics (e. g. they sold their news and their blogging platform to the basically state-owned VK), and then to separate Russian and overseas businesses as much as possible.

the_mitsuhiko 4y ago
Would be a clever move but I have doubts that no longer Yandex has much to say in Russian Yandex. The latter seems very close to the Kremlin now.
0xDEF 4y ago
Yandex is incorporated in the Netherlands and the founders live in Israel. I think they will create several spin-off open source companies (like ClickHouse Inc.) outside Russia to continue doing B2B business with the outside world.
- deepsun 4y ago
  It doesn't matter where the company is incorporated -- when Kremlin wants the access to the data Yandex cannot say no. One cannot operate in Russia and not play by Kremlin rules, especially media companies.
- konart 4y ago
  They did some time ago actually. Like https://nebius.com/about https://nebius.com/about for example. Not to mention that Yandex been operation in countries outside of Russia under different name from the beginning (like https://en.wikipedia.org/wiki/Yango_(ride_sharing) https://en.wikipedia.org/wiki/Yango_(ride_sharing))
- malaya_zemlya 4y ago
  Unforytunately, only one founder, Arkady Volozh, is still alive. Ilya Segalovich has died 10 years ago.

gre 4y ago

github link https://github.com/YTsaurus/YTsaurus https://github.com/YTsaurus/YTsaurus

perryh2 4y ago

[flagged]

maxdo 4y ago
It's a problem of entire Russia. My Asian girlfriend from china was denied to enter a restaurant in Moscow because of her origins... It's absolutely legal to put restrict your rental apartments nationality. In yandex you can ads like " will lease my house only to russians". It's everywhere. If you will ride a taxi in russia, an offensive words to other nationalities/ethnicities is everywhere. The problem is so big, that even non russian/slavic people applied to offend each other. I've been in a taxi when two middle east decent people start screaming to each other "churka" even they are both could apply to this definition. I personally trying not to invest into anything Russian, simply because that society is very, very sick and they are very far away even from recognition of this problem. They think it's their strength...
- TechBro8615 4y ago
  > I've been in a taxi when two middle east decent people start screaming to each other "churka" Another way of interpreting this sort of culture is that they know not to take things too personally and that intent behind words is more important than the words themselves. It's a kind of liberating way of interacting with each other, and not uncommon to see in environments like fraternities or sports teams. Some people will call that culture exclusionary, but I might call those people neurotic. To put it another way, who are you to be offended on behalf of the people calling each other churka? Clearly they're not offended by it, so shouldn't you let them have their fun?
  - 2-718-281-828 4y ago
    [dead]
- konart 4y ago
  > My Asian girlfriend from china was denied to enter a restaurant in Moscow because of her origins... While I'm not questioning your statement - this sound very questionable, considering the number of Chinese tourists in Moscow every year. >It's absolutely legal to put restrict your rental apartments nationality >I personally trying not to invest into anything Russian, simply because that society is very, very sick Do you as a gaijin invest in anything japanese in this case? It is not uncommon to see a "gaijins are not allowed" sign in Japan either. While it is true that Russia has (and will probably have for a long time) some racism problems (just like superiority complex) - I wouldn't say it's as bad as you think it is (especially compared to 90s for example)

sciencesama 4y ago

Who cares ?

konart 4y ago
Same people who use Clickhouse or YDB at least.

IYasha 4y ago

Funny to see 0 of 50 comments on a HN tech article being tech-related...

booi 4y ago
I think what you're seeing is a resurgence of tech ethics where the source and support of a project is as important as the technology itself.
hotstickyballs 4y ago
It’s not technically interesting since a lot of big data solutions already solve this problem so I guess only the geopolitics are left.
- xpl 4y ago
  I doubt about "a lot". Also, "already solve" does not mean "solve better" or even "good enough". It would be very interesting to see some in-depth comparisons with already-existing open source technology (like Hadoop, Hive, Iceberg, ZooKeeper) to get a sense of when and where YT could be more effective.
- hamilyon2 4y ago
  I am obviously biased, but, yes, technically it is very, very interesting. Distributed transactions, kiparis, YQL are interesting. Another aspect, the only other open alternative is Hadoop, hbase and hive. They don't compete with yt on usability and developer experience aspects. Yt is much more polished, despite historical quirks.
l0b0 4y ago
Agreed, but this is presumably also an absolutely massive project hardly anyone here has even used before. So it's not surprising that there are no big tech insights on the day of the release. An `scc` printout might be interesting, but any in-depth analysis is going to take a long time.
- arivkin 4y ago

g9yuayon 4y ago

The github repo has 5 million lines of C++ code (headers included), 1.6 million lines of C code, and even nearly 1 million lines of Scala + Java code. We'd need some serious docs to adopt this technology. The most interesting part of YT is Cypress. I'm particularly interested in how they make their master cluster horizontally scalable.

gritukan 4y ago
Historically, the master server of YTsaurus was a single RSM (replicated state machine) that contained all the meta-information about the cluster. This included the tree of the distributed filesystem, transactions, information about users and tables, placement of chunks, and much more. However, this approach proved to be non-scalable as the memory amount and throughput of the master server soon became insufficient. To address this issue, we implemented Multicell technology. With Multicell, there are multiple RSMs called secondary masters that store information about chunks of the tables and their placement. The primary master still stores information about the distributed filesystem and transactions but is now single and non-sharded. After a few years, the masters became overloaded again, and we implemented Portals. With Portals, one can select a subtree of Cypress and place it in one of the secondary masters. This technology is used nowadays, and home directories of some active users are hosted on secondary masters. However, we anticipate that this approach will also become insufficient in a few years. Therefore, we are currently working on a new technology called Sequoia, which stores information about the Cypress tree shape in horizontally scalable dynamic tables. It is hard to describe all aspects of master server internals in one comment. Therefore, feel free to join our chat at t.me/ytsaurus for further discussion!
- ddorian43 4y ago
  > Therefore, we are currently working on a new technology called Sequoia, which stores information about the Cypress tree shape in horizontally scalable dynamic tables. Why not just use a database for the metadata? Something that can be sharded and has transactions like YugabyteDB/Yandex-ydb/etc?
  - 4y ago

magundu 4y ago

Is it an alternative to Snowflake?

rch 4y ago
I work at Cloudera and I see it as roughly analogous to CDP, but I didn't see anything about hybrid on-prem/cloud deployments.
- arivkin 4y ago
  It's true. We are very close to CDP and Apache Hadoop. We are only about on-prem right now. It's not a secret that we didn't need a cloud deployment at Yandex. But we see and understand the demand for it. And we have a plan:)
  - rch 4y ago
    Interesting! I'm looking forward to seeing what you have lined up.
- deleted 4y ago
  [deleted]
alephone1 4y ago
We provide YQL for running large-scale OLAP SQL queries. In this regard YT can be compared to Snowflake. However, we target on prem deployment in the first place, while Snowflake runs in aws/gcp/azure, and queries are performed over data that sits in S3.

ushakov 4y ago

How many companies exist that will utilise its full capabilities? My bet would be 50-100

arivkin 4y ago
Why do you think so? How many companies use Apache Hadoop?
RobotToaster 4y ago
Most people don't utilise the full capabilities of the tools they use. Sure it would be overkill for a lot of applications, but so is redis, react.js, etc.
xpl 4y ago
My guess it could be successfully used for relatively small (terabyte-scale) workloads. It's the same as when people use k8s not utilizing its full capabilities, only to be able to massively scale up when needed.

VWWHFSfQ 4y ago

It's my understanding that the original Yandex founders and tech left Russia several years ago. Is this new technology? Or is it stuff being extricated from the (now) Kremlin-controlled codebase?

xpl 4y ago
It isn't new, in the article they state that it originated back in 2006. It's been in development since then.

gritukan 4y ago

Hello! I work at YT and would like to answer a question that was asked in a flagged thread about the comparison between YT and Hive and Zookeeper. Both Cypress and Zookeeper are fault-tolerant distributed hierarchical filesystems that can be used for distributed coordination, but Cypress has much richer functionality. Recall that Zookeeper's data model is just a tree consisting of homogeneous nodes that can be either ephemeral or persistent, along with a set of sessions that control the lifetime of ephemeral nodes. This simple model allows to implement multiple primitives of distributed synchronization, such as leader election, exactly-once queue processing, or two-phase commits. However, it is not always easy to integrate Zookeeper with third-party systems. For example, if you want to elect a leader via Zookeeper and use it to insert data into a database, it is mandatory that the instance remains the leader during the commit into the database, which is not easy to implement without races or some additional assumptions. In YTsaurus, transactions permeate our entire system. You can start a transaction and acquire an exclusive lock at some Cypress node (which is a way to make a leader election), and after that, the transaction becomes the leader lease. You can then modify Cypress, run MapReduce or YQL operations using the transaction as a prerequisite, lock some files and tables in the same transaction, and do many other things. Currently, we are working on the ability to use Cypress locks as prerequisites for dynamic table commits. There are many other features in Cypress that are not implemented in Zookeeper, such as symlinks, automatic expiration of unused nodes, and many others. Moreover, Cypress can be sharded using Portals about which I wrote in a previous comment, so this filesystem is scalable unlike Zookeeper. Even without sharding a single primary master of YTsaurus can hold tens of gigabytes of metadata of Cypress while Zookeeper state size is limited with hundreds of megabytes accoring to etcd vs Zookeeper comparision [1]. One major disadvantage of Cypress compared to Zookeeper is the lack of watches, so all changes tracking should be done via short polling. The good news is that Cypress is well-optimized for read queries with the possibility to read from followers and from multiple threads, so this is not a big problem. In the meantime, we are considering the possibility of adding some kind of watches to Cypress. The big difference between Cypress and Zookeeper is the replicated state machine implementation. With all due respect to Zookeeper developers, Zookeeper was implemented over 15 years ago when the world of distributed algorithms was different. Today we see that ZAB (the consensus algorithm used in Zookeeper) has some shortcomings in failover speed and stability. There are multiple reports of Zookeeper being unstable under heavy load. In YTsaurus, we use an in-house library called Hydra for RSM implementation. This is our consensus algorithm very similar to RAFT that has proven itself to be both efficient and fault-tolerant. We use Hydra for master servers, clock servers, and tablet cells (RSMs that store data in dynamic tables). I even had an idea to implement a Zookeeper API using Hydra both to simplify migration to YTsaurus and check Hydra performance and correctness via multiple tests implemented for Zookeeper (Jepsen, for instance), but did not have enough time to finish this project. This comment is already quite long, so I will write about the YT vs Hive comparison in another comment later on. [1] -- https://etcd.io/docs/v3.3/learning/why/ https://etcd.io/docs/v3.3/learning/why/

karsinkk 4y ago

This looks very impressive! As another commenter echoed, the code base is ~5million lines of C++ code, but almost no comments at all. Unless the documentation is excellent, maintenance/open source work is going to be difficult.

xpl 4y ago
The docs, for the reference: https://ytsaurus.tech/docs/en/ https://ytsaurus.tech/docs/en/ P.S. I wonder if LLMs could be used to generate docs and comments for big hairy codebases. Seems that the current generation of LLMs lack context to do it, but maybe it's "just one or two more papers down the line"®...
- klysm 4y ago
  The cost of wrong docs is pretty high. You’d need someone knowledgeable to make corrections

ALLTaken 4y ago

I've read the whole article, it's very long, but it was written in such a refreshingly clean way. Hats-up! I believe the engineers really spent a lot of their time building this from the ground up. I'm thankful that those large companies open-source their software and algorithms.