24 ms·
YDB – An open-source Distributed SQL Database
- adorable_monkey 5y ago
- gtirloni 5y agoI couldn't find anything about the wire protocol being compatible with MySQL or PostgreSQL and it seems to need a specific SDK. This will limit adoption considerably.
- outcoldman 5y agoYeah, seems like their SDK are required https://ydb.tech/en/docs/reference/ydb-sdk/ https://ydb.tech/en/docs/reference/ydb-sdk/
- acatton 5y agoThe wire protocol is grpc https://github.com/ydb-platform/ydb-go-sdk/blob/98938c92a9ff9c547151e258458a68393c7961ef/connection.go#L8 https://github.com/ydb-platform/ydb-go-sdk/blob/98938c92a9ff...
- fomichev3000 5y agoThat's right, wire protocol is grpc. SQL dialect is called YQL. YDB doesn't have MySQL or Postgres compatibility, but we think about it.
- euos 5y ago
- eatonphil 5y agoIt looks [0] like they use PostgreSQL's parser library under the hood which is cool. [0] https://github.com/ydb-platform/ydb/tree/main/ydb/library/yql/parser/pg_wrapper/postgresql/src https://github.com/ydb-platform/ydb/tree/main/ydb/library/yq...
- JeopardyJJJ 5y ago
- nik736 5y agoA bit off topic but what are all those devs at Yandex doing right now?
- the_mitsuhiko 5y agoFrom what I have gathered the atmosphere in Yandex is loaded and complex. Yandex is a huge beneficiary from the war but a lot of people in the company are conflicted about it.
- mardifoufs 5y agoHow does it benefit from the war? I had the impression that they were caught in a pretty bad situation due to their outstanding debts and their suspension from nasdaq so that's very surprising!
- machinekob 5y agoProbably cause all of the other cloud providers are limiting operations in Russia
- gaploid 5y agoI believe cloud revenue in Yandex structure is quite low and main business in search/ads
- euos 5y agoIt is not about revenue. They are basically state corporation.
- vasilia 5y agoSearch/ads and ride/delivery services. As you may know Yandex has very advanced SDC system which was used(SDC robots) by Grubhub in Arizona University. They have delivery services in London/Paris and taxi services in Insrael and couple European countries. As for the Russia I found news(in Russian) that McDonalds was huge part of orders in delivery service. Ads system fully broken because of Google suspension. Without concurents average prices will rise exponentially. Small business will not be able to buy ads and will closed in near feature.
- julienmarie 5y agoIt seems they are doing the same thing they did with ClickHouse in the OLAP space but this time in the OLTP space.
- fomichev3000 5y agoYes, you can think this way. But I need to add, that YDB is also a platform for developing distributed systems that store data. YDB provides a scalable and replicated storage with low latency, a conception of a tablet (that is also used in many systems) that implements distributed consensus. These building blocks are used for persistent queue implementation, block store, KV-tablets. These blocks are hard to develop and they are very good when you need to build something new or optimal for a specific problem. OLTP is an example of such a problem. But yes, we were building YDB to support OLTP workload initially.
- wnolens 5y agoVery cool.
- agnesv 5y agoHow is it compared to ClickHouse?
- gaploid 5y agoClickHouse is OLAP (analytics), this one is OLTP (transactional)
- julienmarie 5y agoHence the focus on multi-AZ. This makes completely sense for transactional.
- gaploid 5y agoI believe that something like AZure CosmosDB, AWS Aurora/DynamoDB but open-sourced, which is great.
- samber 5y agoor cockroachdb and spanner ?
- fomichev3000 5y agoYes, YDB plays in this category.
- c4pt0r 5y agoI think YDB's architecture is more like TiDB
- fomichev3000 5y agoYDB can work in a single AZ and such setups exist. Most of our users want to handle AZ downtime, so it's a major case.
- squarefoot 5y agoTo those condemning Yandex for the war in Ukraine, please keep in mind that they're a Russian company forced to comply with their country laws, as every other company out there. Until we know for sure that their management is really aligned with their government agenda, whatever they say, or don't say, now has zero credibility because it could be extorted from above. During the 2nd gulf war against Iraq, when the death toll began to rise, the US government forbade the reporting of military personnel coffins returning from the war zone, and news sources complied, including those against the war; this happens everywhere every time. What doesn't happen everywhere and every time is a rogue nation attacking their neighbors and killing civilians using laughable pretexts while the real goal is to take control of a very resourceful and industrialized area (Donbas) to be used to help the already poor Russian economy. The enemy is the Russian government and the rich oligarchy behind it, not Russian people that could be either lied and manipulated by ruthless politicians (been there, done that; does Covid19 ring a bell?) or merely intimidated.
- dotdi 5y agoTrue, but on the other hand, people outside of Russia can decide for themselves how to react to and treat companies from unfriendly countries. You mention oligarchs - how can it be ruled out that Yandex is directly or indirectly, fully or partly owned, and even directed, by them?
- app4soft 5y ago> people outside of Russia can decide for themselves how to react to and treat companies from unfriendly countries. Since 2019 Yandex and the FSB (Federal Security Bureau of Russia, ex-KGB) had reached an agreement where the company would provide the required data without handing over the encryption keys.[0] Learn more about Russian war crimes & ongoing invasion of Ukraine: https://en.wikipedia.org/wiki/2022_Russian_invasion_of_Ukraine https://en.wikipedia.org/wiki/2022_Russian_invasion_of_Ukrai... N.B. I'm Ukrainian living in Ukraine. Here is my statement for HN:[1] + Verified Ways to Help Ukraine: https://euromaidanpress.com/2022/02/27/verified-ways-to-help-the-ukrainian-army/ https://euromaidanpress.com/2022/02/27/verified-ways-to-help... + My Patreon: https://patreon.com/app4soft https://patreon.com/app4soft [ Please, DONATE me just right now ] [0] https://en.wikipedia.org/wiki/Yandex#Security https://en.wikipedia.org/wiki/Yandex#Security [1] https://news.ycombinator.com/item?id=30395897 https://news.ycombinator.com/item?id=30395897
- addisonj 5y agoInteresting, the separate compute and storage tiers is another system going that direction which I think is becoming almost the standard at this point, especially for "cloud-native" things designed to run on k8s. From what I can tell (it isn't very explicit on this point) they are avoiding a distributed consensus at the storage layer and instead relying on a single writer/multiple reader model with the single writer being enforced by assignment of the tablets in the compute tier, with the tablet being responsible for writing to multiple storage nodes for durability? (But I might be wrong) Assuming yes this approach, I think, is under utilized and is pretty similar to how Apache Pulsar works (my day job),but I am not sure how many distributed RDBMS have tried it out, will be cool to see how it evolves! It isn't clear how they ensure the assignment of a tablet to only a single compute node, but I think that is an easier problem relative to distributed consensus at the storage tier.
- jeffbee 5y agoAssigning leaders is trivial with something like zookeeper. But in this case it appears that the leader metadata is stored in a table of the database itself, which raises questions of operability if those tablets are unavailable.
- fomichev3000 5y agoYDB doesn't use Zookeeper. The system is built of tablets, every tablet implements distributed consensus algorithm. There are different types of tablets in the system, say SchemeShard is tablet that stores metadata, table schema for instance. DataShard stores table partition data.
- asadawadia 5y agoWhich consensus algorithm? I don't see raft in the codebase
- fomichev3000 5y agoWe have consensus protocol over shared log. Some details you can find in a power point presentation at this page https://2019.hydraconf.com/2019/talks/muxomfgqembsb3st7i3ci/ https://2019.hydraconf.com/2019/talks/muxomfgqembsb3st7i3ci/
- samber 5y agoI found a lot of mentions of "Postgresql" in repository: https://github.com/ydb-platform/ydb/search?q=postgresql https://github.com/ydb-platform/ydb/search?q=postgresql Is it build on top of PG ?
- jenny91 5y agoI'd almost guess so. Yandex is one of the largest production users of pg.
- fomichev3000 5y agoYDB is developed from ground up. We think about postgres compatibility layer, that's why you see 'postgres' in source code.
- gaploid 5y agoit seems not. they have compatibility for querying external PG dbs (YQL query engine), I thing that's the reason for that.
- polskibus 5y agoI would love to see it undergo a Jepsen treatment and a teardown by Andy Pavlo.
- konart 5y agoThey do mention Andy Pavlo in this publication by the way: https://habr.com/ru/company/yandex/blog/660271/ https://habr.com/ru/company/yandex/blog/660271/ saying that YDB was inspired by his and Michael Stonebraker's NewSQL ideas.
- MichaelMoser123 5y agohere is a table that compares YDB to postgress https://db-engines.com/en/system/PostgreSQL%3BYandex+Database https://db-engines.com/en/system/PostgreSQL%3BYandex+Databas... I think they wanted to have a DB that is better tuned to distributed systems. Still don't know, why they do an SQL like query language called YQL (what would that mean in practical terms? Could a common ORM framework like JPA deal with the YQL query language ?)
- orthoxerox 5y agoAs far as I understand, YDB is different enough from regular RDBMS's that they don't provide an ODBC/JDBC/ADO.Net driver.
- SomeCallMeTim 5y agoIf that comparison table is right about "no foreign keys," that's a showstopper for me. :( Back to looking at Couchbase, Yugabase, or Citus for my distributed SQL.
- keredson 5y agothat's very common in distributed databases. even traditional databases, it's very common to not have FKs on large tables, and just handle it in software. indexing billions or more of rows is non-trivial.
- MichaelMoser123 5y agodidn't notice that. Wow that's pretty basic...
- reactor 5y agoI guess its depends, for many use cases, it can be managed at application level. I've parted ways with FK for a long time since it created more hassles than it solved esp when it comes to sharding and replications.
- SomeCallMeTim 5y ago
- manishsharan 5y agoIs it ethical to recommend Clickhouse? I had done a PoC a long while ago before the war but I am not sure it is ethical to recommend this to enterprises given its origins?
- tylerhannan 5y agoQuestions of ethics are interesting... As a disclaimer, I work at ClickHouse looking after DevRel. I can’t speak for what you find appropriate for your needs. But, I believe our perspective is clear. If you didn’t see it: https://clickhouse.com/blog/we-stand-with-ukraine/ https://clickhouse.com/blog/we-stand-with-ukraine/
- manishsharan 5y agoThanks. This is really good to know. Enterprise stake holders are a jumpy lot and this will help.
- zepearl 5y agoThanks - I was wondering about Clickhouse and I feel better after having read your post :)
- zX41ZdbW 5y agoClickHouse has no longer any ties to Russia.
- lizardactivist 5y agoVery impressive project!
- jeffbee 5y agoSeems to stake out some unclaimed territory in between bigtable and vitess, which is cool because I dislike the design of vitess and you can’t download bigtable. Very interested to try this.
- KronisLV 5y agoCurrently seems to support a number of languages: https://ydb.tech/en/docs/reference/ydb-sdk/install https://ydb.tech/en/docs/reference/ydb-sdk/install - Python - Go - .NET - Java - PHP Has some example code here: https://ydb.tech/en/docs/getting_started/sdk https://ydb.tech/en/docs/getting_started/sdk
- flakiness 5y agoAlthough the doc talks about their own SQL dialect "YQL", it seems to be supporting PG SQL as a compatible layer. https://github.com/ydb-platform/ydb/tree/main/ydb/library/yql/sql https://github.com/ydb-platform/ydb/tree/main/ydb/library/yq... It's fascinating to see the PG's prevalence as the de-fact SQL standard.
- isoprophlex 5y agoimo having had to deal with oracle sql, teradata sql, mssql and mysql, postgres as a dialect is such a sane and consistent experience. Also the ecosystem... The official docs basically solve your problems without much fuss, you don't have to rely on horrible vendor-specific forums.
- awild 5y ago> The official docs basically solve your problems without much fuss, you don't have to rely on horrible vendor-specific forums. tdodbc also comes with examples that are so far beyond the scope of a normal persons usecase, they honestly just feel like some greybeard doing the equivalent of a 10 minute long Tony Hawk combo. It's just such a pain in the butt to use, we had to fight teradata all the way till very recently.
- fomichev3000 5y agoWhile you have a good point, I need to say that traditional databases and distributed databases differ in some points. For instance in YDB you can connect to any node of a database, it means that you need some kind of balancing (server side or client side). I'm not talking about SQL dialect right now, but rather how you connect to database and how you handle connection losses or node overloads. YDB SDKs have client side balancing feature to distribute load evenly across database nodes.
- fomichev3000 5y agoPostgres support is in progress actually.
- 5y ago
- peapod91 5y agoAre there any notable differences or benchmarks between YDB vs CockroachDB, Spanner, or Yugabyte?
- fomichev3000 5y agoWe have benchmarks in our roadmap, we understand there are required.
- sirjaz 5y agoI don't see any plans for support of Windows Server. Do you know of any plans for it? Since it still has about 75% of the on-prem server market
- fomichev3000 5y agoThanks for pointing this out. I think it's possible, most of the code is cross platform, we used to build YDB on Windows, so it should not be a problem. I suppose we need to improve a layer that works with storage devices to make it effective, for now it should be very naive implementation.
- porkradish 5y agoI noticed under the limits section of the documentation that a query result can have no more then 1000 rows. Anything more is truncated. I am surprised to see such a low limit. Maybe I misread something?
- gaploid 5y agoThat's something similar to DynamoDB and CosmosDB. I believe the workaround is to have paging on client side to fetch all results.
- kovalad 5y agoIt is correct, there is a limitation for queries executed via ExecuteDataQuery method To retrieve more than 10 000 rows Scan Queries could be used -- more details in documentation https://ydb.tech/en/docs/concepts/scan_query https://ydb.tech/en/docs/concepts/scan_query
- rbranson 5y agoWhat's the core ordering mechanism behind the serializability for arbitrary distributed transactions? This would be something like TrueTime in Spanner or FoundationDB's GRV/commit proxies.
- fomichev3000 5y agoThe core tx ordering mechanism is inspired by Calvin deterministic transactions mechanism. We have some differences, like we have a distributed tx coordinator that allow us to scale number of transactions in flight.
- infologics 5y ago[dead]
- yamrzou 5y agoAre YDB and Clickhouse developed by the same developers at Yandex? Or, at least, do they have the same performance-oriented mindset? Asking because I pretty much like Clickhouse.
- fomichev3000 5y agoThe teams are approaches are different, but I need to say that internal clients are the same. Yandex is doing great things and it needs great tools.
- the-alchemist 5y agoAnyone know a difference between Yugabyte and YDB? Both seem to be in the OLTP space. I actually thought Yugabyte renamed itself YDB. :(
- fomichev3000 5y agoThey are totally different systems. Just they have a first letter 'Y'.
- chalcolithic 5y agoHow much of a stretch it is to say that YDB is a competitor to NDB Cluster?