2 ms·

I found the article very helpful from a technical perspective, and didn't focus on the number too much, as it could easily be swapped and the decision-making pr

I found the article very helpful from a technical perspective, and didn't focus on the number too much, as it could easily be swapped and the decision-making process for when to shard is kinda out of scope. But I hadn't considered this, so thanks for pushing back. Good to keep in mind their incentives. I will say, since their product is a proxy whose interface is a single SQL connection, you should in theory be able to do dev queries through that black box, much the same as application queries? What is so scary here that it would require a hundred engineers?

groundzeros2015 3mo ago
> you should in theory be able to do dev queries through that black box Because it’s a leaky abstraction which is trying to make guarantees over network connections which are extremely difficult to make within the same kernel. A few questions I would start with: - is the system even ACID compliant? In my reading of this article, no. - is my sql feature set limited? Will it enforce all constraints? Or are their cross-shard limitations? For example the article doesn’t discuss transactions or how they would roll back, or how they guarantee a consistent view of data. Next: - how does it respond to error? - how does it respond to load? This is a complex system and complexity breeds bugs. But now you don’t have the tools or procedures to investigate those bugs because you can’t poke the system at desk with tools. You can’t run experiments; you can’t even see all the data.
- jackb4040 3mo ago
  Thank you, good points. I'm just learning about this in real time. It looks like it does support transactions, but they basically destroy the performance benefits: https://vitess.io/docs/faq/sharding/advanced/can-i-use-vitess-to-do-cross-shard-joins-or/ https://vitess.io/docs/faq/sharding/advanced/can-i-use-vites... The more I read the more I'm struggling to understand the benefit of a router like this that sits on top of a monolithic SQL, vs a truly distributed DB like cockroach. Like you I'd love to learn more about the internals of their actual SQL engine, which is just barely touched on in the article. The idea of ripping out the layer of a SQL interpreter that does just enough to route it to a real server tickles my brain in the same way as when I learned how node.js ripped the js interpreter out of a browser.
  - groundzeros2015 3mo ago
    What problem are you trying to solve? What did you try? Unfortunately the SQL language is the simplest part of the database. The concurrency and consistency guarantees are the key technology,

groundzeros2015 3mo ago

> you should in theory be able to do dev queries through that black box Because it’s a leaky abstraction which is trying to make guarantees over network connections which are extremely difficult to make within the same kernel. A few questions I would start with: - is the system even ACID compliant? In my reading of this article, no. - is my sql feature set limited? Will it enforce all constraints? Or are their cross-shard limitations? For example the article doesn’t discuss transactions or how they would roll back, or how they guarantee a consistent view of data. Next: - how does it respond to error? - how does it respond to load? This is a complex system and complexity breeds bugs. But now you don’t have the tools or procedures to investigate those bugs because you can’t poke the system at desk with tools. You can’t run experiments; you can’t even see all the data.

jackb4040 3mo ago
Thank you, good points. I'm just learning about this in real time. It looks like it does support transactions, but they basically destroy the performance benefits: https://vitess.io/docs/faq/sharding/advanced/can-i-use-vitess-to-do-cross-shard-joins-or/ https://vitess.io/docs/faq/sharding/advanced/can-i-use-vites... The more I read the more I'm struggling to understand the benefit of a router like this that sits on top of a monolithic SQL, vs a truly distributed DB like cockroach. Like you I'd love to learn more about the internals of their actual SQL engine, which is just barely touched on in the article. The idea of ripping out the layer of a SQL interpreter that does just enough to route it to a real server tickles my brain in the same way as when I learned how node.js ripped the js interpreter out of a browser.
- groundzeros2015 3mo ago
  What problem are you trying to solve? What did you try? Unfortunately the SQL language is the simplest part of the database. The concurrency and consistency guarantees are the key technology,