3 ms·

You don't allow N O(1) queries, you allow k O(1) queries, where k is a small constant independent from N. It is simply a physical reality that a massively scala

You don't allow N O(1) queries, you allow k O(1) queries, where k is a small constant independent from N. It is simply a physical reality that a massively scalable realtime system cannot run anything other than a small number of log(N) queries. You can't run N O(1) queries or one O(N) query - there is just no way to evaluate it in real time. If you need to present the result of such a query to users, you can run it as an analytical query and cache it, but you will not be able to run it on every request.

justinsb 17y ago
But the users are going to issue those N queries while you're still at the whiteboard talking about big-O notation. Exposing a full query language to the database system means that you have the opportunity to optimize the query fully. Limiting the query language only limits your ability to optimize, so you won't even be able to explain to the user that there's an alternative way to get their answer without bringing down the database. Are there limitations of databases such that it is possible to come up with better query execution strategies than the database can figure out automatically - of course there are! But in that case, fix the query optimizer or fix the database. Don't try to solve the problem by redefining it.
- coffeemug 17y ago
  If the query language is designed with this in mind, the compiler would detect that a non-realtime query is passed to a database marked as realtime and would throw an error. Of course the user could issue N realtime queries from within a host language, but that would involve really trying to work against the database. A good analogy might be CORBA/DCOM vs. SOAP. In DCOM days it was easy to call many different functions while being oblivious that each call requires a network roundtrip. Most programmers made this mistake most of the time. With SOAP, you can still work this way, but it makes the likelihood of this happening in practice significantly lower because it exposes the issue to the programmer. Regarding your argument of a fuller optimization, presumably a well designed query language would be flexible enough to allow the user to express every possible realtime query (assuming this is theoretically possible), so the user doesn't need to combine smaller queries via host-language trickery. People have been trying to fix the optimizer for decades, but most production optimizers still have plenty of edge cases. Perhaps redefining the problem is exactly what the doctor ordered?
  - justinsb 17y ago
    Nice theory, but consider that the N+1 problem is so widespread that it actually has a name, and I think you'll realize that people are a much harder problem than you think. And this isn't even a place where SQL is at all problematic! Perhaps this supports your DCOM argument though, in that it's the abstraction layers that really cause the problem. That said, it's great that you're re-examining the problem; that's how progress is made.
- gruseom 17y ago
  Limiting the query language only limits your ability to optimize This seems backwards. The more general a language is, the harder it is to analyze and transform, and that is what optimizers do.
  - justinsb 17y ago
    It might be hard to optimize a more general language, but it's impossible to optimize something that can't be expressed in a simple language.

justinsb 17y ago

But the users are going to issue those N queries while you're still at the whiteboard talking about big-O notation. Exposing a full query language to the database system means that you have the opportunity to optimize the query fully. Limiting the query language only limits your ability to optimize, so you won't even be able to explain to the user that there's an alternative way to get their answer without bringing down the database. Are there limitations of databases such that it is possible to come up with better query execution strategies than the database can figure out automatically - of course there are! But in that case, fix the query optimizer or fix the database. Don't try to solve the problem by redefining it.

coffeemug 17y ago
If the query language is designed with this in mind, the compiler would detect that a non-realtime query is passed to a database marked as realtime and would throw an error. Of course the user could issue N realtime queries from within a host language, but that would involve really trying to work against the database. A good analogy might be CORBA/DCOM vs. SOAP. In DCOM days it was easy to call many different functions while being oblivious that each call requires a network roundtrip. Most programmers made this mistake most of the time. With SOAP, you can still work this way, but it makes the likelihood of this happening in practice significantly lower because it exposes the issue to the programmer. Regarding your argument of a fuller optimization, presumably a well designed query language would be flexible enough to allow the user to express every possible realtime query (assuming this is theoretically possible), so the user doesn't need to combine smaller queries via host-language trickery. People have been trying to fix the optimizer for decades, but most production optimizers still have plenty of edge cases. Perhaps redefining the problem is exactly what the doctor ordered?
- justinsb 17y ago
  Nice theory, but consider that the N+1 problem is so widespread that it actually has a name, and I think you'll realize that people are a much harder problem than you think. And this isn't even a place where SQL is at all problematic! Perhaps this supports your DCOM argument though, in that it's the abstraction layers that really cause the problem. That said, it's great that you're re-examining the problem; that's how progress is made.
gruseom 17y ago
Limiting the query language only limits your ability to optimize This seems backwards. The more general a language is, the harder it is to analyze and transform, and that is what optimizers do.
- justinsb 17y ago
  It might be hard to optimize a more general language, but it's impossible to optimize something that can't be expressed in a simple language.