4 ms·

"But if the new Moore’s law gives us exponential core-number growth, to exploit those cores we need to grow the number of shards exponentially as well, meaning

"But if the new Moore’s law gives us exponential core-number growth, to exploit those cores we need to grow the number of shards exponentially as well, meaning we’re placing and exponential number of constraints on an application that probably only needs to accommodate a linearly increasing number of users, and that’s bad." I'm no expert on parallel programming, but isn't this sloppy thinking? Like the implication is that we must have an exponential increase in cores in order to accommodate a linear number of users... More generally is anything about this approach really novel? Databases can already run transactions and queries asynchronously. Any language that supports concurrent programming can take advantage of this, even if the database isn't issuing a callback. And ruling out concurrent languages that are capable of implementing this approach, but include some language features the authors don't like seems like a side issue. Not just trying to snark... if someone can explain what I'm missing I'm open to it!

pron 14y ago
> Like the implication is that we must have an exponential increase in cores in order to accommodate a linear number of users... As the article says, you can just ignore those extra cores. But, if you want to do cooler stuff with your data you'd better use them, and if so, sharding is a dead-end. > More generally is anything about this approach really novel? I said it's a repurposing of an old idea, that of the old RDMSs where the application was the database. The idea is not about concurrent transactions but about uniting the application and the database and letting the database run and parallelize the application's logic, and be the engine of scaling. I think most people see the database simply as a data storage component, and try to make it fast and scalable enough. We say, let the database run the application. > And ruling out concurrent languages that are capable of implementing this approach... I haven't ruled out any language. In fact, the demo is in Java, which certainly doesn't qualify. Any language can do this. I only said that in order to go forward, in order for the common programmer to take advantage of modern hardware, we must use languages built specifically for modern hardware. I've named two that I think fit the bill, but if other languages end up becoming the language for the new age - it's all the same.
- threepipeproblm 14y ago
  Thanks for your response. I think I understand your point re: cores, although I wouldn't have used the argument quoted above, instead I would just make the point in your response - we have all these cores, gotta use them. Since I found some things to nitpick I want to say I definitely like the approach you are coming at this from -- in terms of how you apparently view sharding, NoSQL, database as a processing engine, etc. But I still think you are presenting it as something novel (just look at the title) when what you are really doing is promoting a long-established but sorely neglected philosophy... a good philosophy, which needs to be promoted. The novel aspect seems to be supporting callbacks in your Spacebase product, which isn't particularly essential to this philosophy. Thanks for the article.
  - pron 14y ago
    I think callbacks are absolutely essential. How else is the db supposed to schedule and run your business logic?
    - threepipeproblm 14y ago
      Okay I have been thinking about your response for a while... Database engines already possess sophisticated scheduling logic for query processing. We are constantly taking advantage of that, in different ways, when we delegate to the database. And even though we are supposed to have all these business layers and whatnot, I think centralizing logic in a DB is a pretty common practice. So I have been asking myself, is there a significant functional difference between the db doing a callback to the program, versus other methods of interacting with that built-in scheduling engine? My supposition is that in the absence of such a capability, one could always get mostly (or fully) equivalent behavior by some combination of (A) having the app fire queries (and consume their results) asynchronously (B) giving queries some other means of initiating a response, e.g. writing to a table that acts as a message queue. To me it seems like such a transformation wouldn't even change the balance of code between app vs DB all that much, it would just change the calling mechanism, and maybe you need these couple of extra modules to manage your query submissions and asynchronously dequeue stuff. So I admit this qualifies as more than syntactic sugar, but not sure goes much beyond that. Of course, many prized language features don't go beyond that at all; and calling/concurrency idioms are important. I like the idea of centralizing logic in the DB and I can see how your approach allows this in a much more elegant and straightforward manner. So I think I get it. Whether it's a totally new architecture, I dunno... I come out of this thinking if there was a way to improve your article it would perhaps be to avoid the claim that this is allowing us to take advantage of database to "schedule and run" whatever business logic is implemented in the db via the highly optimized query processing facilities. All programs effectively do that in their use of the database, which is why the scheduling capabilities exist in advance of your approach. The core thread of what you are doing/advocating, as I understand it, is something like this. (a) Centralize business logic in the DB. (b) Stop being ashamed of it. (c) Support calling conventions that make it easier. If you made the argument that way, I would have seen it as a slam dunk. And I would expect that to be a controversial argument on HN. Interested in your thoughts. (BTW, I bet one could do these sorts of callbacks using the relatively recent capability of SQL Server to run Stored Procedures written in .NET.)

pron 14y ago

> Like the implication is that we must have an exponential increase in cores in order to accommodate a linear number of users... As the article says, you can just ignore those extra cores. But, if you want to do cooler stuff with your data you'd better use them, and if so, sharding is a dead-end. > More generally is anything about this approach really novel? I said it's a repurposing of an old idea, that of the old RDMSs where the application was the database. The idea is not about concurrent transactions but about uniting the application and the database and letting the database run and parallelize the application's logic, and be the engine of scaling. I think most people see the database simply as a data storage component, and try to make it fast and scalable enough. We say, let the database run the application. > And ruling out concurrent languages that are capable of implementing this approach... I haven't ruled out any language. In fact, the demo is in Java, which certainly doesn't qualify. Any language can do this. I only said that in order to go forward, in order for the common programmer to take advantage of modern hardware, we must use languages built specifically for modern hardware. I've named two that I think fit the bill, but if other languages end up becoming the language for the new age - it's all the same.

threepipeproblm 14y ago
Thanks for your response. I think I understand your point re: cores, although I wouldn't have used the argument quoted above, instead I would just make the point in your response - we have all these cores, gotta use them. Since I found some things to nitpick I want to say I definitely like the approach you are coming at this from -- in terms of how you apparently view sharding, NoSQL, database as a processing engine, etc. But I still think you are presenting it as something novel (just look at the title) when what you are really doing is promoting a long-established but sorely neglected philosophy... a good philosophy, which needs to be promoted. The novel aspect seems to be supporting callbacks in your Spacebase product, which isn't particularly essential to this philosophy. Thanks for the article.
- pron 14y ago
  I think callbacks are absolutely essential. How else is the db supposed to schedule and run your business logic?
  - threepipeproblm 14y ago
    Okay I have been thinking about your response for a while... Database engines already possess sophisticated scheduling logic for query processing. We are constantly taking advantage of that, in different ways, when we delegate to the database. And even though we are supposed to have all these business layers and whatnot, I think centralizing logic in a DB is a pretty common practice. So I have been asking myself, is there a significant functional difference between the db doing a callback to the program, versus other methods of interacting with that built-in scheduling engine? My supposition is that in the absence of such a capability, one could always get mostly (or fully) equivalent behavior by some combination of (A) having the app fire queries (and consume their results) asynchronously (B) giving queries some other means of initiating a response, e.g. writing to a table that acts as a message queue. To me it seems like such a transformation wouldn't even change the balance of code between app vs DB all that much, it would just change the calling mechanism, and maybe you need these couple of extra modules to manage your query submissions and asynchronously dequeue stuff. So I admit this qualifies as more than syntactic sugar, but not sure goes much beyond that. Of course, many prized language features don't go beyond that at all; and calling/concurrency idioms are important. I like the idea of centralizing logic in the DB and I can see how your approach allows this in a much more elegant and straightforward manner. So I think I get it. Whether it's a totally new architecture, I dunno... I come out of this thinking if there was a way to improve your article it would perhaps be to avoid the claim that this is allowing us to take advantage of database to "schedule and run" whatever business logic is implemented in the db via the highly optimized query processing facilities. All programs effectively do that in their use of the database, which is why the scheduling capabilities exist in advance of your approach. The core thread of what you are doing/advocating, as I understand it, is something like this. (a) Centralize business logic in the DB. (b) Stop being ashamed of it. (c) Support calling conventions that make it easier. If you made the argument that way, I would have seen it as a slam dunk. And I would expect that to be a controversial argument on HN. Interested in your thoughts. (BTW, I bet one could do these sorts of callbacks using the relatively recent capability of SQL Server to run Stored Procedures written in .NET.)