3 ms·

Thanks for your response. I think I understand your point re: cores, although I wouldn't have used the argument quoted above, instead I would just make the poi

Thanks for your response. I think I understand your point re: cores, although I wouldn't have used the argument quoted above, instead I would just make the point in your response - we have all these cores, gotta use them. Since I found some things to nitpick I want to say I definitely like the approach you are coming at this from -- in terms of how you apparently view sharding, NoSQL, database as a processing engine, etc. But I still think you are presenting it as something novel (just look at the title) when what you are really doing is promoting a long-established but sorely neglected philosophy... a good philosophy, which needs to be promoted. The novel aspect seems to be supporting callbacks in your Spacebase product, which isn't particularly essential to this philosophy. Thanks for the article.

pron 14y ago
I think callbacks are absolutely essential. How else is the db supposed to schedule and run your business logic?
- threepipeproblm 14y ago
  Okay I have been thinking about your response for a while... Database engines already possess sophisticated scheduling logic for query processing. We are constantly taking advantage of that, in different ways, when we delegate to the database. And even though we are supposed to have all these business layers and whatnot, I think centralizing logic in a DB is a pretty common practice. So I have been asking myself, is there a significant functional difference between the db doing a callback to the program, versus other methods of interacting with that built-in scheduling engine? My supposition is that in the absence of such a capability, one could always get mostly (or fully) equivalent behavior by some combination of (A) having the app fire queries (and consume their results) asynchronously (B) giving queries some other means of initiating a response, e.g. writing to a table that acts as a message queue. To me it seems like such a transformation wouldn't even change the balance of code between app vs DB all that much, it would just change the calling mechanism, and maybe you need these couple of extra modules to manage your query submissions and asynchronously dequeue stuff. So I admit this qualifies as more than syntactic sugar, but not sure goes much beyond that. Of course, many prized language features don't go beyond that at all; and calling/concurrency idioms are important. I like the idea of centralizing logic in the DB and I can see how your approach allows this in a much more elegant and straightforward manner. So I think I get it. Whether it's a totally new architecture, I dunno... I come out of this thinking if there was a way to improve your article it would perhaps be to avoid the claim that this is allowing us to take advantage of database to "schedule and run" whatever business logic is implemented in the db via the highly optimized query processing facilities. All programs effectively do that in their use of the database, which is why the scheduling capabilities exist in advance of your approach. The core thread of what you are doing/advocating, as I understand it, is something like this. (a) Centralize business logic in the DB. (b) Stop being ashamed of it. (c) Support calling conventions that make it easier. If you made the argument that way, I would have seen it as a slam dunk. And I would expect that to be a controversial argument on HN. Interested in your thoughts. (BTW, I bet one could do these sorts of callbacks using the relatively recent capability of SQL Server to run Stored Procedures written in .NET.)
  - pron 14y ago
    Yes, well, you summarize most of my points but one: the importance of callbacks is not just in the database triggering application actions (as would be done with asynchronous queries or the message bus you mention), but by scheduling application code to run on appropriate CPU cores. Our ultimate goal isn't to make a new programming model for its own aesthetic (or non-aesthetic) sake, but in order to take advantage of multiple cores -- namely, for the sake of performance. So the database does not "control" the application -- it actually parallelizes it. And as for SQL server, yes, I guess it's a similar concept but a different kind of database. I advocate that the database should be part of the application; they should be sharing the same heap in the same process, and only then should the applications let the database "drive". That is why I also don't quite agree with your wording of "centralizing logic in the DB" because the DB and the application are one; certainly the DB is not more (or less) centralized than the app. Once you get that, there is no point in being ashamed of it. Just like you let your web-app container parallelize your presentation layer, you let the database parallelize your business logic. In both cases the middleware shares the same process as the app, and the leap of faith required isn't big at all.
    - threepipeproblm 14y ago
      Thanks for your response! I think it addressed my comments well. It does make me think about the fact that almost every major DB today runs out of process and that the market seems to have selected this pretty definitively. It seems like only recently some products are running in-process for performance reasons. But if I understand the trade offs here, only a single application can run in-process with the db. Also the db is not protected from crashing when the app crashes. But this is the 1990's explanation... any thoughts on how that applies to your situation?
      - pron 14y ago
        > almost every major DB today runs out of process and that the market seems to have selected this pretty definitively. True. That's why I've tried to show the performance to be gained by letting the DB become integrated with the app. This is a result of the DB having just the right kind of knowledge about the relationships among domain objects to allow it to parallelize business logic. > But if I understand the trade offs here, only a single application can run in-process with the db. Also the db is not protected from crashing when the app crashes. Well, if the application and DB are one, so what? Note that we're only talking about OLTP DBs here. I don't see much of a reason for analytics DBs to be unified with the app. How often do you need more than one app working with the same OLTP data? However, if you absolutely must, there is no reason why the in-process DB shouldn't store its data in shared memory, thus allowing other DB instances to access the information. There are ways to ensure that if one app crashes the memory is left in a consistent state.

pron 14y ago

I think callbacks are absolutely essential. How else is the db supposed to schedule and run your business logic?

threepipeproblm 14y ago
Okay I have been thinking about your response for a while... Database engines already possess sophisticated scheduling logic for query processing. We are constantly taking advantage of that, in different ways, when we delegate to the database. And even though we are supposed to have all these business layers and whatnot, I think centralizing logic in a DB is a pretty common practice. So I have been asking myself, is there a significant functional difference between the db doing a callback to the program, versus other methods of interacting with that built-in scheduling engine? My supposition is that in the absence of such a capability, one could always get mostly (or fully) equivalent behavior by some combination of (A) having the app fire queries (and consume their results) asynchronously (B) giving queries some other means of initiating a response, e.g. writing to a table that acts as a message queue. To me it seems like such a transformation wouldn't even change the balance of code between app vs DB all that much, it would just change the calling mechanism, and maybe you need these couple of extra modules to manage your query submissions and asynchronously dequeue stuff. So I admit this qualifies as more than syntactic sugar, but not sure goes much beyond that. Of course, many prized language features don't go beyond that at all; and calling/concurrency idioms are important. I like the idea of centralizing logic in the DB and I can see how your approach allows this in a much more elegant and straightforward manner. So I think I get it. Whether it's a totally new architecture, I dunno... I come out of this thinking if there was a way to improve your article it would perhaps be to avoid the claim that this is allowing us to take advantage of database to "schedule and run" whatever business logic is implemented in the db via the highly optimized query processing facilities. All programs effectively do that in their use of the database, which is why the scheduling capabilities exist in advance of your approach. The core thread of what you are doing/advocating, as I understand it, is something like this. (a) Centralize business logic in the DB. (b) Stop being ashamed of it. (c) Support calling conventions that make it easier. If you made the argument that way, I would have seen it as a slam dunk. And I would expect that to be a controversial argument on HN. Interested in your thoughts. (BTW, I bet one could do these sorts of callbacks using the relatively recent capability of SQL Server to run Stored Procedures written in .NET.)
- pron 14y ago
  Yes, well, you summarize most of my points but one: the importance of callbacks is not just in the database triggering application actions (as would be done with asynchronous queries or the message bus you mention), but by scheduling application code to run on appropriate CPU cores. Our ultimate goal isn't to make a new programming model for its own aesthetic (or non-aesthetic) sake, but in order to take advantage of multiple cores -- namely, for the sake of performance. So the database does not "control" the application -- it actually parallelizes it. And as for SQL server, yes, I guess it's a similar concept but a different kind of database. I advocate that the database should be part of the application; they should be sharing the same heap in the same process, and only then should the applications let the database "drive". That is why I also don't quite agree with your wording of "centralizing logic in the DB" because the DB and the application are one; certainly the DB is not more (or less) centralized than the app. Once you get that, there is no point in being ashamed of it. Just like you let your web-app container parallelize your presentation layer, you let the database parallelize your business logic. In both cases the middleware shares the same process as the app, and the leap of faith required isn't big at all.
  - threepipeproblm 14y ago
    Thanks for your response! I think it addressed my comments well. It does make me think about the fact that almost every major DB today runs out of process and that the market seems to have selected this pretty definitively. It seems like only recently some products are running in-process for performance reasons. But if I understand the trade offs here, only a single application can run in-process with the db. Also the db is not protected from crashing when the app crashes. But this is the 1990's explanation... any thoughts on how that applies to your situation?
    - pron 14y ago
      > almost every major DB today runs out of process and that the market seems to have selected this pretty definitively. True. That's why I've tried to show the performance to be gained by letting the DB become integrated with the app. This is a result of the DB having just the right kind of knowledge about the relationships among domain objects to allow it to parallelize business logic. > But if I understand the trade offs here, only a single application can run in-process with the db. Also the db is not protected from crashing when the app crashes. Well, if the application and DB are one, so what? Note that we're only talking about OLTP DBs here. I don't see much of a reason for analytics DBs to be unified with the app. How often do you need more than one app working with the same OLTP data? However, if you absolutely must, there is no reason why the in-process DB shouldn't store its data in shared memory, thus allowing other DB instances to access the information. There are ways to ensure that if one app crashes the memory is left in a consistent state.