3 ms·
> but developers will no longer need to explicitly worry about data model their application uses. There will be a tighter coupling of programming frameworks and
by crazygringo 7y ago
> but developers will no longer need to explicitly worry about data model their application uses. There will be a tighter coupling of programming frameworks and DBMSs such that all database interactions will be transparent (and optimal).
Sorry, but this feels hilariously wrong.
Databases that work performantly are all about making key architectural decisions about which information is indexed and/or denormalized and/or preprocessed, how, and ensuring that queries are written only in forms that take advantage of that design.
It's the difference between queries taking milliseconds or hours on reasonably-sized datasets of, say, 10GB+.
Because it's shockingly easy to write a query that, otherwise, will not only read every row of your database, but be required to read every row once for every row, because that's literally the only way to calculate the result if the right data isn't suitably indexed or preprocessed. (E.g. "find the number of users who have ever ordered the same item (any item) more than once" on a table indexed only by order ID.)
I don't see how that can ever ever be made "transparent", and it certainly doesn't have anything to do with tighter coupling with a programming framework.
- foota 7y agoI don't think it's anyway near reality, but there's actually a number of ways this could work. If you prepackage your queries, or do some sort of profiling guided optimization, the database could learn what it needs to index to answer your queries efficiently. There's already some precedence for this, with how Facebook for instance packages their graphql queries ahead of time to avoid parsing them at runtime. Alternatively, you could maybe imagine models where storage is sharded along with the code for running it, so you say declaratively specify the data you need (count for this grouped by that, etc) and then at runtime the data is pre-aggregated per server hosting a certain shard.
- yazaddaruvala 7y agoRather than specifying a schema+indices or an Elasticsearch like mapping, preprocessing queries to derive the mapping will definitely be the next step. However, I doubt we will ever derive mappings from application code or (retroactive) profile guided optimization. The biggest problem will become deployments. We would need to couple application code deployments to deployments of databases even more than we do today (which is a giant problem). When a new index is needed or new data types we would need to “prime” the datastore in order to get optimal queries from the start. Even if this application-database deployment issue is fixed through automation it’s likely we wouldn’t want it to be coupled. Database index creation or schema change can take hours or longer. Application code needs to be very nimble. In the days of CI we don’t want Feature A’s database migration to block the deployment of Feature B or BugFix C.
- foota 7y agoThis is fair. Although I do think the collocated data + serving thing would be pretty neat.
- roenxi 7y agoThe data model question is - at best - a very poor choice of words. Implicitly assuming a data model is a bad idea as the data model is one of the major choices every program has to contend with. So implicitly assuming it isn't really a desirable goal. It is like types where explicitly having information on types in key places in a program is always desirable and the only question is how pervasive it should be. I also liked "They will instead ask questions about data in a natural language". Humans can't answer questions about data in natural language - when data questions come up they have to be couched in formal terms and statistics. I look forward to the day an AI can answer "what is the best decision we can make with this data?" but it runs into a problem we already have in spades. If the asker of the question isn't thinking in formal terms I question whether they are going to be able to consistently pose meaningful questions and interpret the answers. Applied Math is not easy to implement in industry because a company's ability to perform is limited by its management's understanding. It is easy to nitpick. I'm also glad we have people thinking about the big picture of how data gets handled.