3 ms·
This is probably the most hot topic for database service providers. Operating hundreds of thousands and millions of databases yields a valuable collateral: oper
by rusanu 10y ago
This is probably the most hot topic for database service providers. Operating hundreds of thousands and millions of databases yields a valuable collateral: operational data (what users are doing, metadata, table shapes, query patterns, application behavior, reasons for blocking, IO patterns, memory patterns, everything). Take the tuning wizards of 5 years ago, add machine learning on vast amounts of operational data, add 'shadow' A/B testing (run a secondary instance shadowing the same workload, on a copy of the database, but, say, apply an index recommendation to the shadow instance and measure the impact). When I left Microsoft one year ago, this whole area was seen as a gold mine and there was massive investment into it. You can already see such offerings as Azure SQL Database Advisor[1]. I'm convinced Amazon is doing its own research in this space, specially for Aurora.
We're talking much more than just index optimizations. Think automatic OLTP vs. OLAP pattern detection and migration of data into columnstores or in-memory tables. Automatic detection of cold vs. hot data and migration to-from expensive fast near storage (SSD) vs. cheap remote storage (S3 like). Automatic resizing of instance size. Materialized views. On-the-fly query rewrite. The field is wide open. And you have the best experts in the field looking at your database.
By contrast, the traditional on-premise auto-tuning did not made much progress over the past 10-15 years. There are reasons for this: small operational dataset (one site only), high risk of regression (difficult to do A/B testing), scarcity of experts. The best avenue for on-premise I predict will be the trickle-down effect of migrating service know-how and technologies to on-premise offerings. We're also going to see a push up the stack of the know-how and experience from operating the service making its way into the app stack (think EntityFramework v. 2018?) and helping developers avoid the common pitfalls.
BTW, only 3 days ago Adrian Coyler's morning paper discussion was Self-driving database management systems [2]
[1] https://docs.microsoft.com/en-us/azure/sql-database/sql-database-advisor https://docs.microsoft.com/en-us/azure/sql-database/sql-data...
[2] https://blog.acolyer.org/2017/01/17/self-driving-database-management-systems/ https://blog.acolyer.org/2017/01/17/self-driving-database-ma...
- lobster_johnson 10y agoVery interesting, thank you.