6 ms·

The Future Database

117 points by munchor 4y ago

pdimitar 4y ago
rmbyrro 4y ago
My dream was having a database like AWS DynamoDB, but with analytical capabilities built-in the API.
- azurezyq 4y ago
  I think you are talking about HTAP(https://en.wikipedia.org/wiki/Hybrid_transactional/analytical_processing https://en.wikipedia.org/wiki/Hybrid_transactional/analytica...). There are already decent products on the market. But don't expect 100% analytical capabilities like BigQuery / Snowflake. There are tradeoffs. If your workload fits the model well (mostly OLTP, but occasionally some with heavier aggregations, etc.), it would be awesome. Example: TiDB https://docs.pingcap.com/tidb/dev/explore-htap https://docs.pingcap.com/tidb/dev/explore-htap https://www.vldb.org/pvldb/vol13/p3072-huang.pdf https://www.vldb.org/pvldb/vol13/p3072-huang.pdf
  - dravita 4y ago
    When it comes to HTAP workloads, SingleStoreDB tackled that years ago and has been the leader in that category of workloads. In recent years, the product has expanded to handle broader workloads and use cases. https://www.singlestore.com/resources/research-paper-cloud-native-transactions-and-analytics-in-singlestore/ https://www.singlestore.com/resources/research-paper-cloud-n...
- Scarbutt 4y ago
  Doing application programming in Dynamodb is a nightmare.
  - gnulinux 4y ago
    I personally found DynamoDB enjoyable but the trick is you need to understand DynamoDB gives you exactly no way to have analytics, so you have to have a separate and robust story for analytics e.g. we periodically dump our DynamoDB into a RedshiftDB so we get best of the both worlds. I personally find it enjoyable to program, and had a better experience than when everything was Postgres.
    - rmbyrro 4y ago
      Same experience. From an OLTP standpoint, the limitations to query the data are actually amazing, because it forces you to think about the right questions from the beginning. On more flexible DBMS like Postgres, it's very very easy to shoot yourself in the foot. I find it easier to understand and take into consideration the DynamoBD constraints (explicit) than SQL shortcomings (implicit).
      - Scarbutt 4y ago
        because it forces you to think about the right questions from the beginning. For many apps thinking that you can have the right questions from the beginning is delusional.
        gnulinux 4y ago
        > For many apps thinking that you can have the right questions from the beginning is delusional. Strong disagree, I think you might be approaching this from a traditional angle where analytics questions are also questions of the system. In DynamoDB you only focus on access patterns you need for system to work real-time, without bothering with analytics (as stated above). Any question you want to ask on data outside of the operational code is out of the scope.
- thdxr 4y ago
  been pairing DynamoDB with rockset to solve this
  - rmbyrro 4y ago
    Yeah, I do pairing with analytical DBMS. The issue is the pairing.
  - dravita 4y ago
    Try SingleStoreDB for the combined DynamoDB workloads and analytical workloads in a single database tech, see https://usefathom.com/blog/ditched-dynamodb https://usefathom.com/blog/ditched-dynamodb
- dravita 4y ago
  Here is the database of your dreams: SingleStoreDB. https://www.singlestore.com/customers/fathom/ https://www.singlestore.com/customers/fathom/
o_m 4y ago
The biggest missing piece for future databases is the ability to guarantee you only store your data in Europe. All of these databases are coming from the US, with no self-hosting option, or EU region for their SaaS service. With Schrems II it the EU region should be operated by an European company detached from the US company.
- k__ 4y ago
  Doesn't Upstash allow this?
- dilyevsky 4y ago
  CockroachDB with enterprise license has data locality pin support even at row level - https://www.cockroachlabs.com/blog/regional-by-row/ https://www.cockroachlabs.com/blog/regional-by-row/ I think their serveless product also has that option
throwusawayus 4y ago
bunch of principles here are specific to using managed 3rd-party database, rather than self host // on-prem meanwhile.. lots of apocalyptic posts lately about how vc funding will dry up since public tech stocks have crashed so how safe is it to have your business completely rely on a managed database startup, instead of self host or using a major cloud provider's direct offering? may be unwise these days methinks
samhw 4y ago
I can't make sense of lots of this. - Without self-replicating grey goo, infinite scalability is surely more a property of some kind of networked computer rental business (like AWS) rather than a database. - What does 'serverless' mean exactly? My understanding is that it denotes a stateless application which is executed to serve a request but doesn't run continually as a daemon. Essentially the aforementioned computer rental business provides the event loop and the program provides the event handler. I fail to see how this is compatible with a database, which is definitionally very stateful. (And that encompasses much, much more than just the data.) - 'Intelligence': Databases already are intelligent and self-optimising, and have been at least since MySQL/Postgres: https://dev.mysql.com/doc/refman/8.0/en/cost-model.html#cost-model-operation https://dev.mysql.com/doc/refman/8.0/en/cost-model.html#cost... - 'Fundamentally reliable': This idea could reasonably be described as, uh, 'not novel'. - 'Distributed globally, locally available': As far as I can tell, this collection of words is entirely devoid of any meaning. It sounds like it came out of a random passphrase generator. - 'Scale should not come at the cost of performance'. While technically semantically meaningful, this is not novel or interesting, and I'm pretty sure this has been a pleasant daydream for database designers since databases were stored in punchcards. As far as I can see, this is comparable to saying 'houses should not come at the cost of money'. This feels more like a laundry list of daydreams rather than a meaningful narrowing-down of how future databases will be architected. "It should be infinitely scalable, usable by a toaster, and it shouldn't need a computer to live on. It should be serverless and stateless but also self-optimising and with connection pooling. It should be consistent, available, and, uh, partitions, it should be cool with those too. It should be usable by anyone and perfectly tailored to their needs as well as to the opposite needs. Also..."
- imachine1980_ 4y ago
  What does 'serverless' mean exactly? 1) i think server-less, means ops-less, no manager of server even at scale,maybe you need optimization query but not to deploy databases, migration and that. 3)Scale should not come at the cost of performance is easier said than done, is the same idea automatic cache an thinks like this 2) i think the same i will still use monolith like PostgreSQL use a doubt have more intensive workflows than medium.
TheAceOfHearts 4y ago
If you're truly forward-thinking you'd start planning for multiplanetary databases. Get a leg up on the competition by the time humans reach Mars. The core issues of planetary database systems have mostly been solved already. What about database systems which have to deal with time dilation due to running on spacecraft that travel through space at different velocities? But in all seriousness, one of the core features missing in modern database systems is a way to smoothly handle tracking and deploying changes. Why are online schema changes so difficult?
- setr 4y ago
  > What about database systems which have to deal with time dilation due to running on spacecraft that travel through space at different velocities? I think they already handle this case by accident, because distributed clocks are unreliable and inconsistent anyways.
  - devnulll 4y ago
    The Big Players put atomic clocks in the datacenter, and provide multiple mechanism's of time synchronization. Google Spanner was the first to do this, back in... 2007? https://www.theverge.com/2012/11/26/3692392/google-spanner-atomic-clocks-GPS https://www.theverge.com/2012/11/26/3692392/google-spanner-a... https://cloud.google.com/spanner/docs/true-time-external-consistency https://cloud.google.com/spanner/docs/true-time-external-con... Now, I don't think they worry overmuch about general or special relativity at this point, but if we start putting datacenters in orbit that will become an actual issue. I'll wager we'll start seeing Edge Compute and CDN features on the very large satellite constellations (Starlink, etc) soon. Initially this will preserve bandwidth between satellites and with the ground stations. This will be a big upsell feature. The Quant folks would pay more for those sweet low-latency links...
- candiddevmike 4y ago
  Going further, I'd love to have a declarative schema for my database with some kind of hook layer to allow for data transformations between state. Or figure out how to eliminate the need to carry forward hundreds of schema migrations, such as an easier way to squash them periodically.
jandrewrogers 4y ago
What I struggle with is that half the databases that have been built over the last decade would have claimed these same principles. It isn't so much that these principles are not novel or unique that gives me pause, but the lack of acknowledgement of how many expertly designed databases failed to deliver on them. These principles contain no insights into why anyone should expect this particular database to succeed where similarly qualified people have failed. At a minimum, these principles smuggle in the assumption that several Hard Problems, both theoretical and practical, have been addressed (which is possible but not in evidence). As another way of framing it, these principles seem to be making the unstated assumption that the Future Database only supports a narrow set of data models and workloads. Which would be considerably less interesting than actually reimagining core database architecture and solving hard problems.
jfbaro 4y ago
Great list. As I can add anything here, I will say: - Bitemporal support OOTB (storage would be more expensive, as temporal data needs more disk space) - CoW capabilities OOTB, so it would be super easy (fast and cheap) to create ephemeral database for development purpose. - Charge per request (ms of reads, ms of writes) - for the sake of being more specific about serverless. - AI capabilities that detects the use of the database and suggests indexes or other tweaks to make the database as fast as possible (and cheap), even if schema changes, database size increases or query patterns change - PostgreSQL support (and all its extensions... I know that's a hard one as PS is based on MySQL) - OOTB capabilities for Masking and/or anonymizing of data (PCI, PII, etc) Thanks

The Future Database

117 points by munchor 4y ago

pdimitar 4y ago

rmbyrro 4y ago

My dream was having a database like AWS DynamoDB, but with analytical capabilities built-in the API.

azurezyq 4y ago
I think you are talking about HTAP(https://en.wikipedia.org/wiki/Hybrid_transactional/analytical_processing https://en.wikipedia.org/wiki/Hybrid_transactional/analytica...). There are already decent products on the market. But don't expect 100% analytical capabilities like BigQuery / Snowflake. There are tradeoffs. If your workload fits the model well (mostly OLTP, but occasionally some with heavier aggregations, etc.), it would be awesome. Example: TiDB https://docs.pingcap.com/tidb/dev/explore-htap https://docs.pingcap.com/tidb/dev/explore-htap https://www.vldb.org/pvldb/vol13/p3072-huang.pdf https://www.vldb.org/pvldb/vol13/p3072-huang.pdf
- dravita 4y ago
  When it comes to HTAP workloads, SingleStoreDB tackled that years ago and has been the leader in that category of workloads. In recent years, the product has expanded to handle broader workloads and use cases. https://www.singlestore.com/resources/research-paper-cloud-native-transactions-and-analytics-in-singlestore/ https://www.singlestore.com/resources/research-paper-cloud-n...
Scarbutt 4y ago
Doing application programming in Dynamodb is a nightmare.
- gnulinux 4y ago
  I personally found DynamoDB enjoyable but the trick is you need to understand DynamoDB gives you exactly no way to have analytics, so you have to have a separate and robust story for analytics e.g. we periodically dump our DynamoDB into a RedshiftDB so we get best of the both worlds. I personally find it enjoyable to program, and had a better experience than when everything was Postgres.
  - rmbyrro 4y ago
    Same experience. From an OLTP standpoint, the limitations to query the data are actually amazing, because it forces you to think about the right questions from the beginning. On more flexible DBMS like Postgres, it's very very easy to shoot yourself in the foot. I find it easier to understand and take into consideration the DynamoBD constraints (explicit) than SQL shortcomings (implicit).
    - Scarbutt 4y ago
      because it forces you to think about the right questions from the beginning. For many apps thinking that you can have the right questions from the beginning is delusional.
      - gnulinux 4y ago
        > For many apps thinking that you can have the right questions from the beginning is delusional. Strong disagree, I think you might be approaching this from a traditional angle where analytics questions are also questions of the system. In DynamoDB you only focus on access patterns you need for system to work real-time, without bothering with analytics (as stated above). Any question you want to ask on data outside of the operational code is out of the scope.
thdxr 4y ago
been pairing DynamoDB with rockset to solve this
- rmbyrro 4y ago
  Yeah, I do pairing with analytical DBMS. The issue is the pairing.
- dravita 4y ago
  Try SingleStoreDB for the combined DynamoDB workloads and analytical workloads in a single database tech, see https://usefathom.com/blog/ditched-dynamodb https://usefathom.com/blog/ditched-dynamodb
dravita 4y ago
Here is the database of your dreams: SingleStoreDB. https://www.singlestore.com/customers/fathom/ https://www.singlestore.com/customers/fathom/

o_m 4y ago

The biggest missing piece for future databases is the ability to guarantee you only store your data in Europe. All of these databases are coming from the US, with no self-hosting option, or EU region for their SaaS service. With Schrems II it the EU region should be operated by an European company detached from the US company.

k__ 4y ago
Doesn't Upstash allow this?
dilyevsky 4y ago
CockroachDB with enterprise license has data locality pin support even at row level - https://www.cockroachlabs.com/blog/regional-by-row/ https://www.cockroachlabs.com/blog/regional-by-row/ I think their serveless product also has that option

throwusawayus 4y ago

bunch of principles here are specific to using managed 3rd-party database, rather than self host // on-prem meanwhile.. lots of apocalyptic posts lately about how vc funding will dry up since public tech stocks have crashed so how safe is it to have your business completely rely on a managed database startup, instead of self host or using a major cloud provider's direct offering? may be unwise these days methinks

samhw 4y ago

I can't make sense of lots of this. - Without self-replicating grey goo, infinite scalability is surely more a property of some kind of networked computer rental business (like AWS) rather than a database. - What does 'serverless' mean exactly? My understanding is that it denotes a stateless application which is executed to serve a request but doesn't run continually as a daemon. Essentially the aforementioned computer rental business provides the event loop and the program provides the event handler. I fail to see how this is compatible with a database, which is definitionally very stateful. (And that encompasses much, much more than just the data.) - 'Intelligence': Databases already are intelligent and self-optimising, and have been at least since MySQL/Postgres: https://dev.mysql.com/doc/refman/8.0/en/cost-model.html#cost-model-operation https://dev.mysql.com/doc/refman/8.0/en/cost-model.html#cost... - 'Fundamentally reliable': This idea could reasonably be described as, uh, 'not novel'. - 'Distributed globally, locally available': As far as I can tell, this collection of words is entirely devoid of any meaning. It sounds like it came out of a random passphrase generator. - 'Scale should not come at the cost of performance'. While technically semantically meaningful, this is not novel or interesting, and I'm pretty sure this has been a pleasant daydream for database designers since databases were stored in punchcards. As far as I can see, this is comparable to saying 'houses should not come at the cost of money'. This feels more like a laundry list of daydreams rather than a meaningful narrowing-down of how future databases will be architected. "It should be infinitely scalable, usable by a toaster, and it shouldn't need a computer to live on. It should be serverless and stateless but also self-optimising and with connection pooling. It should be consistent, available, and, uh, partitions, it should be cool with those too. It should be usable by anyone and perfectly tailored to their needs as well as to the opposite needs. Also..."

imachine1980_ 4y ago
What does 'serverless' mean exactly? 1) i think server-less, means ops-less, no manager of server even at scale,maybe you need optimization query but not to deploy databases, migration and that. 3)Scale should not come at the cost of performance is easier said than done, is the same idea automatic cache an thinks like this 2) i think the same i will still use monolith like PostgreSQL use a doubt have more intensive workflows than medium.

TheAceOfHearts 4y ago

If you're truly forward-thinking you'd start planning for multiplanetary databases. Get a leg up on the competition by the time humans reach Mars. The core issues of planetary database systems have mostly been solved already. What about database systems which have to deal with time dilation due to running on spacecraft that travel through space at different velocities? But in all seriousness, one of the core features missing in modern database systems is a way to smoothly handle tracking and deploying changes. Why are online schema changes so difficult?

setr 4y ago
> What about database systems which have to deal with time dilation due to running on spacecraft that travel through space at different velocities? I think they already handle this case by accident, because distributed clocks are unreliable and inconsistent anyways.
- devnulll 4y ago
  The Big Players put atomic clocks in the datacenter, and provide multiple mechanism's of time synchronization. Google Spanner was the first to do this, back in... 2007? https://www.theverge.com/2012/11/26/3692392/google-spanner-atomic-clocks-GPS https://www.theverge.com/2012/11/26/3692392/google-spanner-a... https://cloud.google.com/spanner/docs/true-time-external-consistency https://cloud.google.com/spanner/docs/true-time-external-con... Now, I don't think they worry overmuch about general or special relativity at this point, but if we start putting datacenters in orbit that will become an actual issue. I'll wager we'll start seeing Edge Compute and CDN features on the very large satellite constellations (Starlink, etc) soon. Initially this will preserve bandwidth between satellites and with the ground stations. This will be a big upsell feature. The Quant folks would pay more for those sweet low-latency links...
candiddevmike 4y ago
Going further, I'd love to have a declarative schema for my database with some kind of hook layer to allow for data transformations between state. Or figure out how to eliminate the need to carry forward hundreds of schema migrations, such as an easier way to squash them periodically.

jandrewrogers 4y ago

What I struggle with is that half the databases that have been built over the last decade would have claimed these same principles. It isn't so much that these principles are not novel or unique that gives me pause, but the lack of acknowledgement of how many expertly designed databases failed to deliver on them. These principles contain no insights into why anyone should expect this particular database to succeed where similarly qualified people have failed. At a minimum, these principles smuggle in the assumption that several Hard Problems, both theoretical and practical, have been addressed (which is possible but not in evidence). As another way of framing it, these principles seem to be making the unstated assumption that the Future Database only supports a narrow set of data models and workloads. Which would be considerably less interesting than actually reimagining core database architecture and solving hard problems.

jfbaro 4y ago

Great list. As I can add anything here, I will say: - Bitemporal support OOTB (storage would be more expensive, as temporal data needs more disk space) - CoW capabilities OOTB, so it would be super easy (fast and cheap) to create ephemeral database for development purpose. - Charge per request (ms of reads, ms of writes) - for the sake of being more specific about serverless. - AI capabilities that detects the use of the database and suggests indexes or other tweaks to make the database as fast as possible (and cheap), even if schema changes, database size increases or query patterns change - PostgreSQL support (and all its extensions... I know that's a hard one as PS is based on MySQL) - OOTB capabilities for Masking and/or anonymizing of data (PCI, PII, etc) Thanks