4 ms·

It seems to me that what this is saying is that storage has become so cheap that if another database provides even slight advantages over another for some workl

by jerf 5y ago

It seems to me that what this is saying is that storage has become so cheap that if another database provides even slight advantages over another for some workload it is likely to be deployed and have all the data copied over to it. HN entrepreneurs take note, this also suggests to me that there may be a market for a database (or a "metadatabase") that takes care of this for you. I'd love to be able to have a "relational database" that is also some "NoSQL" databases (since there's a few major useful paradigms there) that just takes care of this for me. I imagine I'd have to declare my schemas, but I'd love it if that's all I had to do and then the DB handled keeping sync and such. Bonus points if you can give me cross-paradigm transactionality, especially in terms of coherent insert sets (so "today's load of data" appears in one lump instantly from clients point of view and they don't see the load in progress). At least at first, this wouldn't have to be best-of-breed necessarily at anything. I'd need good SQL joining support, but I think I wouldn't need every last feature Postgres has ever had out of the box. If such a product exists, I'm all ears. Though I am thinking of this as a unified database, not a collection of databases and products that merely manages data migrations and such. I'm looking to run "CREATE CASSANDRA-LIKE VIEW gotta_go_fast ON SELECT a.x, a.y, b.z FROM ...", maybe it takes some time of course but that's all I really have to do to keep things in sync. (Barring resource overconsumption.)

rmbyrro 5y ago
I'm afraid it's not feasible to develop a single general purpose implementation for that. The amount of complexity to guarantee data integrity while covering all possible use cases will be just unmanageable. I'd be extremely happy to be proven wrong, though...
WatchDog 5y ago
My understanding of the cockroach db architecture, it that it’s essentially two discrete components, a key value store that actually persists the data, and a SQL layer built on top. Although I don’t think it’s recommended or supported to access the key value store directly.
andy_ppp 5y ago
Postgres with Cassandra built in and scaled separately would be really great.
grncdr 5y ago
I think there was a project like this a few years ago (wrapping a relational DB + ElasticSearch into one box) and I thought it was CrateDB, but from looking at their current website I think I'm misremembering. The concept didn't appeal to me very much then, so I never looked into it further. --- To address your larger point, I think Postgres has a better chance of absorbing other datastores (via FDW and/or custom index types) and updating them in sync with it's own transactions (as far as those databases support some sort of atomic swap operation) than a new contender has of getting near Postgres' level of reliability and feature richness.
- neuronexmachina 5y ago
  Were you thinking of ZomboDB? https://github.com/zombodb/zombodb https://github.com/zombodb/zombodb
jgraettinger1 5y ago
> I'd love to be able to have a "relational database" that is also some "NoSQL" databases (since there's a few major useful paradigms there) that just takes care of this for me. I imagine I'd have to declare my schemas, but I'd love it if that's all I had to do and then the DB handled keeping sync and such. You might be interested in what we're building [0] It synchronizes your data systems so that, for example, you can CDC tables from your Postgres DB, transform them in interesting ways, and then materialize the result in a view within Elastic or DynamoDB that updates continuously and with millisecond latency. It will even propagate your sourced SQL schemas into JSON schemas, and from there to, say, equivalent Elastic Search schema. [0]: https://github.com/estuary/flow https://github.com/estuary/flow
mwarkentin 5y ago
AWS tried building this with Glue Elastic Views: https://aws.amazon.com/glue/features/elastic-views/ https://aws.amazon.com/glue/features/elastic-views/ It's been in preview forever though, not sure when it's going to officially launch.

by jerf 5y ago

rmbyrro 5y ago

I'm afraid it's not feasible to develop a single general purpose implementation for that. The amount of complexity to guarantee data integrity while covering all possible use cases will be just unmanageable. I'd be extremely happy to be proven wrong, though...

WatchDog 5y ago

My understanding of the cockroach db architecture, it that it’s essentially two discrete components, a key value store that actually persists the data, and a SQL layer built on top. Although I don’t think it’s recommended or supported to access the key value store directly.

andy_ppp 5y ago

Postgres with Cassandra built in and scaled separately would be really great.

grncdr 5y ago

I think there was a project like this a few years ago (wrapping a relational DB + ElasticSearch into one box) and I thought it was CrateDB, but from looking at their current website I think I'm misremembering. The concept didn't appeal to me very much then, so I never looked into it further. --- To address your larger point, I think Postgres has a better chance of absorbing other datastores (via FDW and/or custom index types) and updating them in sync with it's own transactions (as far as those databases support some sort of atomic swap operation) than a new contender has of getting near Postgres' level of reliability and feature richness.

neuronexmachina 5y ago
Were you thinking of ZomboDB? https://github.com/zombodb/zombodb https://github.com/zombodb/zombodb

jgraettinger1 5y ago

> I'd love to be able to have a "relational database" that is also some "NoSQL" databases (since there's a few major useful paradigms there) that just takes care of this for me. I imagine I'd have to declare my schemas, but I'd love it if that's all I had to do and then the DB handled keeping sync and such. You might be interested in what we're building [0] It synchronizes your data systems so that, for example, you can CDC tables from your Postgres DB, transform them in interesting ways, and then materialize the result in a view within Elastic or DynamoDB that updates continuously and with millisecond latency. It will even propagate your sourced SQL schemas into JSON schemas, and from there to, say, equivalent Elastic Search schema. [0]: https://github.com/estuary/flow https://github.com/estuary/flow

mwarkentin 5y ago

AWS tried building this with Glue Elastic Views: https://aws.amazon.com/glue/features/elastic-views/ https://aws.amazon.com/glue/features/elastic-views/ It's been in preview forever though, not sure when it's going to officially launch.