5 ms·

My dream was having a database like AWS DynamoDB, but with analytical capabilities built-in the API.

azurezyq 4y ago
I think you are talking about HTAP(https://en.wikipedia.org/wiki/Hybrid_transactional/analytical_processing https://en.wikipedia.org/wiki/Hybrid_transactional/analytica...). There are already decent products on the market. But don't expect 100% analytical capabilities like BigQuery / Snowflake. There are tradeoffs. If your workload fits the model well (mostly OLTP, but occasionally some with heavier aggregations, etc.), it would be awesome. Example: TiDB https://docs.pingcap.com/tidb/dev/explore-htap https://docs.pingcap.com/tidb/dev/explore-htap https://www.vldb.org/pvldb/vol13/p3072-huang.pdf https://www.vldb.org/pvldb/vol13/p3072-huang.pdf
- dravita 4y ago
  When it comes to HTAP workloads, SingleStoreDB tackled that years ago and has been the leader in that category of workloads. In recent years, the product has expanded to handle broader workloads and use cases. https://www.singlestore.com/resources/research-paper-cloud-native-transactions-and-analytics-in-singlestore/ https://www.singlestore.com/resources/research-paper-cloud-n...
Scarbutt 4y ago
Doing application programming in Dynamodb is a nightmare.
- gnulinux 4y ago
  I personally found DynamoDB enjoyable but the trick is you need to understand DynamoDB gives you exactly no way to have analytics, so you have to have a separate and robust story for analytics e.g. we periodically dump our DynamoDB into a RedshiftDB so we get best of the both worlds. I personally find it enjoyable to program, and had a better experience than when everything was Postgres.
  - rmbyrro 4y ago
    Same experience. From an OLTP standpoint, the limitations to query the data are actually amazing, because it forces you to think about the right questions from the beginning. On more flexible DBMS like Postgres, it's very very easy to shoot yourself in the foot. I find it easier to understand and take into consideration the DynamoBD constraints (explicit) than SQL shortcomings (implicit).
    - Scarbutt 4y ago
      because it forces you to think about the right questions from the beginning. For many apps thinking that you can have the right questions from the beginning is delusional.
      - gnulinux 4y ago
        > For many apps thinking that you can have the right questions from the beginning is delusional. Strong disagree, I think you might be approaching this from a traditional angle where analytics questions are also questions of the system. In DynamoDB you only focus on access patterns you need for system to work real-time, without bothering with analytics (as stated above). Any question you want to ask on data outside of the operational code is out of the scope.
thdxr 4y ago
been pairing DynamoDB with rockset to solve this
- rmbyrro 4y ago
  Yeah, I do pairing with analytical DBMS. The issue is the pairing.
- dravita 4y ago
  Try SingleStoreDB for the combined DynamoDB workloads and analytical workloads in a single database tech, see https://usefathom.com/blog/ditched-dynamodb https://usefathom.com/blog/ditched-dynamodb
dravita 4y ago
Here is the database of your dreams: SingleStoreDB. https://www.singlestore.com/customers/fathom/ https://www.singlestore.com/customers/fathom/

I think you are talking about HTAP(https://en.wikipedia.org/wiki/Hybrid_transactional/analytical_processing https://en.wikipedia.org/wiki/Hybrid_transactional/analytica...). There are already decent products on the market. But don't expect 100% analytical capabilities like BigQuery / Snowflake. There are tradeoffs. If your workload fits the model well (mostly OLTP, but occasionally some with heavier aggregations, etc.), it would be awesome. Example: TiDB https://docs.pingcap.com/tidb/dev/explore-htap https://docs.pingcap.com/tidb/dev/explore-htap https://www.vldb.org/pvldb/vol13/p3072-huang.pdf https://www.vldb.org/pvldb/vol13/p3072-huang.pdf

dravita 4y ago
When it comes to HTAP workloads, SingleStoreDB tackled that years ago and has been the leader in that category of workloads. In recent years, the product has expanded to handle broader workloads and use cases. https://www.singlestore.com/resources/research-paper-cloud-native-transactions-and-analytics-in-singlestore/ https://www.singlestore.com/resources/research-paper-cloud-n...

Scarbutt 4y ago

Doing application programming in Dynamodb is a nightmare.

gnulinux 4y ago
I personally found DynamoDB enjoyable but the trick is you need to understand DynamoDB gives you exactly no way to have analytics, so you have to have a separate and robust story for analytics e.g. we periodically dump our DynamoDB into a RedshiftDB so we get best of the both worlds. I personally find it enjoyable to program, and had a better experience than when everything was Postgres.
- rmbyrro 4y ago
  Same experience. From an OLTP standpoint, the limitations to query the data are actually amazing, because it forces you to think about the right questions from the beginning. On more flexible DBMS like Postgres, it's very very easy to shoot yourself in the foot. I find it easier to understand and take into consideration the DynamoBD constraints (explicit) than SQL shortcomings (implicit).
  - Scarbutt 4y ago
    because it forces you to think about the right questions from the beginning. For many apps thinking that you can have the right questions from the beginning is delusional.
    - gnulinux 4y ago
      > For many apps thinking that you can have the right questions from the beginning is delusional. Strong disagree, I think you might be approaching this from a traditional angle where analytics questions are also questions of the system. In DynamoDB you only focus on access patterns you need for system to work real-time, without bothering with analytics (as stated above). Any question you want to ask on data outside of the operational code is out of the scope.

thdxr 4y ago

been pairing DynamoDB with rockset to solve this

rmbyrro 4y ago
Yeah, I do pairing with analytical DBMS. The issue is the pairing.
dravita 4y ago
Try SingleStoreDB for the combined DynamoDB workloads and analytical workloads in a single database tech, see https://usefathom.com/blog/ditched-dynamodb https://usefathom.com/blog/ditched-dynamodb

dravita 4y ago

Here is the database of your dreams: SingleStoreDB. https://www.singlestore.com/customers/fathom/ https://www.singlestore.com/customers/fathom/