4 ms·

But why would you buy an operational DB from Databricks? The only thing that makes sense is Databricks flailing to maintain market cap.

antruok 1y ago
In addition to the AI use cases, sometimes you wanna share the data warehouse data in oltp way for fast lookups and high concurrency. Not sure whether Neon will do that but I hope so. One example from Snowflake is hybrid tables which adds rowstore next to columnar. OLAP + OLTP = HTAP
- bbohan 1y ago
  SingleStore been doing that for years . Unistore been strugglin
  - antruok 1y ago
    I wonder why Singlestore has been so unpopular (at least I never hear about it). Quick guess is that HTAP itself isn’t a significant feature requirement, maybe just a cherry on top of other major db features.
    - orangechairs 1y ago
      SingleStore (nee MemSQL) was/is niche -- great fit inside banks in particular.
    - moonikakiss 1y ago
      it was a couple things. 1. It's really really hard to replace anyone's OLTP. 2. OLTP and OLAP are owned by such different teams. Who do you make your champion? 3. The modern HTAP dream is possible without something like SingleStore. You need a columnstore that can keep up with your OLTP tables and provide transactional correctness. Who cares if it's all within one system ps: I worked at SingleStore. https://www.mooncake.dev/blog/htap-is-dead https://www.mooncake.dev/blog/htap-is-dead
ako 1y ago
ETL to bring all your data into Databricks/Snowflake is a lot of effort. Much better if your OLTP data already exists in Databricks and you directly access it from your OLAP layer.
- moonikakiss 1y ago
  With the push towards open table formats (Iceberg) from both Snowflake and Databricks, it's even harder to get your Postgres OLTP tables ready for OLAP. The problem isn't in the CDC / replication tools in the market. The problem is that columnar stores (especially Iceberg) are not designed for the write /upserts patterns of OLTP systems. They just can't keep up... This is a big problem we're hoping to solve at Mooncake [0]. Turn Iceberg into an operational columnstore. So that it can be keep up (<s freshness) with your Postgres. https://www.mooncake.dev/ https://www.mooncake.dev/
  - ako 1y ago
    Is Iceberg involved in every read/write? I thought it was mostly metadata?
    - zhousun 1y ago
      DataFile(parquet) is not enough for table with update/delete, (they are part of iceberg "metadata"). for CDC from OLTP use-cases, the pattern involves rapidly marking rows as deleted/ insert new rows and optimizing small files. This is required for minutes-latency replication. And for second latency replication, it is more involving, you actually need to build layer on top of iceberg to track pk/ apply deletion.

antruok 1y ago

In addition to the AI use cases, sometimes you wanna share the data warehouse data in oltp way for fast lookups and high concurrency. Not sure whether Neon will do that but I hope so. One example from Snowflake is hybrid tables which adds rowstore next to columnar. OLAP + OLTP = HTAP

bbohan 1y ago
SingleStore been doing that for years . Unistore been strugglin
- antruok 1y ago
  I wonder why Singlestore has been so unpopular (at least I never hear about it). Quick guess is that HTAP itself isn’t a significant feature requirement, maybe just a cherry on top of other major db features.
  - orangechairs 1y ago
    SingleStore (nee MemSQL) was/is niche -- great fit inside banks in particular.
  - moonikakiss 1y ago
    it was a couple things. 1. It's really really hard to replace anyone's OLTP. 2. OLTP and OLAP are owned by such different teams. Who do you make your champion? 3. The modern HTAP dream is possible without something like SingleStore. You need a columnstore that can keep up with your OLTP tables and provide transactional correctness. Who cares if it's all within one system ps: I worked at SingleStore. https://www.mooncake.dev/blog/htap-is-dead https://www.mooncake.dev/blog/htap-is-dead

ako 1y ago

ETL to bring all your data into Databricks/Snowflake is a lot of effort. Much better if your OLTP data already exists in Databricks and you directly access it from your OLAP layer.

moonikakiss 1y ago
With the push towards open table formats (Iceberg) from both Snowflake and Databricks, it's even harder to get your Postgres OLTP tables ready for OLAP. The problem isn't in the CDC / replication tools in the market. The problem is that columnar stores (especially Iceberg) are not designed for the write /upserts patterns of OLTP systems. They just can't keep up... This is a big problem we're hoping to solve at Mooncake [0]. Turn Iceberg into an operational columnstore. So that it can be keep up (<s freshness) with your Postgres. https://www.mooncake.dev/ https://www.mooncake.dev/
- ako 1y ago
  Is Iceberg involved in every read/write? I thought it was mostly metadata?
  - zhousun 1y ago
    DataFile(parquet) is not enough for table with update/delete, (they are part of iceberg "metadata"). for CDC from OLTP use-cases, the pattern involves rapidly marking rows as deleted/ insert new rows and optimizing small files. This is required for minutes-latency replication. And for second latency replication, it is more involving, you actually need to build layer on top of iceberg to track pk/ apply deletion.