5 ms·

Postgres data stored in Parquet on S3: LTAP architecture explained

182 points by andrenotgiant 3mo ago

deleted 3mo ago
[deleted]
seobot_dk1289 3mo ago
[flagged]
andrenotgiant 3mo ago
Here's what I don't understand: Part of the value of doing an ETL pipeline via streaming replication is you get the full history of data in a table. An SCD type 2 table where each row also has a valid_from and valid_to timestamp column. How would someone do the same thing with this architecture?
- eveningtree 3mo ago
  Rather than answering directly, I'm thinking about this problem from the other end altogether ever since I saw the dbricks rt demo. Apologies for the rambling response, as I haven't yet finished thinking about this problem... We ended up with 'hot' data in oltp and 'cold/archival' data in olap because the storage size of oltp has always been limited. (1) Limited by computation - there's only so much data that we can store on disks and nvme (2) Limited by wallet - disks and nvme are EXPENSIVE Also, the tight coupling of compute and data didn't help. It limited the size of databases on the individual expensive compute nodes. So, another question will be - What's currently stopping me from keeping the scd history tables right in my oltp db? what's forcing me to copy state into my etl/elt pipeline and the process it into scd into a dedicated olap db? To some extent,the answer is still the same - the oltp cannot scale for the storage size required for keeping historical data. So, I've had to take out the 'cold' historical data and keep it in my olap freezer. Now, if oltp itself is scaling, I'm not gonna bother with the copying step. I'll just prefer to store the history in oltp itself. In my perspective (majorly from handling IoT systems), I need olap for 2 reasons - (1) storage scalability, and (2) analytical processing speed I now consider (1) to be a solved problem As for (2), I'm still not sure how this architecture ends up matching the query processing speeds of column-oriented storages. But again, I need to study more. The SCD pipeline still remains in some form. Either in the form of (1) scd rows that we currently keep (etl pipeline) , or (2) as older lsn rows that simply don't get deleted (existing db engine). I've done quite a lot of experimentation with (2), and it is a pretty solid concept to work with. I've spent quite a lot of years hammering my brain at databases and datastores in general. And I've now got a feeling that this is it. Finally.
- deleted 3mo ago
  [deleted]
- hasyimibhar 3mo ago
  It wouldn't be possible to do this with LTAP architecture since (I'm assuming) the individual logical changes are not visible. But honestly I've always seen SCD type 2 table as a workaround due to lack of data modeling experience in the source database. If you design your tables correctly, you shouldn't need SCD type 2 downstream. For example, if you know your user can change emails, and there might be events from another source that is keyed by user email (e.g. marketing-related events), then naturally you will need some sort of email_history table that has historical mapping of user id to email (you probably need it for audit purposes too). Then in this case there is no need to build SCD type 2 table of user from CDC, it's already there.
  - deleted 3mo ago
    [deleted]
  - TheTaytay 3mo ago
    I think you have a point, and SCD type 2 feels like a workaround, but there is also something to be said for the ability to query every row as it was at any given version. I’m not saying that SCD type 2 is the best solution given there might be a more domain-specific way to do it, but I see it a lot like file-based version control. It’s convenient to be able to examine all files as they existed at any point in time, without having to “model” the ways in which those files might change directly into the domain of the individual files. If you have something like dolt (not affiliated), a version controlled database, you wouldn’t have to slap change dates on anything OR create your historical table. The changes would be implicit in the version history.
    - hasyimibhar 3mo ago
      > If you have something like dolt (not affiliated), a version controlled database, you wouldn’t have to slap change dates on anything OR create your historical table. The changes would be implicit in the version history. Nope, even if I have the ability to see the exact changes of each row, I would still add timestamps everywhere, because timestamp of row change does not equal event timestamp. For example, if I have an order table with status column, and I see a CDC event where status changed from in_progress to completed, I cannot simply assume that the CDC timestamp is the timestamp when order was completed. It's possible that the source database received the event late a few minutes late due to delay upstream, or it's backfilling some missed orders a few days ago. Having a completed_at timestamp (and a bunch of other timestamps for each order lifecycle) would eliminate any ambiguities, and your data analyst will thank you for it. It's the same thing with row history. You cannot simply assume that your row changes are aligned with the logical history of your entity.
- khurs 3mo ago
  Both Iceberg and Delta Lake support 'time travel' so you can query data as it was at a certain date. SELECT count * FROM my_table AS OF "2025-01-01" https://delta.io/blog/2023-02-01-delta-lake-time-travel/ https://delta.io/blog/2023-02-01-delta-lake-time-travel/ https://iceberg.apache.org/docs/latest/spark-queries/#spark-sql-functions https://iceberg.apache.org/docs/latest/spark-queries/#spark-...
  - _zoltan_ 3mo ago
    up to a limited number of snapshots. it's not arbitrary times, it's when there was an atomic snapshot created in the catalog.
    - tomjakubowski 3mo ago
      Don't know about other systems, but in Ducklake every committed transaction creates a new snapshot. https://ducklake.select/docs/stable/duckdb/usage/snapshots https://ducklake.select/docs/stable/duckdb/usage/snapshots
      - _zoltan_ 3mo ago
        because they are making an extra step: instead of the snapshot being actually an object manifest, they store the snapshot metadata in a relation database. I have a use case where we do 100k changes a second. I bet you "their unlimited snapshots because we are cool and using dbs for metadata" will fail in such a use case.
    - jesseryoung 3mo ago
      Every commit in Delta is time travel-able. Defaults in Delta clean up logs in 30 days (and data in 7 days but requires a VACUUM be issued.) You can set those retention periods to any arbitrary time period, but it's typically advised against due to the cost to store all the historical data.
- ignoreusernames 3mo ago
  If safe keeper exposes the changes to the tables somehow, a type2 scd is just a windowed lag over the primary key sorted by the timestamp
  - nikita 3mo ago
    Safekeepers keep a window of WAL in Postgres WAL format and doesn’t have an external API. It streams WAL to pageservers and Postgres read replicas
PunchyHamster 3mo ago
I don't wanna see that S3 bandwidth bill after running some big query
- khurs 3mo ago
  There are self hosted object stores which use the same protocol as S3. One example: https://github.com/minio/minio https://github.com/minio/minio Parquet files are smaller than row based storage in a database (but not those databases with focus on strong compression). And for backup - the files are probably easier to just copy to multiple disks for redundancy, as opposed to database dumps and incremental backups which at the Petabyte scale will be a pain.
  - _zoltan_ 3mo ago
    your reply makes zero sense. there is a reason why people develop for S3: a lot of enterprise data is there. people ingest there from various sources. and it's not just parquet usually, it's multivendor sources writing to an iceberg catalog. nobody will run minio on AWS other than hobby projects and small demos. I regularly work with iceberg datasets in the double digit TB range per dataset. keep that in mind when you think about sizes. databricks, snowflake, large enterprise vendors: they are targeting these sizes.
    - khurs 3mo ago
      > nobody will run minio on AWS other than hobby projects and small demos. You realise not every company uses AWS for any/all its needs? There are datacenters around the world owned by individual companies or co-located. And many companies still have servers on prem. Compute and disks are getting more dense & liquid cooled, so less rack space is needed for same power. And Minio and others can handle Petabytes+ https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/solution-overview-c22-743215.html https://www.cisco.com/c/en/us/products/collateral/servers-un... Backblaze, Cloudflare R2 and other cheaper S3 compatible competitors also exist.
      - 3mo ago
Avalaxy 3mo ago
Super cool stuff. Being able to combine your analytical platform and transactional database into one storage layer without having to set up ETL pipelines in between is really a game changer. Especially since it's just postgres, instead of some proprietary database.
- ah27182 3mo ago
  But much of the layers behind this LTAP architecture are proprietary. The goal for databricks is for customer’s data to never leave their infra.
dsauerbrun 3mo ago
Maybe I'm too stupid to understand the article... How does this achieve performant querying for olap and oltp purposes? Based on my understanding, olap queries will go to the parquet files which are stored in a columnar fashion and oltp style queries will go to a caching layer that sits on top of those parquet files? What's the special sauce here? Seems like they're just caching the data which, for all intents and purposes, seems like the same solution of storing another copy of the data which is what they say they're avoiding.
- conradludgate 3mo ago
  Hi, I work on Lakebase (but not on storage), here's how I understand it. For Lakebase and Neon, our architecture needs the caching layer regardless (what we call Pageservers). Performing reads from S3 directly is too slow so we reconstruct pages and keep them on an nvme server for faster querying. Changing the format on S3 to be Parquet effectively introduces no additional copies over our existing architecture
  - dsauerbrun 3mo ago
    Hmm, if the caching layer doesn't change(I assume it was optimized for olap style queries), and the new parquet format is better for olap... I'm still not understanding how it performs well for oltp reads. I'll give the article another read... Maybe I missed something. Thank you for the response! Really nice to be able to get info straight from people who work on the product
    - nikita 3mo ago
      Recent data plus working set is always in Postgres page format. Historical data when pushed to s3 is in parquet. This happens async - not on the transaction hot path. So older data below certain LSN is on s3 in parquet available to all analytics processing. Hot data is on page servers in page format for OLTP. You can be smart in querying both representations for real time analytical queries
      - 3mo ago
deleted 3mo ago
[deleted]
scritty-dev 3mo ago
So then would LTAP sit to both the left and the right of the medallion architecture? Meaning would you on the left of Bronze use it as an OLTP and to the right of Gold use it as an OLAP? Currently we've been mainly utilizing it to the right of Gold to develop analytic PERN applications that allow us to reuse the RBAC/ACLs set in Unity Catalog, but from this article it seems like that's only half of its utility?
saisrirampur 3mo ago
But why? I’m skeptical of the idea of unifying storage just because it sounds “elegant” or “cool”. It’s not obvious to me how a single storage engine can compete with purpose-built OLTP and OLAP systems like Postgres and ClickHouse, without significant tradeoffs. You also mention removing CDC pipelines. I’m curious if the materialization (conversion across formats) can catchup to an OLTP workload that is heavy (50K+ tps), which is pretty common these days. Also CDC if done right and with care can be magical for users and stays native to the OLTP/OLAP data-store. Third, data Lakes and open formats are suitable for Data Warehousing / Data analyst use-cases than real-time customer facing apps. Sure, you might work on changing that, which is what you are upto, but you’ll always run into tradeoffs, which will make it hard to unleash the best performance, much needed for the latter category.
- nikita 3mo ago
  Conversion is async. The whole point is to never deal with CDC which is error prone and taxing Postgres with occupying a replication slot and burning memory and cpu in the OLTP system.
  - ronfriedhaber 3mo ago
    > CDC which is error prone Just have superior CDC :)
    - saisrirampur 3mo ago
      Exactly! Why unifying storage, which opens up a can of trade-offs.
      - nikita 3mo ago
        If you product is CDC based (peerdb) you don’t want storage to support this :) This architecture is better for OLTP because all maintenance operations are moved to storage AND it has all other benefits such as LTAP that emerge from having a scalable storage.
hasyimibhar 3mo ago
How does LTAP architecture deals with major Postgres upgrade? Is it truly zero-downtime for both upstream and downstream?
chrislusf 3mo ago
Does this use table buckets?
ah27182 3mo ago
[dead]
dzonga 3mo ago
this is an interesting development. OLAP databases have gone this route now known as Lakehouse architecture. figured out soon enough OLTP will go the same way. by eliminating CDC & other data engineering tasks - people will have interesting ways to analyze hot data. RisingWave etc - were already on the frontier of what you can do with hot data - but now more you won't have something separate to run.
ianberdin 3mo ago
Surprisingly, I'm already encountering a second solution that involves storing data chunks on S3 — and this is all within the same week. This is becoming popular. At Playcode, we built what we believe is a revolutionary file system for our Playcode Cloud (https://playcode.io/cloud https://playcode.io/cloud), which enables the creation of full-stack web software. The FS built completely from scratch using Rust. We thought we were the smartest ones around and that nobody else had figured this out. But it turns out Databricks, Neon, and several others have as well. The idea behind a *Bottomless File System* is really cool, and it works very well for us. Essentially, as described here: - There is a *page server* - A *Linux file system* split into chunks (let's call them chunks instead of pages) - A *cache on NVMe* - And of course, *object storage*, where everything is asynchronously synchronized It works quite well, though it has its downsides. One clear advantage is that NVMe drives have become expensive lately, while object storage remains cheap — so the benefits are undeniable. That said, latency is also a factor. On top of that, uplink costs are rising. To run an object storage-backed file system, you need a very strong uplink with consistent speed — 1 Gbps is simply not enough. Ideally, you want *5 to 10 Gbps*, depending on the load. We spend a lot of time optimizing and experimenting with different hosting providers — specifically bare metal hardware. The main challenges are: - *Slow disks* - *Slow uplink* - And as it turns out, *object storage can be unreliable* — unless you're using S3 But AWS hardware is expensive, so nothing in life is ever that simple.
- ianberdin 3mo ago
  I also want to point out that all of this sounds fun and great — until the load kicks in and usage starts to grow. That's when you start seeing: - *Rate limits* from object storage - *Dropped packets* - *Hanging S3 requests* - *Overloaded NVMe drives* — because it turns out they're nowhere near as fast as they seem For example, we recently discovered that the read speed is 5 GB/s, but the *average write speed is only 400 MB/s* — not several gigabytes as expected. Surprise! Who would have thought that Bare Metal could ship such underwhelming drives? And then there's the CPU — which is also easy to kill, for instance, if you're compressing chunks. And so on, and so on. A lot of things surface once you're in *production usage*. On paper, of course, everything looked much simpler.
lmeyerov 3mo ago
I have to wonder: What's the open protocol - some sort of LTAP read/write path - for compute engines to target LTAP / lakebases? Sort of like how iceberg > deltalake, except now one layer up (OLTP+OLAP). LTAP / lakebase poses an interesting gap we have been tracking ever since iceberg as we build gfql (the first oss fully-vectorized property graph compute engine with CPU+GPU targets): There's a natural desire to support both OLAP + OLTP modes for new engines, but also a gap for the cloud-native storage tier we don't want to own. Iceberg etc were exciting as introducing more write-friendly standards-based OLAP backends better than just S3, eg, gives atomicity, and via iceberg, with open governance for code+protocol. LTAP introduces the pattern we can target in theory for OLTP+OLAP backends... but in practice, via a proprietary manner. A potential 'open' form of writers is to support postgres for oltp path writes + spark writers (presumably arrow-flight-friendly) for olap path writes, but that is so hacky.
onchainbuilder 3mo ago
[flagged]

182 points by andrenotgiant 3mo ago

deleted 3mo ago

[deleted]

seobot_dk1289 3mo ago

[flagged]

andrenotgiant 3mo ago

Here's what I don't understand: Part of the value of doing an ETL pipeline via streaming replication is you get the full history of data in a table. An SCD type 2 table where each row also has a valid_from and valid_to timestamp column. How would someone do the same thing with this architecture?

eveningtree 3mo ago
Rather than answering directly, I'm thinking about this problem from the other end altogether ever since I saw the dbricks rt demo. Apologies for the rambling response, as I haven't yet finished thinking about this problem... We ended up with 'hot' data in oltp and 'cold/archival' data in olap because the storage size of oltp has always been limited. (1) Limited by computation - there's only so much data that we can store on disks and nvme (2) Limited by wallet - disks and nvme are EXPENSIVE Also, the tight coupling of compute and data didn't help. It limited the size of databases on the individual expensive compute nodes. So, another question will be - What's currently stopping me from keeping the scd history tables right in my oltp db? what's forcing me to copy state into my etl/elt pipeline and the process it into scd into a dedicated olap db? To some extent,the answer is still the same - the oltp cannot scale for the storage size required for keeping historical data. So, I've had to take out the 'cold' historical data and keep it in my olap freezer. Now, if oltp itself is scaling, I'm not gonna bother with the copying step. I'll just prefer to store the history in oltp itself. In my perspective (majorly from handling IoT systems), I need olap for 2 reasons - (1) storage scalability, and (2) analytical processing speed I now consider (1) to be a solved problem As for (2), I'm still not sure how this architecture ends up matching the query processing speeds of column-oriented storages. But again, I need to study more. The SCD pipeline still remains in some form. Either in the form of (1) scd rows that we currently keep (etl pipeline) , or (2) as older lsn rows that simply don't get deleted (existing db engine). I've done quite a lot of experimentation with (2), and it is a pretty solid concept to work with. I've spent quite a lot of years hammering my brain at databases and datastores in general. And I've now got a feeling that this is it. Finally.
deleted 3mo ago
[deleted]
hasyimibhar 3mo ago
It wouldn't be possible to do this with LTAP architecture since (I'm assuming) the individual logical changes are not visible. But honestly I've always seen SCD type 2 table as a workaround due to lack of data modeling experience in the source database. If you design your tables correctly, you shouldn't need SCD type 2 downstream. For example, if you know your user can change emails, and there might be events from another source that is keyed by user email (e.g. marketing-related events), then naturally you will need some sort of email_history table that has historical mapping of user id to email (you probably need it for audit purposes too). Then in this case there is no need to build SCD type 2 table of user from CDC, it's already there.
- deleted 3mo ago
  [deleted]
- TheTaytay 3mo ago
  I think you have a point, and SCD type 2 feels like a workaround, but there is also something to be said for the ability to query every row as it was at any given version. I’m not saying that SCD type 2 is the best solution given there might be a more domain-specific way to do it, but I see it a lot like file-based version control. It’s convenient to be able to examine all files as they existed at any point in time, without having to “model” the ways in which those files might change directly into the domain of the individual files. If you have something like dolt (not affiliated), a version controlled database, you wouldn’t have to slap change dates on anything OR create your historical table. The changes would be implicit in the version history.
  - hasyimibhar 3mo ago
    > If you have something like dolt (not affiliated), a version controlled database, you wouldn’t have to slap change dates on anything OR create your historical table. The changes would be implicit in the version history. Nope, even if I have the ability to see the exact changes of each row, I would still add timestamps everywhere, because timestamp of row change does not equal event timestamp. For example, if I have an order table with status column, and I see a CDC event where status changed from in_progress to completed, I cannot simply assume that the CDC timestamp is the timestamp when order was completed. It's possible that the source database received the event late a few minutes late due to delay upstream, or it's backfilling some missed orders a few days ago. Having a completed_at timestamp (and a bunch of other timestamps for each order lifecycle) would eliminate any ambiguities, and your data analyst will thank you for it. It's the same thing with row history. You cannot simply assume that your row changes are aligned with the logical history of your entity.
khurs 3mo ago
Both Iceberg and Delta Lake support 'time travel' so you can query data as it was at a certain date. SELECT count * FROM my_table AS OF "2025-01-01" https://delta.io/blog/2023-02-01-delta-lake-time-travel/ https://delta.io/blog/2023-02-01-delta-lake-time-travel/ https://iceberg.apache.org/docs/latest/spark-queries/#spark-sql-functions https://iceberg.apache.org/docs/latest/spark-queries/#spark-...
- _zoltan_ 3mo ago
  up to a limited number of snapshots. it's not arbitrary times, it's when there was an atomic snapshot created in the catalog.
  - tomjakubowski 3mo ago
    Don't know about other systems, but in Ducklake every committed transaction creates a new snapshot. https://ducklake.select/docs/stable/duckdb/usage/snapshots https://ducklake.select/docs/stable/duckdb/usage/snapshots
    - _zoltan_ 3mo ago
      because they are making an extra step: instead of the snapshot being actually an object manifest, they store the snapshot metadata in a relation database. I have a use case where we do 100k changes a second. I bet you "their unlimited snapshots because we are cool and using dbs for metadata" will fail in such a use case.
  - jesseryoung 3mo ago
    Every commit in Delta is time travel-able. Defaults in Delta clean up logs in 30 days (and data in 7 days but requires a VACUUM be issued.) You can set those retention periods to any arbitrary time period, but it's typically advised against due to the cost to store all the historical data.
ignoreusernames 3mo ago
If safe keeper exposes the changes to the tables somehow, a type2 scd is just a windowed lag over the primary key sorted by the timestamp
- nikita 3mo ago
  Safekeepers keep a window of WAL in Postgres WAL format and doesn’t have an external API. It streams WAL to pageservers and Postgres read replicas

PunchyHamster 3mo ago

I don't wanna see that S3 bandwidth bill after running some big query

khurs 3mo ago
There are self hosted object stores which use the same protocol as S3. One example: https://github.com/minio/minio https://github.com/minio/minio Parquet files are smaller than row based storage in a database (but not those databases with focus on strong compression). And for backup - the files are probably easier to just copy to multiple disks for redundancy, as opposed to database dumps and incremental backups which at the Petabyte scale will be a pain.
- _zoltan_ 3mo ago
  your reply makes zero sense. there is a reason why people develop for S3: a lot of enterprise data is there. people ingest there from various sources. and it's not just parquet usually, it's multivendor sources writing to an iceberg catalog. nobody will run minio on AWS other than hobby projects and small demos. I regularly work with iceberg datasets in the double digit TB range per dataset. keep that in mind when you think about sizes. databricks, snowflake, large enterprise vendors: they are targeting these sizes.
  - khurs 3mo ago
    > nobody will run minio on AWS other than hobby projects and small demos. You realise not every company uses AWS for any/all its needs? There are datacenters around the world owned by individual companies or co-located. And many companies still have servers on prem. Compute and disks are getting more dense & liquid cooled, so less rack space is needed for same power. And Minio and others can handle Petabytes+ https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/solution-overview-c22-743215.html https://www.cisco.com/c/en/us/products/collateral/servers-un... Backblaze, Cloudflare R2 and other cheaper S3 compatible competitors also exist.
    - 3mo ago

Avalaxy 3mo ago

Super cool stuff. Being able to combine your analytical platform and transactional database into one storage layer without having to set up ETL pipelines in between is really a game changer. Especially since it's just postgres, instead of some proprietary database.

ah27182 3mo ago
But much of the layers behind this LTAP architecture are proprietary. The goal for databricks is for customer’s data to never leave their infra.

dsauerbrun 3mo ago

Maybe I'm too stupid to understand the article... How does this achieve performant querying for olap and oltp purposes? Based on my understanding, olap queries will go to the parquet files which are stored in a columnar fashion and oltp style queries will go to a caching layer that sits on top of those parquet files? What's the special sauce here? Seems like they're just caching the data which, for all intents and purposes, seems like the same solution of storing another copy of the data which is what they say they're avoiding.

conradludgate 3mo ago
Hi, I work on Lakebase (but not on storage), here's how I understand it. For Lakebase and Neon, our architecture needs the caching layer regardless (what we call Pageservers). Performing reads from S3 directly is too slow so we reconstruct pages and keep them on an nvme server for faster querying. Changing the format on S3 to be Parquet effectively introduces no additional copies over our existing architecture
- dsauerbrun 3mo ago
  Hmm, if the caching layer doesn't change(I assume it was optimized for olap style queries), and the new parquet format is better for olap... I'm still not understanding how it performs well for oltp reads. I'll give the article another read... Maybe I missed something. Thank you for the response! Really nice to be able to get info straight from people who work on the product
  - nikita 3mo ago
    Recent data plus working set is always in Postgres page format. Historical data when pushed to s3 is in parquet. This happens async - not on the transaction hot path. So older data below certain LSN is on s3 in parquet available to all analytics processing. Hot data is on page servers in page format for OLTP. You can be smart in querying both representations for real time analytical queries
    - 3mo ago

deleted 3mo ago

[deleted]

scritty-dev 3mo ago

So then would LTAP sit to both the left and the right of the medallion architecture? Meaning would you on the left of Bronze use it as an OLTP and to the right of Gold use it as an OLAP? Currently we've been mainly utilizing it to the right of Gold to develop analytic PERN applications that allow us to reuse the RBAC/ACLs set in Unity Catalog, but from this article it seems like that's only half of its utility?

saisrirampur 3mo ago

But why? I’m skeptical of the idea of unifying storage just because it sounds “elegant” or “cool”. It’s not obvious to me how a single storage engine can compete with purpose-built OLTP and OLAP systems like Postgres and ClickHouse, without significant tradeoffs. You also mention removing CDC pipelines. I’m curious if the materialization (conversion across formats) can catchup to an OLTP workload that is heavy (50K+ tps), which is pretty common these days. Also CDC if done right and with care can be magical for users and stays native to the OLTP/OLAP data-store. Third, data Lakes and open formats are suitable for Data Warehousing / Data analyst use-cases than real-time customer facing apps. Sure, you might work on changing that, which is what you are upto, but you’ll always run into tradeoffs, which will make it hard to unleash the best performance, much needed for the latter category.

nikita 3mo ago
Conversion is async. The whole point is to never deal with CDC which is error prone and taxing Postgres with occupying a replication slot and burning memory and cpu in the OLTP system.
- ronfriedhaber 3mo ago
  > CDC which is error prone Just have superior CDC :)
  - saisrirampur 3mo ago
    Exactly! Why unifying storage, which opens up a can of trade-offs.
    - nikita 3mo ago
      If you product is CDC based (peerdb) you don’t want storage to support this :) This architecture is better for OLTP because all maintenance operations are moved to storage AND it has all other benefits such as LTAP that emerge from having a scalable storage.

hasyimibhar 3mo ago

How does LTAP architecture deals with major Postgres upgrade? Is it truly zero-downtime for both upstream and downstream?

chrislusf 3mo ago

Does this use table buckets?

ah27182 3mo ago

[dead]

dzonga 3mo ago

this is an interesting development. OLAP databases have gone this route now known as Lakehouse architecture. figured out soon enough OLTP will go the same way. by eliminating CDC & other data engineering tasks - people will have interesting ways to analyze hot data. RisingWave etc - were already on the frontier of what you can do with hot data - but now more you won't have something separate to run.

ianberdin 3mo ago

Surprisingly, I'm already encountering a second solution that involves storing data chunks on S3 — and this is all within the same week. This is becoming popular. At Playcode, we built what we believe is a revolutionary file system for our Playcode Cloud (https://playcode.io/cloud https://playcode.io/cloud), which enables the creation of full-stack web software. The FS built completely from scratch using Rust. We thought we were the smartest ones around and that nobody else had figured this out. But it turns out Databricks, Neon, and several others have as well. The idea behind a *Bottomless File System* is really cool, and it works very well for us. Essentially, as described here: - There is a *page server* - A *Linux file system* split into chunks (let's call them chunks instead of pages) - A *cache on NVMe* - And of course, *object storage*, where everything is asynchronously synchronized It works quite well, though it has its downsides. One clear advantage is that NVMe drives have become expensive lately, while object storage remains cheap — so the benefits are undeniable. That said, latency is also a factor. On top of that, uplink costs are rising. To run an object storage-backed file system, you need a very strong uplink with consistent speed — 1 Gbps is simply not enough. Ideally, you want *5 to 10 Gbps*, depending on the load. We spend a lot of time optimizing and experimenting with different hosting providers — specifically bare metal hardware. The main challenges are: - *Slow disks* - *Slow uplink* - And as it turns out, *object storage can be unreliable* — unless you're using S3 But AWS hardware is expensive, so nothing in life is ever that simple.

ianberdin 3mo ago
I also want to point out that all of this sounds fun and great — until the load kicks in and usage starts to grow. That's when you start seeing: - *Rate limits* from object storage - *Dropped packets* - *Hanging S3 requests* - *Overloaded NVMe drives* — because it turns out they're nowhere near as fast as they seem For example, we recently discovered that the read speed is 5 GB/s, but the *average write speed is only 400 MB/s* — not several gigabytes as expected. Surprise! Who would have thought that Bare Metal could ship such underwhelming drives? And then there's the CPU — which is also easy to kill, for instance, if you're compressing chunks. And so on, and so on. A lot of things surface once you're in *production usage*. On paper, of course, everything looked much simpler.

lmeyerov 3mo ago

I have to wonder: What's the open protocol - some sort of LTAP read/write path - for compute engines to target LTAP / lakebases? Sort of like how iceberg > deltalake, except now one layer up (OLTP+OLAP). LTAP / lakebase poses an interesting gap we have been tracking ever since iceberg as we build gfql (the first oss fully-vectorized property graph compute engine with CPU+GPU targets): There's a natural desire to support both OLAP + OLTP modes for new engines, but also a gap for the cloud-native storage tier we don't want to own. Iceberg etc were exciting as introducing more write-friendly standards-based OLAP backends better than just S3, eg, gives atomicity, and via iceberg, with open governance for code+protocol. LTAP introduces the pattern we can target in theory for OLTP+OLAP backends... but in practice, via a proprietary manner. A potential 'open' form of writers is to support postgres for oltp path writes + spark writers (presumably arrow-flight-friendly) for olap path writes, but that is so hacky.

onchainbuilder 3mo ago

[flagged]