3 ms·
This is a really clever design. The cost estimates are particularly notable: if they're right that's a cost of about $3/day for 6TB/day of written data, 2TB/da
by simonw 1y ago
This is a really clever design.
The cost estimates are particularly notable: if they're right that's a cost of about $3/day for 6TB/day of written data, 2TB/day of deletes and 50K read queries.
Storing all those TBs of data in S3 is where the real cost lies. I think it costs $5520 to store 8TB*30 = 240TB in S3, and if you retain all data your monthly cost goes up by $5520 every month.
- xyzzy_plugh 1y agoI think the idea is that the deletes would eventually be compacted, so it's ultimately half as much, but I digress. The cost isn't that bad all things considered. Hot, durable and available data ain't that cheap, especially in the cloud. Self-hosting is within an order of magnitude.
- shayonj 1y agoI think ideally you could map retention of cold data to file objects itself and using key space naming strategy and lifecycle rules, expire the data that is not needed, thus saving on the storage costs (as much as possible hopefully)
- shayonj 1y agoHere is another take on deletes through by just updating the row groups in Parquet file through multi part upload and UploadPartCopy - https://www.shayon.dev/post/2025/285/mutable-atomic-deletes-with-parquet-backed-columnar-tables-on-s3/ https://www.shayon.dev/post/2025/285/mutable-atomic-deletes-...