4 ms·
> you write new files containing the changes (the log) People are asking for create-if-not-exists specifically to be able to add objects to an ordered log, wit
by yencabulator 2y ago
> you write new files containing the changes (the log)
People are asking for create-if-not-exists specifically to be able to add objects to an ordered log, without needing a separate service for coordination.
S3 cannot be used for this. GCS, for example, can.
- klodolph 2y agoYeah—I know I didn’t mention that. You can build this type of system on top of S3 + CAS. The “append” feature isn’t necessary for functionality, it just improves the cost / performance. The idea of running your own database on top of S3 is, well, it’s gonna be janky. It’s not ideal. You do end up seeing databases running on top of S3 (I’ve seen some), and sometimes it even makes sense.
- yencabulator 2y agoOLAP data in S3 makes perfect sense, and that use case is desperate for create-if-not-exist to enable simpler inserts. https://www.databricks.com/wp-content/uploads/2020/08/p975-armbrust.pdf https://www.databricks.com/wp-content/uploads/2020/08/p975-a...