3 ms·
Time series data can vary and I'd agree that most time series is immutable. There are however use cases in which updates can happen. For instance, at my previou
by Shelnutt2 6y ago
Time series data can vary and I'd agree that most time series is immutable. There are however use cases in which updates can happen. For instance, at my previous job before TileDB Inc, we had a case where 99% of our data was immutable and never updated but a very small amount of data could be updated if there were late arriving parts. In order to get near-realtime data we accepted that sometimes some columns might not be available within the window the datapoint represents. In that case that record might reappear at a later time with the complete and correct values.
Of course there are trade offs, we could have forced a longer waiting period until we were confident the data was finalized. We also could have ignored the updates. In the end we used a system of staging tables for loading the last 24 hours of data before merging out into a more finalized table. This kept the load of updating records in the database down, and still allowed us to achieve our goals.
At the time, several years ago, I was not aware of TileDB else we would have considered it instead of a more traditional database vendor.