4 ms·
The cloud storage option for CH looks like a game changer for time-based data. Any concerns there about accidentally causing "trashing" when cold data is needed
by atombender 3y ago
The cloud storage option for CH looks like a game changer for time-based data. Any concerns there about accidentally causing "trashing" when cold data is needed? I believe the MergeTree system works by splitting the table into parts during insertion that then later get merged together, so you have to careful that during merging, only the "hot" data is touched, otherwise you'll start pulling in cloud storage data that's supposed to be effectively read-only.
- the_duke 3y agoThe merging happens primarily on hot data, I haven't run into any issues there. But there are lots of approaches, depending on your needs. You can (should) define a "cache disk" for S3, which will cache up to X Gb locally to avoid trashing. Another option is is to move data into separate (purely S3 backed) tables after a certain time to avoid accidentally fetching large amounts of data from S3. You can still easily join the data together if needed.