4 ms·

Well: in many cases, compactions can make more files eligible to be garbage collected. So if you want to accelerate the freeing up of space, running a compactio

Well: in many cases, compactions can make more files eligible to be garbage collected. So if you want to accelerate the freeing up of space, running a compaction may serve that goal. It will temporarily increase usage, but once the garbage collector catches up the net effect is to reduce disk usage. Unless I'm mis-remembering how things work. (Caveat: I left Google almost 9 years ago.) And so "run a compaction to free up space" is a reasonable shorthand most of the time... unless you're so close to the edge that you don't have room for the temporary increase.

puzzle 8y ago
You're not misremembering. It's just that timing matters. And GC always got less attention than compactions. My rule of thumb was: if I have a change that will shrink tablets by 10% through e.g. compression and I apply it wholesale with a manual compaction of the entire table, I should assume that in the worst case the table will temporarily take up 190% of current space, until the next GC run or two. If all or most of your data is in one table, this can be a problem. Organic compactions are friendlier because they occur over the course of (typically) days, leaving the GC with plenty of time to clean up.
- londons_explore 8y ago
  The real issue is that Bigtable's used to be in one big shared cell which could handle temporary 190% increases in resources. Now, to increase isolation between services, everyone is off in a partition, and each individual partition is much smaller and therefore can't withstand resource spikes without falling over. And I bet the "emergency loan" functionality for various resources still isn't automated and still has arcane requirements to meet and a delay before it kicks in. Yay - a 15 minute delay. Thats exactly what I need when my entire service is down! /s

4 ms·

Well: in many cases, compactions can make more files eligible to be garbage collected. So if you want to accelerate the freeing up of space, running a compactio

by snewman 8y ago

puzzle 8y ago
You're not misremembering. It's just that timing matters. And GC always got less attention than compactions. My rule of thumb was: if I have a change that will shrink tablets by 10% through e.g. compression and I apply it wholesale with a manual compaction of the entire table, I should assume that in the worst case the table will temporarily take up 190% of current space, until the next GC run or two. If all or most of your data is in one table, this can be a problem. Organic compactions are friendlier because they occur over the course of (typically) days, leaving the GC with plenty of time to clean up.
- londons_explore 8y ago
  The real issue is that Bigtable's used to be in one big shared cell which could handle temporary 190% increases in resources. Now, to increase isolation between services, everyone is off in a partition, and each individual partition is much smaller and therefore can't withstand resource spikes without falling over. And I bet the "emergency loan" functionality for various resources still isn't automated and still has arcane requirements to meet and a delay before it kicks in. Yay - a 15 minute delay. Thats exactly what I need when my entire service is down! /s