4 ms·

Compactions do NOT reduce disk usage. On the contrary, they rewrite the data, so they generate new and presumably smaller SSTables. Which means more disk usage.

by puzzle 8y ago

Compactions do NOT reduce disk usage. On the contrary, they rewrite the data, so they generate new and presumably smaller SSTables. Which means more disk usage. It's the garbage collector on the master that removes files. You can combine the two, but that requires some tooling that compacts a section of the table, then triggers (or waits for, I can't recall) the garbage collector.

shereadsthenews 8y ago
This will not be of general interest to HN, but I disagree. If compaction is behind then commanding a higher compaction rate can free space, and it can also bring down your service. You can also call for higher rates of splits and merges with the same effect. I have no idea of those actions contributed to this outage.
- puzzle 8y ago
  Compactions happen on the tablet server. They only write data. By definition, they increase disk usage, right away. Only the garbage collector, which needs global state and thus runs on the master, can actually delete files in Colossus and bring down disk usage. It doesn't always run as frequently as one would think or hope. It definitely doesn't run immediately after one tablet has been compacted. I remember one incident where a team got an alert that they were at 90% of quota or so. They compacted data too aggressively and they actually hit 100% before the GC could do its job. That's why I mentioned that you should compact a fraction of the table, then let GC run. Google being Google, I'm sure such a tool has been written, in several variants, by N different teams.
  - shereadsthenews 8y ago
    As long as we're splitting hairs, only the Colossus curator can really delete the file and if the quota manager is behind you can still run out of space.
    - londons_explore 8y ago
      The quota manager can be behind in the opposite direction and let you go way over quota too. Yay - free unlimited disk space for all! (for ~5 minutes!)
snewman 8y ago
Well: in many cases, compactions can make more files eligible to be garbage collected. So if you want to accelerate the freeing up of space, running a compaction may serve that goal. It will temporarily increase usage, but once the garbage collector catches up the net effect is to reduce disk usage. Unless I'm mis-remembering how things work. (Caveat: I left Google almost 9 years ago.) And so "run a compaction to free up space" is a reasonable shorthand most of the time... unless you're so close to the edge that you don't have room for the temporary increase.
- puzzle 8y ago
  You're not misremembering. It's just that timing matters. And GC always got less attention than compactions. My rule of thumb was: if I have a change that will shrink tablets by 10% through e.g. compression and I apply it wholesale with a manual compaction of the entire table, I should assume that in the worst case the table will temporarily take up 190% of current space, until the next GC run or two. If all or most of your data is in one table, this can be a problem. Organic compactions are friendlier because they occur over the course of (typically) days, leaving the GC with plenty of time to clean up.
  - londons_explore 8y ago
    The real issue is that Bigtable's used to be in one big shared cell which could handle temporary 190% increases in resources. Now, to increase isolation between services, everyone is off in a partition, and each individual partition is much smaller and therefore can't withstand resource spikes without falling over. And I bet the "emergency loan" functionality for various resources still isn't automated and still has arcane requirements to meet and a delay before it kicks in. Yay - a 15 minute delay. Thats exactly what I need when my entire service is down! /s

by puzzle 8y ago

shereadsthenews 8y ago

This will not be of general interest to HN, but I disagree. If compaction is behind then commanding a higher compaction rate can free space, and it can also bring down your service. You can also call for higher rates of splits and merges with the same effect. I have no idea of those actions contributed to this outage.

puzzle 8y ago
Compactions happen on the tablet server. They only write data. By definition, they increase disk usage, right away. Only the garbage collector, which needs global state and thus runs on the master, can actually delete files in Colossus and bring down disk usage. It doesn't always run as frequently as one would think or hope. It definitely doesn't run immediately after one tablet has been compacted. I remember one incident where a team got an alert that they were at 90% of quota or so. They compacted data too aggressively and they actually hit 100% before the GC could do its job. That's why I mentioned that you should compact a fraction of the table, then let GC run. Google being Google, I'm sure such a tool has been written, in several variants, by N different teams.
- shereadsthenews 8y ago
  As long as we're splitting hairs, only the Colossus curator can really delete the file and if the quota manager is behind you can still run out of space.
  - londons_explore 8y ago
    The quota manager can be behind in the opposite direction and let you go way over quota too. Yay - free unlimited disk space for all! (for ~5 minutes!)

snewman 8y ago

Well: in many cases, compactions can make more files eligible to be garbage collected. So if you want to accelerate the freeing up of space, running a compaction may serve that goal. It will temporarily increase usage, but once the garbage collector catches up the net effect is to reduce disk usage. Unless I'm mis-remembering how things work. (Caveat: I left Google almost 9 years ago.) And so "run a compaction to free up space" is a reasonable shorthand most of the time... unless you're so close to the edge that you don't have room for the temporary increase.

puzzle 8y ago
You're not misremembering. It's just that timing matters. And GC always got less attention than compactions. My rule of thumb was: if I have a change that will shrink tablets by 10% through e.g. compression and I apply it wholesale with a manual compaction of the entire table, I should assume that in the worst case the table will temporarily take up 190% of current space, until the next GC run or two. If all or most of your data is in one table, this can be a problem. Organic compactions are friendlier because they occur over the course of (typically) days, leaving the GC with plenty of time to clean up.
- londons_explore 8y ago
  The real issue is that Bigtable's used to be in one big shared cell which could handle temporary 190% increases in resources. Now, to increase isolation between services, everyone is off in a partition, and each individual partition is much smaller and therefore can't withstand resource spikes without falling over. And I bet the "emergency loan" functionality for various resources still isn't automated and still has arcane requirements to meet and a delay before it kicks in. Yay - a 15 minute delay. Thats exactly what I need when my entire service is down! /s