3 ms·
This will not be of general interest to HN, but I disagree. If compaction is behind then commanding a higher compaction rate can free space, and it can also bri
by shereadsthenews 8y ago
This will not be of general interest to HN, but I disagree. If compaction is behind then commanding a higher compaction rate can free space, and it can also bring down your service. You can also call for higher rates of splits and merges with the same effect. I have no idea of those actions contributed to this outage.
- puzzle 8y agoCompactions happen on the tablet server. They only write data. By definition, they increase disk usage, right away. Only the garbage collector, which needs global state and thus runs on the master, can actually delete files in Colossus and bring down disk usage. It doesn't always run as frequently as one would think or hope. It definitely doesn't run immediately after one tablet has been compacted. I remember one incident where a team got an alert that they were at 90% of quota or so. They compacted data too aggressively and they actually hit 100% before the GC could do its job. That's why I mentioned that you should compact a fraction of the table, then let GC run. Google being Google, I'm sure such a tool has been written, in several variants, by N different teams.
- shereadsthenews 8y agoAs long as we're splitting hairs, only the Colossus curator can really delete the file and if the quota manager is behind you can still run out of space.
- londons_explore 8y agoThe quota manager can be behind in the opposite direction and let you go way over quota too. Yay - free unlimited disk space for all! (for ~5 minutes!)