3 ms·

> On Monday 11 March 2019, Google SREs were alerted to a significant increase in storage resources for metadata used by the internal blob service. On Tuesday 12

by dap 8y ago

> On Monday 11 March 2019, Google SREs were alerted to a significant increase in storage resources for metadata used by the internal blob service. On Tuesday 12 March, to reduce resource usage, SREs made a configuration change which had a side effect of overloading a key part of the system for looking up the location of blob data. The increased load eventually lead to a cascading failure. Any idea what this means? What kind of configuration change would reduce storage but overload the system? I imagine turning on compression could do that, but I'd expect it would be easy to reverse. (Edit: Maybe it was the cascading failure that made it more serious?)

bostik 8y ago
> What kind of configuration change would reduce storage but overload the system? Maybe the increased storage usage was due to wrong kind of load rebalancing? I can easily see that a change to reduce _object_ hit rate in any given location, might just increase the _metadata_ hit rate. If storage is balanced by spreading the load to workers that keep their data nearby (for reduced latency), then an unexpected increase in these workers could also cause an increase in required storage. If you want to balance the load on these properly, you'll have to do more location data lookups. Oops. Now your metadata's metadata is the chokepoint. Massive read contention on blob store address lookups --> cascading failure.
rhizome 8y ago
At a high level it sounds to me like they're paying Complexity Debt.
puzzle 8y ago
Two straightforward changes in Bigtable or Spanner are changing tablet compression settings or tuning compaction more aggressively. Edit: given the mention of a job that got stopped, perhaps they had a tool that manually forced compaction of data in order to adopt new compression settings or purge deleted rows. Normally, compactions would happen on the tablet servers on a cycle of N days. Another possible explanation is that they tweaked the garbage collector, which runs on the master.
- shereadsthenews 8y ago
  These seem really likely. Being a bigtable operator is exactly like being Mickey Mouse in the Sorcerer's Apprentice. You want buckets of water? Okay.

by dap 8y ago

bostik 8y ago

> What kind of configuration change would reduce storage but overload the system? Maybe the increased storage usage was due to wrong kind of load rebalancing? I can easily see that a change to reduce _object_ hit rate in any given location, might just increase the _metadata_ hit rate. If storage is balanced by spreading the load to workers that keep their data nearby (for reduced latency), then an unexpected increase in these workers could also cause an increase in required storage. If you want to balance the load on these properly, you'll have to do more location data lookups. Oops. Now your metadata's metadata is the chokepoint. Massive read contention on blob store address lookups --> cascading failure.

rhizome 8y ago

At a high level it sounds to me like they're paying Complexity Debt.

puzzle 8y ago

Two straightforward changes in Bigtable or Spanner are changing tablet compression settings or tuning compaction more aggressively. Edit: given the mention of a job that got stopped, perhaps they had a tool that manually forced compaction of data in order to adopt new compression settings or purge deleted rows. Normally, compactions would happen on the tablet servers on a cycle of N days. Another possible explanation is that they tweaked the garbage collector, which runs on the master.

shereadsthenews 8y ago
These seem really likely. Being a bigtable operator is exactly like being Mickey Mouse in the Sorcerer's Apprentice. You want buckets of water? Okay.