4 ms·
> somehow a containerd boltdb on that host got corrupted, so when the machine bounced, containerd refused to come back online. We use containerd as a cache Hey
by kunley 3y ago
> somehow a containerd boltdb on that host got corrupted, so when the machine bounced, containerd refused to come back online. We use containerd as a cache
Hey, even if I can feel sympathetic for the course of unfortunate events, it's hard to not to comment:
if you're using a cache, you should invalidate it on failure!
- tptacek 3y agoIt's a read-through cache. This wasn't a cache invalidation issue. It's a systems-level state corruption problem that just happened to break a system used primarily as a cache.
- kunley 3y agoWhat I meant is that if the compromised host was unable to use broken boltdb cache, the cache should be zeroed and repopulated. Was it really hours of such cache rebuild vs hours of trying to fix the boltdb? Btw I am happy I got only small amounts of data in any of bolt databases...
- tptacek 3y agoThis isn't a boltdb we designed. It's just containerd. I am probably not doing the outage justice, because "blitz and repopulate" is a time-honored strategy here.