On your first point, you’re correct. Directors scan their databases on start to rebuild their cache and reschedule the jobs that need to get retried. The scan i

On your first point, you’re correct. Directors scan their databases on start to rebuild their cache and reschedule the jobs that need to get retried. The scan is usually quick for data that was recently inserted since they’re likely in cache, it make take a couple of minutes to scan everything. Because we keep the database to a small size we can cap how long this operation will take. We also scan the database in reverse order (using the primary key, thanks to the rough ordering of KSUIDs), to help reschedule he most recent jobs first, which is possible because the scan can happen concurrently with tye jobs processing. On your second point, archiving actually happens in the “drainers”, not the director. It’s the component that picks up unused databases and flush their data back into the set of running directors. We initially built archiving into the directors but it turned out to steal too much resources away from job processing so we moved it out into the drainers, which actually gives us opportunities to do it more efficiently (for example creating larger archives, which helps with compression as well). Sorry if we cut some details off of the post, there is plenty more to tell about this system but it’s a lot for a single blog post ;)