Without really deep experience (my FPGA time ended 15ish years ago), I would say that granularity and frequency of the swapped out parts need to hit a sweet spo

Without really deep experience (my FPGA time ended 15ish years ago), I would say that granularity and frequency of the swapped out parts need to hit a sweet spot and you need to be able to hide that latency. If I understand you correctly, you tsrget the latter? But as with all pipelining you need do have an early trigger or lookahead.