Sounds challenging. If it's something that happened a lot, you could keep the test suite in a separate repo with API levels, such that tests need newer APIs g

Sounds challenging. If it's something that happened a lot, you could keep the test suite in a separate repo with API levels, such that tests need newer APIs go in a higher numbered folder (think database migrations in web frameworks) and create a simple run script.