I've been using evolutionary techniques with Pareto front optimization to deal with this tradeoff. If two (or more) objectives are in direct conflict, but other

I've been using evolutionary techniques with Pareto front optimization to deal with this tradeoff. If two (or more) objectives are in direct conflict, but otherwise don't dominate one another, we simply take them all. Then, the problem becomes one of making sure you have enough resources to maintain the frontier across the generations. If you do have to collapse the frontier, you can use things like crowding score to maintain diversity. Example use case - minimizing total neuron activations in a spiking neural simulation while simultaneously maximizing activation coverage and output correctness scores. You don't necessarily know if the cheaper to simulate, but less correct, candidate represents a better evolutionary path until you go down it a few steps. This also gives you an idea of what your horizon looks like on the fitness landscape. If your Pareto front is small, you are typically deep into exploitation. A simple forward path. If it is large, you are putting more resources into exploration, but you aren't giving up exploitation - it just takes longer to get through the jungle bits. This can be used as a heuristic to inform restarting or other strategic corrective measures in the evolutionary algorithm. If we are stuck in jungle for too long, we might decide to abandon the run and pick new hyperparameters.