It's easy to switch the LLM API, but in practice this requires having a strong eval suite so that the expected behavior of whatever is built on top changes with

It's easy to switch the LLM API, but in practice this requires having a strong eval suite so that the expected behavior of whatever is built on top changes within acceptable limits. It's really the implications of the LLM switch that matter.