Start with 7B and refine as much as you possibly can there. Your value is going to be determined by the variable iteration_time, where your final equation invol

Start with 7B and refine as much as you possibly can there. Your value is going to be determined by the variable iteration_time, where your final equation involves 1/iteration_time. Any lessons learned there will effectively scale. When you cannot get any improvements over several days/weeks of different experiments and have converged somewhat, then move up to the next model size, and do that to convergence. Then repeat this loop as you go. Same concept as mipmapping, just with resources. How you use your resources is more important than the ones you have. I've made the vast majority of my own big discoveries with a T4 or a single A100, generally speaking, and I've done this for years. In terms of providers, I like Lambda the best, personally, but I do a shocking amount of work in Colab Pro due to its iterative nature. I believe I've had GPU availability issues for both of them, however.