Now, consider that this is the cost of the final model reported in the paper. This doesn't account for all the iterations of trying out e.g. different model arc

Now, consider that this is the cost of the final model reported in the paper. This doesn't account for all the iterations of trying out e.g. different model architectures, hyperparameter sweeps, etc. The true cost of the experimentation is likely at least an order of magnitude higher.