Yeah, I am talking about ~10% performance differences for a specific model (I believe it is when benchmarking vanilla seq2seq model (from AIAYN paper) with smal

Yeah, I am talking about ~10% performance differences for a specific model (I believe it is when benchmarking vanilla seq2seq model (from AIAYN paper) with small batch size and no gradient accumulation). NVLink is about ~$100, so for these cases, I guess you shouldn't expect it to be "more than buying another card" type of improvements.