3 ms·
Because new CUDA versions often come with new bugs, so as a researcher with a publication deadline, you're incentivised to never upgrade. I still vividly rememb
by fxtentacle 3y ago
Because new CUDA versions often come with new bugs, so as a researcher with a publication deadline, you're incentivised to never upgrade. I still vividly remember how messy it was to downgrade a large TensorFlow project to a previous version which still supported an older CUDA because the current CUDA cuBLAS couldn't do matrix multiplication without silent memory corruption. (I believe it was 9.x to 10.0 that broke it) And we lacked the connections to directly talk to any engineer at NVIDIA, which meant that nobody even confirmed the bugs existence after we submitted a GitHub issue. Eventually, we found a way to trigger a crash on Google Colab using the bug and then someone from Google filed it again with NVIDIA and then things finally got fixed. About 6 months after the research paper was finished.