4 ms·
Very sad, shouldve used an agnostic framework instead of CUDA
by brcmthrowaway 3y ago
Very sad, shouldve used an agnostic framework instead of CUDA
- jsheard 3y agoIt's only ~1000 LoC, seems like a pretty good case study to port over to other runtimes and show they can stand up to CUDA.
- geph2021 3y agoAs far as I can tell, its optional dependency is Open MP, not CUDA. Doesn't seem directly dependent on CUDA.
- gpderetta 3y agoYes, a quick skim of the code only shows openmp dependency. The C/CUDA reference might have meant to be C/OMP . Although I wonder if it would work well with GCC PTX OMP offloading.
- dnlzro 3y agoThe plan is to eventually implement with CUDA: "Currently, I am working on [...] direct CUDA implementation, which will be significantly faster and probably come close to PyTorch."
- robrenaud 3y agoAre there any strong LLMs trained without CUDA?
- blackeyeblitzar 3y agoYes, there are several. See this blog post from Databricks describing the landscape of LLMs trained on AMD hardware for example: https://www.databricks.com/blog/training-llms-scale-amd-mi250-gpus https://www.databricks.com/blog/training-llms-scale-amd-mi25... The most interesting one IMO is OLMo from AI2, which is truly open. You can read their blog post about it (https://blog.allenai.org/hello-olmo-a-truly-open-llm-43f7e7359222 https://blog.allenai.org/hello-olmo-a-truly-open-llm-43f7e73...) but basically it is open everything - they released everything you need to reproduce their weights (training data, training code, evaluation code, and weights) with a friendly (Apache) license.
- ZoomerCretin 3y agoGemini and Gemma were trained on Google's TPUs.
- deleted 3y ago[deleted]
- exe34 3y agoLooking forward to your patches!