4 ms·
How large is the set of binaries needed to do this training job? The current pytorch + CUDA ecosystem is so incredibly gigantic and manipulating those container
by localhost 2y ago
How large is the set of binaries needed to do this training job? The current pytorch + CUDA ecosystem is so incredibly gigantic and manipulating those container images is painful because they are so large. I was hopeful that this would be the beginnings of a much smaller training/fine-tuning stack?
- karpathy 2y agoThat is 100% my intention and hope and I think we are very close to deleting all of that. Right now on master, I am already only using Python for the tokenization preprocessing. In principle the requirements for llm.c should be extremely minimal. I think this a few days of work that is high on my mind. Biggest problem right now is finding a place that can host the 135GB of tokens for FineWeb100B. Will probably use S3 or something. Related see: https://github.com/karpathy/llm.c/issues/482 https://github.com/karpathy/llm.c/issues/482
- metadat 2y agoCould this be a good case for a torrent?