4 ms·

Conceptually there should be a predictable tradeoff curve between memory size and execution time. This could be quite useful (e.g. if 100hrs is ok, so are 200hr

Conceptually there should be a predictable tradeoff curve between memory size and execution time. This could be quite useful (e.g. if 100hrs is ok, so are 200hrs), after all you don't train such a model every day. But in practice this curve is probably very non-linear in the low end. This writeup shows nicely how lumpy the various steps of loading kernels, model etc https://huggingface.co/docs/transformers/perf_train_gpu_one https://huggingface.co/docs/transformers/perf_train_gpu_one

PaulHoule 3y ago
Memory size mostly depends on the batch size. With a smaller batch size you usually learn more per sample but it takes more time per sample. When I was trying to make a foundation model for clinical notes around 2015 with LSTM I found a batch size of 1 worked best on a CPU but a much larger batch size was ideal in GPU. That is, the CPU would not be faster in terms of samples/second if I increased the batch size but the GPU would get better. What I really cared about was calendar time not sample efficiency, the CPU did best when I optimized for sample efficiency, the GPU sped up enough with large batches that I didn’t mind showing the network more samples.