3 ms·
Very nice teaching comments, I am learning new stuff that also applies to transformers. I have been wondering for a while now: how do you benchmark GPU code ac
by gillesjacobs 4y ago
Very nice teaching comments, I am learning new stuff that also applies to transformers.
I have been wondering for a while now: how do you benchmark GPU code accurately? Huggingface/transformers used to have a benchmarking library that uses python, pytorch and cuda libraries to benchmark memory usage, inference and epoch times but it has rightly been deprecated for being inaccurate. They recommend using external 3rd party tooling to benchmark GPU processes. Thing is: I can't seem to find good 3rd party benchmarking tools.
- tysam_and 4y agoI pumped my fist and involuntarily said "YEEEHEEESSSSSS" quite loudly when I read your first compliment, you are the first comment on that and I really, really, really, appreciate it. Woop woop! I am so freaking exhausted from all the commenting here and on Reddit (self-inflicted!), so I'll just give you a gem that has saved my bacon multiple times. You are correct that profiling in this world for GPU stuff is basically absolute crap. There is one exception that I have found (finally) at long last: with torch.profiler.profile( schedule=torch.profiler.schedule(wait=10, warmup=15, active=5), on_trace_ready=torch.profiler.tensorboard_trace_handler('./log/resnet18'), record_shapes=True ) as profiler: # training loop # do a step profiler.step() if steps > [number of wait+warmup+active steps]: break # oh, and don't forget to set your epoch count to 0 for this so you don't loop over and over again Then load up tensorboard point at that json. And now to close it out, Steve Jobs style: It's that simple. Hope that helps. Much love and 'Night! ::) :D :D :D :D :))))
- KeplerBoy 4y agoThe best way to benchmark CUDA code is probably using nvtx annotations and nsight systems.