3 ms·
I pumped my fist and involuntarily said "YEEEHEEESSSSSS" quite loudly when I read your first compliment, you are the first comment on that and I really, really,
by tysam_and 4y ago
I pumped my fist and involuntarily said "YEEEHEEESSSSSS" quite loudly when I read your first compliment, you are the first comment on that and I really, really, really, appreciate it. Woop woop!
I am so freaking exhausted from all the commenting here and on Reddit (self-inflicted!), so I'll just give you a gem that has saved my bacon multiple times. You are correct that profiling in this world for GPU stuff is basically absolute crap. There is one exception that I have found (finally) at long last:
with torch.profiler.profile(
schedule=torch.profiler.schedule(wait=10, warmup=15, active=5),
on_trace_ready=torch.profiler.tensorboard_trace_handler('./log/resnet18'),
record_shapes=True
) as profiler:
# training loop
# do a step
profiler.step()
if steps > [number of wait+warmup+active steps]:
break
# oh, and don't forget to set your epoch count to 0 for this so you don't loop over and over again
Then load up tensorboard point at that json. And now to close it out, Steve Jobs style: It's that simple.
Hope that helps. Much love and 'Night! ::) :D :D :D :D :))))