7 ms·
What Every User Should Know About Mixed Precision Training in PyTorch (2022)
- olives 3y agoIs it just me or does the font on the pytorch website look incredibly strange? (I'm on chrome)
- atomlib 3y agoYou mean adrenochrome? Looks normal to me. Edge and Firefox, Windows 11.
- Shrezzing 3y agoAt 100% zoom on Firefox and Edge, the tops & bottoms of lower case letters have some very strange thinning/bolding going on. The top of the `e` char in "Syed Ahmed" is thinner (maybe 1px in height) than the lower curve of the same `e` character (maybe 3px in height). It looks like they have different font-weights for the top and bottom of the characters somehow. Zooming in to 125% the effect goes away, and the font-weight at the top and bottom appear equally thick.
- blitzar 3y agoSame for me at 100%. Something to do with pixel sizes etc making it look like is drawn with roblox.
- keithalewis 3y agoIt is incredibly strange that you are asking the question here instead of asking the pytorch website maintainers.
- hatthew 3y agoYeah I'm getting that on windows, but not mac (both chrome, standard low-ppi displays with no scaling).
- aborsy 3y agoIs there a reason for choosing PyTorch over tensorflow?
- dog436zkj3p7 3y agoYes, many. At this point you should be asking the question whether there's ever a reason to choose tensorflow at all.
- armcat 3y agoOne reason is that overall there are more PyTorch based ML projects out there, which translates to larger exploration space and wider support base. Around the beginning of 2021 PyTorch overtook TensorFlow as the ML framework of choice, see https://trends.google.com/trends/explore?date=today%205-y&q=pytorch,tensorflow&hl=en https://trends.google.com/trends/explore?date=today%205-y&q=...
- complex1314 3y agoIf you add keras to the mix it seems to be the most popular of all https://trends.google.com/trends/explore?date=today%205-y&q=pytorch,tensorflow,Keras&hl=en https://trends.google.com/trends/explore?date=today%205-y&q=...
- blitzar 3y agoI thought tensorflow was dead and PyTorch the new king.
- logicchains 3y agoPyTorch has a very good record of backwards compatibility compared to Tensorflow; your code is much less likely to be broken/deprecated if you use PyTorch.
- imtringued 3y agoPytorch is both faster in terms of performance and ease of development.
- NeutralForest 3y ago
- jerpint 3y agoI wonder if future versions of PyTorch will automatically apply mixed precision if not specified, the article makes it seem like a no brainer to use them by default
- hatthew 3y agoIt can still be complicated by many things. Some parts of a module may not work well in lower precision and need to be in higher precision. If you ever create new tensors within the forward pass, you need to manually adjust your code to automatically use the right datatype. You still need to figure out which low-precision dtype you want to use in your model (and perhaps use different ones in different parts of your model). Etc.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- brutus1213 3y agoIs there any tooling that lets one profile the individual parts of the compute graph? In CUDA land, I can imagine how to build such a thing. No clue how it would work on Mobile.
- hatthew 3y agoIs the pytorch profiler what you're thinking of? https://pytorch.org/tutorials/intermediate/tensorboard_profiler_tutorial.html https://pytorch.org/tutorials/intermediate/tensorboard_profi...