13 ms·
>PyTorch/Nvidia GPUs easily overtaking TensorFlow/Google TPUs. TF lost to PyTorch, and this is Google’s fault - TF APIs are both insane and badly documented.
by vl 3y ago
>PyTorch/Nvidia GPUs easily overtaking TensorFlow/Google TPUs.
TF lost to PyTorch, and this is Google’s fault - TF APIs are both insane and badly documented.
But nothing comes close to performance of Google’s TPU exaflop mega-clusters. Nvidia is not even in the same ballpark.
- jdlyga 3y agoI can speak from experience on this. Getting started with TensorFlow was very complicated with sparse documentation, so we dropped the idea of using it.
- vl 3y agoI had to use TF when I worked at G, when I left I immediately started to use PyTorch and never looked back.
- belval 3y agoThere is a first mover handicap there though. TF1.0 included a bunch of things that were harder to understand like tf.Session(). PyTorch was inspired from the good parts and "we will eager-everything". Internally I'm sure there was a lot of debate in the TF team that culminated with TF2.0, but by that time the damage was done and people saw PyTorch as easier.
- bitL 3y agoI think the main problem was debugging tensors on the fly, impossible with TF/Keras, but completely natural to PyTorch. Most researchers needed to sequentially observe what is going on in tensors (histograms etc.) and even doing backprop for their newly constructed layers by hand and that was difficult with TF.
- mirker 3y agoNah, TF has had dynamic execution since TF2 and it’s still losing users, it seems. The execution model and API is simply more complicated. What’s a session, placeholder, constant, tensor, …? PyTorch was sold as numpy with GPU support and it is pretty close to that. JAX is an attempt to approach language simplicity and purity.
- disgruntledphd2 3y agoNope, Pytorch was inspired by the Lua version of Torch which well predates Tensor flow. To be fair, basically every other DL framework made the same mistake though. Also, tensorflow was a total nightmare to install while Pytorch was pretty straightforward, which definitely shouldn't be discounted.
- andyjohnson0 3y ago> Also, tensorflow was a total nightmare to install while Pytorch was pretty straightforward, which definitely shouldn't be discounted. I think this is a very important point, and I remember sweating blood trying to build a standalone tf environment (admittedly on windows) in the past. I'm impressed by how much simpler and smoother the process has recently become. I do prefer Keras to Pytorch though - but thats just me
- throwaway2037 3y agotensorflow was a total nightmare to install while Pytorch was pretty straightforward Hat tip for this comment. On HN, I read some great commentary about "time to achieve first HTTP 200 with your REST API". Regarding installed software libraries, lower friction to achieve "Hello, World!" is important.
- gillesjacobs 3y agoI have used both but ended up dropping TF for PyTorch after 2018. Mainly it was the larger PyTorch ecosystem in my field (NLP) and clear API design and documentation that did it for me. However, TF was still a valid contender and it was not clearcut back in 2016-17 which framework was better.
- joseph_grobbles 3y ago[dead]
- tdullien 3y agoTF in its first version was stellarly misdesigned. It was infuriatingly difficult to use, particularly if you were of the "I just want to write code and have it autodiffed + SGDed" school, I found it crazy to use Python to manually construct a computational graph...
- metadat 3y agoWhat's the meaning of SDG in this context? Edit: Hypothesis: Stochastic Gradient Descent
- ragebol 3y agoParent typed SGD, which means Stochastic Gradient Descent. An optimization method.
- dekhn 3y agoYou need something to construct a graph. Why not pick a well-known language already used in scientific computing and stats/data science? The other options are: pick a lesser known language (lua, julia) or a language not traditionally used for scientific computing (php, ruby), or a compiled language most researchers don't know (C++), or a raw config file format (which you would then use code to generate). What's really crazy is using Pure, Idiomatic Python which is then Traced to generate a graph (what Jax does). I want my model definitions to be declarative, not implict in the code.
- whymauri 3y agoPython is the least of my concerns with Tensorflow... especially TF 1.0. What a mess it was, and kinda still is.
- 1024core 3y agoThere's a reason why the TL behind TF (Rajat Monga) got booted out.
- piecerough 3y agoWhat's this based on?
- nl 3y agoThe post says the same thing: > Unfortunately, this early lead would be completely squandered within a few short years, with PyTorch/Nvidia GPUs easily overtaking TensorFlow/Google TPUs. ML was, and frankly is, still too nascent to have significant technical barriers to entry. The sustained eye-popping funding for AI companies generated a surge in supply, with the number of ML researchers growing ~25% YoY for the past decade. I taught myself enough ML to blend in with the researchers at Brain over a relatively short 2 years, and so have many others. Nobody, not even Google, can afford to throw money into a bottomless pit.
- sytelus 3y agoNVidia A100 DGX Superpod is equivalent to exaflop TPU pod. No?
- vl 3y agoPerhaps NVidia is close now. A bit hard to say without specific hardware info. Google’s were already available 5-6 years ago. And probably current versions are even faster. They have super fast optical interconnects in torus or hyper-torus configuration that allow synchronous weight updates on 1k+ TPUs. This leads to dramatically lower training times and less noise, which leads to better-performing models. I.e. you can’t even train model to the same level on traditional GPUs. Once they started to get deployed, models that trained for 3 weeks on 30 GPUs were trained in 30 minutes on 1k TPU cluster. All this reiterated main point in the article - Google had tremendous lead and wasted it due to the lack of vision and product execution ability.
- rerx 3y agoGPU cluster scaling has come a long way. Just check out the scaling plot here: https://github.com/NVIDIA/Megatron-LM https://github.com/NVIDIA/Megatron-LM
- why_only_15 3y agoYou are not correct about TPUs being drastically better for GPUs at this. If you look at public benchmarks, both have a similar cost per hardware flop ($0.88/hr for 312tflops A100 on GCP, $0.97/hr for 275tflops TPUv4) and both achieve similar model flop:hardware flop ratios (40-60%). An existence proof that GPU mega-clusters are possible is that GPT-4 cost ~$100m over ~3 months, so ~100m a100-hours / (3 months * 30 days/month * 24 hours/day = 2160 hours) = ~45k a100s collaborating, which is the equivalent of ~10 TPUv4 pods on a single training run.
- vl 3y agoI assume/suspect internally Google has v5 already. The thing about TPU clusters that they have hyper-torus optical interconnect between TPUs. This allows for extremely efficient weight updates. To replicate this with A100s you need very custom hardware/software deployment. But to be fair, I don’t know what is latest and greatest available from NVidia or other clouds in this area right now. EDIT: Looks like NVidia has NVSwitch, which provides interconnect for 256 GPUs. Pretty cool!
- AaronFriel 3y agoNVidia acquired Mellanox in 2019 in order to own the interconnect - exactly the custom hardware/software stack you thought needed.
- rerx 3y agoExternal NVSwitch (for more than 8 GPUs) isn't available for purchase yet. But even now you typically buy your A100 or H100 training GPUs in servers where each GPU has an individual 400 Gbps Mellanox networking adapter, then connect them in a sophisticated switched network that provides full throughput between all GPUs of the cluster. The deployment is not that custom, typically you would let Nvidia and your vendors implement their reference design for you: https://www.nvidia.com/en-us/data-center/dgx-superpod/ https://www.nvidia.com/en-us/data-center/dgx-superpod/ Then run an open-source software stack on top of CUDA etc. I don't know how well the TPU hyper-torus interconnect performs, but the networking topology seems to be less general than switched NVLink or InfiniBand.