3 ms·
what are your thoughts on https://github.com/tensorflow/runtime https://github.com/tensorflow/runtime ?
by The_rationalist 5y ago
what are your thoughts on https://github.com/tensorflow/runtime https://github.com/tensorflow/runtime ?
- erwincoumans 5y agoFor Academic Papers (the context of this HN topic), JAX and PyTorch makes more sense. A new runtime could be useful in production.
- dekhn 5y agotfrt underlies JAX. In many ways it's the system TF was supposed to have a 5 years ago.
- erwincoumans 5y agoYes indeed: https://blog.tensorflow.org/2022/02/tfrt-progress-update.html?m=1 https://blog.tensorflow.org/2022/02/tfrt-progress-update.htm... The ML Pathways infra is more complicated, and about more efficient scheduling many ops (in a graph) across many accelerators (rather than a small model on a single local GPU)
- dekhn 5y agosure, I know :) I used to work on TPUs at Google in Platforms, much of my time was spent working with Jax and other teams/researchers who found novel ways to break TPU hardware. What continues to surprise me is that for all his cleverness, Jeff Dean and the rest of the TF leadership spend the last 10 years basically recreating MPI-style high performance computing, but threw away all the learning and rebuilt every bit (except the matrix libraries) from scratch. TF started with parameter servers (every machine has its own copy of weights and periodically contributes them to a common model, asynchronously) to models that are sharded by data input and model structure that is mapped to the TPU topology (TPUv4 is a 3D wrapped torus). Really not that different from the T3E I used in the 90s.