At a high level, this looks similar Spark barrier mode for Tensorflow / Horovod. Except this system relies on etcd, which k8s folk know has some limitations an

At a high level, this looks similar Spark barrier mode for Tensorflow / Horovod. Except this system relies on etcd, which k8s folk know has some limitations and admin costs... For a modeling-focused project, one will still probably do better with a multi-gpu machine versus elastic complexity. https://medium.com/plumbersofdatascience/whats-new-in-spark-2-4-121162f1c385 https://medium.com/plumbersofdatascience/whats-new-in-spark-...