Use of those tools (along with MLlib and virtually anything relying on a py4j bridge) was precisely the setup I tested and found to have unacceptably poor perfo

Use of those tools (along with MLlib and virtually anything relying on a py4j bridge) was precisely the setup I tested and found to have unacceptably poor performance when taken across our range of both large and small workloads, in addition to many problems with deep inflexibility in controlling the runtime environment on a per-project basis (our most critical requirement). See my other comment below with a link to a previous discussion.