Once my architect wanted everything on python so I obliged, because of the size of the data I went with spark, but since we were using python I had to interface

Once my architect wanted everything on python so I obliged, because of the size of the data I went with spark, but since we were using python I had to interface with it using pyspark, which still uses python 2, after a few months we start having all this problems with the py4j bridge in spark collapsing and us having to restart the service every few hours. Fed up with the situation I rewrote the thing in Java/Scala and we haven’t touched the service ever since. My point here is that using the right tool for the task can mitigate a myriad of issues, all the python knowledge in the world was not good when working with a java tool. What is needed is balance, not every language is good for everything, not every framework can do it all and certainly not one engineer knows it’s all (either the individual contributor or the manager)