Our experience was as a Python shop who was backed into a corner to use Apache Pig for our Hadoop batch jobs. We decided to rewrite some of those jobs from Pig

Our experience was as a Python shop who was backed into a corner to use Apache Pig for our Hadoop batch jobs. We decided to rewrite some of those jobs from Pig to PySpark, and though there was a little bit of a learning curve and some sharp edges, the development experience is so much better than Pig that my team is generally happy with the switch.