We had to come up with some tasks for interns to attempt over three week period. We had some spaghetti data preprocessing written in java and parameterized by

We had to come up with some tasks for interns to attempt over three week period. We had some spaghetti data preprocessing written in java and parameterized by hand modified xml, orchestrated on a map reduce job. Every year the job needed to be extended to handle some new fields and it was an inevitable clusterfuck that required a full month to figure out. I personally stayed well away, it was clearly somoene's vacation work. But it was suggested that these turnaround times were not good, the individual jobs took hours to run and nobody was setup to debug locally. I estimated that given the data size (100k rows LOL) joined to dozens of rows (LOL!) should take no more than few minutes in python (both to write the code and run it (POWER OF PYTHON!) and there wasn't a really good reason why we should need hundred nodes and complicated data orchestration. It was obviously decided to rewrite into spark jobs because this was "BIG DATA" and we needed scale (100K rows LOL never going to increase). Somehow the rewrite never finished because the jobs were running out of memory (8GB LOL!). At the end I asked why didn't they just write the join in python? I was told it wouldn't scale and that this was BIG DATA. I personally have come to the conclusion that vast segments of IT are working on kickbacks and nepotism and maybe 9/10 of workers should be fired, but thats just the shitty places I've worked so probably that philosophy doesn't SCALE!!!!!