One issue that was very time consuming for me until recently was that on machines we had mining data upstream across many clusters of vm's, they would sometimes

One issue that was very time consuming for me until recently was that on machines we had mining data upstream across many clusters of vm's, they would sometimes spit out mal formatted files that would need to be fed in downstream for indexing. Each file had about 100k objects, and there would be less 10 objects with errors on avg if they were mal formatted. It was easy (but annoying) to do at first but scaling up the clusters made it exponentially harder to do by hand in the terminal. It was conceptually easy to solve once I abstracted the problem a bit: I had a rough idea of where I was getting the data from on the mining servers (urls, ocr'd pdfs , etc), and since there are a bunch of libraries out there for parsing json that give an idea of where the errors are occurring in the files by the byte, after that it was a combination of traversing file forwards and backwards (in memory, luckily files are only ~20mb each, but crashed every text editor I tried before trying to do search and replace) looking for any data within the file that could help me reconstruct the mal formatted object or remove the object if not enough information was available (which if I didn't want to do if my error occurrence wasn't ~10/100,000, I could use the other objects near it to reconstruct the object from inference).