It sometimes takes time to discover the best approach to writing code that handles similar inputs. I once worked on an ingestion pipeline that was supposed to c

It sometimes takes time to discover the best approach to writing code that handles similar inputs. I once worked on an ingestion pipeline that was supposed to clean up data sent to us by a person whose sole job was to sit in front of a Windows PC running three pieces of software: a) a terminal running a text mode app extracting data from a mainframe b) Excel c) Outlook Their sole job was to copy data from the terminal, paste it into Excel on a daily basis, save it as a CSV file, and send it as an attachment to the address monitored by a script that was responsible to running data processing pipeline. Because of the manual nature of the job and the way data was presented in the terminal there were errors, which were really unpredictable. I was not allowed to talk to the person doing this job or visit them in their office so it took me three months to find out that what was shown on the terminal screen was essentially a 80x25 version of a punchcard and that position of the fields mattered. Sometimes the user would not copy the whole screen, sometimes there would be an extra character added while pasting data into Excel (not always a stray "v"), sometimes a gremlin character would be added between the mainframe and the terminal (there must have been a serial connection somewhere). Forget proper encoding, JSON or XML, it was raw data, really raw. When I started working on the pipeline it would break on every incorrectly formatted record, when I finished the job and left it was only barfing on 5% bad records and those that it could not process would be neatly put aside an emailed to the support people responsible for dealing with this client. There was a lot of repetition initially, but I then discovered patterns (random offsets, encoding errors, extra characters) and that allowed me to build a set of generic classes to quickly implement problem-specific handlers.