Fascinating, thanks. In all of the large projects I've worked on, the CSV format variants and their inconsistencies have caused disasters. Who knew that Pandas

Fascinating, thanks. In all of the large projects I've worked on, the CSV format variants and their inconsistencies have caused disasters. Who knew that Pandas and Spark handle CSV settings differently by default and spark has a hard time with newlines in its CSV output. CSVs inconsistencies result in a lot of data corruption, doubly so in large teams. Replacing CSV with flat JSON or parquet depending on the use case has been a good move for avoiding these issues. The risks of CSV are usually just too high.