I'm currently processing a huge archive of files where someone was too lazy to bother using the proper encoding. out of the first 700K files I've run through I'

I'm currently processing a huge archive of files where someone was too lazy to bother using the proper encoding. out of the first 700K files I've run through I've dumped 3000 into a secondary queue because they've been so bastardized the mojibake is completely indecipherable. It continues to astonish me that programmers who spend hours arguing the relative merits of ECC RAM and database ACID implementations are so quick to destroy data by lazily defaulting to ASCII.