In a sense, it is simple: simple to understand. Not simple to solve. Like to win a race Usain Bolt, it's simple, run faster! Fortunately, Python is well equi

In a sense, it is simple: simple to understand. Not simple to solve. Like to win a race Usain Bolt, it's simple, run faster! Fortunately, Python is well equipped for that. If you open a file with Python that you know it might contains mixed encoded text, you can use try/except to inform the user or open in binary mode, and just store the binary. But my favorite way of doing it is: open('file', error=strategy) Strategy can be: - "ignore": undecodable text is skipped - "replace": undecodable text is replace with "?" - "surrogateescape" (you need to use utf8): undecodable text is decoded to a special representation which makes no human sense, but can en rencoded back to it's original value. It's kinda ironic because people bashed Python for separating bytes/text, forcing them to deal with encoding correctly in Python 3. After all, this problem of "slamming bytes together" comes from languages that treat text as a bytes array, allowing this stupid mistake.