5 ms·
> I have had them failing showing there was a problem in some text extraction, which boiled down to the fact that lowercasing a Turkish I in python at least res
by chriswarbo 4y ago
> I have had them failing showing there was a problem in some text extraction, which boiled down to the fact that lowercasing a Turkish I in python at least results in more than one character, so positions in the text after it had been lowercased were not the same as before. I had absolutely not considered that.
Oof, this reminds me of an obscure problem caught by a property test (ScalaTest + ScalaCheck) a few years ago: we had to parse files containing fixed-width fields, with optional parts, padded with whitespace, etc. The customer couldn't provide any documentation for the format, so we had to figure it out based on a thousand files captured off their system.
One day a colleague ran a build on their machine, and hit a test failure. It turned out that they were using a different JVM (it was written for JDK11, but they were using JDK14); we were generating strings which include/exclude whitespace based on the 'Char::isWhitespace' method; but our parser combinators were matching whitespace using regular expressions.
This discrepancy caused the test failure, since ScalaCheck had generated a string containing a "mongolian vowel separator". That character was considered as whitespace in older Unicode standards, but not newer ones. The Char::isWhitespace method seemed to take that into account, but the regexp matcher wasn't (e.g. see https://unicode-explorer.com/c/180E https://unicode-explorer.com/c/180E )