3 ms·
Most input by humans is unicode. Most people do not understand unicode. Don't try to sanitize it other than checking it's valid utf8 - which hopefully your prog
by the8472 7y ago
Most input by humans is unicode. Most people do not understand unicode. Don't try to sanitize it other than checking it's valid utf8 - which hopefully your programming language's string type/http parameter deserializer/DB engine already do for you.
For example some people think that stripping ZWNJ, ZWJ or other kinds of spaces is a thing they ought to do because it confuses their markup parsers or can be used to encode hidden information in posts or stuff like that. Guess what, it breaks emoji, arabic, some asian languages and a bunch of other things.
If you only sanitize your output and realize you made a mistake you can easily fix it by changing your output sanitzing algorithm. If you santizied your input you threw away data and can't fix your problem.
- nine_k 7y agoSee also: wtf-8 encoding, used by Mozilla for potentially broken Unicode input.
- klodolph 7y agoWTF-8 is for a very specific application—to encode in an 8-bit sequence a 16-bit sequence which is nominally UTF-16 but may contain unpaired surrogates. It’s not used for weird user inputs, instead, it’s used for e.g. interop with the Win32 API.