4 ms·
Remind HN: Unicode hacks
Just a friendly reminder that some Unicode characters[1] look like spaces and should be taken into account when writing filtering/trimming functions. Of course it's not a big deal but something to keep in mind to prevent stuff like usernames who are basically a bunch of spaces.
[1] http://www.cs.tut.fi/~jkorpela/chars/spaces.html
- citricsquid 16y ago
- citricsquid 16y agoSeems ALT+0173 works here as a "blank" character. I'm not sure of its exact purpose, but I've never seen it dealt with and often use it as "nothing". The only solution I've seen to properly sanitising Unicode characters is just to disable them entirely and print their name.
- alanh 16y agoIf you break my “typographer’s quotes” by overzealously sanitizing when you don’t absolutely need to, you’ll end up 6' under a † if you know what I mean. ;-)
- antileet 16y agoIt's called a "Soft Hyphen" http://www.cs.tut.fi/~jkorpela/shy.html http://www.cs.tut.fi/~jkorpela/shy.html While it makes sense to make usernames consist of only alphanumeric characters (something people are used to - thanks to email), I don't see why you'd need to "sanitize" Unicode characters.
- someone_here 16y ago┌( ಠ_ಠ)┘ How dare you take my unicode from me.
- deleted 16y ago[deleted]
- olalonde 16y agoSeems like Twitter is "vulnerable" to U+00A0 tweets: http://twitter.com/#!/olivierll/status/7852651047817216 http://twitter.com/#!/olivierll/status/7852651047817216 For those who are wondering, you can type Unicode codes directly from your keyboard (Ubuntu: Ctrl-Shift-u, other OS: http://en.wikipedia.org/wiki/Unicode_input http://en.wikipedia.org/wiki/Unicode_input)
- alanh 16y agoOn OS X this particular character is alt+space.
- Bootvis 16y agoA more innocent trick with unicode and twitter is squeezing extra characters in a tweet by using unicode ligatures: http://en.wikipedia.org/wiki/List_of_precomposed_Latin_characters_in_Unicode#Ligatures http://en.wikipedia.org/wiki/List_of_precomposed_Latin_chara... Unfortunately the amount of ligatures is small but it might come in handy.
- citricsquid 16y agohttp://twitter.com/#!/citricsquid/status/7864128295145472 http://twitter.com/#!/citricsquid/status/7864128295145472 Same with 0173, although mine seems to produce nothing, whereas yours is a line break (I think?)
- tptacek 16y agoThis is a classic web security problem; most famously, WinAPI systems have a "flattening" function that would convert things like PRIME U+2032 into ASCII 0x27 (the tick that terminates SQL statements). Database engines can also interpret character sets differently than the rest of the app stack, leading to similar problems. UTF-7 cursed Wordpress for something like a year in which multiple preauth SQL injection flaws were discovered. The answer to these problems is whitelist filtering and neutralization; if a character isn't known-safe, substitute its HTML entity alternative. If you're writing blacklist filters that need to know what spaces are, you're already playing to lose.
- RyanMcGreal 16y agoI just want to drop a thank-you for your dedication to good security practice and steady generosity with advice - in what is likely an intimidating topic for many developers (at least it is for me).
- perlgeek 16y agoSorry, whitelisting isn't the answer to SQL injection - bind parameters are. With bind parameters you can pass data out of band, and the DB engine never tries to parse it as SQL.
- leif 16y agoYou're right, of course, that you should never use user-input data as code. SQL bind parameters solve this problem for SQL. I'm sure there exist other things without such an easy solution, in which case you do need to sanitize with a whitelist.
- tptacek 16y agoI wasn't trying to be prescriptive about SQL injection. But, it always skeeves me out when people knee-jerk out "parameterized queries" as the answer to SQL injection. Yes, they're better and safer and you should use them wherever you can. But they don't "solve" SQL injection; for instance, there are query fragments that can't be parameterized (which is why you still find SQL injection in sortable table headers and in pagination and in custom query builders). Be careful.
- olalonde 16y agoFor more details on the potential visual spoofs: http://unicode.org/reports/tr36/#visual_spoofing http://unicode.org/reports/tr36/#visual_spoofing
- VMG 16y agoInteresting - just tested it in python and everything is removed with str.strip(), except "\ufeff", which also has zero width. >>> print("\ufeff#") # >>> print(len("\ufeff#".strip())) 2
- stwe 16y agoThere are also other unicode hacks like changing text direction (U+200F).
- stwe 16y agoIt used to have funny effects on websites (browser name in title bar spelled backwards), but it doesn't seem to work now. The above comment contains the unicode character three times.