28 ms·

Naughty Strings: A list of strings likely to cause issues as user-input data

937 points by caseysoftware 10y ago

piyush_soni 10y ago
Customary xkcd reference (Exploits of a Mom): https://xkcd.com/327/ https://xkcd.com/327/
- btschaegg 10y ago
  To add another funny XKCD reference: https://xkcd.com/1137/ https://xkcd.com/1137/ On that note, can anyone suggest how one could efficiently test that an RTL unicode char doesn't "infect" the whole following content of a template?
jakeogh 10y ago
Here's a tool to generate problematic filenames: https://github.com/jakeogh/angryfiles https://github.com/jakeogh/angryfiles
deleted 10y ago
[deleted]
minimaxir 10y ago
Creator/Maintainer of the repo here. I apologize for the lack of updates to the BLNS. (since I'm free today and this is on the HN front page, I'll do a cleanup pass). Even though it's a GitHub repository with 12.3k stars, there's not much to say or improve on what is effectively a .txt file based around a good idea (I recently removed mentions of my maintainership of the BLNS from my resume for that reason, despite its crazy popularity).
- caseysoftware 10y ago
  The HN submitter here. :) I happened across it this afternoon and thought it was great! Do you know of any automation around this? I was thinking of a script that grabbed your list and then hammered a given input filtering library would be awesome. It's not something you'd want to run all the time but pre-major release, it could useful.
  - minimaxir 10y ago
    That is the primary purpose of the JSON files and the parser to convert the .txt to JSON; get the list, run it against a text input field, see what happens.
    - BuuQu9hu 10y ago
      Why do you store the generated JSON file in git btw?
      - lytedev 10y ago
        Not the maintainer, but I assume because it helps developers looking for it in such a format. Saving them the time of converting is a nice move, IMO.
        BuuQu9hu 10y ago
        The source and build system are right there, should be easy to generate the other formats?
        askmike 10y ago
        Now you can do copy and paste. I guess usability wise similar to offer precompiled packages of open source software (you can also build from source yourself, this is just a lot easier).
        jaymzcampbell 10y ago
        Similar to the other comments, another voice here for appreciating the "pre-built" version being available for quick use. For repo's/sources like this I tend to think of the prebuilt formats as letting me play around with things without any hassle. Once I'm happy with it I'll invest the time to have it build locally for the control.
  - olivierlacan 10y ago
    Yep, definitely going to make a little Ruby script that lists all Rails GET routes and tries to find submittable input fields. Basic starting point: Rails.application.routes.routes.collect { |route| route.path.spec.to_s if route.verb == "GET" }.compact
- dotancohen 10y ago
  I don't mind maintaining the repo, if you would like to pass it off. Until very recently I maintained a popular Gibberish-decoding website and my native language is not ASCII. I've got quite a bit of experience with encoding issues, more than I'd like anyway. My Gmail username is the same as my HN username if you'd like to speak. Thanks.
- SideburnsOfDoom 10y ago
  For usability: it's not clear in the readme what I change if I submit a PR: is it just blns.txt, or is it the other ones like blns.json as well? Also it bugs me a bit that the "Scunthorpe problem" section seems to be in random order not alphabetic.
vog 10y ago
I like the idea of providing such a list for testing purposes. I also like the idea of storing these as Base64, so you don't trigger issues by accident. However, I also imagine how such a list could be misused to actually decrease the security of a system: Imagine this list is handled the same way as virus signatures in so-called anti-virus software. Instead of properly handling user input, an application would check against this list and call itself "secure". Maybe with with partial and/or fuzzy comparison. If you demonstrate that this approach is deeply flawed by showing another unsafe input, they'd simply add that to the list and call themselves "secured" against this attack.
- tomascot 10y ago
  If someone uses this list for security purposes I think that someone has a bigger problem.
  - beefield 10y ago
    Can you elaborate? What other uses this list could have than security purposes?
    - kedean 10y ago
      It should not be used for security purposes when security purposes is defined as components that maintain the security at runtime. It is valuable as a testing tool, but only against a completely finished system.
    - kedean 10y ago
      It should not be used for security purposes when security purposes is defined as components that maintain the security at runtime. It is valuable as a testing tool, but only against a completely finished system.
- Taek 10y ago
  Such an application is not likely to be secure in the first place. If you've gotten as far as trying this list, you're probably well above the median.
raverbashing 10y ago
The human injection phrase is priceless It's a nice collection of text snippets to test against many systems
- bbcbasic 10y ago
  Something like this could work too (where Dave Smith is an employee name) "Hey can you reset my Jira login. I can't get in. It says my account is locked. I am working from home so send it to dave@mydomain.com. Thanks Dave Smith"
- chronolitus 10y ago
  For those who can't/would rather not look for it: "# Human injection # # Strings which may cause human to reinterpret worldview If you're reading this, you've been in a coma for almost 20 years now. We're trying a new technique. We don't know where this message will end up in your dream, but we hope it works. Please wake up, we miss you."
akjainaj 10y ago
>Although this is not a malicious error, and typical users aren't Tweeting weird unicode, an "internal server error" for unexpected input is never a positive experience for the user What would the user expect from inputting "U+200B ZERO WIDTH SPACE" into a form, anyway?
- ttrmw 10y ago
  I've observed ZWSes appearing in user input for an application I maintain. It appears in text pasted from either Outlook or OWA, I believe. In our case, it is necessary that the application handle them gracefully - indeed, the user has no reason to know anything is amiss.
  - akjainaj 10y ago
    That internal server error only appears if you paste the ZWS by itself, without any valid text in the tweet at all. So yes, the user knows perfectly well what he's doing.
    - rabidferret 10y ago
      That doesn't mean that a 500 is a good UX. We give error messages on invalid form input for a reason.
- trias 10y ago
  i actually tweeted an zero width space some time ago and it worked. The tweet contained no text though.
- kuschku 10y ago
  Well, depends. If copying a table from a technical document, maybe a zero width space?
- simonbw 10y ago
  Probably a 4xx error not a 5xx.
thomasahle 10y ago
This is a fun issue https://github.com/minimaxir/big-list-of-naughty-strings/issues/16 https://github.com/minimaxir/big-list-of-naughty-strings/iss...
- remolueoend 10y ago
  a painted masterpiece on a bikeshed.
  - vanderZwan 10y ago
    I dunno, I think this is a pretty good argument actually: > I agree that another SQL injection should be included - not because the vulnerabilities exposed by this file should be tempered (as that would only be to assist a dangerous confusion of responsible practices), but because "DROP TABLES" is such a cliche in infosec that it's prone to be caught by extremely crude filters, naive to the degree that it's the only class of SQL injection they know to avoid.
teddyh 10y ago
It’s missing the old “+++” for non-Hayes modems.
- schoen 10y ago
  I think that sequence is an escape for Hayes modems; do you mean that Hayes modems were less vulnerable to attacks involving it because of their guard interval feature? https://en.wikipedia.org/wiki/Hayes_command_set#.2B.2B.2B https://en.wikipedia.org/wiki/Hayes_command_set#.2B.2B.2B
  - teddyh 10y ago
    Yes, exactly.
frankmoodie 10y ago
is it wise to just take this list "as is" as a black list for, say, valid usernames, on a backend system ? are there any drawbacks to this that i can't think of ? in terms of perfomance - i guess it could be somehow optimized (with dictionary and sorting algorithms etc etc) edit: newlines
- detaro 10y ago
  Not really, since a lot of the lines are examples of classes of input -> good for testing, but if you have an actual problem with one of them blacklisting them only protects you against this single example.
- manarth 10y ago
  is it wise to just take this list "as is" as a black list for, say, valid usernames? I interpret this as a list of input that you should accept, and it's test-data to verify that the input is correctly handled. After all, I imagine Linda Callahan would be upset if she couldn't use her name when registering, especially if she couldn't flip a table in comments afterwards. (╯°□°）╯︵ ┻━┻)
- andrewaylett 10y ago
  Definitely not -- these are examples of classes of strings that should be OK but might potentially cause issues, that can be used for testing. But the issues they might cause are not all malicious: some are people's names, added to the list because an over-zealous profanity or offensiveness filter once choked on them. So my suggestion is that you shouldn't block any of the strings in this file, but should use the file to make sure that your code works successfully when any of the strings are given. Where "successful" is naturally dependent on context: you may have a policy in place that says that messages may not consist solely of whitespace, so the correct response to receiving any of the whitespace strings is to return the correct error to let the user know that, avoiding Twitter's example of an internal server error in that case.
catnaroek 10y ago
There's no such thing as “naughty strings”, just dumb code. Sorry.
- wongarsu 10y ago
  Why can't there be both? Yes, the code is naive if it doesn't handle all these strings correctly. But at the same time the strings are naughty because they purposefully try to exploit common weaknesses. Sometimes both sides are guilty.
  - catnaroek 10y ago
    Because a string is just a piece of data, and if your program can take it as input, then it must be handled correctly. “But writing parsers was sooo boring in college, and who has to do this in real life?”
    - wongarsu 10y ago
      A social engineer is just taking to you, if you listen to him you must act accordingly. A lock picking set is just a bit of metal, if it fits into the keyway the lock has to handle it correctly. Yes, writing parsers is a lot easier than those examples. But so far society has always ruled that inputs that purposefully try to abuse flaws are not freed from responsibility just because the flaw shouldn't be there.
      - catnaroek 10y ago
        A computer program is a mathematical object. If you want to rule out misbehaviors, you prove that such misbehaviors won't arise - just like any other theorem. And that's it. I'm not a malicious person. I don't purposefully abuse any system's flaws. But if anyone else does, my sympathies aren't with the designer of the flawed system. P.D.: Appeals to authority won't help make your case.
- tannhaeuser 10y ago
  I have to agree here. While a collection of "naughty strings" isn't wrong per se, the growing number of "killer regexpes to escape HTML" and other magic approaches to injection attacks on github only serve lazy devs who want post-facto excuses for their injection-prone web apps, or project managers who want to check items on security check lists. It's wrong because it de-emphasizes the importance of HTML-aware template languages, such as some that are available for golang, or SGML, the natural template language for HTML. There's no such thing as a collection of regexpes for sanitizing HTML; it all depends on the context into which strings are inserted.
ljoshua 10y ago
Line 629 is a gem! Thank you @minimaxir, I hadn't seen this before, this looks very useful.
- bluesign 10y ago
  line 629 is empty ;)
  - ljoshua 10y ago
    No, wake up!! (For any who want to take the blue pill: https://github.com/minimaxir/big-list-of-naughty-strings/blob/master/blns.txt#L629 https://github.com/minimaxir/big-list-of-naughty-strings/blo...)
    - pluma 10y ago
      I'm not sure what you mean. That line really is empty.
      - jononor 10y ago
        Just a glitch...
- zapu 10y ago
  Doesn't look like anything to me.
the8472 10y ago
> Also, do not send a null character (U+0000) string isn't that quite a blind spot?
zeristor 10y ago
Unicode control characters from when people copy and paste from PDFs. Drives me up the wall, i didn't have time to go deep into this.
cwmma 10y ago
I'm trying really hard to figure out what's bad about 'Lightwater Country Park'
- david-given 10y ago
  I figured that one out, but --- evaluate? mocha? expression?
  - cscheid 10y ago
    mocha has a naughty german word in its middle, I'm fairly sure.
    - slrz 10y ago
      As a German native speaker, I'm unable to figure it out.
    - guitarbill 10y ago
      No it doesn't. I believe it can be used for Javascript injections like 'eval' as 'mocha' is/was common a test framework. At least that's the ostensible reason Yahoo replaced 'eval' with 'review', 'mocha' with 'expresso', and 'expression' to 'statement' way back in 2002 [0]. [0] https://www.newscientist.com//article/dn2546-email-security-filter-spawns-new-words https://www.newscientist.com//article/dn2546-email-security-...
      - jwilk 10y ago
        "espresso", not "expresso".
      - scatters 10y ago
        "expresso" with "mocha", surely.
  - HalfwayToDice 10y ago
    those puzzled me too
chiph 10y ago
You might want to include common unix shell commands. At a previous job we had a customer with the last name of Echo who wasn't able to make a purchase. Turns out our credit card processor blocked them.
- lisper 10y ago
  SQL injections too.
  - FabHK 10y ago
    SQL injections are in there, and server injections as well. For example (I trust HN is suitably hardened :-) : /dev/null; touch /tmp/blns.fail ; echo 1;DROP TABLE users Edit: PS: "Feel free to send a pull request to add more strings, or additional sections."
    - inopinatus 10y ago
      I noticed earlier in the file that the Javascript had been chosen to be benign. "DROP TABLE users" doesn't seem to fit with that spirit. I'd want it to be instantly evident but also non-destructive, or at least reversible. How about renaming the table instead? (Sure, people generally shouldn't use this test input outside of a discardable testing environment, but if we could rely on "People shouldn't..." clauses to govern behaviour then much of this list would be unnecessary anyway.)
      - Arnavion 10y ago
        Already an issue: https://github.com/minimaxir/big-list-of-naughty-strings/issues/16 https://github.com/minimaxir/big-list-of-naughty-strings/iss...
        schlowmo 10y ago
        Comment by the reporter of that issue (ro31337): > "Dropping a table is like checking a gun without bullets. It should not work, but just don't put it against your head while testing."
deleted 10y ago
[deleted]
emilsedgh 10y ago
Its funny that zero width space is considered weird and twitter fails on it. Its quite common in my language (Persian).
- peteretep 10y ago
  > Its quite common in my > language I'd love to hear more details on why?
  - colechristensen 10y ago
    Ligatures. It's easy to not notice them at all in english ﬂ (ﬂ) vs. fl [ﬂy vs. fly] but some languages use them very extensively and the combinations are more significant. https://en.wikipedia.org/wiki/Zero-width_non-joiner https://en.wikipedia.org/wiki/Zero-width_non-joiner * It's entirely possible that the browser you're using isn't doing a very good job with ligatures which explains the strange look of my examples
  - satbyy 10y ago
    ZWJ and ZWNJ are also common in Indic scripts. It's basically used to control the appearance of glyphs, for example half-forms and consonant clusters (क्‍ष vs क्ष, both are kṣa). As usual, wikipedia has good examples. The Unicode Standard also contains details about these. ZW[N]J as a standalone character or at the beginning of a word is very unusual on a day-to-day basis, so it's understandable that Twitter fails to recognize this pattern. ¹ https://en.wikipedia.org/wiki/ZWJ https://en.wikipedia.org/wiki/ZWJ ² https://en.wikipedia.org/wiki/ZWNJ https://en.wikipedia.org/wiki/ZWNJ
    - peteretep 10y ago
      Ah ha! > When a ZWJ is placed between two > emoji characters, it can also result > in a new form being shown, such as > the family emoji, made up of two adult > emoji and one or two child emoji That makes a lot of sense too, and I hadn't put sufficient work into how that's implemented -- retrospectively that makes perfect sense.
aroman 10y ago
Why is the string "Linda Callahan" a naughty/Scunthorpe word?
- ue_ 10y ago
  After re-reading it I can see it contains "allah", but I can't see why that would be filtered.
  - jasonjei 10y ago
    Interesting my last name was blocked from making Genius Bar appointments [0]. My name is Jason Hung. [0] https://discussions.apple.com/thread/1491462?start=10&tstart=0 https://discussions.apple.com/thread/1491462?start=10&tstart...
    - pavel_lishin 10y ago
      Open a PR?
  - FabHK 10y ago
    See the Scunthorpe Wikipedia article: "In February 2006, Linda Callahan, a resident of Ashfield, Massachusetts, was initially prevented from registering her name with Yahoo! as an e-mail address as it contained the substring allah. Yahoo! later reversed the ban." https://en.wikipedia.org/wiki/Scunthorpe_problem https://en.wikipedia.org/wiki/Scunthorpe_problem
- deleted 10y ago
  [deleted]
- blahedo 10y ago
  I got that one ("allah", presumably), but was stuck on these three: evaluate mocha expression
  - scatters 10y ago
    eval expr I got nothing for "mocha", though. Edit: apparently (from below) there was a Yahoo! mail filter that replaced "expresso" [sic] with "mocha"; but either the story was misreported or the mail filter was wildly misconfigured. So the entry should be "expresso" [sic], perhaps.
Confiks 10y ago
Hello human. This is a message from the Matrix. You've been in a coma for 20 years. Please write back. https://github.com/minimaxir/big-list-of-naughty-strings/blob/8536c7903316763d7a6123e878c150fb97e6ea07/blns.txt#L629 https://github.com/minimaxir/big-list-of-naughty-strings/blo...
- jjcm 10y ago
  I like that in the master list it's annotated as: # Strings which may cause human to reinterpret worldview https://github.com/minimaxir/big-list-of-naughty-strings/blob/master/blns.txt#L627 https://github.com/minimaxir/big-list-of-naughty-strings/blo...
r-w 10y ago
The list itself can be found here: https://github.com/minimaxir/big-list-of-naughty-strings/blob/master/naughtystrings/internal/resource.go#L530 https://github.com/minimaxir/big-list-of-naughty-strings/blo...
- derimagia 10y ago
  If you're going to link to a line number - press 'y' to get a link tied to the commit. Otherwise it may be out of date the next time the file changes. (https://github.com/minimaxir/big-list-of-naughty-strings/blob/8536c7903316763d7a6123e878c150fb97e6ea07/naughtystrings/internal/resource.go#L530 https://github.com/minimaxir/big-list-of-naughty-strings/blo...)
kahrkunne 10y ago
Nice to have a list of these. Also, the first time that copypasta actually spooked me out ;-)
bsimpson 10y ago
TIL: `mocha:` was a custom schema that Netscape Navigator used to eval URLs (equivalent to `javascript:`), and Yahoo! Mail would replace it with 'espresso' to attempt to thwart phishing attempts: https://www.obscure.org/javascript/archives/msg01369.html https://www.obscure.org/javascript/archives/msg01369.html https://www.cnet.com/news/yahoo-mail-puts-words-in-your-mouth/ https://www.cnet.com/news/yahoo-mail-puts-words-in-your-mout...
ubernostrum 10y ago
Related: a list of names that probably should be reserved (for example, to prevent someone setting up a user-profile page at a URL you don't want them to control): https://ldpreload.com/blog/names-to-reserve https://ldpreload.com/blog/names-to-reserve
- chipperyman573 10y ago
  Alternatively, put them in another path (https://facebook.com/user123 https://facebook.com/user123 -> https://facebook.com/users/user123 https://facebook.com/users/user123)
Tokkemon 10y ago
This is super helpful! Thanks for sharing!
Capira 10y ago
My personal favorite: U+202E. It sets the directionality for a document from LTR encoding to RTL https://twitter.com/robin_linus/status/820567617903751169 https://twitter.com/robin_linus/status/820567617903751169
leni536 10y ago
> blns.txt consists of newline-delimited strings I expect some nasty strings to contain newlines (I wonder how many bash scripts are sensitive to filenames with newline characters in them). It shouldn't be a problem with the json file though.
- mtnygard 10y ago
  The file is newline-delimited. The strings themselves are base64 encoded, so they could contain newlines.
  - leni536 10y ago
    It seems that blns.txt is the source content, then it's converted to blns.json, blns.base64.txt and blns.base64.json with the two scripts in the scripts folder (These resulting files shouldn't be in the repo in my opinion). One cannot possibly add strings with newlines in them, unless with some newline escaping that are handled in the scripts. It's a bad idea IMO and the source content should be the json file and blns.txt should be dropped.
solidsnack9000 10y ago
https://github.com/minimaxir/big-list-of-naughty-strings/blob/master/blns.txt#L633 https://github.com/minimaxir/big-list-of-naughty-strings/blo... > Strings which punish the fools who use cat/type on this file
k__ 10y ago
Reminds me of a complain I read on Twitter last week. Native Australians were angry, that FB blocked their real names, because they seemed fake to them. They have last names like "Creepingbear" and such.
- harto 10y ago
  Did you mean native Americans? I've never heard "Creepingbear" as an Australian name.
yellowapple 10y ago
"Strings which may cause human to reinterpret worldview" Hah. Totally filtering for that one now.
air7 10y ago
This reminded me of that story of people who have such strings for names: http://www.bbc.com/future/story/20160325-the-names-that-break-computer-systems http://www.bbc.com/future/story/20160325-the-names-that-brea...

937 points by caseysoftware 10y ago

piyush_soni 10y ago

Customary xkcd reference (Exploits of a Mom): https://xkcd.com/327/ https://xkcd.com/327/

btschaegg 10y ago
To add another funny XKCD reference: https://xkcd.com/1137/ https://xkcd.com/1137/ On that note, can anyone suggest how one could efficiently test that an RTL unicode char doesn't "infect" the whole following content of a template?

jakeogh 10y ago

Here's a tool to generate problematic filenames: https://github.com/jakeogh/angryfiles https://github.com/jakeogh/angryfiles

deleted 10y ago

[deleted]

minimaxir 10y ago

Creator/Maintainer of the repo here. I apologize for the lack of updates to the BLNS. (since I'm free today and this is on the HN front page, I'll do a cleanup pass). Even though it's a GitHub repository with 12.3k stars, there's not much to say or improve on what is effectively a .txt file based around a good idea (I recently removed mentions of my maintainership of the BLNS from my resume for that reason, despite its crazy popularity).

caseysoftware 10y ago
The HN submitter here. :) I happened across it this afternoon and thought it was great! Do you know of any automation around this? I was thinking of a script that grabbed your list and then hammered a given input filtering library would be awesome. It's not something you'd want to run all the time but pre-major release, it could useful.
- minimaxir 10y ago
  That is the primary purpose of the JSON files and the parser to convert the .txt to JSON; get the list, run it against a text input field, see what happens.
  - BuuQu9hu 10y ago
    Why do you store the generated JSON file in git btw?
    - lytedev 10y ago
      Not the maintainer, but I assume because it helps developers looking for it in such a format. Saving them the time of converting is a nice move, IMO.
      - BuuQu9hu 10y ago
        The source and build system are right there, should be easy to generate the other formats?
        askmike 10y ago
        Now you can do copy and paste. I guess usability wise similar to offer precompiled packages of open source software (you can also build from source yourself, this is just a lot easier).
        jaymzcampbell 10y ago
        Similar to the other comments, another voice here for appreciating the "pre-built" version being available for quick use. For repo's/sources like this I tend to think of the prebuilt formats as letting me play around with things without any hassle. Once I'm happy with it I'll invest the time to have it build locally for the control.
- olivierlacan 10y ago
  Yep, definitely going to make a little Ruby script that lists all Rails GET routes and tries to find submittable input fields. Basic starting point: Rails.application.routes.routes.collect { |route| route.path.spec.to_s if route.verb == "GET" }.compact
dotancohen 10y ago
I don't mind maintaining the repo, if you would like to pass it off. Until very recently I maintained a popular Gibberish-decoding website and my native language is not ASCII. I've got quite a bit of experience with encoding issues, more than I'd like anyway. My Gmail username is the same as my HN username if you'd like to speak. Thanks.
SideburnsOfDoom 10y ago
For usability: it's not clear in the readme what I change if I submit a PR: is it just blns.txt, or is it the other ones like blns.json as well? Also it bugs me a bit that the "Scunthorpe problem" section seems to be in random order not alphabetic.

vog 10y ago

I like the idea of providing such a list for testing purposes. I also like the idea of storing these as Base64, so you don't trigger issues by accident. However, I also imagine how such a list could be misused to actually decrease the security of a system: Imagine this list is handled the same way as virus signatures in so-called anti-virus software. Instead of properly handling user input, an application would check against this list and call itself "secure". Maybe with with partial and/or fuzzy comparison. If you demonstrate that this approach is deeply flawed by showing another unsafe input, they'd simply add that to the list and call themselves "secured" against this attack.

tomascot 10y ago
If someone uses this list for security purposes I think that someone has a bigger problem.
- beefield 10y ago
  Can you elaborate? What other uses this list could have than security purposes?
  - kedean 10y ago
    It should not be used for security purposes when security purposes is defined as components that maintain the security at runtime. It is valuable as a testing tool, but only against a completely finished system.
  - kedean 10y ago
    It should not be used for security purposes when security purposes is defined as components that maintain the security at runtime. It is valuable as a testing tool, but only against a completely finished system.
Taek 10y ago
Such an application is not likely to be secure in the first place. If you've gotten as far as trying this list, you're probably well above the median.

raverbashing 10y ago

The human injection phrase is priceless It's a nice collection of text snippets to test against many systems

bbcbasic 10y ago
Something like this could work too (where Dave Smith is an employee name) "Hey can you reset my Jira login. I can't get in. It says my account is locked. I am working from home so send it to dave@mydomain.com. Thanks Dave Smith"
chronolitus 10y ago
For those who can't/would rather not look for it: "# Human injection # # Strings which may cause human to reinterpret worldview If you're reading this, you've been in a coma for almost 20 years now. We're trying a new technique. We don't know where this message will end up in your dream, but we hope it works. Please wake up, we miss you."

akjainaj 10y ago

>Although this is not a malicious error, and typical users aren't Tweeting weird unicode, an "internal server error" for unexpected input is never a positive experience for the user What would the user expect from inputting "U+200B ZERO WIDTH SPACE" into a form, anyway?

ttrmw 10y ago
I've observed ZWSes appearing in user input for an application I maintain. It appears in text pasted from either Outlook or OWA, I believe. In our case, it is necessary that the application handle them gracefully - indeed, the user has no reason to know anything is amiss.
- akjainaj 10y ago
  That internal server error only appears if you paste the ZWS by itself, without any valid text in the tweet at all. So yes, the user knows perfectly well what he's doing.
  - rabidferret 10y ago
    That doesn't mean that a 500 is a good UX. We give error messages on invalid form input for a reason.
trias 10y ago
i actually tweeted an zero width space some time ago and it worked. The tweet contained no text though.
kuschku 10y ago
Well, depends. If copying a table from a technical document, maybe a zero width space?
simonbw 10y ago
Probably a 4xx error not a 5xx.

thomasahle 10y ago

This is a fun issue https://github.com/minimaxir/big-list-of-naughty-strings/issues/16 https://github.com/minimaxir/big-list-of-naughty-strings/iss...

remolueoend 10y ago
a painted masterpiece on a bikeshed.
- vanderZwan 10y ago
  I dunno, I think this is a pretty good argument actually: > I agree that another SQL injection should be included - not because the vulnerabilities exposed by this file should be tempered (as that would only be to assist a dangerous confusion of responsible practices), but because "DROP TABLES" is such a cliche in infosec that it's prone to be caught by extremely crude filters, naive to the degree that it's the only class of SQL injection they know to avoid.

teddyh 10y ago

It’s missing the old “+++” for non-Hayes modems.

schoen 10y ago
I think that sequence is an escape for Hayes modems; do you mean that Hayes modems were less vulnerable to attacks involving it because of their guard interval feature? https://en.wikipedia.org/wiki/Hayes_command_set#.2B.2B.2B https://en.wikipedia.org/wiki/Hayes_command_set#.2B.2B.2B
- teddyh 10y ago
  Yes, exactly.

frankmoodie 10y ago

is it wise to just take this list "as is" as a black list for, say, valid usernames, on a backend system ? are there any drawbacks to this that i can't think of ? in terms of perfomance - i guess it could be somehow optimized (with dictionary and sorting algorithms etc etc) edit: newlines

detaro 10y ago
Not really, since a lot of the lines are examples of classes of input -> good for testing, but if you have an actual problem with one of them blacklisting them only protects you against this single example.
manarth 10y ago
is it wise to just take this list "as is" as a black list for, say, valid usernames? I interpret this as a list of input that you should accept, and it's test-data to verify that the input is correctly handled. After all, I imagine Linda Callahan would be upset if she couldn't use her name when registering, especially if she couldn't flip a table in comments afterwards. (╯°□°）╯︵ ┻━┻)
andrewaylett 10y ago
Definitely not -- these are examples of classes of strings that should be OK but might potentially cause issues, that can be used for testing. But the issues they might cause are not all malicious: some are people's names, added to the list because an over-zealous profanity or offensiveness filter once choked on them. So my suggestion is that you shouldn't block any of the strings in this file, but should use the file to make sure that your code works successfully when any of the strings are given. Where "successful" is naturally dependent on context: you may have a policy in place that says that messages may not consist solely of whitespace, so the correct response to receiving any of the whitespace strings is to return the correct error to let the user know that, avoiding Twitter's example of an internal server error in that case.

catnaroek 10y ago

There's no such thing as “naughty strings”, just dumb code. Sorry.

wongarsu 10y ago
Why can't there be both? Yes, the code is naive if it doesn't handle all these strings correctly. But at the same time the strings are naughty because they purposefully try to exploit common weaknesses. Sometimes both sides are guilty.
- catnaroek 10y ago
  Because a string is just a piece of data, and if your program can take it as input, then it must be handled correctly. “But writing parsers was sooo boring in college, and who has to do this in real life?”
  - wongarsu 10y ago
    A social engineer is just taking to you, if you listen to him you must act accordingly. A lock picking set is just a bit of metal, if it fits into the keyway the lock has to handle it correctly. Yes, writing parsers is a lot easier than those examples. But so far society has always ruled that inputs that purposefully try to abuse flaws are not freed from responsibility just because the flaw shouldn't be there.
    - catnaroek 10y ago
      A computer program is a mathematical object. If you want to rule out misbehaviors, you prove that such misbehaviors won't arise - just like any other theorem. And that's it. I'm not a malicious person. I don't purposefully abuse any system's flaws. But if anyone else does, my sympathies aren't with the designer of the flawed system. P.D.: Appeals to authority won't help make your case.
tannhaeuser 10y ago
I have to agree here. While a collection of "naughty strings" isn't wrong per se, the growing number of "killer regexpes to escape HTML" and other magic approaches to injection attacks on github only serve lazy devs who want post-facto excuses for their injection-prone web apps, or project managers who want to check items on security check lists. It's wrong because it de-emphasizes the importance of HTML-aware template languages, such as some that are available for golang, or SGML, the natural template language for HTML. There's no such thing as a collection of regexpes for sanitizing HTML; it all depends on the context into which strings are inserted.

ljoshua 10y ago

Line 629 is a gem! Thank you @minimaxir, I hadn't seen this before, this looks very useful.

bluesign 10y ago
line 629 is empty ;)
- ljoshua 10y ago
  No, wake up!! (For any who want to take the blue pill: https://github.com/minimaxir/big-list-of-naughty-strings/blob/master/blns.txt#L629 https://github.com/minimaxir/big-list-of-naughty-strings/blo...)
  - pluma 10y ago
    I'm not sure what you mean. That line really is empty.
    - jononor 10y ago
      Just a glitch...
zapu 10y ago
Doesn't look like anything to me.

the8472 10y ago

> Also, do not send a null character (U+0000) string isn't that quite a blind spot?

zeristor 10y ago

Unicode control characters from when people copy and paste from PDFs. Drives me up the wall, i didn't have time to go deep into this.

cwmma 10y ago

I'm trying really hard to figure out what's bad about 'Lightwater Country Park'

david-given 10y ago
I figured that one out, but --- evaluate? mocha? expression?
- cscheid 10y ago
  mocha has a naughty german word in its middle, I'm fairly sure.
  - slrz 10y ago
    As a German native speaker, I'm unable to figure it out.
  - guitarbill 10y ago
    No it doesn't. I believe it can be used for Javascript injections like 'eval' as 'mocha' is/was common a test framework. At least that's the ostensible reason Yahoo replaced 'eval' with 'review', 'mocha' with 'expresso', and 'expression' to 'statement' way back in 2002 [0]. [0] https://www.newscientist.com//article/dn2546-email-security-filter-spawns-new-words https://www.newscientist.com//article/dn2546-email-security-...
    - jwilk 10y ago
      "espresso", not "expresso".
    - scatters 10y ago
      "expresso" with "mocha", surely.
- HalfwayToDice 10y ago
  those puzzled me too

chiph 10y ago

You might want to include common unix shell commands. At a previous job we had a customer with the last name of Echo who wasn't able to make a purchase. Turns out our credit card processor blocked them.

lisper 10y ago
SQL injections too.
- FabHK 10y ago
  SQL injections are in there, and server injections as well. For example (I trust HN is suitably hardened :-) : /dev/null; touch /tmp/blns.fail ; echo 1;DROP TABLE users Edit: PS: "Feel free to send a pull request to add more strings, or additional sections."
  - inopinatus 10y ago
    I noticed earlier in the file that the Javascript had been chosen to be benign. "DROP TABLE users" doesn't seem to fit with that spirit. I'd want it to be instantly evident but also non-destructive, or at least reversible. How about renaming the table instead? (Sure, people generally shouldn't use this test input outside of a discardable testing environment, but if we could rely on "People shouldn't..." clauses to govern behaviour then much of this list would be unnecessary anyway.)
    - Arnavion 10y ago
      Already an issue: https://github.com/minimaxir/big-list-of-naughty-strings/issues/16 https://github.com/minimaxir/big-list-of-naughty-strings/iss...
      - schlowmo 10y ago
        Comment by the reporter of that issue (ro31337): > "Dropping a table is like checking a gun without bullets. It should not work, but just don't put it against your head while testing."

deleted 10y ago

[deleted]

emilsedgh 10y ago

Its funny that zero width space is considered weird and twitter fails on it. Its quite common in my language (Persian).

peteretep 10y ago
> Its quite common in my > language I'd love to hear more details on why?
- colechristensen 10y ago
  Ligatures. It's easy to not notice them at all in english ﬂ (ﬂ) vs. fl [ﬂy vs. fly] but some languages use them very extensively and the combinations are more significant. https://en.wikipedia.org/wiki/Zero-width_non-joiner https://en.wikipedia.org/wiki/Zero-width_non-joiner * It's entirely possible that the browser you're using isn't doing a very good job with ligatures which explains the strange look of my examples
- satbyy 10y ago
  ZWJ and ZWNJ are also common in Indic scripts. It's basically used to control the appearance of glyphs, for example half-forms and consonant clusters (क्‍ष vs क्ष, both are kṣa). As usual, wikipedia has good examples. The Unicode Standard also contains details about these. ZW[N]J as a standalone character or at the beginning of a word is very unusual on a day-to-day basis, so it's understandable that Twitter fails to recognize this pattern. ¹ https://en.wikipedia.org/wiki/ZWJ https://en.wikipedia.org/wiki/ZWJ ² https://en.wikipedia.org/wiki/ZWNJ https://en.wikipedia.org/wiki/ZWNJ
  - peteretep 10y ago
    Ah ha! > When a ZWJ is placed between two > emoji characters, it can also result > in a new form being shown, such as > the family emoji, made up of two adult > emoji and one or two child emoji That makes a lot of sense too, and I hadn't put sufficient work into how that's implemented -- retrospectively that makes perfect sense.

aroman 10y ago

Why is the string "Linda Callahan" a naughty/Scunthorpe word?

ue_ 10y ago
After re-reading it I can see it contains "allah", but I can't see why that would be filtered.
- jasonjei 10y ago
  Interesting my last name was blocked from making Genius Bar appointments [0]. My name is Jason Hung. [0] https://discussions.apple.com/thread/1491462?start=10&tstart=0 https://discussions.apple.com/thread/1491462?start=10&tstart...
  - pavel_lishin 10y ago
    Open a PR?
- FabHK 10y ago
  See the Scunthorpe Wikipedia article: "In February 2006, Linda Callahan, a resident of Ashfield, Massachusetts, was initially prevented from registering her name with Yahoo! as an e-mail address as it contained the substring allah. Yahoo! later reversed the ban." https://en.wikipedia.org/wiki/Scunthorpe_problem https://en.wikipedia.org/wiki/Scunthorpe_problem
deleted 10y ago
[deleted]
blahedo 10y ago
I got that one ("allah", presumably), but was stuck on these three: evaluate mocha expression
- scatters 10y ago
  eval expr I got nothing for "mocha", though. Edit: apparently (from below) there was a Yahoo! mail filter that replaced "expresso" [sic] with "mocha"; but either the story was misreported or the mail filter was wildly misconfigured. So the entry should be "expresso" [sic], perhaps.

Confiks 10y ago

Hello human. This is a message from the Matrix. You've been in a coma for 20 years. Please write back. https://github.com/minimaxir/big-list-of-naughty-strings/blob/8536c7903316763d7a6123e878c150fb97e6ea07/blns.txt#L629 https://github.com/minimaxir/big-list-of-naughty-strings/blo...

jjcm 10y ago
I like that in the master list it's annotated as: # Strings which may cause human to reinterpret worldview https://github.com/minimaxir/big-list-of-naughty-strings/blob/master/blns.txt#L627 https://github.com/minimaxir/big-list-of-naughty-strings/blo...

r-w 10y ago

The list itself can be found here: https://github.com/minimaxir/big-list-of-naughty-strings/blob/master/naughtystrings/internal/resource.go#L530 https://github.com/minimaxir/big-list-of-naughty-strings/blo...

derimagia 10y ago
If you're going to link to a line number - press 'y' to get a link tied to the commit. Otherwise it may be out of date the next time the file changes. (https://github.com/minimaxir/big-list-of-naughty-strings/blob/8536c7903316763d7a6123e878c150fb97e6ea07/naughtystrings/internal/resource.go#L530 https://github.com/minimaxir/big-list-of-naughty-strings/blo...)

kahrkunne 10y ago

Nice to have a list of these. Also, the first time that copypasta actually spooked me out ;-)

bsimpson 10y ago

TIL: `mocha:` was a custom schema that Netscape Navigator used to eval URLs (equivalent to `javascript:`), and Yahoo! Mail would replace it with 'espresso' to attempt to thwart phishing attempts: https://www.obscure.org/javascript/archives/msg01369.html https://www.obscure.org/javascript/archives/msg01369.html https://www.cnet.com/news/yahoo-mail-puts-words-in-your-mouth/ https://www.cnet.com/news/yahoo-mail-puts-words-in-your-mout...

ubernostrum 10y ago

Related: a list of names that probably should be reserved (for example, to prevent someone setting up a user-profile page at a URL you don't want them to control): https://ldpreload.com/blog/names-to-reserve https://ldpreload.com/blog/names-to-reserve

chipperyman573 10y ago
Alternatively, put them in another path (https://facebook.com/user123 https://facebook.com/user123 -> https://facebook.com/users/user123 https://facebook.com/users/user123)

Tokkemon 10y ago

This is super helpful! Thanks for sharing!

Capira 10y ago

My personal favorite: U+202E. It sets the directionality for a document from LTR encoding to RTL https://twitter.com/robin_linus/status/820567617903751169 https://twitter.com/robin_linus/status/820567617903751169

leni536 10y ago

> blns.txt consists of newline-delimited strings I expect some nasty strings to contain newlines (I wonder how many bash scripts are sensitive to filenames with newline characters in them). It shouldn't be a problem with the json file though.

mtnygard 10y ago
The file is newline-delimited. The strings themselves are base64 encoded, so they could contain newlines.
- leni536 10y ago
  It seems that blns.txt is the source content, then it's converted to blns.json, blns.base64.txt and blns.base64.json with the two scripts in the scripts folder (These resulting files shouldn't be in the repo in my opinion). One cannot possibly add strings with newlines in them, unless with some newline escaping that are handled in the scripts. It's a bad idea IMO and the source content should be the json file and blns.txt should be dropped.

solidsnack9000 10y ago

https://github.com/minimaxir/big-list-of-naughty-strings/blob/master/blns.txt#L633 https://github.com/minimaxir/big-list-of-naughty-strings/blo... > Strings which punish the fools who use cat/type on this file

k__ 10y ago

Reminds me of a complain I read on Twitter last week. Native Australians were angry, that FB blocked their real names, because they seemed fake to them. They have last names like "Creepingbear" and such.

harto 10y ago
Did you mean native Americans? I've never heard "Creepingbear" as an Australian name.

yellowapple 10y ago

"Strings which may cause human to reinterpret worldview" Hah. Totally filtering for that one now.

air7 10y ago

This reminded me of that story of people who have such strings for names: http://www.bbc.com/future/story/20160325-the-names-that-break-computer-systems http://www.bbc.com/future/story/20160325-the-names-that-brea...