3 ms·

I wrote a tool a while back that's excellent at this. It produces a 1MB file that you can do a straightforward search against and use to blacklist 161,000 diffe

I wrote a tool a while back that's excellent at this. It produces a 1MB file that you can do a straightforward search against and use to blacklist 161,000 different common passwords. https://github.com/robsheldon/bad-passwords-index https://github.com/robsheldon/bad-passwords-index I want people to use it. I gave it the most permissible possible license. Please people, use it. I'll even update it Real Soon Now.

Hondor 10y ago
I like your license choice. Is there a reason you chose Unlicense over CC0? I've never heard of it before. The text is a bit shorter than CC0 which is nice. Creative Commons recommends against using CC licenses for software so as not to step on the toes of the GPL which doesn't seem applicable if you really want public domain. [1] [1] https://creativecommons.org/2011/04/15/using-cc0-for-public-domain-software/ https://creativecommons.org/2011/04/15/using-cc0-for-public-...
- thaumaturgy 10y ago
  Oof. I don't remember exactly, but I do remember spending a lot of time (too much time) deciding between various license options. I think in the end I chose Unlicense primarily because it's easier to find Unlicensed projects on Github.
PerfectElement 10y ago
Thank you for this. I was looking for something similar the other day.
semi-extrinsic 10y ago
> There are no fancypants libraries, frameworks, or nonsense required. Use your favorite language with the index, I don't care what it is. ...unless you like XML. If you like XML, I'm judging you. My hat's off to you, good sir.
013 10y ago
"bad_passwords.txt" seems to be 1 line long? Am I missing something.. edit: Oh, "Convert your user's password to lowercase and do a simple string search against the index. No regular expressions are necessary." That makes sense, apologies.
jedimastert 10y ago
Out of curiosity, what packing algorithm did you use?
- thaumaturgy 10y ago
  Calling it an algorithm may be giving it too much credit. The packing program reads in a file, sorts the passwords into clusters based on their length, maintaining a popularity count as it goes. Then it prunes any passwords that occur less than some number of times in the input file ("4" by default, so it doesn't spend a lot of time churning on unpopular passwords). Next, starting with the longest passwords, it works its way down the list, merging the popularity count of shorter passwords into the most popular longer passwords which contain the entire shorter password. e.g., "password" gets merged into "password1234". Finally it starts writing the output. Starting with the most popular password, it searches up to N passwords ahead in the list (40 by default), looking for any passwords that begin with the same characters that the current password ends with. So it finds "123456", and writes "password123456" to the output file. I think the only thing I can do to optimize it further is to re-cluster the passwords by their starting character during the output stage, to improve its chances of always finding two passwords that can be glued together. It's a crude brute-force approach but it works okay. I just looked at the code, it's not that bad, I really will release it with the next update.
  - simcop2387 10y ago
    What I'd do is sort the popular passwords by length, start with the longest. Then as you're adding new passwords to the end, check if it's already in there. Then you'll go from phrases to words with a lot of them being duplicates. At least if you wanted to keep it as a long string. I'd still probably approach this as a bloom filter as others suggested since it'd let you have a larger number of entries for similar memory footprint.
phonon 10y ago
Why not make a Django-specific version, like https://github.com/timboss/Django-Common-Password-Validation https://github.com/timboss/Django-Common-Password-Validation
- thaumaturgy 10y ago
  Because then it would be Django-specific. There are a few approaches that would work better for specific environments. I could make a gigantic Bloom filter for example, with lookup functions in a few languages. But my goal was to make something that any programmer could use in any environment. Substring searching is a basic programming skill and just about everything supports it. (COBOL74 needs external functions to do it.)
  - phonon 10y ago
    No, I meant in addition; if you want it to be used widely, a few tweaks (if that?) and you could make a Django version (Django already supports the feature, it just uses a short (1000) wordlist by default). So https://github.com/timboss/Django-Common-Password-Validation https://github.com/timboss/Django-Common-Password-Validation is just a a longer (10,000) wordlist, for example. If nothing else, it makes what you worked on more discoverable.
joshuak 10y ago
Good, but might I suggest a larger set via bloom filter.
- thaumaturgy 10y ago
  A Bloom filter is absolutely the right way to do this sort of thing, except that I had as a firm requirement, "something that any programmer of any skill level can use in any environment." The file is small enough, and the approach straightforward enough, that for instance somebody could write a trivial Wordpress or Joomla or Drupal plugin for it, and they wouldn't have to think about Bloom filters or have ready-made lookup functions. I placed greater value on adoptability than cleverness.

Hondor 10y ago

I like your license choice. Is there a reason you chose Unlicense over CC0? I've never heard of it before. The text is a bit shorter than CC0 which is nice. Creative Commons recommends against using CC licenses for software so as not to step on the toes of the GPL which doesn't seem applicable if you really want public domain. [1] [1] https://creativecommons.org/2011/04/15/using-cc0-for-public-domain-software/ https://creativecommons.org/2011/04/15/using-cc0-for-public-...

thaumaturgy 10y ago
Oof. I don't remember exactly, but I do remember spending a lot of time (too much time) deciding between various license options. I think in the end I chose Unlicense primarily because it's easier to find Unlicensed projects on Github.

PerfectElement 10y ago

Thank you for this. I was looking for something similar the other day.

semi-extrinsic 10y ago

> There are no fancypants libraries, frameworks, or nonsense required. Use your favorite language with the index, I don't care what it is. ...unless you like XML. If you like XML, I'm judging you. My hat's off to you, good sir.

013 10y ago

"bad_passwords.txt" seems to be 1 line long? Am I missing something.. edit: Oh, "Convert your user's password to lowercase and do a simple string search against the index. No regular expressions are necessary." That makes sense, apologies.

jedimastert 10y ago

Out of curiosity, what packing algorithm did you use?

thaumaturgy 10y ago
Calling it an algorithm may be giving it too much credit. The packing program reads in a file, sorts the passwords into clusters based on their length, maintaining a popularity count as it goes. Then it prunes any passwords that occur less than some number of times in the input file ("4" by default, so it doesn't spend a lot of time churning on unpopular passwords). Next, starting with the longest passwords, it works its way down the list, merging the popularity count of shorter passwords into the most popular longer passwords which contain the entire shorter password. e.g., "password" gets merged into "password1234". Finally it starts writing the output. Starting with the most popular password, it searches up to N passwords ahead in the list (40 by default), looking for any passwords that begin with the same characters that the current password ends with. So it finds "123456", and writes "password123456" to the output file. I think the only thing I can do to optimize it further is to re-cluster the passwords by their starting character during the output stage, to improve its chances of always finding two passwords that can be glued together. It's a crude brute-force approach but it works okay. I just looked at the code, it's not that bad, I really will release it with the next update.
- simcop2387 10y ago
  What I'd do is sort the popular passwords by length, start with the longest. Then as you're adding new passwords to the end, check if it's already in there. Then you'll go from phrases to words with a lot of them being duplicates. At least if you wanted to keep it as a long string. I'd still probably approach this as a bloom filter as others suggested since it'd let you have a larger number of entries for similar memory footprint.

phonon 10y ago

Why not make a Django-specific version, like https://github.com/timboss/Django-Common-Password-Validation https://github.com/timboss/Django-Common-Password-Validation

thaumaturgy 10y ago
Because then it would be Django-specific. There are a few approaches that would work better for specific environments. I could make a gigantic Bloom filter for example, with lookup functions in a few languages. But my goal was to make something that any programmer could use in any environment. Substring searching is a basic programming skill and just about everything supports it. (COBOL74 needs external functions to do it.)
- phonon 10y ago
  No, I meant in addition; if you want it to be used widely, a few tweaks (if that?) and you could make a Django version (Django already supports the feature, it just uses a short (1000) wordlist by default). So https://github.com/timboss/Django-Common-Password-Validation https://github.com/timboss/Django-Common-Password-Validation is just a a longer (10,000) wordlist, for example. If nothing else, it makes what you worked on more discoverable.

joshuak 10y ago

Good, but might I suggest a larger set via bloom filter.

thaumaturgy 10y ago
A Bloom filter is absolutely the right way to do this sort of thing, except that I had as a firm requirement, "something that any programmer of any skill level can use in any environment." The file is small enough, and the approach straightforward enough, that for instance somebody could write a trivial Wordpress or Joomla or Drupal plugin for it, and they wouldn't have to think about Bloom filters or have ready-made lookup functions. I placed greater value on adoptability than cleverness.