7 ms·

Show HN: 5+ Billion Passwords in Order of Most Popular

184 points by berzerk0 9y ago

deleted 9y ago
[deleted]
berzerk0 9y ago
Having issues with the Seedbox, torrents are down temporarily. The main page contains links to Mega.NZ alternative downloads. Will be fixed shortly, apologies for the inconvenience.
- kaslai 9y ago
  Is there really any reason to multiply the downloads by 4 just to have different compression methods? It seems to me like that just needlessly dilutes the seed swarm and wastes space, since pretty much any modern archive reader can unpack all the provided formats. Sure, specialized command-line utils can't, but if you're using one of those then you probably know which one to use for a given format.
  - berzerk0 9y ago
    You know, you're not the first person to bring that up - and I am having seed problems. Perhaps next rev I'll only include the .7z (smallest) and greppable formats.
- infinisil 9y ago
  Have you considered using IPFS for distribution?
  - berzerk0 9y ago
    Hadn't heard of that before, I'll look into it
- berzerk0 9y ago
  Torrents are back! Mostly. Two out of the 12 aren't, but every wordlist can be downloaded via torrent in at least 3 compressed formats, including .7z
smaili 9y ago
Not to sound negative, but is this something we should really be exposing? It feels like the only ones who gain from this are hackers and password crackers, no?
- ignoramous 9y ago
  To quote a certain Alfred Charles Hobbs: "Rogues are very keen in their profession, and know already much more than we can teach them." Also see: Security through Obscurity.
  - pvg 9y ago
    This doesn't have anything to do with 'security through obscurity'.
    - ignoramous 9y ago
      It might not be. When someone raises concerns around disclosing weak passwords or weakness in crypto, or security holes in general [0], I kind of like to think they are trying to think abt the problem in the wrong way. STO is a good framework, I think, to help people steer in the right direction, and hence the 'also see' label. [0] Google's notorious Project Zero, for instance.
      - pvg 9y ago
        STO is a good framework It's a fairly specific term so trotting it out for everything makes it pointless and obscures its actual meaning. Every time someone misunderstands something about information security (or someone thinks they do), someone else comes along and says 'Security through obscurity!' or 'Trusting trust!' or something about source code. It turns these things from terminology and concepts into magical-sounding incantations that really just mean 'I disagree/don't like this'. This is analogous to the more familiar overuse of 'ad hominem' and 'straw man' and so on. It's not a good thing.
        9y ago
gfody 9y ago
you should package this up as a bloom filter and simple js routine web developers could use to do client-side checks to validate passwords. edit: on 2nd thought looks like a bloom filter for 5B entries at p=0.01 would be ~5GB, so not exactly convenient
- berzerk0 9y ago
  The largest list is 20GB, but it's not the only one. Popularity was based on how many they appeared in files that had all duplicates removed (in reference to themselves) The smallest file had passwords that appeared 75+ times, and the largest file had passwords that appeared 2+ times. The top 195 Thousand (which appeared 25+ times in analysis) clocks in at 803kb as a text file with nothing but the passwords themselves
- surement 9y ago
  Password validation is stupid and annoying. At best this should be used to display a warning that can be ignored.
jacquesm 9y ago
I don't understand: Why do you assume that password checkers keep their lists in alphabetically sorted form rather than just to load the whole thing into a db table with an index on it?
- berzerk0 9y ago
  As I was looking around for the files to make this project, on SecLists, Weakpass, and Hashes.org, most of the files were in alphabetical order. This was especially true for the larger files.
  - jacquesm 9y ago
    Yes, but it doesn't matter what order they are in if you put them in a DB. I don't know anybody that would do a sequential scan over a file in production. All you've done is shuffled around the most frequent cases to the beginning, which is great if that's what you are looking for, but now something else occupies the last slot and that case is just as bad as before. The real solution is to see these files as input to an indexed table so that you can get to the entry you want in log(n) time.
    - neuroid 9y ago
      I don't know anybody that would do a sequential scan over a file in production Well, that's pretty much how one would try to crack a password using a wordlist. EDIT: If the goal is to crack a bunch of properly hashed (PBKDF2, scrypt, etc.) and salted passwords then a lookup table is not very practical.
      - jacquesm 9y ago
        If you're trying to crack 'a password' you're doing it wrong. Alternatively, you're doing something illegal. That's not why these lists are made public (though I can appreciate the fact that they are 'dual use').
      - edoceo 9y ago
Mz 9y ago
•These lists are for LAWFUL, ETHICAL AND EDUCATIONAL PURPOSES ONLY. Yeah, like that is going to stop people from doing nefarious things with this info. If you feel the need to post this screechy, all caps disclaimer, maybe rethink your project entirely? Geez.
lucasgonze 9y ago
This list is immediately useful for validating user-created new passwords. Just stop with the bizarre rules about having uppercase, lowercase, symbols, numbers, length, etc. Instead require a string not in the top 10K (or 100K, or 1M) most popular.
- bradleyjg 9y ago
  Since the list is only rarely updated, this seems a perfect application Botelho's for minimal perfect hashing algorithm[1]. At about 8 bits per item storage and constant lookup it would be quite practical to use the top 32 Million list (appeared at least 10 times). [1] http://cmph.sourceforge.net/papers/tr06.pdf http://cmph.sourceforge.net/papers/tr06.pdf
  - berzerk0 9y ago
    Speaking of rarely updated - version two will come out mid-July at the latest. I'm also going to attempt to attribute the sources to given countries or at least language families. The list as it stands has a heavily US-centric bias. I plan to grep through it and pull out things like Cyrillic or Arabic characters, and then repeat the process using only those characters. That's not to say that everyone who uses Arabic or Cyrillic characters is guaranteed to use them, but it's more specific. If "password" is #2 in the world, perhaps пароль (password) will be the #2 most popular password with Russian Cyrillic Characters
    - lucasgonze 9y ago
      I wonder about how to package your data as developer tools. There should be libraries in any language used to develop systems which validate user-entered passwords.
      - berzerk0 9y ago
        Dropbox has their zxcvbn method which incorporates a list of 30k of the most common passwords https://github.com/dropbox/zxcvbn https://github.com/dropbox/zxcvbn
Roritharr 9y ago
Somehow I wonder if life is significantly different for people named Daniel.
- fasteo 9y ago
  Or Alexander or Victoria. It's weird that these first names appear in the Top196-probable.txt file.
bbcbasic 9y ago
If you pick the 5 billionth one you've probably picked a gooden.
- dang 9y ago
  We've banned this account for abusing the site, including posting many unsubstantive comments after we asked you to stop. Please don't create accounts to break the site rules with.
macscam 9y ago
Wow this is so useful for um security

184 points by berzerk0 9y ago

deleted 9y ago

[deleted]

Having issues with the Seedbox, torrents are down temporarily. The main page contains links to Mega.NZ alternative downloads. Will be fixed shortly, apologies for the inconvenience.

kaslai 9y ago
Is there really any reason to multiply the downloads by 4 just to have different compression methods? It seems to me like that just needlessly dilutes the seed swarm and wastes space, since pretty much any modern archive reader can unpack all the provided formats. Sure, specialized command-line utils can't, but if you're using one of those then you probably know which one to use for a given format.
- berzerk0 9y ago
  You know, you're not the first person to bring that up - and I am having seed problems. Perhaps next rev I'll only include the .7z (smallest) and greppable formats.
infinisil 9y ago
Have you considered using IPFS for distribution?
- berzerk0 9y ago
  Hadn't heard of that before, I'll look into it
berzerk0 9y ago
Torrents are back! Mostly. Two out of the 12 aren't, but every wordlist can be downloaded via torrent in at least 3 compressed formats, including .7z

smaili 9y ago

Not to sound negative, but is this something we should really be exposing? It feels like the only ones who gain from this are hackers and password crackers, no?

ignoramous 9y ago
To quote a certain Alfred Charles Hobbs: "Rogues are very keen in their profession, and know already much more than we can teach them." Also see: Security through Obscurity.
- pvg 9y ago
  This doesn't have anything to do with 'security through obscurity'.
  - ignoramous 9y ago
    It might not be. When someone raises concerns around disclosing weak passwords or weakness in crypto, or security holes in general [0], I kind of like to think they are trying to think abt the problem in the wrong way. STO is a good framework, I think, to help people steer in the right direction, and hence the 'also see' label. [0] Google's notorious Project Zero, for instance.
    - pvg 9y ago
      STO is a good framework It's a fairly specific term so trotting it out for everything makes it pointless and obscures its actual meaning. Every time someone misunderstands something about information security (or someone thinks they do), someone else comes along and says 'Security through obscurity!' or 'Trusting trust!' or something about source code. It turns these things from terminology and concepts into magical-sounding incantations that really just mean 'I disagree/don't like this'. This is analogous to the more familiar overuse of 'ad hominem' and 'straw man' and so on. It's not a good thing.
      - 9y ago

gfody 9y ago

you should package this up as a bloom filter and simple js routine web developers could use to do client-side checks to validate passwords. edit: on 2nd thought looks like a bloom filter for 5B entries at p=0.01 would be ~5GB, so not exactly convenient

berzerk0 9y ago
The largest list is 20GB, but it's not the only one. Popularity was based on how many they appeared in files that had all duplicates removed (in reference to themselves) The smallest file had passwords that appeared 75+ times, and the largest file had passwords that appeared 2+ times. The top 195 Thousand (which appeared 25+ times in analysis) clocks in at 803kb as a text file with nothing but the passwords themselves
surement 9y ago
Password validation is stupid and annoying. At best this should be used to display a warning that can be ignored.

jacquesm 9y ago

I don't understand: Why do you assume that password checkers keep their lists in alphabetically sorted form rather than just to load the whole thing into a db table with an index on it?

berzerk0 9y ago
As I was looking around for the files to make this project, on SecLists, Weakpass, and Hashes.org, most of the files were in alphabetical order. This was especially true for the larger files.
- jacquesm 9y ago
  Yes, but it doesn't matter what order they are in if you put them in a DB. I don't know anybody that would do a sequential scan over a file in production. All you've done is shuffled around the most frequent cases to the beginning, which is great if that's what you are looking for, but now something else occupies the last slot and that case is just as bad as before. The real solution is to see these files as input to an indexed table so that you can get to the entry you want in log(n) time.
  - neuroid 9y ago
    I don't know anybody that would do a sequential scan over a file in production Well, that's pretty much how one would try to crack a password using a wordlist. EDIT: If the goal is to crack a bunch of properly hashed (PBKDF2, scrypt, etc.) and salted passwords then a lookup table is not very practical.
    - jacquesm 9y ago
      If you're trying to crack 'a password' you're doing it wrong. Alternatively, you're doing something illegal. That's not why these lists are made public (though I can appreciate the fact that they are 'dual use').
    - edoceo 9y ago

Mz 9y ago

•These lists are for LAWFUL, ETHICAL AND EDUCATIONAL PURPOSES ONLY. Yeah, like that is going to stop people from doing nefarious things with this info. If you feel the need to post this screechy, all caps disclaimer, maybe rethink your project entirely? Geez.

lucasgonze 9y ago

This list is immediately useful for validating user-created new passwords. Just stop with the bizarre rules about having uppercase, lowercase, symbols, numbers, length, etc. Instead require a string not in the top 10K (or 100K, or 1M) most popular.

bradleyjg 9y ago
Since the list is only rarely updated, this seems a perfect application Botelho's for minimal perfect hashing algorithm[1]. At about 8 bits per item storage and constant lookup it would be quite practical to use the top 32 Million list (appeared at least 10 times). [1] http://cmph.sourceforge.net/papers/tr06.pdf http://cmph.sourceforge.net/papers/tr06.pdf
- berzerk0 9y ago
  Speaking of rarely updated - version two will come out mid-July at the latest. I'm also going to attempt to attribute the sources to given countries or at least language families. The list as it stands has a heavily US-centric bias. I plan to grep through it and pull out things like Cyrillic or Arabic characters, and then repeat the process using only those characters. That's not to say that everyone who uses Arabic or Cyrillic characters is guaranteed to use them, but it's more specific. If "password" is #2 in the world, perhaps пароль (password) will be the #2 most popular password with Russian Cyrillic Characters
  - lucasgonze 9y ago
    I wonder about how to package your data as developer tools. There should be libraries in any language used to develop systems which validate user-entered passwords.
    - berzerk0 9y ago
      Dropbox has their zxcvbn method which incorporates a list of 30k of the most common passwords https://github.com/dropbox/zxcvbn https://github.com/dropbox/zxcvbn

Roritharr 9y ago

Somehow I wonder if life is significantly different for people named Daniel.

fasteo 9y ago
Or Alexander or Victoria. It's weird that these first names appear in the Top196-probable.txt file.

bbcbasic 9y ago

If you pick the 5 billionth one you've probably picked a gooden.

dang 9y ago
We've banned this account for abusing the site, including posting many unsubstantive comments after we asked you to stop. Please don't create accounts to break the site rules with.

macscam 9y ago

Wow this is so useful for um security