37 ms·
300M Freely Downloadable Pwned Passwords
- captn3m0 9y agoGoing to generate a bloom-filter from this dataset tonight. Troy mentions some arguments against torrents, but it is better to have a authoritative torrent than none, imo.
- developer2 9y agoIs a bloom filter worth it in this case? With the optimal "k" hash functions of 10 and a "p" error rate of 0.001% (false positives of approximately 1 in 1000), a bloom filter for the 306,259,512 items will take 538 MB. Increasing the error rate to 0.01% (1 in 100) is still 358 MB. That's a sizeable filter to maintain in memory (then again... RAM is cheap). I'd probably just shove the passwords into a database, limiting the index prefix to the first X characters to reduce index size.
- captn3m0 9y agoDistributing a 538 MB file (which can be compressed further) is much easier.
- wongarsu 9y agoWhat are the actual use cases where this size difference matters? I distributing to a general audience, 0.5GB and 10GB isn't that much of a difference, and most people are more equipped for handling lists of strings than for handling bloom filters.
- bradleyjg 9y ago>which can be compressed further Can it? I think of a bloom filter as similar to a lossy compression scheme and wouldn't expect it to be further compressible to any significant extent using a general purpose lossless scheme. Similar to how general purpose compressors generally don't do very well with mp3s or jpgs.
- Deimorz 9y agoReducing the size by ~95% in exchange for a 0.001% error rate seems like a pretty nice tradeoff to be able to make for some uses. The nature of the data means it can never really be "perfect" anyway (there are certainly some password breaches that exist but aren't included in the list), so massively reducing the resources required in exchange for a bit of artificial error seems pretty reasonable to me.
- bradleyjg 9y agoA signed minimal perfect hash function may be a better bet. You can get down to around 2.68 bits per key plus w sign bits where the false positive rate is 2^-w. For false positive rate of 2^-9 (0.00195) that's 447 MB, which is slightly less than an optimal bloom filter for the same number of items, and lookups will be considerably faster. Construction time and the fact that you can't add to it without rebuilding the whole thing are the downsides. But given the application I don't think they matter much. http://sux4j.di.unimi.it/docs/it/unimi/dsi/sux4j/mph/MinimalPerfectHashFunction.html http://sux4j.di.unimi.it/docs/it/unimi/dsi/sux4j/mph/Minimal...
- hedora 9y agoIt's also a bummer there are only sha1's in the file. It would be good to block things within hamming distance 2 of a leaked password (so p@ssw0rd€ would also be blacklisted...)
- Piskvorrr 9y agoYou could go the other way, and check likely candidates from the new password - e.g. if user enters "hunter2", you check hunter and hunter0 and hunter1 and hunter3 and hunt3r2.
- vanni 9y agoOr better a cookoo filter: http://dl.acm.org/citation.cfm?doid=2674005.2674994 http://dl.acm.org/citation.cfm?doid=2674005.2674994
- danbruc 9y agoIf it doesn't have to be really fast, you can simply binary search fetching no more than 29 lines from the file. Or you can interpolate the expected location of the hash and read a block around that location. This could get you down to reading only one or two disk blocks if the actual position is never more than half a disk block away from the interpolated position. It could be more but given that we are dealing with cryptographic hashes I would expect the interpolated position to never be too far away from the actual position.
- peteretep 9y agoI wonder how we force change with individual companies? Today I had to sign up for a UPS account. The password length was set to max 27 characters, and the form had disabled paste in the password field. Who do we lobby to get them to fail their next PCI-DSS compliance test?
- swiley 9y agoWell for that you can probably turn off JavaScript or use the web inspector to enable paste.
- peteretep 9y agoI can. My wife, who I've taught to use a password manager, probably can't. And neither of these excuses a 27 character limit that strongly suggests my password is being stored unencrypted somewhere.
- keteb 9y agoNot sure how that suggests that at all. User inputs have some sort of cap (don't want someone using a 2GB string as their password). So naturally there's a conversation at some point about "what's our maximum password size". If that conversation starts off at 1000 characters you're fine, but more often then not it looks more like: "Make the requirement 8-12 characters" "12 is too short" "fine make it longer, like" "Ok" [18 char implementation] "Hey, Bob in accounting says he uses 20 char passwords" "Fine, bump it by another 50%" [27 char implementation] [no further internal complaints] [Specs never updated or reviewed again]
- vacri 9y agouse X windows - middle-click-to-paste usually gets around these 'no paste here' thingies. :)
- tracker1 9y agoIt may well be encrypted, but not hashed...
- 9y ago
- r_singh 9y agoIs it safe to test my password on this website? (because I just did)
- gordonmorris 9y agoFrom the article: "It goes without saying (although I say it anyway on that page), but don't enter a password you currently use into any third-party service like this! I don't explicitly log them and I'm a trustworthy guy but yeah, don't." Safe: probably. Good practice: no.
- fredoliveira 9y agoIf you only have one password to try, I'd say you have more important issues to tackle. Install a password manager, start using generated passwords, and stop using 'your password'.
- coldsmoke 9y agoInteresting - "correct horse battery staple"[0] is flagged as not being in the data set. I was sure someone should have used that by now. [0] https://xkcd.com/936/ https://xkcd.com/936/
- r0f1 9y agoWhat would be the best data structure for using this in, say, a Python script? I imagine just putting it into a dictionary (hash map etc.) won't work because of the size.
- fanf2 9y agoBloom filter :-)
- drchickensalad 9y agoIf you want it lossless, a trie can be great. Not sure if one implemented in Python would be worth it though; should be a package with it in C.
- pjf 9y agoSomeone should apply deep learning to this and check how it compares with brute-forcing passwords. E.g. https://github.com/thoppe/5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8 https://github.com/thoppe/5baa61e4c9b93f3f0682250b6cf8331b7e...
- rocqua 9y agoCan't, as the passwords aren't available in plain-text. Only as sha-1 hashes.
- RomanPushkin 9y agoThat's the point here. We know 300M passwords leaked, but we can't investigate these passwords and learn from this data. It's sad that researchers should look into black market if they want to get plain passwords. I'm pretty sure everyone who needs these passwords can find them.
- j_s 9y agoWould it make sense to host the file on a cheap OVH/Scaleway VPS with unlimited bandwidth? I guess CloudFlare doing it for free beats that though!
- tomschlick 9y ago"Unlimited" as in, "as soon as you start becoming a problem we drop you".
- pfg 9y agoOVH lets you use the advertised bandwidth 24/7, if you wish to do so. Naturally, this doesn't replace a CDN, and their peering isn't the best, so some routes might be congested and won't be that fast. I have no experience with Scaleway on this, but based on what I've heard about them in the past, I imagine their policy is roughly the same.
- j_s 9y agoAFAIK these services are operate by ISPs with lopsided peering agreements.
- nischalsamji 9y agoHIBP provides a REST API to check if a password has been found in a breach, Is there a disadvantage of using it in applications and restricting users not to use the breached password?
- lancefisher 9y agoIt's not ideal to send every new user's password to a 3rd party service.
- Piskvorrr 9y agoHence the downloadable file, and even a suggestion to use that as an in-house checker. It's in the article.
- edelans 9y agoyou can still send the SHA1
- wongarsu 9y agoWithout salt, meaning the majority of passwords can be reversed with brute-forcing or rainbow tables. The second google result for rainbow tables lets me download software and tables to efficiently reverse any sha1 whos plaintext fits [a-zA-Z0-9]{1,9} or [a-z0-9]{1,10}. That's likely the majority of passwords an attacker would observe
- deleted 9y ago[deleted]
- nicpottier 9y agoI used one password for a decade or so in the 90s and early naughts and though I've since moved on to use LastPass and two factor for everything this is the first time that password appears in one of these databases. Guessing it was in MySpace.. Ironically I used another password for sites I trusted less and that one isn't in there.
- colinbartlett 9y agoI do agree with Troy that this could be useful to send to relatives and family: > I'm envisaging more tech-savvy people using this service to demonstrate a point to friends, relatives and co-workers: "you see, this password has been breached before, don't use it!" But I can't be the only one whose family would be baffled by the term "pwned". I wish it said something like "Your password has been hacked!" which we all know not to be technically correct but would resonate a lot more.
- Piskvorrr 9y agoToo alarmist, and in wrong tense: people would start freaking out "by typing my password into this box, my password has been hacked". Resonance is not always a good thing, see Tacoma Narrows bridge ;)
- oliwarner 9y agoAn interesting element to this is how resistant some people are too using torrents for legitimate purposes, even as a backup mirror. It's something we've come to embrace in the Linux world. Much faster than a single server and saves bandwidth at individual sites. Surprised this pragmatism hasn't reached the rest of you yet.
- jff 9y agoHaving an http option makes it better for those on restrictive work networks--I'm downloading the file because I want to experiment with it, but I certainly wouldn't want a call from Networking asking why I'm torrenting stuff.
- oliwarner 9y agoYes, I understand the reason for keeping a slow legacy option available. Even in your situation, if you're doing this for work and IT calls you up, you just tell them what you're doing. "I'm downloading a very large file over Bittorrent because it's 50% faster than the HTTP download and I'd like to do some work today. Is that all? Thanks. Bye." In some places you'd get that call from IT just for downloading a large file. Getting a call doesn't mean you're doing something wrong, they're just checking it's you, not malware, and that it's for work. If they haven't already blocked it, have at it.
- devy 9y agoCan Troy or someone contact Google (bq-public-data@google.com) and push this to the GCP's BigQuery public dataset[1] for hosting and easier look up your password via SQL in BigQuery rather than some 3rd party site? [1] https://cloud.google.com/bigquery/public-data/ https://cloud.google.com/bigquery/public-data/
- cobookman 9y agoI work for Google cloud. Will ping internally about us hosting it.
- dredmorbius 9y agoThank you.
- aj7 9y agoIf I test my passwords, aren't they also now pwned?
- Mithorium 9y agoIn a word: Yes
- stephengillie 9y agoNot if you grep locally. How big is this data set? It can't be much bigger than a AAA video game download.
- ocdtrekkie 9y agoIt's 11.9 GB of text (5.3 GB zipped). So smaller than quite a few video game downloads.
- sleepybrett 9y agofairly low compression rate for zipped ascii, but i guess it is mostly a giant pile of nearly random strings.
- edelans 9y agotest the SHA1 of your passwords, not your passwords, and you are safe.
- jaclaz 9y agoI don't get it. >Do not send any password you actively us to a third-party service - even this one. So I can only test password that I am not using (and by extension that I am not going to use in the future). >oh no - pwned! >This password has previously appeared in a data breach and should never be used. If you've ever used it anywhere before, change it immediately! If I cannot (shouldn't) submit any password I am actively using, what does it matter if I used it before? Now I already changed it.
- model_m_warrior 9y agoI believe the idea is to ensure no one can use the listing to brute force.
- thephyber 9y agoI think gp is complaining that the second you type your password into the form, you've "used it", hence you should change it. The gp makes a good point, but that's also why you can submit the `sha1($your_password)` instead. The only question is why did Troy allow un-hashed passwords to be submitted.
- jaclaz 9y agoMaybe - even better - if you could submit only - say - first 8 characters of the SHA1 (and NOT the complete hash) and provide - still say - max 10 "whole" hashes found with that 8 char beginning (if more than 10 ask for a ninth char). I mean, here is the SHA1 of my password (not really): d012f68144ed0f121d3cc330a17eec528c2e7d59 This site: https://hashkiller.co.uk/sha1-decrypter.aspx https://hashkiller.co.uk/sha1-decrypter.aspx >We have a total of just over 312.072 billion unique decrypted SHA1 hashes since August 2007. Took exactly 221 ms to reverse it to "pippo".
- excitom 9y ago>If a password is not found in the Pwned Passwords set, it'll result in a response like this: Wait, so I test my password to see if it's "good" and now you have a copy of a password I will be using. Am I just being paranoid?
- joatmon-snoo 9y agoThis is Troy we're talking about - I strongly doubt he'd do anything like that without full disclosure.
- thephyber 9y agoI think the point of gp is to say that once you submit your password, you have no control of where it goes. Maybe a malicious copy of the website exists at lots of LevenshteinDist=1 domains. Accidentally typo the domain and get pwned, thinking you are submitting it to an ethical security researcher's tool, but actually getting phished.
- mnem 9y agoFrom the article: "It goes without saying (although I say it anyway on that page), but don't enter a password you currently use into any third-party service like this! I don't explicitly log them and I'm a trustworthy guy but yeah, don't."
- rrauenza 9y agoYou can post the sha1sum instead. $ sha1sum SooperSekretPassw0rd^D SooperSekretPassw0rddc0d3504b259a92dce59b850969601d12c06a75f -
- deleted 9y ago[deleted]
- 13of40 9y agoIf you're on Windows, you can calculate it in PowerShell like this: $password = "foobar" ([Security.Cryptography.SHA1CryptoServiceProvider]::Create().ComputeHash([Text.Encoding]::ASCII.GetBytes($password)) | %{'{0:x2}' -f $_}) -join ""
- oelmekki 9y agoI really would love we be done with passwords altogether. We're asking non power users to make their password unique, and then make it complicated, and then remember all of them in their head, not on a post-it. Nobody can do that, not even us who are telling them to do that. And then, we explain to them they're dumb if they didn't do that. Currently, my way to generate a new password is this : `pwgen | md5sum`. And then, I use "lost password" everywhere (but for my mailbox, obviously), that is, the rare times my browser is not already prefilling the login form. This makes me wonder why we don't just go with that : generate a random password for the user in registration form, allow the browser to save it. On the login form, check if fields are prefilled. If not, only display an email field and send an auth link as mail. User clicking it (once, and fast enough) is logged in. You still have to remember your mailbox password, but that's the only one, quite akin the root password of a server.
- mbid 9y agoAs others have pointed out, the use case for pasting plain text passwords is not quite clear. Maybe it would be a good idea to allow searching for hashes only, or at least hash the password in js on the client. Also, I'm genuinely curious as to why SHA-1 is used and not SHA-256. Surely the one-time additional cost of using SHA-256 would've been negligible for Troy? If at some point somebody manages to do preimage attacks on SHA-1, I have to assume my password is broken if I've submitted its hash to his API. Although I guess you'd have to actually be able to enumerate preimages, preferably from small to big. Still, I don't understand why Troy doesn't account for the possibility by using a hash function widely considered to be stronger.
- tracker1 9y agoI believe the sources for the data breaches were mentioned in the article, so if someone wanted to get those sources anyway, it wouldn't be a big deal.
- bigiain 9y agoHe stops well short of saying "here's a link to the pastebin dumps!", but yeah - it wouldn't take too much google-fu to build your own version of this - perhaps not a 300M entry one, but I doubt it'd take more than a weekend to get halfway there if you wanted too.
- Deimorz 9y agoOne really unfortunate aspect of the passwords being hashed is that there's no info available about their lengths. Knowing the lengths could allow you to reduce the size considerably when you enforce a minimum password length. For example, if I have a site that requires passwords to be at least 10 chars long, I don't need any of the data for breached passwords that are shorter than 10 characters. People can't possibly use them anyway, so that's probably a huge chunk of the data that's completely useless to be storing and checking.
- royce 9y agoI've cracked* just under 99% of them so far (including the 14 million added in Update 1). Statistics are here: https://gist.github.com/roycewilliams/b1de2afbfe5cb71bea16c94042b9bbfc https://gist.github.com/roycewilliams/b1de2afbfe5cb71bea16c9... Regardless of composition, the top 12 lengths are: 8: 32% (102260862) 10: 14% (45084047) 9: 13% (41525797) 7: 10% (33632055) 6: 06% (20211176) 11: 05% (18275968) 12: 04% (14052958) 15: 02% (8291459) 13: 02% (8042452) 14: 01% (6321198) 16: 01% (4201765) 5: 00% (3054291) In other words, requiring a minimum length of 12 would make 80% of the passwords in the corpus inapplicable. ... and the top 12 masks are: ?l?l?l?l?l?l?l?l,47823614 ?l?l?l?l?l?l?d?d,7005728 ?d?d?d?d?d?d?d?d,6212778 ?l?l?l?l?l?l?l?l?l?l,6023602 ?l?l?l?l?l?l?l?l?l,5379482 ?l?l?l?l?l?l?l?d?d,5169013 ?l?l?l?l?l?l?l?l?d?d,5090400 ?l?l?l?l?l?l?l,4998896 ?d?d?d?d?d?d?d,4798329 ?l?l?l?l?d?d?d?d,4798124 ?d?d?d?d?d?d?d?d?d?d,4754401 ?l?l?l?l?l?l?d?d?d?d,4377841 Almost 48 million of them are 8 lower-case characters. * And to be clear, "cracked" is an overstatement. Many of his sources are public. Simply using those sources as wordlists makes "cracking" these like shooting fish in a barrel.
- deleted 9y ago[deleted]
- Deimorz 9y agoOh, very cool, thanks for posting. I had actually written a really basic cracker and started seeing if I could figure out how many of them belonged to shorter passwords, but you're doing a much, much better job of it than I am (I was just brute-force generating short passwords, no wordlists or anything). Are you planning to make a blog post or anything "final" with the info you find out, or will you just keep updating those gists?
- deleted 9y ago[deleted]
- danso 9y agoI'm enjoying typing in things that I've never used as passwords (my own name, for example) to see if anyone else at some point thought these things would make for good passwords.
- Freaky 9y agoQuick script to binary-search for passwords locally: https://gist.github.com/Freaky/4cb7ce8c107c3da2e4a8210356e8da25 https://gist.github.com/Freaky/4cb7ce8c107c3da2e4a8210356e8d...
- _wldu 9y agoMany of these passwords are one or two characters in length. I think the 300 million number is inflated for publicity. Who allows a password that only has one character? Go here and type the character 'a': https://haveibeenpwned.com/ https://haveibeenpwned.com/
- Freaky 9y agoThere are 9120 printable ASCII strings of 1-2 characters in length. Cunning, including that many to bulk out the list :)
- _wldu 9y ago;) OK. But there are a ton of 3 and 4 char passwords too. I mean, what sort of site would allow that? It's just hard to believe these were actually passwords. BTW, I upvoted your comment. It made me laugh. Point granted.
- bigiain 9y agoFWIW - I do that all the time. If someone's website forces me to set up an account without me already being convinced there's enough benefit to me in return for my personal information, they're likely to get a signup for test@example.com with password "foo". (And if they then respond with "please click the confirmation link in the email we just sent", I'll sign up again with $sitename@$spare-domain-I-own.com and pick up the link from the spam filtered catchall account). I probably do this at least once a month when there's hints something useful in a web forum I want to read, but I'm not (yet or ever) convinced I'll ever become a member of that forum's community...
- oxplot 9y agoWoha, wait a second: > Each of the 306 million passwords is being provided as a SHA1 hash. That's it? Without any salting? This would make it trivial to recover the plain text using rainbow tables.
- bigiain 9y agoTroy explains where the majority of them came from - it'd be even more trivial to Google for the dumps yourself. It's not like everyone who is curious cant go findabout stuff like this doesn't have the Rockyou dump already, and there's easily enough links to start your own list here: https://www.google.com/search?q=password+lists https://www.google.com/search?q=password+lists
- misticdeveloper 9y agoI'm confused. If a website salted their hashes, wouldn't it not matter if he password alone was "pwned"?
- lilyball 9y agoIf a website salts their hashes, that only helps protect their users if that website is hacked and their password database stolen. But 306 million passwords have already been exposed by data breaches at other sites. And users have a tendency to reuse passwords across multiple sites. Just because your website wasn't hacked doesn't mean an attacker can't go look up one of your users in someone else's data breach and try the same password on your site.