8 ms·
I found it on some sketchy download site that I had to use jdownloader for and solve many captchas based on a tip from a past HN story. After that, you'll find
by Natsu 6y ago
I found it on some sketchy download site that I had to use jdownloader for and solve many captchas based on a tip from a past HN story.
After that, you'll find that the data is poorly and inconsistently encoded (lots of ugly BOMs), a bunch of files are split in weird ways (that I had to concatenate then give sensible names to). Figuring out what order they go in (they're not all split on a record boundary!) may take some translation, too. After that, a bunch of them have bad, broken CVS headers and you have to find something that can manage huge files to edit the first couple of lines.
Then you find that all of them use different delimiters (colon or comma), some of them quote each entry, and they each have a different set of data that doesn't match up at all. I'm still trying to figure out what the data in each file is and see if it can be normalized into one schema.
So it's about like downloading a phonebook for half the world with some extra mystery data that has no description where you have to guess what it even means.
Interestingly, I checked for a family member who deleted their FB a few years ago and they're not in there. It'd be interesting to see if there are any accounts that deleted before 2019 in the data or not. I suspect not, but this isn't enough of a test to prove it.
- beowulfey 6y agoreally? i only checked one of the country zips but it was 6 colon-delimited txt files that were easy to grep to see if any numbers i knew were in there
- Natsu 6y agoThey're all over the place. There are several dozen files, each named after a country. Some contain a single text file that's normal, others have different delimiters, encodings and other nonsense. The last record of some of the split files is split across files, etc. This is not particularly high quality data, I'm sure a lot of people didn't look at the data very much and just ignored a few lines of errors on import or did simple greps as you say because it's a pain in the rear to find editors that even can handle multi-GB files, etc. Even good converters like iconv can bomb out on, I think it was Qatar (which was also little endian UTF-16 with a BOM), etc. But yes, it's all over the place. I love how one of them decided to number things 1, 2, III, 4, though maybe that was the translator, I dunno, I never learned to read Arabic letters...
- 3np 6y agoI’ve had my phone number linked to FB since forever and my info is not in the leak. Neither is my brother. My mom is, though.
- reaperducer 6y agoI'm not in it, but my cat is. I wonder what kind of spam he'll start getting.
- wyldfire 6y agoDoes anyone...that you might know of...have the tidied-up/demultiplexed/transmogrified version of the dump available?
- Natsu 6y agoI haven't seen one or it would've saved me a ton of trouble. I think it might be quite a while before I finish putting this into an sqlite database given all the schema nonsense and the weekend being almost over now.
- Natsu 6y agoThere seems to be a lot of data that includes the delimiters in what's supposed to be data, just to make it even more helpful. Because yeah, let's just store a lot of data with commas in a CSV file, that won't make it obnoxious to parse or anything...