4 ms·
To save folks a click, the dataset itself has been made available here: https://zenodo.org/records/15170676 https://zenodo.org/records/15170676 It's 118 gigaby
by pavel_lishin 1y ago
To save folks a click, the dataset itself has been made available here: https://zenodo.org/records/15170676 https://zenodo.org/records/15170676
It's 118 gigabytes of JSON.
- 1123581321 1y agoIs there a faster way to download than the link there? It's steady, but a roughly 11 hour download.
- DaSHacka 1y agoI'm sure someone will make a magnet for it eventually
- aehgwdyw 1y agomagnet:?xt=urn:btih:19db177fa7f13515e11c23e7c694419e875adfd8&xt=urn:btmh:1220ff0a57b459dae436d6c425721e04240aad55545a56bbfb5371d8c21ce125d7a9&dn=dataset.zst
- ghodawalaaman 1y agohmm they should have uploaded a torrent link to download the data, I wonder why they didn't do that?
- deleted 1y ago[deleted]
- diggan 1y ago> It's 118 gigabytes of JSON. 118.0 GB of ZST compressed JSON (https://zenodo.org/records/15170676 https://zenodo.org/records/15170676). The actual uncompressed JSON would most likely be much, much larger.
- squigz 1y agoI imagine it can be reduced a fair bit simply by stripping out unneeded fields from the messages; I imagine it can be reduced even further by removing unneeded messages entirely (i.e., bot welcome messages), deduping messages, etc
- hampus 1y agoI downloaded it and decompressed it — it's approximately 2.1 terabytes in size.
- kedaiapps 1y agohey there, would you mind sharing the file please?
- diggan 1y agoI managed to download the file, but since they restricted the downloads, it seems they hid the checksums too. Could you possibly share the md5/sha256 of the .zst file?
- philipkglass 1y agosha256sum dataset.zst 0196416253fab4bce08504737bc81215927d9afdc6ccc81f75345518109266a4 dataset.zst
- aehgwdyw 1y agomagnet:?xt=urn:btih:19db177fa7f13515e11c23e7c694419e875adfd8&xt=urn:btmh:1220ff0a57b459dae436d6c425721e04240aad55545a56bbfb5371d8c21ce125d7a9&dn=dataset.zst
- topcatto 1y ago[dead]
- paulirish 1y agoHere's the top 500 servers in the dataset: https://pastebin.com/pANc3WkT https://pastebin.com/pANc3WkT
- deleted 1y ago[deleted]
- 01HNNWZ0MV43FF 1y agoAnyone who has it, please post the SHA384 or BLAKE3 (BLAKE3 will be way faster on 100+ GB) so I can verify it if I get a torrent later. Zenodo requires a sign-in, and I won't auth with GitHub as they want some WILD permissions on my GitHub.
- philipkglass 1y agoWhen I started the download this morning I was able to use wget without authentication, but I see that Zenodo has added a login-wall now. The decompressed data appears to be JSONL, but at least the version I downloaded has a little binary garbage at the front. The first readable JSON object has the author "Fortnite Germany". Size as .zst: 117,962,356,699 bytes ZST SHA384: b8863645654610f1fde2859bb20bd87d913865af7791e0ec33741402944d5b9bdfdaaf65c2dc610730efb01f446e2588
- polarix 1y agoIt is not just a login wall, they have restricted access even for logged in users, presumably to only the uploaders. A magnet would be nice.
- philipkglass 1y agoAbout the decompressed data: zstdcat dataset.zst | sha384sum 0812f3876a7e319081f596a5545321e5c8e8def501add3a4f5ff039568fe59aa5d4ac5d2c3e549532f529bd09b887596 zstdcat dataset.zst | wc 2059116741 22128178392 2099550453760 If you can post your email and you have a sftp server or other accessible means to receive this large file, I'll contact you and then maybe you can help distribute it more widely. (Offer also applies to anyone else reading this thread.)
- aehgwdyw 1y agomagnet:?xt=urn:btih:19db177fa7f13515e11c23e7c694419e875adfd8&xt=urn:btmh:1220ff0a57b459dae436d6c425721e04240aad55545a56bbfb5371d8c21ce125d7a9&dn=dataset.zst
- 1y ago
- naikrovek 1y agoSite says the file is restricted for me. Dang.