4 ms·
> It's 118 gigabytes of JSON. 118.0 GB of ZST compressed JSON (https://zenodo.org/records/15170676 https://zenodo.org/records/15170676). The actual uncompresse
by diggan 1y ago
> It's 118 gigabytes of JSON.
118.0 GB of ZST compressed JSON (https://zenodo.org/records/15170676 https://zenodo.org/records/15170676). The actual uncompressed JSON would most likely be much, much larger.
- squigz 1y agoI imagine it can be reduced a fair bit simply by stripping out unneeded fields from the messages; I imagine it can be reduced even further by removing unneeded messages entirely (i.e., bot welcome messages), deduping messages, etc
- hampus 1y agoI downloaded it and decompressed it — it's approximately 2.1 terabytes in size.
- kedaiapps 1y agohey there, would you mind sharing the file please?
- diggan 1y agoI managed to download the file, but since they restricted the downloads, it seems they hid the checksums too. Could you possibly share the md5/sha256 of the .zst file?
- philipkglass 1y agosha256sum dataset.zst 0196416253fab4bce08504737bc81215927d9afdc6ccc81f75345518109266a4 dataset.zst
- aehgwdyw 1y agomagnet:?xt=urn:btih:19db177fa7f13515e11c23e7c694419e875adfd8&xt=urn:btmh:1220ff0a57b459dae436d6c425721e04240aad55545a56bbfb5371d8c21ce125d7a9&dn=dataset.zst
- topcatto 1y ago[dead]