4 ms·
About your data dumps: you're about to get hammered, so please share them in torrent! This would be much better for the thousands of people wanting to bootstrap
by Aissen 15y ago
About your data dumps: you're about to get hammered, so please share them in torrent! This would be much better for the thousands of people wanting to bootstrap.
Also, I understand json is very easy to use, etc. But those big dumps cry for a binary format. Or at least add zlib/lzma compression so people don't waste bandwidth on uuencoded binary data in json.
- brianwhitman 15y agothe dumps are in a format that other code understands (fastingest.py) The code data is compressed using zlib (and then base64'd.) It's all on s3-- in our experience, big data dumps like this get relatively little traffic after the original hype dies down and a torrent doesn't make sense. We're pretty sure amazon can handle the load :)
- JonnieCache 15y agoS3 has bittorrent support built in, it's trivial to enable. But you probably knew that.