3 ms·
If you unpack all of https://files.pushshift.io/reddit/comments/ https://files.pushshift.io/reddit/comments/ you have many Tb of JSONs that are just dumps of AP
by wongarsu 7y ago
If you unpack all of https://files.pushshift.io/reddit/comments/ https://files.pushshift.io/reddit/comments/ you have many Tb of JSONs that are just dumps of API responses that slowly change schema over the years. It's also an incredibly useful dataset.
In the end CPUs are fast enough and compression algorithms good enough that I would argue it doesn't really matter what format you use for storage, as long as it's reasonably easy to read back.
- et2o 7y agoIn the case of genomics, there have been at this point decades of work developing high performance file formats and there are large ecosystems of tools around them. Lots of bioinformatics is really manipulating these files. So using a supported file format makes a big difference.