3 ms·

What sort of sizes are we talking about? I'm thinking if it would be possible to "crowdfund" the storage-costs for a requester-pays s3 bucket for it.

• marginalia_nu 4y ago
  I'm probably producing around 250-500 Gb data/month at this point.
  • SahAssar 4y ago
    What's the cumulative size for the index to date? I'm not rich by any measure, but if it's within reach I'd probably fund the storage costs.
  • justinclift 4y ago
    Any idea how compressible that is? If it's something that compresses really well (eg text data in a database), then live compression filesystems (eg ZFS, likely others) could potentially help make that workable.
    • marginalia_nu 4y ago
      The data is either already compressed or dense binary soup, so no luck.
  • indrora 4y ago
    Reach out to Jason Scott -- textfiles.com -- and see if he knows anyone who would be interested. He might know some folks.