3 ms·
Text is tiny, but database rows get expensive far faster than the raw storage costs. For one thing, you basically can't run any database for small data like th
by londons_explore 2y ago
Text is tiny, but database rows get expensive far faster than the raw storage costs.
For one thing, you basically can't run any database for small data like this direct from hard disk. Even for ancient data, where say a 100 byte chat message is read once per year on average, a 12 TB hard drive stores 120000000000 messages needs to provide 3800 IOPS to handle that load. No hard drive does that.
- 0cf8612b2e1e 2y agoSince the data is all cold and immutable, do you even need the database anymore? Especially if you are locking down access to paid plans only (paying clients with performance expectations would still require live db). Shard the old data by community/age into separate parquet files you can retrieve as needed. You have it just in case, but then you are just paying for storage.