Like I said, even with compression, gzip in your example, you can only load up so much data into the physical ram modules. Reduce the list to domains and cut ou

Like I said, even with compression, gzip in your example, you can only load up so much data into the physical ram modules. Reduce the list to domains and cut out dupes, minimize the schema, kick least recently used entries, do whatever you can to drop memory usage, but it's a n+1 problem if memory usage is linked to an unknown sized set of block lists. A buffered pager that streams from disk to shifts through the list as it iterates and checks is going to need to take place if you can't hold it all in memory. Though, even with limited memory, if you're reading off fast storage, like nvme, the small amount of memeory shouldn't really matter much, as even reading the files from disk should be plenty fast enough for your average internet connection imo