3 ms·
Well, you could implement an LPM trie that focusses on FQDN labels; for example. Which would greatly reduce the memory usage compared to what AdGuard Home does
by cookiengineer 3y ago
Well, you could implement an LPM trie that focusses on FQDN labels; for example. Which would greatly reduce the memory usage compared to what AdGuard Home does currently with their per-rulelist-slices.
If you represent LPM tries as JSON and order them by alphabet, you'll get even benefits of gzip on HDD to save lots of space. Just a suggestion and how I would implement it with an eBPF module, not meant as a critic.
- wutwutwat 3y agoLike I said, even with compression, gzip in your example, you can only load up so much data into the physical ram modules. Reduce the list to domains and cut out dupes, minimize the schema, kick least recently used entries, do whatever you can to drop memory usage, but it's a n+1 problem if memory usage is linked to an unknown sized set of block lists. A buffered pager that streams from disk to shifts through the list as it iterates and checks is going to need to take place if you can't hold it all in memory. Though, even with limited memory, if you're reading off fast storage, like nvme, the small amount of memeory shouldn't really matter much, as even reading the files from disk should be plenty fast enough for your average internet connection imo