3 ms·

This isn't an issue with needing a fork that uses less memory. You're loading hundreds of thousands, if not millions of urls into memory so they can be compared

This isn't an issue with needing a fork that uses less memory. You're loading hundreds of thousands, if not millions of urls into memory so they can be compared to every single dns request happening. Text, even compressed, requires a certain amount of bytes to store, and there is no way to use less memory beyond that point, it's literally, physically impossible. Are there memory gains in adguard? Probably. But I doubt enough to magically make your 128MB of memory defy physics and hold more than 128MB of bytes. Doesn't matter what those bytes are, or how efficiently you made them, that physical rock can only ever handle 128MB of those bytes

cookiengineer 3y ago
Well, you could implement an LPM trie that focusses on FQDN labels; for example. Which would greatly reduce the memory usage compared to what AdGuard Home does currently with their per-rulelist-slices. If you represent LPM tries as JSON and order them by alphabet, you'll get even benefits of gzip on HDD to save lots of space. Just a suggestion and how I would implement it with an eBPF module, not meant as a critic.
- wutwutwat 3y ago
  Like I said, even with compression, gzip in your example, you can only load up so much data into the physical ram modules. Reduce the list to domains and cut out dupes, minimize the schema, kick least recently used entries, do whatever you can to drop memory usage, but it's a n+1 problem if memory usage is linked to an unknown sized set of block lists. A buffered pager that streams from disk to shifts through the list as it iterates and checks is going to need to take place if you can't hold it all in memory. Though, even with limited memory, if you're reading off fast storage, like nvme, the small amount of memeory shouldn't really matter much, as even reading the files from disk should be plenty fast enough for your average internet connection imo

cookiengineer 3y ago

Well, you could implement an LPM trie that focusses on FQDN labels; for example. Which would greatly reduce the memory usage compared to what AdGuard Home does currently with their per-rulelist-slices. If you represent LPM tries as JSON and order them by alphabet, you'll get even benefits of gzip on HDD to save lots of space. Just a suggestion and how I would implement it with an eBPF module, not meant as a critic.

wutwutwat 3y ago
Like I said, even with compression, gzip in your example, you can only load up so much data into the physical ram modules. Reduce the list to domains and cut out dupes, minimize the schema, kick least recently used entries, do whatever you can to drop memory usage, but it's a n+1 problem if memory usage is linked to an unknown sized set of block lists. A buffered pager that streams from disk to shifts through the list as it iterates and checks is going to need to take place if you can't hold it all in memory. Though, even with limited memory, if you're reading off fast storage, like nvme, the small amount of memeory shouldn't really matter much, as even reading the files from disk should be plenty fast enough for your average internet connection imo