3 ms·
We ran into similar issues with aggressive crawling. What helped was rate limiting combined with making intent explicit at the entry point, instead of letting r
by Simplita 9mo ago
We ran into similar issues with aggressive crawling. What helped was rate limiting combined with making intent explicit at the entry point, instead of letting requests fan out blindly. It reduced both load and unexpected edge cases.
- cheshire_cat 9mo agoWhat do you mean by "making intent explicit at the entry point"?
- deleted 9mo ago[deleted]