2 ms·
Yup. Recently built something that needs to do low volume scraping. About 40% success rate - rest hits bot detection even on first try
by Havoc 9mo ago
Yup. Recently built something that needs to do low volume scraping. About 40% success rate - rest hits bot detection even on first try
- ninjagoo 9mo agoDid you have rate limits built in? Ultimately scraping tools will need to mimic humans. Ironic. I wonder if bots/ai will need to build their own specialized internet for faster sharing of data, with human centered interfaces to human spaces.
- fc417fc802 9mo agoIPFS and IPNS already exist.
- Havoc 9mo agoYeah it gets blocked literally on first fetch so not a rate limit issue.