2 ms·
> For the 404s (assuming the status code isn't a 4xx) Most are a 4xx code, I checked myself, some may be 301/302 redirect to 4xx not being handled properly by
by KomoD 3y ago
> For the 404s (assuming the status code isn't a 4xx)
Most are a 4xx code, I checked myself, some may be 301/302 redirect to 4xx not being handled properly by their crawler
- phgn 3y agoGood point. We're using https://crawlee.dev https://crawlee.dev, I think there's a way to handle more status codes as errors... Right now it only excludes pages based on the text content: https://github.com/lindylearn/aboutideasnow/blob/main/apps/api/src/common/filter.ts https://github.com/lindylearn/aboutideasnow/blob/main/apps/a...