To your first point, I don't see the issue. Your crawler hits a URL and gets 301. It's either redirected to a URL already in its canonical database, and therefo

To your first point, I don't see the issue. Your crawler hits a URL and gets 301. It's either redirected to a URL already in its canonical database, and therefore knows that the redirected URL is garbage, or it gets something new-- congratulations! 404 is for a resource which can't be found; if the resource can be found, 404 is wrong for scripts as much as for users. Your second point is valid, but a little past the point, I think. My only actions after seeing a 404 are to check my spelling, hit back, close the tab, or start from the site root. Maybe it would be good for my browser to redirect to / or something by default, assuming there isn't a page loaded; I'm open to ideas. I'm just saying, it's never been helpful to me to download a 1.2mb transparent png that says "Sorry your browser doesn't know what 404 means." (Or another example ripped from real life: I literally just now hit a link to a video on a third-tier media site and was rewarded with a completely different video, with little text at the top that says "Error: Not found". This is what people really use custom 404s for.)