4 ms·

I'd be surprised if search engines don't penalize websites that 404 previously crawled pages. They'd want to avoid the risk of a page disappearing and causing a

I'd be surprised if search engines don't penalize websites that 404 previously crawled pages. They'd want to avoid the risk of a page disappearing and causing a poor user experience.

rpgbr 3y ago
Not necessarily 404. They could redirect (301) to another, better ranked similar content.
- lolinder 3y ago
  That doesn't work for a news site—unless they're doing a ton of duplicate stories for some reason, "similar" isn't good enough if someone is looking for an article to go with a specific headline they found through search.
causality0 3y ago
I don't know why you're being downvoted. Link rot is a cancer we should all work against.
- ghaff 3y ago
  Websites get redesigned, companies go out of business, etc. It's a nice principle that links should live on forever, but it's not very practical for a decentralized web (which many profess to be in favor of).
  - _dain_ 3y ago
    >Websites get redesigned, companies go out of business these two things are not remotely the same. if you are redesigning your website then it's your responsibility to make the old links redirect to the new pages. if you don't do that, it's a broken redesign.
- TZubiri 3y ago
  Yes. Things should not die, we should all live forever, as bits and bytes
telmo 3y ago
They clearly couldn't care less about user experience. This is all about selling ads. The web was a beautiful thing for a moment in time, but the rotting corpse that is left after "adtech" did its thing is an anti-user monstrosity, to a degree that is almost comical.
- justrealist 3y ago
  > an anti-user monstrosity The web today has 6 billion users instead of maybe 50mm in the golden age you're reminiscing about.
  - callalex 3y ago
    Can that be attributed to advancements of the web, or just the incredible advancements and price drops in the devices used to access it?
snowwrestler 3y ago
It depends on the pages. Search engines actually reward sites for 404ing previously crawled pages, if the pages had little value in the search index. Whether that is done on purpose, or is an unintended side effect of other ranking decisions, is up for debate.

I'd be surprised if search engines don't penalize websites that 404 previously crawled pages. They'd want to avoid the risk of a page disappearing and causing a poor user experience.

rpgbr 3y ago

Not necessarily 404. They could redirect (301) to another, better ranked similar content.

lolinder 3y ago
That doesn't work for a news site—unless they're doing a ton of duplicate stories for some reason, "similar" isn't good enough if someone is looking for an article to go with a specific headline they found through search.

causality0 3y ago

I don't know why you're being downvoted. Link rot is a cancer we should all work against.

ghaff 3y ago
Websites get redesigned, companies go out of business, etc. It's a nice principle that links should live on forever, but it's not very practical for a decentralized web (which many profess to be in favor of).
- _dain_ 3y ago
  >Websites get redesigned, companies go out of business these two things are not remotely the same. if you are redesigning your website then it's your responsibility to make the old links redirect to the new pages. if you don't do that, it's a broken redesign.
TZubiri 3y ago
Yes. Things should not die, we should all live forever, as bits and bytes

telmo 3y ago

They clearly couldn't care less about user experience. This is all about selling ads. The web was a beautiful thing for a moment in time, but the rotting corpse that is left after "adtech" did its thing is an anti-user monstrosity, to a degree that is almost comical.

justrealist 3y ago
> an anti-user monstrosity The web today has 6 billion users instead of maybe 50mm in the golden age you're reminiscing about.
- callalex 3y ago
  Can that be attributed to advancements of the web, or just the incredible advancements and price drops in the devices used to access it?

snowwrestler 3y ago

It depends on the pages. Search engines actually reward sites for 404ing previously crawled pages, if the pages had little value in the search index. Whether that is done on purpose, or is an unintended side effect of other ranking decisions, is up for debate.