4 ms·
Ah yeah, that makes sense. and there's a balance here. IMO, Google puts far too much weight on recent content (and makes it impossible to find that one post you
by cptcobalt 4y ago
Ah yeah, that makes sense. and there's a balance here. IMO, Google puts far too much weight on recent content (and makes it impossible to find that one post you read in 2007 and you remember the vague keywords for), but also staleness of content is also an issue.
I've built native support for RSS and JSON feeds for discovery on high-quality sites, and I'm also tracking estimated content refresh rate to know when to re-fetch.
(I'm at a point where I've given myself too many signals to flag against, and finding the right tuning while starting to build an index, where I'm also actively evolving the schema and figuring out how to prioritize content I return in SERPs, is a problem. My current thinking is just a series of differentiated set of rules that populate the crawl queue for different reasons—seeking appropriate breadth—but then I'll definitely hit the re-crawl issue, particularly around sites I'm scoring as high quality.)
- marginalia_nu 4y ago> Ah yeah, that makes sense. and there's a balance here. IMO, Google puts far too much weight on recent content (and makes it impossible to find that one post you read in 2007 and you remember the vague keywords for), but also staleness of content is also an issue. There's actually a paradox with new content that should make it less interesting to crawl, which is that the odds content will vanish or change is inversely proportional to its age. If something has been around for 10 years, it's a fairly safe bet it will be tomorrow. If something has been around for 10 hours, it's a coin-toss. Aggressively seeking out fresh content is probably a waste. If anything, fresh content should be regarded with suspicion. Maybe probe it a few days later to make sure it's still there before adding it to the index.