4 ms·

I've started to build a search engine as a hobby side project (I'm three weekends in. It's very much inspired by marginalia, but with different content, curatio

I've started to build a search engine as a hobby side project (I'm three weekends in. It's very much inspired by marginalia, but with different content, curation, and UX goals. I have beefy hardware at home that can see it through.) My early crawl architecture plans actually mirror this new scheme, so it's delightful to have a bit of confirmation I'm on a mildly correct track. One of my long-running side projects has been to do a lot of web-scraping on a website with continually evolving content, that's related to the economics of a side business, and yeah—we hit a lot of those growing pains of a monolithic scraper (poor debugging, unexplained re-crawling, waiting too long for a crawl plan, rare race conditions w/ multiple scrapers), averted by this model. I kinda want to know how the big G does their crawling—I watch all the different googlebots hit my many websites, but I'm curious to see their architecture of prioritization and analysis—but a lot of the analysis is definitely the resulting sauce that should be kept somewhat proprietary and secret.

marginalia_nu 4y ago
I've noted my search engine index starts to get noticeably stale after about two months, and right now it takes 2 or so weeks to crawl. Extrapolating, that might mean that it's sustainable up to about 4x the size of my current index. Right now a lot of this work is manual. Kinda not long-term viable, but for now it's not too bad. Kick off a script every now and again and come back a week later and see how it went. After that, there needs to be some sort of prioritization. Maybe a job that randomly probes URLs and flags dead/changed links for priority re-crawling. I think using RSS feeds to detect new links is also a tool that might be useful.
- cptcobalt 4y ago
  Ah yeah, that makes sense. and there's a balance here. IMO, Google puts far too much weight on recent content (and makes it impossible to find that one post you read in 2007 and you remember the vague keywords for), but also staleness of content is also an issue. I've built native support for RSS and JSON feeds for discovery on high-quality sites, and I'm also tracking estimated content refresh rate to know when to re-fetch. (I'm at a point where I've given myself too many signals to flag against, and finding the right tuning while starting to build an index, where I'm also actively evolving the schema and figuring out how to prioritize content I return in SERPs, is a problem. My current thinking is just a series of differentiated set of rules that populate the crawl queue for different reasons—seeking appropriate breadth—but then I'll definitely hit the re-crawl issue, particularly around sites I'm scoring as high quality.)
  - marginalia_nu 4y ago
    > Ah yeah, that makes sense. and there's a balance here. IMO, Google puts far too much weight on recent content (and makes it impossible to find that one post you read in 2007 and you remember the vague keywords for), but also staleness of content is also an issue. There's actually a paradox with new content that should make it less interesting to crawl, which is that the odds content will vanish or change is inversely proportional to its age. If something has been around for 10 years, it's a fairly safe bet it will be tomorrow. If something has been around for 10 hours, it's a coin-toss. Aggressively seeking out fresh content is probably a waste. If anything, fresh content should be regarded with suspicion. Maybe probe it a few days later to make sure it's still there before adding it to the index.
- rozap 4y ago
  I'm beginning to think that there's no right answer, because the ideal crawling rules depend a lot on what type on content you're seeking, which obviously can't be known up front. I'm working on a federated search tool. I've built a few different sectors (I've called them "realms") that I care about (programming, automotive, racing, fediverse) and found that I need pretty different heuristics depending. Example: car forums circa ~2005 are a treasure trove of valuable information, but 17 year old posts about programming are (in general) less interesting. That informs a lot about how URLs need or do not need to be re-crawled.
  - marginalia_nu 4y ago
    Yeah, probably. Could also be the answer isn't more complicated than to offer up the ability to control the search a bit more. Especially with forums it's easy to figure out the post time since there's only some half-dozen forum softwares. Maybe just add the option to filter by year or something would go a long way. I do think offering different filters like that is probably a good option to having the search engine try to mind-read what you want based on spying on your historical queries and machine learning haruspicy.
    - rozap 4y ago
      Yea, I don't think the idea has any real value. It kind of falls in between a bookmarking tool and a search engine. But forming search engines from trees of other search engines is kind of cool, and I have stumbled onto knowledge that I wouldn't have otherwise found. I'm not sure what the balance is between mind reading and offering too many knobs to a user is, though.

marginalia_nu 4y ago

I've noted my search engine index starts to get noticeably stale after about two months, and right now it takes 2 or so weeks to crawl. Extrapolating, that might mean that it's sustainable up to about 4x the size of my current index. Right now a lot of this work is manual. Kinda not long-term viable, but for now it's not too bad. Kick off a script every now and again and come back a week later and see how it went. After that, there needs to be some sort of prioritization. Maybe a job that randomly probes URLs and flags dead/changed links for priority re-crawling. I think using RSS feeds to detect new links is also a tool that might be useful.

cptcobalt 4y ago
Ah yeah, that makes sense. and there's a balance here. IMO, Google puts far too much weight on recent content (and makes it impossible to find that one post you read in 2007 and you remember the vague keywords for), but also staleness of content is also an issue. I've built native support for RSS and JSON feeds for discovery on high-quality sites, and I'm also tracking estimated content refresh rate to know when to re-fetch. (I'm at a point where I've given myself too many signals to flag against, and finding the right tuning while starting to build an index, where I'm also actively evolving the schema and figuring out how to prioritize content I return in SERPs, is a problem. My current thinking is just a series of differentiated set of rules that populate the crawl queue for different reasons—seeking appropriate breadth—but then I'll definitely hit the re-crawl issue, particularly around sites I'm scoring as high quality.)
- marginalia_nu 4y ago
  > Ah yeah, that makes sense. and there's a balance here. IMO, Google puts far too much weight on recent content (and makes it impossible to find that one post you read in 2007 and you remember the vague keywords for), but also staleness of content is also an issue. There's actually a paradox with new content that should make it less interesting to crawl, which is that the odds content will vanish or change is inversely proportional to its age. If something has been around for 10 years, it's a fairly safe bet it will be tomorrow. If something has been around for 10 hours, it's a coin-toss. Aggressively seeking out fresh content is probably a waste. If anything, fresh content should be regarded with suspicion. Maybe probe it a few days later to make sure it's still there before adding it to the index.
rozap 4y ago
I'm beginning to think that there's no right answer, because the ideal crawling rules depend a lot on what type on content you're seeking, which obviously can't be known up front. I'm working on a federated search tool. I've built a few different sectors (I've called them "realms") that I care about (programming, automotive, racing, fediverse) and found that I need pretty different heuristics depending. Example: car forums circa ~2005 are a treasure trove of valuable information, but 17 year old posts about programming are (in general) less interesting. That informs a lot about how URLs need or do not need to be re-crawled.
- marginalia_nu 4y ago
  Yeah, probably. Could also be the answer isn't more complicated than to offer up the ability to control the search a bit more. Especially with forums it's easy to figure out the post time since there's only some half-dozen forum softwares. Maybe just add the option to filter by year or something would go a long way. I do think offering different filters like that is probably a good option to having the search engine try to mind-read what you want based on spying on your historical queries and machine learning haruspicy.
  - rozap 4y ago
    Yea, I don't think the idea has any real value. It kind of falls in between a bookmarking tool and a search engine. But forming search engines from trees of other search engines is kind of cool, and I have stumbled onto knowledge that I wouldn't have otherwise found. I'm not sure what the balance is between mind reading and offering too many knobs to a user is, though.