5 ms·
The Evolution of Marginalia's Crawling
- mariusor 4y agoI'd be curious if the new architecture can be adapted to use different content types, other than HTML. I think the Fediverse (even if it can be crawled as the normal web pages) it would benefit for a custom crawler that can jump from inbox to inbox and understands (at least some) ActivityPub jsonld.
- marginalia_nu 4y agoYeah, that should be fairly doable. The beauty of the design, since it's mediated by a portable language of JSON-objects, is that you can in principle replace or extend any of the steps, including support for other protocols (like gemini:// ?) or content-types.
- sfsdfsdfasfasd 4y agoIt's very simple to write a basic ActivityPub crawler (I have done this), but you'd go from outbox to outbox, not inbox to inbox. Regardless, it's all just JSON, so it is more straightforward to crawl than the web. I tried writing a crawler to map the Fediverse (to discover homeservers) but I discovered quickly that most of the content in the Fediverse is extremist content, both left-wing and right-wing -- it's basically all tankies and Nazis, with a few techies in a bubble using mastodon.social, who think Fediblock is a solution to this problem. Oh right, and there are a ton of pedo instances too, mostly based in Japan where "loli" is legal. Due to the way ActivityPub publishes federated content, I didn't feel comfortable running a homeserver, because a pedo instance could federate illegal content to my homeserver and get me arrested and charged for possession of content I did not request and do not want to store on my computers, and if I'm not monitoring what content is getting federated to my timeline, I might not even know it's there. Too risky. Demoralized, I abandoned the project, and I don't think much about ActivityPub anymore.
- mariusor 4y ago> because a pedo instance could federate illegal content to my homeserver and get me arrested and charged for possession of content I did not request and do not want to store on my computers, and if I'm not monitoring what content is getting federated to my timeline, I might not even know it's there. Too risky. I also work in the activitypub space and my solution to avoiding unlawful content is to make the "federating" relationship between two servers a mutual one. One asks to follow, the other replies favourably (or not). This puts a damper on the "wild west" of everyone federating with everyone, but ensures content is curated at a basic level by a SysOP so there are no surprises.
- toa697 4y agoOut of curiosity, whats the crawl speed of both marginalia crawlers? I had an inspiration to try and spin my own crawler after reading some other posts on the marginalia search. (it runs very dumbly, just pulling links from an ever increasing in-memory set) And on a single thread with asynchronous web requests and a massive pool of async workers (10k, ram is cheap on a personal machine). I've been able to reach around 300-400 requests per second, pulling the page, parsing for <a> tags, and throwing the href on the stack to search. I find the use of that many bespoke threads to be really surprising. Both because of the increased complexity of threads over async code, and my (possibly naive) expectation that web traffic will always out-bottleneck cpu bound tasks like HTML parsing/lexing/tagging etc. I'll admit that I've been dragging my feet on implementing any proper parsing of my own, so I don't have any comparison to draw from. (Tried SQLite, clogged up my async code too much with blocking ops and im not excited to try a second time yet)
- marginalia_nu 4y agoIn practice, maybe 40-50rps (peaking at 100) for the first design, and 300rps for the second. Although I'm serving search engine traffic from the same machine, so I'm trying to leave ample bandwidth for that. If I go too fast the NAT starts dropping packets and refusing connections, and that's not great for crawling or serving.
- billyhoffman 4y ago@marginalia, Can you talk more about how you prioritize the URL frontier?
- marginalia_nu 4y agoI use a combination of incoming links and the average ranking of these linking sites to add new sites to the crawl queue. It's not super sophisticated, and I think it matters less the bigger the crawl is.
- stinkytaco 4y agoHow do you keep it "indie"? Even fairly small sites probably link to Forbes or the Atlantic once in a while. Do you have a specific "block" list to keep large, commercial sites out of the results?
- marginalia_nu 4y agoI'll crawl those sites too. Most of them will be weeded out in the processing stage, where I exclude websites that have too much heavy duty javascript and tracking and so on. I only index about 20% of the documents I fetch. Some still slip by, but the ranking algorithm takes care of the rest. I'm using personalized pagerank[1] biased toward a set of real human websites, which turns out to rather aggressively promote human websites. [1] http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf (see ch. 6)
- cptcobalt 4y agoI've started to build a search engine as a hobby side project (I'm three weekends in. It's very much inspired by marginalia, but with different content, curation, and UX goals. I have beefy hardware at home that can see it through.) My early crawl architecture plans actually mirror this new scheme, so it's delightful to have a bit of confirmation I'm on a mildly correct track. One of my long-running side projects has been to do a lot of web-scraping on a website with continually evolving content, that's related to the economics of a side business, and yeah—we hit a lot of those growing pains of a monolithic scraper (poor debugging, unexplained re-crawling, waiting too long for a crawl plan, rare race conditions w/ multiple scrapers), averted by this model. I kinda want to know how the big G does their crawling—I watch all the different googlebots hit my many websites, but I'm curious to see their architecture of prioritization and analysis—but a lot of the analysis is definitely the resulting sauce that should be kept somewhat proprietary and secret.
- marginalia_nu 4y agoI've noted my search engine index starts to get noticeably stale after about two months, and right now it takes 2 or so weeks to crawl. Extrapolating, that might mean that it's sustainable up to about 4x the size of my current index. Right now a lot of this work is manual. Kinda not long-term viable, but for now it's not too bad. Kick off a script every now and again and come back a week later and see how it went. After that, there needs to be some sort of prioritization. Maybe a job that randomly probes URLs and flags dead/changed links for priority re-crawling. I think using RSS feeds to detect new links is also a tool that might be useful.
- cptcobalt 4y agoAh yeah, that makes sense. and there's a balance here. IMO, Google puts far too much weight on recent content (and makes it impossible to find that one post you read in 2007 and you remember the vague keywords for), but also staleness of content is also an issue. I've built native support for RSS and JSON feeds for discovery on high-quality sites, and I'm also tracking estimated content refresh rate to know when to re-fetch. (I'm at a point where I've given myself too many signals to flag against, and finding the right tuning while starting to build an index, where I'm also actively evolving the schema and figuring out how to prioritize content I return in SERPs, is a problem. My current thinking is just a series of differentiated set of rules that populate the crawl queue for different reasons—seeking appropriate breadth—but then I'll definitely hit the re-crawl issue, particularly around sites I'm scoring as high quality.)
- marginalia_nu 4y agoKinda wish there was more software architecture discussions. It's hard to get right, and has a huge effect on what you're able to do with your software, but it feels like many of us (myself very much included) are just sort of winging it as we go along based on fads, hunches and whatever random assortment of experiences we've had professionally, which can't be very many bigger projects no matter who you are or how long you've worked where.
- haskellandchill 4y agoI feel like there are already more software architecture discussions out there than can be read by a single person. they're just not indexed well ;)
- password4321 4y agoIs there any value in starting with Common Crawl? https://hn.algolia.com/?query=author%3Amarginalia_nu%20common%20crawl&sort=byDate&type=comment https://hn.algolia.com/?query=author%3Amarginalia_nu%20commo... > https://news.ycombinator.com/item?id=32205535#32211292 https://news.ycombinator.com/item?id=32205535#32211292 > It's simply too unwieldy. It's far easier (and cheaper) to do my own crawling at a manageable scale, than it is to work with CC's datasets. Is there any way to contribute to Common Crawl beyond donating? https://commoncrawl.org/big-picture/what-you-can-do/ https://commoncrawl.org/big-picture/what-you-can-do/
- marginalia_nu 4y agoYeah I stand by that. I just don't see what Common Crawl would actually help me with, other than making my own data more stale given it would take about as long to download the CC dataset as my own crawl takes to perform (i.e. ~200h). As it stands, crawling isn't the hard part of building a search engine. Don't get me wrong, if you're doing data science and want to access a crawl data set, Common Crawl is amazing.
- haskellandchill 4y agosomething about this search is a winner for me, it's killed it for a few terms I've thrown in like "applicative languages" and "low level"
- ramyarjahani 4y ago
- ramyarjahani 4y ago[dead]