3 ms·

I have been thinking about this. I’d like to index the blogosphere and some knowledge base sites and stuff and exclude most of the Top1000 websites. I imagine

by sneak 2y ago

I have been thinking about this. I’d like to index the blogosphere and some knowledge base sites and stuff and exclude most of the Top1000 websites. I imagine I could do it on consumer hardware for less than $10-20k. Perhaps common crawl has done much of the heavy lifting already and I just have an indexing task.

jddj 2y ago
Plugging https://search.marginalia.nu/ https://search.marginalia.nu/
- marginalia_nu 2y ago
  Yeah I ran that on like $5k worth of consumer hardware initially. The only caveat is that you want a lot of RAM, and since consumer machines are mostly limited to 128 GB, that's a bit of a hard limit on how much you can do. In terms of storage space and compute, it's relatively fine. I'm on a single ~$17k server now, probably utilizing about 20% of it's capacity.
  - timeon 2y ago
    Naive question: would lower-level language help here?
    - marginalia_nu 2y ago
      The problem domain is sprawling and varied, so you ideally want a bit of everything. You definitely need to do a bunch of custom low level programming, as a lot of standard tools can't cope with the data volumes involved. Like I'm not even storing known URLs in a DBMS anymore, the table would be too slow to update. High level language with the ability to make downcalls to a low level language is probably a sane compromise.
PeterStuer 2y ago
Robust crawling with all the gratuitous anti-bot infra out there (requesting the rss feed that is specifically meant to be consumed by machine for a public gov site is 'protected' by Cloudflare's anti-botting. Seriously?) takes way more work than you think.
- marginalia_nu 2y ago
  The way to get around anti-bot stuff as a search engine crawler is to be upfront about what you are doing, as well as to respect robots.txt, and rate limits, and use a gratuitous crawl delay (1 sec is the bare minimum). You can register with cloudflare as a search engine crawler and they largely will let your traffic through. If you get blocked by individual sites, you can usually just email them explaining the situation and get unblocked. I'm having very little issues with this.
6510 2y ago
I haven't looked at it for over a decade but the p2p search engine YaCy is very old and it worked just fine. Something similar shouldn't be to hard to make. We are spoiled with tools now. The sales pitch is simple, you download the crawler, point it at your own blog index it and build an index of pages that your blog links to. Then index the pages those pages link to. etc You simply crank up the depth whenever you like it. If you are a half decent blogger you have articles that link to most of the important websites that fit the subject of your blog. You put a search box/page on your website that connects to your desktop client, your visitors can search with options: - articles on this blog, - related pages you've linked to, - 1-5 depth to broaden the topical search (but less related articles) - search other instances It scales so well because searching your own blog is the most important, linked pages is pretty nice to have, deeper crawls are still useful but much less important and searching other instances, the anti climax if you like, is great but the least important. The most crappy hardware can do 50 000 per day, if you run it slowly in the background [say] 100 pages on average per day is still 36 500 every year. More usual is to be excited about the new found tool and run it for a few hours the first day. You are initially shocked how useful it is. The next day you crawl a few more pages until you get bored with it. You look again after a while and do one more good crawl. Few years later and you have an oddly large index. You might want to run it automatically when your rss updates. If you use it once in a while it is easy to ban some instances full of spam. YaCy checks all results returned by other nodes by fetching the html and looking for the keywords on the page. This worked well. A very stale index may reflect poorly on the node but it may also be full of material that is important to you. You would get crusty results at times but this is a feature not a bug. There is no man behind the curtain who is the big decider what you may and may not look at. If your client is not running the search box/page on your blog only does p2p but it is likely able to still search your domain. What is a lot of posts for a blog is not a lot for a crawler. You can glue all kinds of products onto this. Besides a db YaCy keeps the full text of all pages crawled but only the text. If users want a feature that cant be done for free you can sell it to them. If someone has a website that is hard to index they can customize their crawler themselves or pay to have it done. If you want to throw money at it and have a blog search engine you can limit the results by things that have an rss or atom feed.

by sneak 2y ago

jddj 2y ago

Plugging https://search.marginalia.nu/ https://search.marginalia.nu/

marginalia_nu 2y ago
Yeah I ran that on like $5k worth of consumer hardware initially. The only caveat is that you want a lot of RAM, and since consumer machines are mostly limited to 128 GB, that's a bit of a hard limit on how much you can do. In terms of storage space and compute, it's relatively fine. I'm on a single ~$17k server now, probably utilizing about 20% of it's capacity.
- timeon 2y ago
  Naive question: would lower-level language help here?
  - marginalia_nu 2y ago
    The problem domain is sprawling and varied, so you ideally want a bit of everything. You definitely need to do a bunch of custom low level programming, as a lot of standard tools can't cope with the data volumes involved. Like I'm not even storing known URLs in a DBMS anymore, the table would be too slow to update. High level language with the ability to make downcalls to a low level language is probably a sane compromise.

PeterStuer 2y ago

Robust crawling with all the gratuitous anti-bot infra out there (requesting the rss feed that is specifically meant to be consumed by machine for a public gov site is 'protected' by Cloudflare's anti-botting. Seriously?) takes way more work than you think.

marginalia_nu 2y ago
The way to get around anti-bot stuff as a search engine crawler is to be upfront about what you are doing, as well as to respect robots.txt, and rate limits, and use a gratuitous crawl delay (1 sec is the bare minimum). You can register with cloudflare as a search engine crawler and they largely will let your traffic through. If you get blocked by individual sites, you can usually just email them explaining the situation and get unblocked. I'm having very little issues with this.

6510 2y ago

I haven't looked at it for over a decade but the p2p search engine YaCy is very old and it worked just fine. Something similar shouldn't be to hard to make. We are spoiled with tools now. The sales pitch is simple, you download the crawler, point it at your own blog index it and build an index of pages that your blog links to. Then index the pages those pages link to. etc You simply crank up the depth whenever you like it. If you are a half decent blogger you have articles that link to most of the important websites that fit the subject of your blog. You put a search box/page on your website that connects to your desktop client, your visitors can search with options: - articles on this blog, - related pages you've linked to, - 1-5 depth to broaden the topical search (but less related articles) - search other instances It scales so well because searching your own blog is the most important, linked pages is pretty nice to have, deeper crawls are still useful but much less important and searching other instances, the anti climax if you like, is great but the least important. The most crappy hardware can do 50 000 per day, if you run it slowly in the background [say] 100 pages on average per day is still 36 500 every year. More usual is to be excited about the new found tool and run it for a few hours the first day. You are initially shocked how useful it is. The next day you crawl a few more pages until you get bored with it. You look again after a while and do one more good crawl. Few years later and you have an oddly large index. You might want to run it automatically when your rss updates. If you use it once in a while it is easy to ban some instances full of spam. YaCy checks all results returned by other nodes by fetching the html and looking for the keywords on the page. This worked well. A very stale index may reflect poorly on the node but it may also be full of material that is important to you. You would get crusty results at times but this is a feature not a bug. There is no man behind the curtain who is the big decider what you may and may not look at. If your client is not running the search box/page on your blog only does p2p but it is likely able to still search your domain. What is a lot of posts for a blog is not a lot for a crawler. You can glue all kinds of products onto this. Besides a db YaCy keeps the full text of all pages crawled but only the text. If users want a feature that cant be done for free you can sell it to them. If someone has a website that is hard to index they can customize their crawler themselves or pay to have it done. If you want to throw money at it and have a blog search engine you can limit the results by things that have an rss or atom feed.