5 ms·

This is pretty awesome, and the speed is great. I really like obscure search engines, because their (comparatively :P) tiny indexes mean they'll return signifi

by i336_ 11y ago

This is pretty awesome, and the speed is great. I really like obscure search engines, because their (comparatively :P) tiny indexes mean they'll return significantly different data to what say Google or Bing would return. If there's one thing I would definitely vote for with projects like this, it would be... an API. In a world where virtually every search engine is built around the idea of controlled, non-open access, a predictable, documented, free-access API would be a breath of fresh air, and you'd get people seriously interested in this even though it's small. (You'd definitely need rate limiting though because there are always the hangers-on with the low IQs. >.>) You may (or may not, no idea) also be interested in the Common Crawl corpus - http://commoncrawl.org/the-data/get-started/ http://commoncrawl.org/the-data/get-started/ - which contains a few PB of data collected over the past few years (sorted by date). Access via HTTP is only moderately involved: read the line immediately above "Data Format." Of potentially much less interest is a 2012 80TB snapshot Archive.org are (or at least were) offering; they seeded their indexer off the Alexa top 1m and boxed up whatever came back. https://blog.archive.org/2012/10/26/80-terabytes-of-archived-web-crawl-data-available-for-research/ https://blog.archive.org/2012/10/26/80-terabytes-of-archived... It's apparently only available on request (I expect a search engine would definitely qualify, especially considering the data is 4 years old now - although I definitely want to see what's in it if you do decide to chase this - serious :P). Finally, search for "cheetah project" - the 2nd top-level result is messed up, apparently the Chinese (Unicode) got scrambled. Just a note; you may already be aware.

deusu 11y ago
Non-ASCII characters are indeed messed up. I wrote the HTML parser about 15 years ago. Back then my programming style was so bad, that I'm afraid nothing short of a complete rewrite of the parser will be able to fix it. The Archive.org crawl is way too old to be of any use. And the Common Crawl corpus is only marginally bigger than what I crawl myself. There are currently just over a billion pages in my search-index and I can recrawl those about every 45 days. Crawling right now at about 650 URLs/s and 200mbit/s. Only about 60% of URLs crawled end up in the index. The rest are errors, timeouts, redirections, etc. I have been thinking about adding an API. Rate-limiting is a must as you have correctly said. I'm already in a constant fight against SEOs who try to scrape the search-results and who apparently don't see anything wrong in making 10+ queries/second. The API would definitly not be a free-for-all, but sort of like "send me an email and describe what you want to do with it, and if I like it I'll give you an API-key".Together with a primer on what not to do of course. :) If people do more than a certain amount of queries/month, then it would also have to be paid.
- i336_ 11y ago
  Ah. 2001 was... the era of HTML4, more or less, so it's still seeing the majority of page content, but... no Unicode. I see. I figured old data was uninteresting, but TIL that crawling data yourself is not impossible. I mean you have to come from a thousand IPs, browser UAs and cookie states to find everything thanks to the sad state of things, but wow, I didn't realize 200Mbps could do so much. :D As for the errors et al, that makes me think, I wonder what would happen if you saved that data and folded it into the seed list (with maybe 3 or 4 URLs up the chain leading to that link) for next run. Agh... so you already have leeches :( all I can suggest is maybe mixing up your HTML, or maybe putting your site behind CloudFlare and, if there's an option(?), setting the attack detection sensitivity all the way up. I have no idea if they can do this. 10 queries/sec... that's... not going to be too great on the hardware :( I mean these people can just crawl the info themselves and get more up-to-date content >.< wow By "free" API, I don't mean free-for-all "here, have at my bandwidth/hardware" xD - I mean more in the sense that curious shy CompSci types can maybe autogenerate themselves a basic-level API key to experiment with ultra-low-rate data requests here and there, and request a threshold upgrade on their key if they think they have an interesting/justifiable use for the data. And it would be great if this could be a revenue stream! Have you ever featured the site on HN? PS. Constructive criticism on design: I would personally alter the page styles a little; it currently looks nothing like what I associate "search engine homepage" with based on what I've learned to expect, and the page style, while very nice, makes me feel like the site is a blog, not a dynamic search engine.
  - deusu 11y ago
    Crawler works from a single IP. User-Agent is fixed to the robot's UA. Cookies are totally ignored. The search-engine works with just 2 servers. Crawler/Indexer and Webserver/queries. Crawler is a root-server with 1gbit/s connection hosted in a datacenter. Webserver sits here at home with 200mbit downstream and 20mbit upstream. I use the Alexa top-1-million sites as seed-list for the crawler. The errors that do appear during the crawl are either sites that have an outage or more likely simply dead-links. Oh, and URLs that turn out to be blocked by robots.txt. There are a lot of sites out there which block anything but Google and Bing from crawling them. Cloudflare is not an option for me. It would let Cloudflare know what my users are searching for. VERY big no-no. :) I can filter out 99% of automated queries. Luckily they are still pretty dumb at the moment and give me enough fixed clues to identify them. I like your idea of keeping the API free with a very low request-rate. That could work. I would have to find a way that they can't just generate many API-keys though. Using captchas for API-key requests won't stop them from doing that. I posted a "Show HN" about a year ago. Brought in about 1500 extra visitor that day. Got up to 9th place on the HN homepage that day. New webdesign is already done. I have a German site too. https://deusu.de https://deusu.de which actually gets 90% of traffic. That site already has the new design.

by i336_ 11y ago

deusu 11y ago

Non-ASCII characters are indeed messed up. I wrote the HTML parser about 15 years ago. Back then my programming style was so bad, that I'm afraid nothing short of a complete rewrite of the parser will be able to fix it. The Archive.org crawl is way too old to be of any use. And the Common Crawl corpus is only marginally bigger than what I crawl myself. There are currently just over a billion pages in my search-index and I can recrawl those about every 45 days. Crawling right now at about 650 URLs/s and 200mbit/s. Only about 60% of URLs crawled end up in the index. The rest are errors, timeouts, redirections, etc. I have been thinking about adding an API. Rate-limiting is a must as you have correctly said. I'm already in a constant fight against SEOs who try to scrape the search-results and who apparently don't see anything wrong in making 10+ queries/second. The API would definitly not be a free-for-all, but sort of like "send me an email and describe what you want to do with it, and if I like it I'll give you an API-key".Together with a primer on what not to do of course. :) If people do more than a certain amount of queries/month, then it would also have to be paid.

i336_ 11y ago
Ah. 2001 was... the era of HTML4, more or less, so it's still seeing the majority of page content, but... no Unicode. I see. I figured old data was uninteresting, but TIL that crawling data yourself is not impossible. I mean you have to come from a thousand IPs, browser UAs and cookie states to find everything thanks to the sad state of things, but wow, I didn't realize 200Mbps could do so much. :D As for the errors et al, that makes me think, I wonder what would happen if you saved that data and folded it into the seed list (with maybe 3 or 4 URLs up the chain leading to that link) for next run. Agh... so you already have leeches :( all I can suggest is maybe mixing up your HTML, or maybe putting your site behind CloudFlare and, if there's an option(?), setting the attack detection sensitivity all the way up. I have no idea if they can do this. 10 queries/sec... that's... not going to be too great on the hardware :( I mean these people can just crawl the info themselves and get more up-to-date content >.< wow By "free" API, I don't mean free-for-all "here, have at my bandwidth/hardware" xD - I mean more in the sense that curious shy CompSci types can maybe autogenerate themselves a basic-level API key to experiment with ultra-low-rate data requests here and there, and request a threshold upgrade on their key if they think they have an interesting/justifiable use for the data. And it would be great if this could be a revenue stream! Have you ever featured the site on HN? PS. Constructive criticism on design: I would personally alter the page styles a little; it currently looks nothing like what I associate "search engine homepage" with based on what I've learned to expect, and the page style, while very nice, makes me feel like the site is a blog, not a dynamic search engine.
- deusu 11y ago
  Crawler works from a single IP. User-Agent is fixed to the robot's UA. Cookies are totally ignored. The search-engine works with just 2 servers. Crawler/Indexer and Webserver/queries. Crawler is a root-server with 1gbit/s connection hosted in a datacenter. Webserver sits here at home with 200mbit downstream and 20mbit upstream. I use the Alexa top-1-million sites as seed-list for the crawler. The errors that do appear during the crawl are either sites that have an outage or more likely simply dead-links. Oh, and URLs that turn out to be blocked by robots.txt. There are a lot of sites out there which block anything but Google and Bing from crawling them. Cloudflare is not an option for me. It would let Cloudflare know what my users are searching for. VERY big no-no. :) I can filter out 99% of automated queries. Luckily they are still pretty dumb at the moment and give me enough fixed clues to identify them. I like your idea of keeping the API free with a very low request-rate. That could work. I would have to find a way that they can't just generate many API-keys though. Using captchas for API-key requests won't stop them from doing that. I posted a "Show HN" about a year ago. Brought in about 1500 extra visitor that day. Got up to 9th place on the HN homepage that day. New webdesign is already done. I have a German site too. https://deusu.de https://deusu.de which actually gets 90% of traffic. That site already has the new design.