4 ms·

Non-ASCII characters are indeed messed up. I wrote the HTML parser about 15 years ago. Back then my programming style was so bad, that I'm afraid nothing short

by deusu 11y ago

Non-ASCII characters are indeed messed up. I wrote the HTML parser about 15 years ago. Back then my programming style was so bad, that I'm afraid nothing short of a complete rewrite of the parser will be able to fix it. The Archive.org crawl is way too old to be of any use. And the Common Crawl corpus is only marginally bigger than what I crawl myself. There are currently just over a billion pages in my search-index and I can recrawl those about every 45 days. Crawling right now at about 650 URLs/s and 200mbit/s. Only about 60% of URLs crawled end up in the index. The rest are errors, timeouts, redirections, etc. I have been thinking about adding an API. Rate-limiting is a must as you have correctly said. I'm already in a constant fight against SEOs who try to scrape the search-results and who apparently don't see anything wrong in making 10+ queries/second. The API would definitly not be a free-for-all, but sort of like "send me an email and describe what you want to do with it, and if I like it I'll give you an API-key".Together with a primer on what not to do of course. :) If people do more than a certain amount of queries/month, then it would also have to be paid.

i336_ 11y ago
Ah. 2001 was... the era of HTML4, more or less, so it's still seeing the majority of page content, but... no Unicode. I see. I figured old data was uninteresting, but TIL that crawling data yourself is not impossible. I mean you have to come from a thousand IPs, browser UAs and cookie states to find everything thanks to the sad state of things, but wow, I didn't realize 200Mbps could do so much. :D As for the errors et al, that makes me think, I wonder what would happen if you saved that data and folded it into the seed list (with maybe 3 or 4 URLs up the chain leading to that link) for next run. Agh... so you already have leeches :( all I can suggest is maybe mixing up your HTML, or maybe putting your site behind CloudFlare and, if there's an option(?), setting the attack detection sensitivity all the way up. I have no idea if they can do this. 10 queries/sec... that's... not going to be too great on the hardware :( I mean these people can just crawl the info themselves and get more up-to-date content >.< wow By "free" API, I don't mean free-for-all "here, have at my bandwidth/hardware" xD - I mean more in the sense that curious shy CompSci types can maybe autogenerate themselves a basic-level API key to experiment with ultra-low-rate data requests here and there, and request a threshold upgrade on their key if they think they have an interesting/justifiable use for the data. And it would be great if this could be a revenue stream! Have you ever featured the site on HN? PS. Constructive criticism on design: I would personally alter the page styles a little; it currently looks nothing like what I associate "search engine homepage" with based on what I've learned to expect, and the page style, while very nice, makes me feel like the site is a blog, not a dynamic search engine.
- deusu 11y ago
  Crawler works from a single IP. User-Agent is fixed to the robot's UA. Cookies are totally ignored. The search-engine works with just 2 servers. Crawler/Indexer and Webserver/queries. Crawler is a root-server with 1gbit/s connection hosted in a datacenter. Webserver sits here at home with 200mbit downstream and 20mbit upstream. I use the Alexa top-1-million sites as seed-list for the crawler. The errors that do appear during the crawl are either sites that have an outage or more likely simply dead-links. Oh, and URLs that turn out to be blocked by robots.txt. There are a lot of sites out there which block anything but Google and Bing from crawling them. Cloudflare is not an option for me. It would let Cloudflare know what my users are searching for. VERY big no-no. :) I can filter out 99% of automated queries. Luckily they are still pretty dumb at the moment and give me enough fixed clues to identify them. I like your idea of keeping the API free with a very low request-rate. That could work. I would have to find a way that they can't just generate many API-keys though. Using captchas for API-key requests won't stop them from doing that. I posted a "Show HN" about a year ago. Brought in about 1500 extra visitor that day. Got up to 9th place on the HN homepage that day. New webdesign is already done. I have a German site too. https://deusu.de https://deusu.de which actually gets 90% of traffic. That site already has the new design.
  - i336_ 11y ago
    1Gbps for the crawler totally explains it: I can see that doing 600 URL/sec. xD I didn't think of CloudFlare being able to see the traffic... and wow, I never even processed that aspect of their service. But of course... How good is Google's "[ ] I'm not a robot" checkbox thingy at weeding out bots? And perhaps you could use multiple captcha systems...? (Or are actual people tasked to do signups?!) I shudder to think of such an idea, but linking API keys to <popular login-with/connect-with-this-site API> may be an alternative. (One thing that comes to mind is that, if someone authenticates using Reddit - which they can do without releasing any account info - is that you could check their (public, but unfakeable) karma counts and use that as a measure of confidence, in addition to the standard account age metric used everywhere.) The new design is nice :D And if it's been a year (!), another Show HN sometime would certainly be fine.
Hixie 11y ago
The WHATWG had a spec-compliant HTML parser in Pascal, if you need it. Currently only does UTF-8 but that should be fixable.

by deusu 11y ago

i336_ 11y ago

Ah. 2001 was... the era of HTML4, more or less, so it's still seeing the majority of page content, but... no Unicode. I see. I figured old data was uninteresting, but TIL that crawling data yourself is not impossible. I mean you have to come from a thousand IPs, browser UAs and cookie states to find everything thanks to the sad state of things, but wow, I didn't realize 200Mbps could do so much. :D As for the errors et al, that makes me think, I wonder what would happen if you saved that data and folded it into the seed list (with maybe 3 or 4 URLs up the chain leading to that link) for next run. Agh... so you already have leeches :( all I can suggest is maybe mixing up your HTML, or maybe putting your site behind CloudFlare and, if there's an option(?), setting the attack detection sensitivity all the way up. I have no idea if they can do this. 10 queries/sec... that's... not going to be too great on the hardware :( I mean these people can just crawl the info themselves and get more up-to-date content >.< wow By "free" API, I don't mean free-for-all "here, have at my bandwidth/hardware" xD - I mean more in the sense that curious shy CompSci types can maybe autogenerate themselves a basic-level API key to experiment with ultra-low-rate data requests here and there, and request a threshold upgrade on their key if they think they have an interesting/justifiable use for the data. And it would be great if this could be a revenue stream! Have you ever featured the site on HN? PS. Constructive criticism on design: I would personally alter the page styles a little; it currently looks nothing like what I associate "search engine homepage" with based on what I've learned to expect, and the page style, while very nice, makes me feel like the site is a blog, not a dynamic search engine.

deusu 11y ago
Crawler works from a single IP. User-Agent is fixed to the robot's UA. Cookies are totally ignored. The search-engine works with just 2 servers. Crawler/Indexer and Webserver/queries. Crawler is a root-server with 1gbit/s connection hosted in a datacenter. Webserver sits here at home with 200mbit downstream and 20mbit upstream. I use the Alexa top-1-million sites as seed-list for the crawler. The errors that do appear during the crawl are either sites that have an outage or more likely simply dead-links. Oh, and URLs that turn out to be blocked by robots.txt. There are a lot of sites out there which block anything but Google and Bing from crawling them. Cloudflare is not an option for me. It would let Cloudflare know what my users are searching for. VERY big no-no. :) I can filter out 99% of automated queries. Luckily they are still pretty dumb at the moment and give me enough fixed clues to identify them. I like your idea of keeping the API free with a very low request-rate. That could work. I would have to find a way that they can't just generate many API-keys though. Using captchas for API-key requests won't stop them from doing that. I posted a "Show HN" about a year ago. Brought in about 1500 extra visitor that day. Got up to 9th place on the HN homepage that day. New webdesign is already done. I have a German site too. https://deusu.de https://deusu.de which actually gets 90% of traffic. That site already has the new design.
- i336_ 11y ago
  1Gbps for the crawler totally explains it: I can see that doing 600 URL/sec. xD I didn't think of CloudFlare being able to see the traffic... and wow, I never even processed that aspect of their service. But of course... How good is Google's "[ ] I'm not a robot" checkbox thingy at weeding out bots? And perhaps you could use multiple captcha systems...? (Or are actual people tasked to do signups?!) I shudder to think of such an idea, but linking API keys to <popular login-with/connect-with-this-site API> may be an alternative. (One thing that comes to mind is that, if someone authenticates using Reddit - which they can do without releasing any account info - is that you could check their (public, but unfakeable) karma counts and use that as a measure of confidence, in addition to the standard account age metric used everywhere.) The new design is nice :D And if it's been a year (!), another Show HN sometime would certainly be fine.

Hixie 11y ago

The WHATWG had a spec-compliant HTML parser in Pascal, if you need it. Currently only does UTF-8 but that should be fixable.