4 ms·

I will publish the index for download in a few weeks. I'm currently working on the documentation. Oh, and I will publish the raw crawl-data too. Everything toge

by deusu 10y ago

I will publish the index for download in a few weeks. I'm currently working on the documentation. Oh, and I will publish the raw crawl-data too. Everything together is about 2.5tb. There is also a free API in beta-test right now. Will probably be ready for official release next week.

wila 10y ago
Thank you for your work. Keep going on it. Wired you a small donation as I think it is important to have alternatives.
- deusu 10y ago
  Thank you!
kowdermeister 10y ago
That's great news, thanks for the info. Sorry for sounding harsh, for being a side project this is impressive. Have you also published the ranking mechanism? That way people might contribute you to improve it.
- deusu 10y ago
  It's all open-source. So, yes.
webtechgal 10y ago
It would be great if you can share (at least) some information about the kind of hosting setup you're using, how much of bandwidth and how long it took to crawl and index the 2B pages.
- deusu 10y ago
  4 servers in total. 2 are used for crawling, index-building and raw-data storage. Quadcore, 32gb RAM, 4tb HDD and 1gbit/s internet connection on each of these. They are rented and in a big data-center. Crawling uses "only" about 200-250mbit/s of bandwidth. 2 servers for webserver and queries. Quadcore, 32gb RAM. One with 2x512gb SSD, the other with only 1x512gb SSD. These servers are here at home. I have cable internet with 200mbit/s down, 20mbit/s up. Static IPs obviously. A full crawl currently takes about 3 months.
adrianratnapala 10y ago
I think such publication is very important. I have no idea at all if open source search engines could ever work. But if they can, I think a big part of it would be separation of the crawl index and the UI / prioritisation etc. Different people can work on those two ends of the problem and apply different philosophies. Search only forums? Reject porn using XYZ method? Great! But they can all use the same database, or pick from the a common community of databases.

by deusu 10y ago

wila 10y ago

Thank you for your work. Keep going on it. Wired you a small donation as I think it is important to have alternatives.

deusu 10y ago
Thank you!

kowdermeister 10y ago

That's great news, thanks for the info. Sorry for sounding harsh, for being a side project this is impressive. Have you also published the ranking mechanism? That way people might contribute you to improve it.

deusu 10y ago
It's all open-source. So, yes.

webtechgal 10y ago

It would be great if you can share (at least) some information about the kind of hosting setup you're using, how much of bandwidth and how long it took to crawl and index the 2B pages.

deusu 10y ago
4 servers in total. 2 are used for crawling, index-building and raw-data storage. Quadcore, 32gb RAM, 4tb HDD and 1gbit/s internet connection on each of these. They are rented and in a big data-center. Crawling uses "only" about 200-250mbit/s of bandwidth. 2 servers for webserver and queries. Quadcore, 32gb RAM. One with 2x512gb SSD, the other with only 1x512gb SSD. These servers are here at home. I have cable internet with 200mbit/s down, 20mbit/s up. Static IPs obviously. A full crawl currently takes about 3 months.

adrianratnapala 10y ago

I think such publication is very important. I have no idea at all if open source search engines could ever work. But if they can, I think a big part of it would be separation of the crawl index and the UI / prioritisation etc. Different people can work on those two ends of the problem and apply different philosophies. Search only forums? Reject porn using XYZ method? Great! But they can all use the same database, or pick from the a common community of databases.