2 ms·
So I am building my own web search engine
- skillplayed 3mo agoI am building my own search engine with its own index. The reason for doing so is that most privacy focused search engines function as a middleman between Google/Bing and/or fill their results with AI slop. Any feedback is highly appreciated! I just crossed 110mln results indexed today and am crawling ~500 per second.
- n1xis10t 3mo agoWow, so ~43 million a day, ~15 billion a year? That’s sick. How much space do you have on the server, and how many pages do you intend to scale to?
- not_your_vase 3mo agoNo results for "porn". Huhh, are you indexing the same internet that we are using to post these messages? Multi-word search is doing some interesting things. When I search for "4chan magic ball" I'd expect a picture with a magic 8-ball, but instead I am getting almost random, unrelated results. Similar for "vhdl flip flop" search, I'd expect to see a flip-flop implementation. Do you plan to keep it online for the long term, or is it more like for your current amusement?
- skillplayed 3mo agoHaha, by default safe search is set to on, if you turn it off it should no longer filter those results. Image search currently only uses wikimedia if that's what you mean. The index has ~120mln pages now and I expect results to get quite a lot better at around the 500mln mark. Regarding the long term I am not sure, for now it indeed is most amusing to work at.
- nenecmrf 3mo agohave my own search engine as well https://search.inquir.org/ https://search.inquir.org/ trying it on hn now https://aleksandr-kubarskii-0931e2-hn-8439d2.inquir.org/ https://aleksandr-kubarskii-0931e2-hn-8439d2.inquir.org/