3 ms·
"... it's constantly under siege by metasearch engines." How does one know it's metasearch engines. Which ones are they. Have there been any discussions with
by 1vuio0pswjnm7 4y ago
"... it's constantly under siege by metasearch engines."
How does one know it's metasearch engines. Which ones are they. Have there been any discussions with them.
If a web search engine is truly noncommercial and one wants as may folks to to use it as possible, is it feasible to let others copy the crawler, the index and front end. It's like a fisherman who gives way his catch. As word spreads, he will be overwhelmed. Unless he teaches others how to do what he does, how to fish.
The problem with Google and all the Google wannabes is that they want all traffic to pass through their servers, for commercial reasons. In the beginning Google claimed their search engine would be in the academic realm. Presumably others could copy it. This turned out to be a lie. Now people cannot not seem to separate the idea of web search engine from potential money printing machine. The fisherman does not want to help anyone else to learn how to fish.
We need a noncommercial web search engine that can be replicated. It does not have to be a Google clone. A great way to stop anyone from working on web search engines is to suggest any new search engine must be as good as or better than Google, i.e., it must be huge, it must be commercial, and it must be a black box. On the contrary, a new noncommercial search engine can have humble beginnings and does not have to be anywhere near Google quality. IMO, there should be noncommercial web search engines with low traffic. They could all be instances of the same crawler, same index and same front end, they could be variations, they could be totally different. But IMO there should not be one set of servers controlled by one entity hoarding 90% of noncommercial search traffic.
- daoudc 4y agoThis is what we're building at https://mwmbl.org https://mwmbl.org
- marginalia_nu 4y ago> How does one know it's metasearch engines. Which ones are they. Have there been any discussions with them. SearXNG supports marginalia but uses the demo key. Any instance that enables marginalia support will spam the API with this key by default. I don't really mind, it's just that they need to get a key to be useful. The shared global ratelimit of the public key is (IIRC) 15 queries/minute because it's a demo key that's not intended to actually be used in any applications, just to try out the API. Maybe I should set up a daily rotating key or something to prevent this problem. I'd be all for having more search engine instances. Problem is you need specialized hardware to run a search engine, even a small one like mine. I've paid $5000 out of my own pocket to build the server I'm using, plus give or take $200/month in network and power. I'm crazy enough to take that loss. I don't expect many other people will do the same. There's really not much money in running a publicly accessible search engine.
- deleted 4y ago[deleted]
- justinclift 4y ago> SearXNG supports marginalia but uses the demo key. Yeah. Looking over the GitHub issue you raised about it, they didn't seem to really understand the problem before they closed it. I've just asked a question about it (same thing, different wording), which might work: https://github.com/searxng/searxng/discussions/2412 https://github.com/searxng/searxng/discussions/2412
- marginalia_nu 4y agoWell that issue was more about some weird bug they seem to have that causes like a dozen identical queries to fire at the same time. I ended up just putting in this: https://github.com/MarginaliaSearch/MarginaliaSearch/blob/master/code/services-satellite/api-service/src/main/java/nu/marginalia/api/svc/ResponseCache.java https://github.com/MarginaliaSearch/MarginaliaSearch/blob/ma...
- justinclift 4y agoAhhh cool, that makes sense. Good comment ascii art btw. ;D
- marginalia_nu 4y agoI like SearX(NG) and don't mind they are using my API, but I wish they would be more open to dialog. Would get much more use out of my search engine if they did.
- 1vuio0pswjnm7 4y agoHope that happens.