3 ms·

Ask HN: Open Source search engine proposal. Would you pay for data files?

8 points by fxtentacle 5y ago

frogperson 5y ago
If spammers/advertisers are willing to pay for adwords, then why would adding a fee to index data keep them out? wouldn't some sort of group moderation be more effective?
- fxtentacle 5y ago
  Thanks for your comment, because that shows me that I need to better explain this concept. The idea is that you have configuration files where you - the user - write down all the domains, companies, etc, that you want to get search results from. So if you never put that spammer's domain into your local config file, you will never see their results. But since managing all those config files by yourself might be a lot of work, and because less technical people probably can't do it, you can alternatively subscribe to someone else's configuration file so that whatever they add to their config is added to your config automatically, too.
fxtentacle 5y ago
dang, I hope having the "Ask HN" prefix is OK here. It's basically a very long lead-up to the actual Ask HN question: Would you be willing to pay for the creation of search index data files? (But for people to have an opinion, I thought that some context is needed)
- fsflover 5y ago
  > Would you be willing to pay for the creation of search index data files? Those can also be created by crowdsourcing: https://yacy.net https://yacy.net.
  - fxtentacle 5y ago
    Fully agree. With an open file format, people using yacy to create their own index would be welcome and actually help all of us towards having better search. But I believe that for kicking off such a project, it first needs to exceed a critical mass of search coverage on mainstream targets. That's why I am interested to see if it could be executed while being financially self-sufficient. Taking on VCs early would probably undermine the project, so bootstrapping seems like the way to go. And that means you need paying customers from day 1. Relying on friendly volunteers, on the other hand, seems like quite the risky strategy. EDIT: BTW, looking at their screenshots, it looks like Yacy is planning for a very different scale. I was actually considering indexing pages in the 10mio to 50mio URL range. But all their screenshots show <10K URLs, so that would be mostly niche results.
    - fsflover 5y ago
      < But all their screenshots show <10K URLs, so that would be mostly niche results. Those are just demonstrations. According to the live stats, >10mio URLs is totally possible: https://yacy.searchlab.eu/Network.html https://yacy.searchlab.eu/Network.html.
      - fxtentacle 5y ago
        It appears that you're the only one who read the text and commented, so apparently interest for federated search is very low on HN.
        fsflover 5y ago
        It might also be that it was just a bad timing. A large number of submissions do not get to the main page just due to a bad luck. Later, some of them are resubmitted and get the deserved attention. See also: https://news.ycombinator.com/pool https://news.ycombinator.com/pool.
- fxtentacle 5y ago
  In case someone else wonders in the future. Apparently, submitting it like this, it did not show up in https://news.ycombinator.com/ask https://news.ycombinator.com/ask
  - fsflover 5y ago
    This page is only for submissions without links (AFAIK even if they do not have "Ask HN").
mrpf1ster 5y ago
I like the idea and would gladly pay money for a open source federated search engine that isn’t awful. Lately I have been admiring the efforts to make the web less “corporatized” with the open, decentralized projects of the fediverse (but not all the useless blockchain stuff that’s usually associated with “decentralized”). I think this project is a necessary step towards the future of the web, but I can’t comment on the financial viability of it.
- fxtentacle 5y ago
  I'm kind of approaching this from the other side: For my use case, the "corporatized" search results tend to be awful, because they include whoever wants to be included. But most people who SEO optimize heavily and spend a lot on AdWords are precisely the kind of people I do not want to see.
fxtentacle 5y ago
FYI changed the URL to https://elefound.com/posts/elefound-the-imaginary-open-source-web-search/ https://elefound.com/posts/elefound-the-imaginary-open-sourc... for the tech angle so that I can add a 2nd article exploring the business angle [1]. [1] https://news.ycombinator.com/item?id=30374611 https://news.ycombinator.com/item?id=30374611
everforward 5y ago
It depends on the size and performance of tooling. If I can query the data files from in browser, then I'd totally pay for them. I suspect they'll be large though, and require actual infrastructure to support. Probably a mid-sized VPS at minimum. At that point, I don't think I'd want to buy them anymore. It's going to cost more than the new crop of premium paid search engines, and I still have to do the maintenance myself. It makes more sense to me as B2B operation than a B2C, as long as infra is required.

8 points by fxtentacle 5y ago

If spammers/advertisers are willing to pay for adwords, then why would adding a fee to index data keep them out? wouldn't some sort of group moderation be more effective?

fxtentacle 5y ago
Thanks for your comment, because that shows me that I need to better explain this concept. The idea is that you have configuration files where you - the user - write down all the domains, companies, etc, that you want to get search results from. So if you never put that spammer's domain into your local config file, you will never see their results. But since managing all those config files by yourself might be a lot of work, and because less technical people probably can't do it, you can alternatively subscribe to someone else's configuration file so that whatever they add to their config is added to your config automatically, too.

fxtentacle 5y ago

dang, I hope having the "Ask HN" prefix is OK here. It's basically a very long lead-up to the actual Ask HN question: Would you be willing to pay for the creation of search index data files? (But for people to have an opinion, I thought that some context is needed)

fsflover 5y ago
> Would you be willing to pay for the creation of search index data files? Those can also be created by crowdsourcing: https://yacy.net https://yacy.net.
- fxtentacle 5y ago
  Fully agree. With an open file format, people using yacy to create their own index would be welcome and actually help all of us towards having better search. But I believe that for kicking off such a project, it first needs to exceed a critical mass of search coverage on mainstream targets. That's why I am interested to see if it could be executed while being financially self-sufficient. Taking on VCs early would probably undermine the project, so bootstrapping seems like the way to go. And that means you need paying customers from day 1. Relying on friendly volunteers, on the other hand, seems like quite the risky strategy. EDIT: BTW, looking at their screenshots, it looks like Yacy is planning for a very different scale. I was actually considering indexing pages in the 10mio to 50mio URL range. But all their screenshots show <10K URLs, so that would be mostly niche results.
  - fsflover 5y ago
    < But all their screenshots show <10K URLs, so that would be mostly niche results. Those are just demonstrations. According to the live stats, >10mio URLs is totally possible: https://yacy.searchlab.eu/Network.html https://yacy.searchlab.eu/Network.html.
    - fxtentacle 5y ago
      It appears that you're the only one who read the text and commented, so apparently interest for federated search is very low on HN.
      - fsflover 5y ago
        It might also be that it was just a bad timing. A large number of submissions do not get to the main page just due to a bad luck. Later, some of them are resubmitted and get the deserved attention. See also: https://news.ycombinator.com/pool https://news.ycombinator.com/pool.
fxtentacle 5y ago
In case someone else wonders in the future. Apparently, submitting it like this, it did not show up in https://news.ycombinator.com/ask https://news.ycombinator.com/ask
- fsflover 5y ago
  This page is only for submissions without links (AFAIK even if they do not have "Ask HN").

mrpf1ster 5y ago

I like the idea and would gladly pay money for a open source federated search engine that isn’t awful. Lately I have been admiring the efforts to make the web less “corporatized” with the open, decentralized projects of the fediverse (but not all the useless blockchain stuff that’s usually associated with “decentralized”). I think this project is a necessary step towards the future of the web, but I can’t comment on the financial viability of it.

fxtentacle 5y ago
I'm kind of approaching this from the other side: For my use case, the "corporatized" search results tend to be awful, because they include whoever wants to be included. But most people who SEO optimize heavily and spend a lot on AdWords are precisely the kind of people I do not want to see.

fxtentacle 5y ago

FYI changed the URL to https://elefound.com/posts/elefound-the-imaginary-open-source-web-search/ https://elefound.com/posts/elefound-the-imaginary-open-sourc... for the tech angle so that I can add a 2nd article exploring the business angle [1]. [1] https://news.ycombinator.com/item?id=30374611 https://news.ycombinator.com/item?id=30374611

everforward 5y ago

It depends on the size and performance of tooling. If I can query the data files from in browser, then I'd totally pay for them. I suspect they'll be large though, and require actual infrastructure to support. Probably a mid-sized VPS at minimum. At that point, I don't think I'd want to buy them anymore. It's going to cost more than the new crop of premium paid search engines, and I still have to do the maintenance myself. It makes more sense to me as B2B operation than a B2C, as long as infra is required.