5 ms·

Isn't the issue, of course, that spammers have no incentive to game other search engines since they're not worth the time? Any search engine that gets big will

by klbarry 16y ago

Isn't the issue, of course, that spammers have no incentive to game other search engines since they're not worth the time? Any search engine that gets big will have the problem.

bambax 16y ago
Came here to say this but you said it before! ;-) Also, "crowd-sourced curated lists of websites" sound like the old Yahoo directory of yore. They will either become obsolete very quickly or spammers will find a way to penetrate and dominate them.
- jerf 16y ago
  I believe the Yahoo lists were actually centralized. Hypothetically a fully-distributed approach could work better, in the style of Wikipedia. Realistically, well, I have no idea but it's worth a try.
  - ehsanul 16y ago
    I agree. One particular try failed [1], but that doesn't mean it can't work. [1] http://en.wikipedia.org/wiki/Wikia_Search http://en.wikipedia.org/wiki/Wikia_Search
    - dasil003 16y ago
      Not only that, but Yahoo didn't think search was important, so it's not even a real try of enhancing search with curation. The truth is, if some company comes up with a better search engine, whatever ideas behind it are not going to sound like an obvious win up front—if they did then Google would already be doing that. Instead they'll have to create a search engine that is better, but somehow antithetical to Google's business model so that they can't just copy it, because there's no way for a startup to come up with enough resources to stay materially ahead of Google in pure search. And of course that's only half the battle; then you have to be better enough that users can be bothered to switch (or a browser deal coup). Personally I haven't found the spam problem to be nearly as bad as the echo chamber makes out. I think silicon valley types just have a good imagination about how good it could be.
  - rmc 16y ago
    Wikipedia isn't really decrentralised. It's all centralised in wikipedia.org. It's not easy to 'fork it off'/.
- jshen 16y ago
  I've been wanting to build a decentralized curated social search (buzzword soup I know). The idea is that security has to be based on trust, and adversarial search is a security problem. Here's my idea, it starts with delicious style bookmarks and a social graph. the search engine indexes the pages you've bookmarked so your bookmarks are searchable. AND, you can expand your search to include your friends bookmarks. Obviously it wouldn't be comprehensive, but it would work really well for shared interests like programming.
  - sorbus 16y ago
    But it would be useless if you're the first person in your social graph to have a need for particular information; similarly, for it to be very useful you would want your users to push almost every website they visit into it, or any website which includes useful information (otherwise you would get too many searches filled with junk, or just with nothing). It's also useless if a user doesn't have friends using it (as compared to other programs; see below), and so bootstrapping up would be a bit hard. Essentially, you're talking about something like Chrome's history search (it indexes the content of every page you visit, and allows you to do full-text searches of them), but with the ability to expand the search to other users indexes. You'll want to make it entirely painless for users to add pages - a keyboard shortcut at most - and integrate into the browser as much as possible. It's also probably a good idea to add some mechanism to identify users with reliable indexes and allow others to use them; a karma system could be used for this (add karma if a result is useful, remove it if it's not - similar to how HN works with comments). Users with really low karma (indicating that they're pushing lots of spam sites into the service) would have their results biased against in full-index searches, or outright removed without them being aware (similar to how users on HN can be killed, so that the see all their posts normally but no one else does). Disclaimer: I'm not quite awake, so take this advice with a huge grain of salt. It's just my thoughts on the idea.
    - jshen 16y ago
      " It's also probably a good idea to add some mechanism to identify users with reliable indexes and allow others to use them" Yup, social graph wasn't the right choice of words on my part. I was thinking about something like twitters "follow". For example, you could go around a follow a bunch of well known programmers. I don't like the karma idea, at least not at a global level. I think it has to be about trust, at an individual level. These are the people I trust, search their bookmarks.
- didip 16y ago
  It could have been del.icio.us with its army of users tagging keywords on every URL. The way del.icio.us does curation is smart in many level. Spammers need to create many accounts to affect tags they care about.
cdr 16y ago
Sure, it's an arms race. The issue is that Google hasn't been trying to keep up.
- SoftwareMaven 16y ago
  Google's current incentives align with those content sites more than the searcher. Those sites provide a lot of revenue for Google. Until Google starts losing searchers as a result of those sites (unlikely, because my mom is unlikely to care that eHow is massively gaming things), those sites are going to stick around.
protomyth 16y ago
True, but another part of the issue is a pure algorithmic approach can be gamed, inside a search company human curation doesn't scale, and community curation has scaling problems / seeding problems / can be gamed. I do think think that some human curation, if for nothing else to mark sites as copy spam, might be workable. Pattern matching is still humanity's territory.

by klbarry 16y ago

Isn't the issue, of course, that spammers have no incentive to game other search engines since they're not worth the time? Any search engine that gets big will have the problem.

bambax 16y ago

Came here to say this but you said it before! ;-) Also, "crowd-sourced curated lists of websites" sound like the old Yahoo directory of yore. They will either become obsolete very quickly or spammers will find a way to penetrate and dominate them.

jerf 16y ago
I believe the Yahoo lists were actually centralized. Hypothetically a fully-distributed approach could work better, in the style of Wikipedia. Realistically, well, I have no idea but it's worth a try.
- ehsanul 16y ago
  I agree. One particular try failed [1], but that doesn't mean it can't work. [1] http://en.wikipedia.org/wiki/Wikia_Search http://en.wikipedia.org/wiki/Wikia_Search
  - dasil003 16y ago
    Not only that, but Yahoo didn't think search was important, so it's not even a real try of enhancing search with curation. The truth is, if some company comes up with a better search engine, whatever ideas behind it are not going to sound like an obvious win up front—if they did then Google would already be doing that. Instead they'll have to create a search engine that is better, but somehow antithetical to Google's business model so that they can't just copy it, because there's no way for a startup to come up with enough resources to stay materially ahead of Google in pure search. And of course that's only half the battle; then you have to be better enough that users can be bothered to switch (or a browser deal coup). Personally I haven't found the spam problem to be nearly as bad as the echo chamber makes out. I think silicon valley types just have a good imagination about how good it could be.
- rmc 16y ago
  Wikipedia isn't really decrentralised. It's all centralised in wikipedia.org. It's not easy to 'fork it off'/.
jshen 16y ago
I've been wanting to build a decentralized curated social search (buzzword soup I know). The idea is that security has to be based on trust, and adversarial search is a security problem. Here's my idea, it starts with delicious style bookmarks and a social graph. the search engine indexes the pages you've bookmarked so your bookmarks are searchable. AND, you can expand your search to include your friends bookmarks. Obviously it wouldn't be comprehensive, but it would work really well for shared interests like programming.
- sorbus 16y ago
  But it would be useless if you're the first person in your social graph to have a need for particular information; similarly, for it to be very useful you would want your users to push almost every website they visit into it, or any website which includes useful information (otherwise you would get too many searches filled with junk, or just with nothing). It's also useless if a user doesn't have friends using it (as compared to other programs; see below), and so bootstrapping up would be a bit hard. Essentially, you're talking about something like Chrome's history search (it indexes the content of every page you visit, and allows you to do full-text searches of them), but with the ability to expand the search to other users indexes. You'll want to make it entirely painless for users to add pages - a keyboard shortcut at most - and integrate into the browser as much as possible. It's also probably a good idea to add some mechanism to identify users with reliable indexes and allow others to use them; a karma system could be used for this (add karma if a result is useful, remove it if it's not - similar to how HN works with comments). Users with really low karma (indicating that they're pushing lots of spam sites into the service) would have their results biased against in full-index searches, or outright removed without them being aware (similar to how users on HN can be killed, so that the see all their posts normally but no one else does). Disclaimer: I'm not quite awake, so take this advice with a huge grain of salt. It's just my thoughts on the idea.
  - jshen 16y ago
    " It's also probably a good idea to add some mechanism to identify users with reliable indexes and allow others to use them" Yup, social graph wasn't the right choice of words on my part. I was thinking about something like twitters "follow". For example, you could go around a follow a bunch of well known programmers. I don't like the karma idea, at least not at a global level. I think it has to be about trust, at an individual level. These are the people I trust, search their bookmarks.
didip 16y ago
It could have been del.icio.us with its army of users tagging keywords on every URL. The way del.icio.us does curation is smart in many level. Spammers need to create many accounts to affect tags they care about.

cdr 16y ago

Sure, it's an arms race. The issue is that Google hasn't been trying to keep up.

SoftwareMaven 16y ago
Google's current incentives align with those content sites more than the searcher. Those sites provide a lot of revenue for Google. Until Google starts losing searchers as a result of those sites (unlikely, because my mom is unlikely to care that eHow is massively gaming things), those sites are going to stick around.

protomyth 16y ago

True, but another part of the issue is a pure algorithmic approach can be gamed, inside a search company human curation doesn't scale, and community curation has scaling problems / seeding problems / can be gamed. I do think think that some human curation, if for nothing else to mark sites as copy spam, might be workable. Pattern matching is still humanity's territory.