6 ms·

I like the idea of searching a curated list of domains, but I'm not sure that doing the curation yourself is the best approach considering the huge number of us

by p-e-w 4y ago

I like the idea of searching a curated list of domains, but I'm not sure that doing the curation yourself is the best approach considering the huge number of useful but niche websites in existence. I wonder if simply parsing all of Wikipedia (dumps are available, and so are parsers capable of handling them) and building a list of all domains used in external links would do the trick. Wikipedia already has substantial quality control mechanisms, and the resulting list should be essentially free of blog spam and other low-quality content. Wikipedia also maintains "official website" links for many topics, which can be obtained by parsing the infobox from the article.

richardsocher 4y ago
That's our strategy at you.com - we start with the most popular sites, crawl them and build apps for them, eg. you.com/apps and let users vote in search results. Full disclosure: I'm the CEO.
- ngshiheng 4y ago
  > you.com/apps and let users vote what is stopping companies/users from abusing/gaming on this system with bots?
  - richardsocher 4y ago
    Richard, CEO at you.com here. Users vote on their own preferences... not on other users' preferences. so i dont see how it will be gamed unless we start incorporating user preferences into global ones - which we would only do if we thought it worked better and isn't being gamed.
- agileAlligator 4y ago
  Your search actually performed better than Google for me on a random query. I queried both engines "What is the specific heat of alcohol?", Google threw up a rich search answer that linked to some random Mexican site that is clearly exploiting SEO [1], you.com linked me to answers.com (which is more trustworthy than random mexican website). [1]: http://elempresario.mx/sites/default/files/scith/specific-heat-of-alcohol.html http://elempresario.mx/sites/default/files/scith/specific-he...
  - charcircuit 4y ago
    When I search Google I get a rich search answer of http://hyperphysics.phy-astr.gsu.edu/hbase/Tables/sphtt.html http://hyperphysics.phy-astr.gsu.edu/hbase/Tables/sphtt.html
- rawgabbit 4y ago
  I just learned about you.com from this thread. It looks very promising.
mormegil 4y ago
The "official website" links can be retrieved in a machine-readable way from Wikidata. E.g. a completely random silly example of official websites of embassies: https://w.wiki/5T3R https://w.wiki/5T3R
- 12907835202 4y ago
  Is there documentation for rolling your own? I've been considering building my own search engine for a while for my niche topic which has <50 websites and blogs on the web. I can't tell how useful this will be but it'd be fun to give it a go.
  - abujazar 4y ago
    Elastic App Search would be well suited for something like that. It comes with a built in crawler.
  - mormegil 4y ago
    Your own what? Your own query? Sure; it's just a SPARQL query over the Wikidata data model. The documentation portal for the query service is at https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/Wikidata_Query_Help https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/... though you'd need some familiarity with Wikidata, its properties, etc. E.g. the "wdt:P856" in my query is the "official website" property on Wikidata: https://m.wikidata.org/wiki/Property:P856 https://m.wikidata.org/wiki/Property:P856
- Karrot_Kream 4y ago
  Only problem is that Wikidata is still incomplete when compared to Wikipedia itself. But yeah it's "trivial" to search it.
mormegil 4y ago
Oh, and more to the point: This is a role Wikipedia explicitly renounced, isn't it. When it became so big and the Google PageRank gave it high importance, the spam became unbearable so Wikipedia decided it needs to change the incentives and it applied the rel=nofollow to all external links, so that it could stop working as an unpaid manual spam filter for the whole internet. Sure, your new search might ignore the rel=nofollow but if you ever become big enough, the incentives of spammers would lead to a bigger spamming pressure on Wikipedia...
- p-e-w 4y ago
  That's easily fixed by relying only on protected or high-profile pages. Those already deal (mostly successfully) with spam and NPOV violations on a daily basis, so piggybacking on those protection mechanisms should yield a fairly high-quality pool of curated external links.
  - Cthulhu_ 4y ago
    While that sounds good in theory, who is to say those that can edit protected and high profile pages aren't in SEO spammers' pockets? I mean the edit history is public and there's plenty of people that actually pay attention to edits and the like so they would be found out soon enough, but still. I'm sure this is an ongoing discussion when e.g. political figures' pages are protected as well - who becomes the gatekeeper, and what is their political angle?
    - p-e-w 4y ago
      Sure, but at that point you're simply discussing Wikipedia's quality control system, which may be an interesting discussion, but has nothing to do with search engines per se. Considering that Wikipedia has become a pillar of most scientific work (imagine writing a math or computer science paper without Wikipedia – utterly unthinkable), it's safe to say that knowledgeable people have collectively decided that its quality control mechanisms are "good enough", or at least better than those of any other resource of comparable depth and breadth. And that puts Wikipedia's link pool leaps and bounds ahead of whatever dark magic current search engines are using, which mostly seems to be "funnel the entire web through some (easily gamed) heuristic".
marginalia_nu 4y ago
I've looked into this, and found wikipedia's links not to be super useful. Wikipedia prefers references that don't change, so books primarily, and beyond that websites that don't change, so academic journals where you're getting paywalled, and WaybackMachine archives of websites (even if they are still live). You aren't getting much use out of wikipedia.
groffee 4y ago
The problem with only using curated lists is that you kill discoverability of new sites, but it does have promise like we've seen with Braves 'goggles'.

by p-e-w 4y ago

richardsocher 4y ago

That's our strategy at you.com - we start with the most popular sites, crawl them and build apps for them, eg. you.com/apps and let users vote in search results. Full disclosure: I'm the CEO.

ngshiheng 4y ago
> you.com/apps and let users vote what is stopping companies/users from abusing/gaming on this system with bots?
- richardsocher 4y ago
  Richard, CEO at you.com here. Users vote on their own preferences... not on other users' preferences. so i dont see how it will be gamed unless we start incorporating user preferences into global ones - which we would only do if we thought it worked better and isn't being gamed.
agileAlligator 4y ago
Your search actually performed better than Google for me on a random query. I queried both engines "What is the specific heat of alcohol?", Google threw up a rich search answer that linked to some random Mexican site that is clearly exploiting SEO [1], you.com linked me to answers.com (which is more trustworthy than random mexican website). [1]: http://elempresario.mx/sites/default/files/scith/specific-heat-of-alcohol.html http://elempresario.mx/sites/default/files/scith/specific-he...
- charcircuit 4y ago
  When I search Google I get a rich search answer of http://hyperphysics.phy-astr.gsu.edu/hbase/Tables/sphtt.html http://hyperphysics.phy-astr.gsu.edu/hbase/Tables/sphtt.html
rawgabbit 4y ago
I just learned about you.com from this thread. It looks very promising.

mormegil 4y ago

The "official website" links can be retrieved in a machine-readable way from Wikidata. E.g. a completely random silly example of official websites of embassies: https://w.wiki/5T3R https://w.wiki/5T3R

12907835202 4y ago
Is there documentation for rolling your own? I've been considering building my own search engine for a while for my niche topic which has <50 websites and blogs on the web. I can't tell how useful this will be but it'd be fun to give it a go.
- abujazar 4y ago
  Elastic App Search would be well suited for something like that. It comes with a built in crawler.
- mormegil 4y ago
  Your own what? Your own query? Sure; it's just a SPARQL query over the Wikidata data model. The documentation portal for the query service is at https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/Wikidata_Query_Help https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/... though you'd need some familiarity with Wikidata, its properties, etc. E.g. the "wdt:P856" in my query is the "official website" property on Wikidata: https://m.wikidata.org/wiki/Property:P856 https://m.wikidata.org/wiki/Property:P856
Karrot_Kream 4y ago
Only problem is that Wikidata is still incomplete when compared to Wikipedia itself. But yeah it's "trivial" to search it.

mormegil 4y ago

Oh, and more to the point: This is a role Wikipedia explicitly renounced, isn't it. When it became so big and the Google PageRank gave it high importance, the spam became unbearable so Wikipedia decided it needs to change the incentives and it applied the rel=nofollow to all external links, so that it could stop working as an unpaid manual spam filter for the whole internet. Sure, your new search might ignore the rel=nofollow but if you ever become big enough, the incentives of spammers would lead to a bigger spamming pressure on Wikipedia...

p-e-w 4y ago
That's easily fixed by relying only on protected or high-profile pages. Those already deal (mostly successfully) with spam and NPOV violations on a daily basis, so piggybacking on those protection mechanisms should yield a fairly high-quality pool of curated external links.
- Cthulhu_ 4y ago
  While that sounds good in theory, who is to say those that can edit protected and high profile pages aren't in SEO spammers' pockets? I mean the edit history is public and there's plenty of people that actually pay attention to edits and the like so they would be found out soon enough, but still. I'm sure this is an ongoing discussion when e.g. political figures' pages are protected as well - who becomes the gatekeeper, and what is their political angle?
  - p-e-w 4y ago
    Sure, but at that point you're simply discussing Wikipedia's quality control system, which may be an interesting discussion, but has nothing to do with search engines per se. Considering that Wikipedia has become a pillar of most scientific work (imagine writing a math or computer science paper without Wikipedia – utterly unthinkable), it's safe to say that knowledgeable people have collectively decided that its quality control mechanisms are "good enough", or at least better than those of any other resource of comparable depth and breadth. And that puts Wikipedia's link pool leaps and bounds ahead of whatever dark magic current search engines are using, which mostly seems to be "funnel the entire web through some (easily gamed) heuristic".

marginalia_nu 4y ago

I've looked into this, and found wikipedia's links not to be super useful. Wikipedia prefers references that don't change, so books primarily, and beyond that websites that don't change, so academic journals where you're getting paywalled, and WaybackMachine archives of websites (even if they are still live). You aren't getting much use out of wikipedia.

groffee 4y ago

The problem with only using curated lists is that you kill discoverability of new sites, but it does have promise like we've seen with Braves 'goggles'.