6 ms·

At least you get indexed by DDG. Curvio has been live for 2 months now, Google has indexed over 2000 of our pages, and we're the #1 result for over 100 differen

by tansey 15y ago

At least you get indexed by DDG. Curvio has been live for 2 months now, Google has indexed over 2000 of our pages, and we're the #1 result for over 100 different Google queries. Meanwhile, DDG still thinks you're searching for "Cuervo" [1] and has not even indexed our homepage, which has hundreds of backlinks at this point. [1] http://duckduckgo.com/?q=curvio http://duckduckgo.com/?q=curvio

sjs382 15y ago
Yeah, getting indexed seems like a problem with DDG sometimes. Is there a way to submit a sitemap.xml or something similar?
- deleted 15y ago
  [deleted]
- tansey 15y ago
  Nope. They explicitly say they do not plan to accept sitemaps.
  - oldstrangers 15y ago
    Kind of silly, but then again I've never tried to make a search engine.
_delirium 15y ago
I think that might boil down to Bing apparently not having indexed you. From what I can tell, DDG gets a decent proportion of its search results from the Yahoo BOSS API, which I believe is these days mostly feeding Bing data. The results there look pretty similar, anyway: http://www.bing.com/search?q=curvio http://www.bing.com/search?q=curvio
- AznHisoka 15y ago
  This is the problem... In the short run, DDG may be winning small victories. In the long run, they definitely need to build their own index. They need to at least start now, then turn the switch on when it's ready. Of course it's gonna require major infrastructure, but they are building a search engine, so that's a given.
  - minwcnt5 15y ago
    Building their own index is only worthwhile if they can build a good ranking algorithm to go along with it. And they're at a huge disdvantage in doing so because they don't track users to improve quality whereas Bing does.
    - alecco 15y ago
      it could be an open source project, i'd be interested
      - TomAnthony 15y ago
        You could start by playing around with this data: http://commoncrawl.org/ http://commoncrawl.org/ The ways you can interact with it currently are limited, but I think that'll improve.
        alecco 15y ago
        That project looks amazing. But it'd be nicer to have some sort of compressed raw dump to use different tools.

by tansey 15y ago

sjs382 15y ago

Yeah, getting indexed seems like a problem with DDG sometimes. Is there a way to submit a sitemap.xml or something similar?

deleted 15y ago
[deleted]
tansey 15y ago
Nope. They explicitly say they do not plan to accept sitemaps.
- oldstrangers 15y ago
  Kind of silly, but then again I've never tried to make a search engine.

_delirium 15y ago

I think that might boil down to Bing apparently not having indexed you. From what I can tell, DDG gets a decent proportion of its search results from the Yahoo BOSS API, which I believe is these days mostly feeding Bing data. The results there look pretty similar, anyway: http://www.bing.com/search?q=curvio http://www.bing.com/search?q=curvio

AznHisoka 15y ago
This is the problem... In the short run, DDG may be winning small victories. In the long run, they definitely need to build their own index. They need to at least start now, then turn the switch on when it's ready. Of course it's gonna require major infrastructure, but they are building a search engine, so that's a given.
- minwcnt5 15y ago
  Building their own index is only worthwhile if they can build a good ranking algorithm to go along with it. And they're at a huge disdvantage in doing so because they don't track users to improve quality whereas Bing does.
  - alecco 15y ago
    it could be an open source project, i'd be interested
    - TomAnthony 15y ago
      You could start by playing around with this data: http://commoncrawl.org/ http://commoncrawl.org/ The ways you can interact with it currently are limited, but I think that'll improve.
      - alecco 15y ago
        That project looks amazing. But it'd be nicer to have some sort of compressed raw dump to use different tools.