5 ms·

Hey, great project - the more competition in this space the better. To be honest, at the moment the algorithm doesn't return any sensible results for anything (

by Closi 5y ago

Hey, great project - the more competition in this space the better. To be honest, at the moment the algorithm doesn't return any sensible results for anything (at least that I can find), but I hope that you can find a way past this as it's a great place to have a project. I've included some search terms below that I've tried - I've not cherrypicked these and believe they are indicative of current performance. Some of these might be the size of the index - however I suspect it's actually how the search is being parsed/ranked (in particular I think the top two examples show that). > Search "best car brands" Expected: Car Reviews Returns a page showing the best mobile phone brands. then... > Then searching "Best Mobile Phone" Expected: The article from the search above. Returns a gizmodo page showing the best apps to buy... "App Deals: Discounted iOS iPhone, iPad, Android, Windows Phone Apps" > Searching "What is a test?" Expected result: Some page describing what a test is, maybe wikipedia? Returns "Test could confirm if Brad Pitt does suffer from face blindness" > Searching "Duck Duck Go" Expected result: DDG.com Returns "There be dragons? Why net neutrality groups won't go to Congress" > Searching "Google" Expected result: Google.com Returns: An article from the independent, "Google has just created the world’s bluest jeans"

daoudc 5y ago
Thanks for the feedback! I'll take a look at your examples and see if I can improve the rankings.
- devoutsalsa 5y ago
  The first thing I do to test a search engine is to search for my own username on various public sites to see if it can find me. It didn’t find me. But keep it up and I’m sure I’ll be in there eventually (or maybe a overestimate how interesting I am, hehe).
  - rmbyrro 5y ago
    I get this is your usual testikg case for search engines, but if you've read their README you'd have seen its inaproppriate for the project at the current stage.
- GraemeMeyer 5y ago
  Fun idea. It seems to be getting stuck on the first word you enter. e.g. you get the same results for “London” as you do for “London cats”, “London cat rescue” and “London test”.
clay-dreidels 5y ago
What does a search engine algorithm look like, and where can I find examples to build from?
- marginalia_nu 5y ago
  Depends on which algorithm you are looking for, but these are commonly used: * Okapi BM25 for determining the relevance of a result to a query. * TF-IDF for determining the relevance of a term to a document. * PageRank for ranking domains.
rmbyrro 5y ago
I guess that's the real problem. People like to wonder what would be the "ideal world" in a search engine. It may be wishful thinking, I don't know. It seems really hard to produce quality search results. Takes a lot of investment. Makes it an expensive product. But no one wants to pay. So selling ads it's the only way forward. Maybe there's a way to convince people to pay what it takes? I dunno...
- bigyikes 5y ago
  I would gladly pay $5 a month for a Google-quality search service that doesn’t track me. I’ve been using Duck Duck Go for most of this year, but frequently find myself falling back to !g because Google’s results really are much better. I wonder how much money Google search makes per the average user. Is it more than $5/mo?
  - danuker 5y ago
    > because Google’s results really are much better. Or maybe they're average, but you only see the ones where DDG fails. Next time also try Yandex and Baidu.
  - _benj 5y ago
    I've been enjoying a lot neeva, not affiliated at all, just a happy user :-) I think they are $4.95/mo or something? haven't payed a cent yet since there are a few discounts that they do to prompt you to learn how to use it (I really liked that, and def made me more likely to stick with it!)
  - berkut 5y ago
    Likewise, I'd pay that as well (I'd like no ads as well, but tracking is the main issue for me). Similarly, I do have DDG as my main search on all machines and devices just out of principle, but its region-aware searching (I'm in NZ, and often only want NZ results) are very close to useless in my experience (with NZ as the region ticked, it will still return results from .ca and .co.uk domains, which I would have hoped would be almost trivial to remove), and Google seems much better in this area (but not perfect). Similarly, there's often technical/programming things I'll search for that DDG doesn't have indexed at all, and Google does. Google also seems a lot better at ignoring spelling differences (color/colour, favorite/favourite) than DDG, which is often (but not always!) useful.
gjm11 5y ago
I was curious and tried a bunch of other searches, with similarly disappointing results. My searches were a bit more esoteric than Closi's. "langlands program" (pure mathematics thing): yup, top result is indeed related to the Langlands program, though it isn't obviously what anyone would want as their first result for that search. Not bad. "asmodeus" (evil spirit in one of the deuterocanonical books of the Bible, features extensively in later demonology, name used for an evil god in Dungeons & Dragons, etc.): completely blank page, no results, no "sorry, we have no results" message, nothing. Not good. "clerihew" (a kind of comic biographical short poem popular in the late 19th / early 20th century): completely blank page. Not good. "marlon brando" (Hollywood actor): first few results are at least related to the actor -- good! -- but I'd have expected to see something like his Wikipedia or IMDB page near the top, rather than the tangentially related things I actually god. "b minor mass" (one of J S Bach's major compositions): nothing to do with Bach anywhere in the results; putting quotation marks around the search string doesn't help. "top quark" (fundamental particle): results -- of which there were only 7 -- do seem to be about particle physics, and in some cases about the top quark, but as with Marlon Brando they're not exactly the results one would expect. "ferrucio busoni" (composer and pianist): blank page. "dry brine goose" (a thing one might be interested in doing at this time of year): five results, none relevant; top two were about Untitled Goose Game. "alphazero" (game-playing AI made by Google): blank page. Putting a space in results in lots of results related to the word "alpha", none of which has anything to do with AlphaZero. OK, let's try some more mainstream things. "harry potter": blank page. Wat. Tried again; did give some results this time. They are indeed relevant to Harry Potter, though the unexpected first-place hit is Eric Raymond's rave review of Eliezer Yudkowsky's "Harry Potter and the Methods of Rationality", which I am fairly sure is not what Google gives as its first result for "harry potter" :-). "iphone 12" (confession: I couldn't remember what the current generation was, and actually this is last year's): top results are all iPhone-related, but first one is about the iPhone 6, second is from 2007, this is about the iPhone 6, fourth is from 2007, fifth is about the iPhone 4S, etc. "pfizer vaccine": does give fairly relevant-looking results, yay.
- daoudc 5y ago
  Thanks for the detailed feedback! I think most of most of the problems here are because we have a really small index right now. Increasing the number of documents is our top priority. I agree that some kind of feedback when there are no results would be a good idea.

by Closi 5y ago

daoudc 5y ago

Thanks for the feedback! I'll take a look at your examples and see if I can improve the rankings.

devoutsalsa 5y ago
The first thing I do to test a search engine is to search for my own username on various public sites to see if it can find me. It didn’t find me. But keep it up and I’m sure I’ll be in there eventually (or maybe a overestimate how interesting I am, hehe).
- rmbyrro 5y ago
  I get this is your usual testikg case for search engines, but if you've read their README you'd have seen its inaproppriate for the project at the current stage.
GraemeMeyer 5y ago
Fun idea. It seems to be getting stuck on the first word you enter. e.g. you get the same results for “London” as you do for “London cats”, “London cat rescue” and “London test”.

clay-dreidels 5y ago

What does a search engine algorithm look like, and where can I find examples to build from?

marginalia_nu 5y ago
Depends on which algorithm you are looking for, but these are commonly used: * Okapi BM25 for determining the relevance of a result to a query. * TF-IDF for determining the relevance of a term to a document. * PageRank for ranking domains.

rmbyrro 5y ago

I guess that's the real problem. People like to wonder what would be the "ideal world" in a search engine. It may be wishful thinking, I don't know. It seems really hard to produce quality search results. Takes a lot of investment. Makes it an expensive product. But no one wants to pay. So selling ads it's the only way forward. Maybe there's a way to convince people to pay what it takes? I dunno...

bigyikes 5y ago
I would gladly pay $5 a month for a Google-quality search service that doesn’t track me. I’ve been using Duck Duck Go for most of this year, but frequently find myself falling back to !g because Google’s results really are much better. I wonder how much money Google search makes per the average user. Is it more than $5/mo?
- danuker 5y ago
  > because Google’s results really are much better. Or maybe they're average, but you only see the ones where DDG fails. Next time also try Yandex and Baidu.
- _benj 5y ago
  I've been enjoying a lot neeva, not affiliated at all, just a happy user :-) I think they are $4.95/mo or something? haven't payed a cent yet since there are a few discounts that they do to prompt you to learn how to use it (I really liked that, and def made me more likely to stick with it!)
- berkut 5y ago
  Likewise, I'd pay that as well (I'd like no ads as well, but tracking is the main issue for me). Similarly, I do have DDG as my main search on all machines and devices just out of principle, but its region-aware searching (I'm in NZ, and often only want NZ results) are very close to useless in my experience (with NZ as the region ticked, it will still return results from .ca and .co.uk domains, which I would have hoped would be almost trivial to remove), and Google seems much better in this area (but not perfect). Similarly, there's often technical/programming things I'll search for that DDG doesn't have indexed at all, and Google does. Google also seems a lot better at ignoring spelling differences (color/colour, favorite/favourite) than DDG, which is often (but not always!) useful.

gjm11 5y ago

I was curious and tried a bunch of other searches, with similarly disappointing results. My searches were a bit more esoteric than Closi's. "langlands program" (pure mathematics thing): yup, top result is indeed related to the Langlands program, though it isn't obviously what anyone would want as their first result for that search. Not bad. "asmodeus" (evil spirit in one of the deuterocanonical books of the Bible, features extensively in later demonology, name used for an evil god in Dungeons & Dragons, etc.): completely blank page, no results, no "sorry, we have no results" message, nothing. Not good. "clerihew" (a kind of comic biographical short poem popular in the late 19th / early 20th century): completely blank page. Not good. "marlon brando" (Hollywood actor): first few results are at least related to the actor -- good! -- but I'd have expected to see something like his Wikipedia or IMDB page near the top, rather than the tangentially related things I actually god. "b minor mass" (one of J S Bach's major compositions): nothing to do with Bach anywhere in the results; putting quotation marks around the search string doesn't help. "top quark" (fundamental particle): results -- of which there were only 7 -- do seem to be about particle physics, and in some cases about the top quark, but as with Marlon Brando they're not exactly the results one would expect. "ferrucio busoni" (composer and pianist): blank page. "dry brine goose" (a thing one might be interested in doing at this time of year): five results, none relevant; top two were about Untitled Goose Game. "alphazero" (game-playing AI made by Google): blank page. Putting a space in results in lots of results related to the word "alpha", none of which has anything to do with AlphaZero. OK, let's try some more mainstream things. "harry potter": blank page. Wat. Tried again; did give some results this time. They are indeed relevant to Harry Potter, though the unexpected first-place hit is Eric Raymond's rave review of Eliezer Yudkowsky's "Harry Potter and the Methods of Rationality", which I am fairly sure is not what Google gives as its first result for "harry potter" :-). "iphone 12" (confession: I couldn't remember what the current generation was, and actually this is last year's): top results are all iPhone-related, but first one is about the iPhone 6, second is from 2007, this is about the iPhone 6, fourth is from 2007, fifth is about the iPhone 4S, etc. "pfizer vaccine": does give fairly relevant-looking results, yay.

daoudc 5y ago
Thanks for the detailed feedback! I think most of most of the problems here are because we have a really small index right now. Increasing the number of documents is our top priority. I agree that some kind of feedback when there are no results would be a good idea.