9 ms·

Ask HN: Are there any purely keyword-based search engines with no NLP?

• blauditore 9y ago
  Not sure if that's what you're looking for, but SymbolHound[1] takes input queries literally, which is useful for e.g. code search. [1]: http://symbolhound.com/ http://symbolhound.com/
  • gregknicholson 9y ago
    That looks useful. Thanks! I can use it via DuckDuckGo too: https://duckduckgo.com/bang?q=symbolhound https://duckduckgo.com/bang?q=symbolhound
• ChuckMcM 9y ago
  No :-). For what its worth, Google never worked that way either, it used Page Rank initially to sort results by social importance. I would guess that the last keyword based engine of any merit was AltaVista. AV had the issue of returning all pages with your words but it had no notion of sorting them (at least initially, you can read their patent as well) These days its fairly easy to code up an n-gram index and host it on a moderate server. If you have a corpus of documents you can see how well it works. The simplest corpus is to just down load all of Wikipedia. Last time I checked it fit on a 2 TB disk drive. You could also use the Common Crawl database and index it as you would like, or talk to the Internet Archive project for some sort of collaborative project. However I will warn you that of the people who have done this experiment (and it was a new hire task at both Blekko and Google where I worked) folks quickly discovered it wasn't very useful.
  • bogomipz 9y ago
    Since you appear to have a background that includes search and Google I thought I would ask you or anyone else a tangential question - with more than 30 Trillion sites[1] on the internet how is the crawling ever completed before it needs to be re-indexed again? I'm imagining that "the frontier" a crawler needs to crawl is actually a distributed queue and that crawlers are massively parallelized. I'm also imagining the frontier is bucketed by its indexing frequency i.e daily, weekly monthly etc. Is that close? Might you have any resources on how this is architected at large search providers? [1] https://venturebeat.com/2013/03/01/how-google-searches-30-trillion-web-pages-100-billion-times-a-month/ https://venturebeat.com/2013/03/01/how-google-searches-30-tr...
    • jedberg 9y ago
      The crawl is never done. It's constantly crawling. And you're right, it's just a queue of websites that's been found, and when a crawling machine has an available slot, it crawls the next site (obviously that's simplified). Certain pages get crawled more often than others. As a website owner, you can tell Google how often your content changes, which they will use as a clue for how often to recrawl your site. If you're really big (like for example Reddit), you actually lose that control in Google dashboard -- they fully control the crawl rate. From experience, I can tell you that they are crawling large sites, like Reddit, continuously. Even 7 years ago, when I last worked there, they were crawling Reddit so much that we had to set up a separate server infrastructure just to respond to Google's requests, because their access patterns were so different from every other user.
      • owenversteeg 9y ago
        That's fascinating. Is there any way to contact Google to get them to change their crawl patterns? Or a more efficient way to directly send them the site updates as they happen? I imagine a "direct line" to Google would save resources both on Google's side and yours, but I've never ran a site big enough for that to be a problem so I wouldn't know.
        • himlion 9y ago
  • giancarlostoro 9y ago
    To be fair wikipedia compressed without images is pretty small. A couple tens of gigs no more than 80. You can download it to your phone and there are apps to view it all. Not sure how larger it is decompressed but 2 TB sounds like overkill. But yeah wikipedia would be a good way to test it. Also did you mean every single wikipedia site or just the English one? I'm thinking just the English one's sufficient.
    • tomcooks 9y ago
      I know what you mean with "sufficient",but English pages are often lacking informations on a lot of subjects related to countries outside the anglosphere. I still don't understand why wikipedia articles aren't translated according to a common script, at least where there are featured translations (marked a star)
      • zeckalpha 9y ago
        Feel free to make that happen.
    • WalterGR 9y ago
      Also, did he mean the entire revision history of Wikipedia, or just the most recent versions of pages?
      • giancarlostoro 9y ago
        That would definitely explain 2TB haha but yeah, if you pull the recent English version without images it's like ... I wanna say 30GB off the top of my head. As for anyone curious: https://dumps.wikimedia.org/backup-index.html https://dumps.wikimedia.org/backup-index.html
        • 3131s 9y ago
          The most recent was 13.9GB compressed and ~65GB uncompressed. The version without images, talk pages, and revision history is 'enwiki-20180320-pages-articles.xml.bz2' here: https://dumps.wikimedia.org/enwiki/20180320/ https://dumps.wikimedia.org/enwiki/20180320/ The article text contains Wikipedia markup which is a bit difficult to remove but not impossible, there are some existing projects for doing that. DBPedia would have the raw text, but it's not nearly as current.
  • bluecat22 9y ago
    Can you elaborate why setting up a simple search engine based on common crawl was not useful?
    • rpedela 9y ago
      Basic keyword search is great at recall but precision (top 10) gets worse as the number of documents increases. Given the size of the web, basic keyword search tends to perform poorly in terms of relevance. Common Crawl is large enough to see this problem. I think what OP and several people in this thread actually want is Google search minus synonyms and the ability to specify advanced syntax like AND and NEAR queries. I believe that would go a long way to satisfying someone who says they just want "keyword search".
      • ChuckMcM 9y ago
        You are exactly right, and specific keyword search allows you to weaponize a search engine so Google doesn't let you do it. (they have been victimized in the past for letting people specify things that could pull out social security numbers, for example.
  • burfog 9y ago
    Google did work that way. Page rank was used for ranking, which is fine. Page rank did not override the search query, which is what it now does. Our complaint is that the results we get from our queries simply do not match the queries we are making.
• bobosha 9y ago
  a purely keyword search would not very useful. BTW semantic search engines can do exact keyword matches enclosing your query using double quotes e.g. https://www.google.com/search?q="keyword+search+engine" https://www.google.com/search?q="keyword+search+engine".
  • efdee 9y ago
    Sadly, results #1 and #2 link to a page where the phrase "keyword search engine" isn't even present. So much for exact keyword matches.
  • mc32 9y ago
    But keywords with ranges would be useful in many cases. keyword1 w5 keyword2 [...] [find pages where keyword1 is within 5 words' distance of keyword2]
    • earenndil 9y ago
      Google can do this. keyword1 AROUND(5) keyword2.
      • exebook 9y ago
        This gives me pages with the word "around" instead
• dest 9y ago
  Yacy?
• neuralk 9y ago
  I was wishing for old-school keyword search just this morning! Nice synchronicity. I must completely disagree with the other posters claiming that keyword searches are not useful. For niche research, they are extremely helpful or even necessary. Google and Bing have reached the point where it is impossible to do real, niche academic research on them. For instance, I had a very specific thing I was trying to look up involving medicine, religion, and Marco Polo. Try searching for "marco polo doctors" on Google and witness it giving very counterintuitive, one-sided results that may align with the current zeitgeist of interest from people searching Google, but diverge completely with the aim of literal, precise keyword search needed by academics. I did work to improve or hone the search down, looking up Kublai Khan, doctors, atheism brings up blogspam articles on doctors and atheism, but scant results on the 13th century Mongol emperor's religious medical interest. Trying to narrow the search further by including variations on Cambulac, Cambaliech, trying to find any info beyond surface-level on John of Montecorvino and his retinue... all is impossible with search engines in 2018.
  • aldoushuxley001 9y ago
    Kind of off topic, but did you ever find out Kublai Khan's religious medicinal interests? I'd be curious to know.
    • neuralk 9y ago
      Not yet! I'm still on the hunt, but now I am searching for books. I suspect it will require checking out a few tomes from library to find what I'm looking for.
      • mitchdoogle 9y ago
        It might be easier to email a professor who's an expert in Khan to point you in the right direction. If you're lucky, they might know exactly the information you want.
  • deleted 9y ago
    [deleted]
• roadbeats 9y ago
  Kozmos' (https://getkozmos.com https://getkozmos.com) public search engine is keyword based. You can search about 1 million pages -including their content-, sorted as how much they're liked (a.k.a bookmarked) by the users. You can watch my 1-minute pitch in which I mention why Kozmos' search functionality is more interesting: https://youtube.com/watch?v=ETjeEz5Dk_M https://youtube.com/watch?v=ETjeEz5Dk_M We have academician/researchers users who love Kozmos. My goal is to improve search with deep learning, keeping the sorting algorithm same (like count) though. I'm walking towards this goal slowly but surely! P.S If you're into this topic and live in Europe (Berlin), let's have coffee!
  • deleted 9y ago
    [deleted]
• metastable 9y ago
  Have you tried Google’s “Verbatim” mode? It’s listed under search tools and claims to do exactly this, iirc.
  • alok-g 9y ago
    It used to work better.
  • tgb 9y ago
    intext:MYSEARCHTERM is what Google taught a few years ago when they had a "google poweruser web course" thing. My understanding is that these days it either doesn't work consistently or makes Google think you're a spambot and will shut you out after just a couple searches.
• bagswatchesus 9y ago
  The simplest corpus is to just down load all of Wikipedia. Last time I checked it fit on a 2 TB disk drive. http://www.lvbagsen.com/louis-vuitton-handbags/men.html http://www.lvbagsen.com/louis-vuitton-handbags/men.html
• elorant 9y ago
  This can't work for a simple reason. Spam. What's stopping anyone from flooding a page with specific keywords which might be completely unrelated to the content of the page, or the page itself providing too little real info on the searched keyword. It's exactly for reasons like these that Google considers a variety of factors for ranking pages.
  • gregknicholson 9y ago
    OK, so some sites are “garbage in”. I'd like an option to see the “garbage out”, unfiltered. Then maybe I'll appreciate the clever filtering.
• ILikeConemowk 9y ago
  If Google is an option, put your keywords in quotes.
  • 0xcde4c3db 9y ago
    Google sometimes ignores this, for reasons that nobody seems to actually understand.
    • Falling3 9y ago
      Really? I haven't seen that before; do you have an example?
      • 0xcde4c3db 9y ago
        Exact occurrences seem to vary over time, per user/account, the presence/absence of other features being used (e.g. time range or site: operator), and possibly other factors. It's periodically reported on the Google Search and Assistant Help Forum, and there's a 100+-post thread [1] about it. The behavior is like something in Google's stack is incorrectly matching a cached set of results that had the same set of terms without quotation marks. [1] https://productforums.google.com/forum/#!topic/websearch/6gHVUEl8y1k https://productforums.google.com/forum/#!topic/websearch/6gH...
    • KMag 9y ago
      I used to be an engineer on Google's search indexing team. I don't have any specific inside information in this case, but knowing the Google mindset, I'm sure they would have pulled up a large unbiased frequency-weighted sample of search queries with quotes in them. The would have then spun up a custom web search front-end that used an algorithm that sometimes ignored quotes, and ran all of the sampled queries through both the regular front-end and the custom front-end. They then would have manually looked through the affected queries. They would have used this to determine whether the change seemed to better serve the user's intent on average. Of course, this involves a bit of subjective inferring of the user's intent. It don't like it, but I'm sure that for the average query using quotes, it does better find what a user is looking for. That's why Google does this. On the other hand, there's a lot of observation bias: I'm much less likely to notice Google is ignoring quotes when it works out well. It's mostly in the frustrating cases that I really notice and remember noticing that Google sometimes ignores quotes. As for what black magic allows them to sometimes determine that you don't really want quotes, you're guesses would probably be as good as mine. I'm just pretty sure that on average, this black magic has a positive impact on search quality.
• ivank 9y ago
  This only finds things that are in some kind of RSS or Atom feed, but inoreader (with a Professional account) can keyword-search every public article that they have in their index. It doesn't return results quickly, and there is often blogspam, but it does find interesting things as well. A screenshot of the search options: https://i.imgur.com/fIzl2cp.png https://i.imgur.com/fIzl2cp.png
• joeseeder 9y ago
  duckduckgo.com ?
  • akvadrako 9y ago
    No, I use DDG every day and it's worse! They very often explicitly (it even tells you) ignores keywords unless you put every word in double quotes.
    • lancewiggs 9y ago
      " unless you put every word in double quotes." Then at least you can make it use the keywords.
• wyck 9y ago
  Nothing has changed with Google (Bing and Duckduckgo are the same too more or less). 10 years ago it still did essentially the same thing, it's up to you to use the more advanced features to filter the results. For example if you type "marco polo doctor's -doctor -who" or "marco polo doctors group:science". Google operators for example: https://en.wikipedia.org/wiki/Google_Search#Search_syntax https://en.wikipedia.org/wiki/Google_Search#Search_syntax Cheat Sheet: https://www.searchlaboratory.com/wp-content/uploads/2012/11/searchoperators.jpg https://www.searchlaboratory.com/wp-content/uploads/2012/11/...
  • WalterGR 9y ago
    Google has absolutely changed in the past 10 years. Have you tried looking for something obscure recently? Google increasingly over time takes more and more liberties with your search terms, regardless of operator use. I notice that one of the pages you link to is from 2012. Again, the old ways just don't cut it any more. Since it doesn't even mention Google's "Verbatim" search option, it suggests to me that while its content might be technically correct, it's useless to hand-wave towards it as the cure for contemporary complaints about Google's results.
    • wyck 9y ago
      Sorry to be clear, I meant that to find information you always had to do bit of digging, it's true the algorithm has changed a lot.
      • gkya 9y ago
        Not only the algorithm. Recently, using double quotes or plus sign before terms became useless, as it just lists results with the specific word you marked crossed over when there are not much exact hits.
        • Izkata 9y ago
          > Recently, using [..] plus sign before terms became useless Unless it came back at some point, it wasn't very recent - it was intentionally removed in 2011 due to their using "+" for their social network.
• lettergram 9y ago
  Oh man... This is literally my whole business, and I wrote about this here: https://austingwalters.com/is-search-solved/ https://austingwalters.com/is-search-solved/ Hinting at why search was broken. Essentially, search providers are for the general, meaning if you type "whales" it'll bring you to the wikipedia. This is because probably 80% of the time you're looking for wikipedia. It uses NLP to determine when you say "I want to know about whales" because it works in the general case. If you want an exact match do "I want to know about whales" and it'll look for that exact phrase. Now my business, is actually the reverse - not looking for the general case, but identifying the niche - i.e. "what an expert would want": https://projectpiglet.com/ https://projectpiglet.com/ This lets me build a financial advisor which is averaging over 100% YoY returns because it identifies and tracks specific topics (as opposed to just wikipedia changes). So if you go on there and type "Iran", you'll get a lot of search results about Iran, but also about Isreal, Jordan and the like because it identifies Iran being associated in the graph. This works great for investing because you want to know about the related topics (you may not even realized you wanted to know about). Now, that's NLP. But it works for my customers, because exact matches typically are not what people want. They want the Niche, the general, or occasionally as in your case an exact match (if they can think of the right words). Luckily you have quotes "search phrase", in my system I always assume you mean to type "search phrase", so I always look for an exact match. But I still apply NLP to the results, because that's the value.
• paulcole 9y ago
  You really don't want this at all. If you think keyword optimized spam is bad today wait until it's not filtered out.
• EamonnMR 9y ago
  wiby.me seems to do just that.
• pooya13 9y ago
  Kind of off topic but I recently realized that after searching for the exact same thing multiple times in a short period google was giving me different results (presumably assuming I had not found what I was looking for). Which was kind of annoying since I was just using google as the entry point to those top results.
• yesimahuman 9y ago
  My first startup needed that, specifically querying for exact hits for names, companies, etc. The inference that Google did at the time was too inaccurate for this use case (not to mention my use case being against their ToS). I ended up building a custom engine with Lucene/Solr and creating a custom Tokenizer to be more strict
• drharby 9y ago
  Hows duck duck go in this realm?
  • dleslie 9y ago
    It honours a limited but useful search syntax. https://duck.co/help/results/syntax https://duck.co/help/results/syntax
  • gregknicholson 9y ago
    I use DuckDuckGo. I have been frustrated several times by it trying to be too clever at second-guessing me. In those instances my plan B is Google, and I find that sometimes Google has more useful results. (Of course, since I only use Google when DDG's results are poor, I would expect to see Google's results be superior a lot of the time, irrespective of whether their results are generally better or worse.)
• 3131s 9y ago
  Here's one for code that I always hoped would grow, but I'm not sure if it's been updated for a few years. http://symbolhound.com/ http://symbolhound.com/
• saintPirelli 9y ago
  If you can deal with the German UI you could check out labarama.com. I wouldn't know it if I hadn't met the developers on a night out. It's an altruistic project from Austria that keeps to the basics.
  • jhoechtl 9y ago
    So you know more about these guys? I have heard some rumour that the ones behind this search engine are living in constant fear being raid by Google henchmans
• OriPekelman 9y ago
  CommonSearch https://github.com/commonsearch/ https://github.com/commonsearch/ was a project to try and build a search engine where you could "Explain" (in the SQL sense) the result, based on Common Crawl. Open Source and transparent. But it did not seem to have gathered much enthusiasm. Which I find sad. If you have some loose change on you.. a bit of processing on 71TB of data.. and you got yourself an index precisely like you want it. Anyway, without "some" NLP no search engine is going to be very useful. You need to know how to tokenize.. at a minimum. For many languages, this is not as trivial as it is for English.
• z3t4 9y ago
  i still add meta keywords in the hopes of some day they'll be used by search engines ...
• bluecat22 9y ago
  Look into commoncrawl.org which provides a free web index which you can query against. Now that cloud is available, you could in theory download the index and load it into Google's big query or AWS and run your experiments.
• nercht12 9y ago
  ask.com used to work that way. That was more or less why Google replaced Askjeeves as the top engine: Google would throw at you everything whereas Ask would give you only exact matches. It seems to have changed now, though. Too little too late.
• iagovar 9y ago
  Did you tried qwant? (best use lite.qwant.com the main sain is a bit overloaded with info) They have their own crawler apparently.