3 ms·

Yes, I use "scrapy crawl <spider> -o <spider>.json" (see bootstrap.sh). Sphinx can not index JSON files directly, but it can index an XML stream written to stdo

Yes, I use "scrapy crawl <spider> -o <spider>.json" (see bootstrap.sh). Sphinx can not index JSON files directly, but it can index an XML stream written to stdout by a given command. So I have written a script (sphinx/xmlpipe.py), that reads the JSON files generated by scrapy and writes the crawled recipes in sphinx's XML format to stdout. There are alternatives to Sphinx? ;) * Sphinx is ridiculous fast, as you can see when searching. But even building the index takes only 340ms (from which 220ms are spend by the python script that generates the XML) for 1699 recipes on my 3 years old notebook. * Sphinx don't require a RDBMS to index documents from. It can index documents from any source. You just need to write a simple script that brings the documents in the XML format expected by sphinx. * Sphinx is not only a full text search engine. It is also a multi-value store. You can add extra information like the title and url to indexed documents. And so you don't need an additional database. * I need the ability to limit a fulltext search to sentence boundaries. I don't know if there are other fulltext search engines that can do that.

tharshan09 14y ago
I was quite surprised at how fast it was on the searches. It sounds really great and im sure I can put up with xml, I just dont like having to return HTML - but is that you preference and the way you have done it? It looks like its just return dict of results, so no reason why I cant form a json response I guess. How does a typical query look like? and how are you using the sentence boundaries (tbh im sure sure what it even is). I guess for fast text search this is perfect but im guessing any computation it cant do?
- wallunit 14y ago
  For example, if you enter the ingredients "rye whiskey" and "vermouth", following query is generated: @ingredients (rye SENTENCE whiskey) | vermouth The SENTENCE operator, basically works like the & operator, just that both operands must occur in the same sentence. That is very helpful, since the index field "ingredients" contains a list of all ingredients of the recipe separated by an "!". However Sphinx can not only do full text search. You can also filter and sort by attributes and complex expressions, that involve any attribute, the relevance from the fulll-text search, arithmetics and some built-in functions. And Sphinx does that much faster than any RDBMS does. I never managed to generate a sphinx query that took a measurable amount of time, on my 3 years old notebook. ;) The sphinx index doesn't contain any HTML. However I return HTML, from the WSGI app, that serves the AJAX calls. So I guess that is what you are talking about. It just seemed to me, it would be simpler to generate the HTML for the search results on the server-side with Python than in Javascript. By the way if you want to use Sphinx for your own project, and you have the data to index already in a MySQL or PostgreSQL database, there is no need to write a script to generate XML. Sphinx can index data from MySQL and PostgreSQL databases directly. However I was just saying that, thanks to xmlpipe support, you don't need an RDBMS just to use Sphinx. And that thanks to index attributes, Sphinx can completely replace an RDBMS in a lot of cases.
  - tharshan09 14y ago
    Great I think you have me sold :) I want to give this a try since a standard db is what I would normally use to keep the data. This would be an interesting experience. I will be reading up on sphinx and mysql etc, and do any future data scraping straight to sphinx through xmlpipe. Last question :) - Do you have any useful links you can provide on the matter of sphinx etc? Thanks.
    - wallunit 14y ago
      http://sphinxsearch.com/docs/2.0.6/ http://sphinxsearch.com/docs/2.0.6/
      - tharshan09 14y ago
        So i have been trying to get the sphinx search working on OSX today - I have indexed the search. I can search stuff in the mysql port using SQL queries but when I search using the test api scripts in php or python or the search command line tool - it always returns empty results. Not sure what is going on.
        tharshan09 14y ago
        nvm works perfectly on my vps. I guess my local machine is just a bit messed up. Dam MAMP.

tharshan09 14y ago

I was quite surprised at how fast it was on the searches. It sounds really great and im sure I can put up with xml, I just dont like having to return HTML - but is that you preference and the way you have done it? It looks like its just return dict of results, so no reason why I cant form a json response I guess. How does a typical query look like? and how are you using the sentence boundaries (tbh im sure sure what it even is). I guess for fast text search this is perfect but im guessing any computation it cant do?

wallunit 14y ago
For example, if you enter the ingredients "rye whiskey" and "vermouth", following query is generated: @ingredients (rye SENTENCE whiskey) | vermouth The SENTENCE operator, basically works like the & operator, just that both operands must occur in the same sentence. That is very helpful, since the index field "ingredients" contains a list of all ingredients of the recipe separated by an "!". However Sphinx can not only do full text search. You can also filter and sort by attributes and complex expressions, that involve any attribute, the relevance from the fulll-text search, arithmetics and some built-in functions. And Sphinx does that much faster than any RDBMS does. I never managed to generate a sphinx query that took a measurable amount of time, on my 3 years old notebook. ;) The sphinx index doesn't contain any HTML. However I return HTML, from the WSGI app, that serves the AJAX calls. So I guess that is what you are talking about. It just seemed to me, it would be simpler to generate the HTML for the search results on the server-side with Python than in Javascript. By the way if you want to use Sphinx for your own project, and you have the data to index already in a MySQL or PostgreSQL database, there is no need to write a script to generate XML. Sphinx can index data from MySQL and PostgreSQL databases directly. However I was just saying that, thanks to xmlpipe support, you don't need an RDBMS just to use Sphinx. And that thanks to index attributes, Sphinx can completely replace an RDBMS in a lot of cases.
- tharshan09 14y ago
  Great I think you have me sold :) I want to give this a try since a standard db is what I would normally use to keep the data. This would be an interesting experience. I will be reading up on sphinx and mysql etc, and do any future data scraping straight to sphinx through xmlpipe. Last question :) - Do you have any useful links you can provide on the matter of sphinx etc? Thanks.
  - wallunit 14y ago
    http://sphinxsearch.com/docs/2.0.6/ http://sphinxsearch.com/docs/2.0.6/
    - tharshan09 14y ago
      So i have been trying to get the sphinx search working on OSX today - I have indexed the search. I can search stuff in the mysql port using SQL queries but when I search using the test api scripts in php or python or the search command line tool - it always returns empty results. Not sure what is going on.
      - tharshan09 14y ago
        nvm works perfectly on my vps. I guess my local machine is just a bit messed up. Dam MAMP.