3 ms·
If you have a really huge dataset go away from sphinx, it takes forever to index and doesn't offer incremental updates, as well it crashes regularly
by postit 16y ago
If you have a really huge dataset go away from sphinx, it takes forever to index and doesn't offer incremental updates, as well it crashes regularly
- brndnhy 16y agoNo, there is no incremental mode to the indexer, but that doesn't mean it can't be done. Create a counter table to keep track of what records you've indexed and cron frequent indexer runs using the latest records as offsets. If you have documents that frequently change, maintain your full index less often and instead merge the two on a more regular basis. Sphinx's indexer speed is one of its advantages, but it's largely dependent on the efficiency of your SQL and underlying MySQL indices. Perhaps something else is indirectly influencing your indexing performance.
- getsat 16y agoWhat size dataset are you talking about? I've never seen Sphinx crash even when indexing hundreds of gigs of documents. Are there any relevant bug reports?
- thorax 16y agoYeah, we use a large dataset (so does craigslist, etc). I've not seen crashes. I'd be curious to hear more about what you ran into. The indexing speed is so very fast-- I can't say enough good things about sphinx.