3 ms·
The text you display back to your user doesn't/shouldn't have to depend on what you index in your information retrieval system. At my last job, we used to inde
by inertiatic 7y ago
The text you display back to your user doesn't/shouldn't have to depend on what you index in your information retrieval system.
At my last job, we used to index large documents on Solr.
The largest chunks of them were indexed in non-stored fields.
Which means they were searchable but you couldn't retrieve the actual text.
This drastically cut down on the index size and the resources we needed to support it.
Then after scoring you'd return the top hits to the main application as IDs, which could retrieve everything needed (full text included) along with all the other information they'd retrieve for the document and generate the actual output.