3 ms·

Thanks for your nice feedback. 3. About "parallel query processing".... I mean, each thread (index server) is responsible for only one shard/segment/partition

Thanks for your nice feedback. 3. About "parallel query processing".... I mean, each thread (index server) is responsible for only one shard/segment/partition. You need a local broker (1 per PC/node) to read client queries and dispatch them to all index server threads. Each thread processes the query on its own partition and sends back the results to the broker. The broker collects and merges the results and sends the query answer to the client. The client can be another global broker controlling serveral multicores PCs. Most of this is implemented in the the TurboPFor inverted index demo. 9. What about "global terms dictionary" ? Or 'global document-id-->numeric-id' dictionary ? I think, term dictionaries are not a problem, you can have one lexicon per PC/node (common to all local partitions) accessed from all index server threads. 10. What about gpu/fpga/asic,you know of any paper/way that can be used for a search-engine and provides enough value compared to just adding more cpu/ram ? For maximum performance, you must have the whole index in the GPU own memory, otherwise the memory transfer will be the bottleneck. All query processing like decoding and intersections must be done on the GPU.

ddorian43 10y ago
3. I was thinking more of mmaping the index, and assuming it's smaller than ram, it will always be in ram. This way all cores have access to the same index and can satisfy all queries (and not having to keep separate data structures for each core + 1 less merge). Or you mean it's faster this way (how, meaning local-memory faster than mmap in-memory?)?
- powturbo 10y ago
  Yes, you can simply mmap the index file. The index server have not to read the index at the start. The goal of query processing is to decrease the query latency, this is why it is better/necessary to do partitioning. Each query can be processed simultaneously by all threads. Intersections are very costly for long posting lists.

ddorian43 10y ago

3. I was thinking more of mmaping the index, and assuming it's smaller than ram, it will always be in ram. This way all cores have access to the same index and can satisfy all queries (and not having to keep separate data structures for each core + 1 less merge). Or you mean it's faster this way (how, meaning local-memory faster than mmap in-memory?)?

powturbo 10y ago
Yes, you can simply mmap the index file. The index server have not to read the index at the start. The goal of query processing is to decrease the query latency, this is why it is better/necessary to do partitioning. Each query can be processed simultaneously by all threads. Intersections are very costly for long posting lists.