3 ms·

Thanks, I really appreciate it. Already follow Lemire. I think I've seen you also on encode.ru though that's only for compression. There's also https://groups.g

Thanks, I really appreciate it. Already follow Lemire. I think I've seen you also on encode.ru though that's only for compression. There's also https://groups.google.com/forum/#!forum/mechanical-sympathy https://groups.google.com/forum/#!forum/mechanical-sympathy . Regarding the answers: 3. About "parallel query processing". You mean like using multiple cores for 1 query ? Since data is in ram I thought it would be more efficient for single thread ? Or do you mean parallel on the segments (like lucene does) ? 5. I meant what do you do to put food on the table ? Like Lemming is professor. Are you employed/contractor/business ?(just curious) 9. What about "global terms dictionary" ? Or 'global document-id-->numeric-id' dictionary ? Like a service where you send term/doc-id(say url) and get back the numeric-id? Or would those make the posting lists not compresable ? (just thinking ways of lowering overhead on each shard so they take more documents) 10. What about gpu/fpga/asic ,you know of any paper/way that can be used for a search-engine and provides enough value compared to just adding more cpu/ram ? Or they're limited too much by low memory? If yes what about having a ssd(nvme?) on the gpu like AMD intends to? Or using NVME for a ram extension, since it 'acts' like a slower+persistent ram ?

powturbo 10y ago
Thanks for your nice feedback. 3. About "parallel query processing".... I mean, each thread (index server) is responsible for only one shard/segment/partition. You need a local broker (1 per PC/node) to read client queries and dispatch them to all index server threads. Each thread processes the query on its own partition and sends back the results to the broker. The broker collects and merges the results and sends the query answer to the client. The client can be another global broker controlling serveral multicores PCs. Most of this is implemented in the the TurboPFor inverted index demo. 9. What about "global terms dictionary" ? Or 'global document-id-->numeric-id' dictionary ? I think, term dictionaries are not a problem, you can have one lexicon per PC/node (common to all local partitions) accessed from all index server threads. 10. What about gpu/fpga/asic,you know of any paper/way that can be used for a search-engine and provides enough value compared to just adding more cpu/ram ? For maximum performance, you must have the whole index in the GPU own memory, otherwise the memory transfer will be the bottleneck. All query processing like decoding and intersections must be done on the GPU.
- ddorian43 10y ago
  3. I was thinking more of mmaping the index, and assuming it's smaller than ram, it will always be in ram. This way all cores have access to the same index and can satisfy all queries (and not having to keep separate data structures for each core + 1 less merge). Or you mean it's faster this way (how, meaning local-memory faster than mmap in-memory?)?
  - powturbo 10y ago
    Yes, you can simply mmap the index file. The index server have not to read the index at the start. The goal of query processing is to decrease the query latency, this is why it is better/necessary to do partitioning. Each query can be processed simultaneously by all threads. Intersections are very costly for long posting lists.

powturbo 10y ago

Thanks for your nice feedback. 3. About "parallel query processing".... I mean, each thread (index server) is responsible for only one shard/segment/partition. You need a local broker (1 per PC/node) to read client queries and dispatch them to all index server threads. Each thread processes the query on its own partition and sends back the results to the broker. The broker collects and merges the results and sends the query answer to the client. The client can be another global broker controlling serveral multicores PCs. Most of this is implemented in the the TurboPFor inverted index demo. 9. What about "global terms dictionary" ? Or 'global document-id-->numeric-id' dictionary ? I think, term dictionaries are not a problem, you can have one lexicon per PC/node (common to all local partitions) accessed from all index server threads. 10. What about gpu/fpga/asic,you know of any paper/way that can be used for a search-engine and provides enough value compared to just adding more cpu/ram ? For maximum performance, you must have the whole index in the GPU own memory, otherwise the memory transfer will be the bottleneck. All query processing like decoding and intersections must be done on the GPU.

ddorian43 10y ago
3. I was thinking more of mmaping the index, and assuming it's smaller than ram, it will always be in ram. This way all cores have access to the same index and can satisfy all queries (and not having to keep separate data structures for each core + 1 less merge). Or you mean it's faster this way (how, meaning local-memory faster than mmap in-memory?)?
- powturbo 10y ago
  Yes, you can simply mmap the index file. The index server have not to read the index at the start. The goal of query processing is to decrease the query latency, this is why it is better/necessary to do partitioning. Each query can be processed simultaneously by all threads. Intersections are very costly for long posting lists.