3 ms·

Just a quick non-thought-through idea but would it be possible to build an index in a way that allows clients to download only parts of it based on what they se

Just a quick non-thought-through idea but would it be possible to build an index in a way that allows clients to download only parts of it based on what they search? I.e. the search client normalizes the query in some way and then requests only the relevant index pages. The index would probably be a lot larger on the server but if disk space is not an issue...? (Though at some point you have to ask yourself what the benefits of such an approach are compared to a search server.)

jil 6y ago
I keep wanting to try that! Sharding the index would be really cool, and would definitely allow for a seemingly infinite index size. The barrier, as you were getting to, would be making sure index shard downloads could keep up with typing speed. Definitely something I'll try experimenting with in the future, so... watch this space, perhaps?
lgierth 6y ago
Merkle Search Trees: Efficient State-Based CRDTs in Open Networks https://hal.inria.fr/hal-02303490/document https://hal.inria.fr/hal-02303490/document Peer-to-Peer Ordered Search Indexes https://0fps.net/2020/12/19/peer-to-peer-ordered-search-indexes/ https://0fps.net/2020/12/19/peer-to-peer-ordered-search-inde... (which adds useful context about the above) Sorry for just dropping these links, I should already be asleep :)
- westurner 6y ago
  > Merkle Search Trees: Efficient State-Based CRDTs in Open Networks https://hal.inria.fr/hal-02303490/document https://hal.inria.fr/hal-02303490/document https://scholar.google.com/scholar?cites=7160577141569533185&as_sdt=5,43&sciodt=0,43&hl=en https://scholar.google.com/scholar?cites=7160577141569533185... ... "Merkle Hash Grids Instead of Merkle Trees" (2020) https://scholar.google.com/scholar?cluster=13503894708682701593&hl=en&as_sdt=5,43&sciodt=0,43 https://scholar.google.com/scholar?cluster=13503894708682701... Browser-side "Blockchain Certificate Transparency" applications need to support at least exact key lookup by domain/SAN and then also by cert fingerprint value; but the whole CT chain with every cert issue and revocation event is impractically large in terms of disk space. https://github.com/amark/gun#history https://github.com/amark/gun#history may also be practically useful.

jil 6y ago

I keep wanting to try that! Sharding the index would be really cool, and would definitely allow for a seemingly infinite index size. The barrier, as you were getting to, would be making sure index shard downloads could keep up with typing speed. Definitely something I'll try experimenting with in the future, so... watch this space, perhaps?

lgierth 6y ago

Merkle Search Trees: Efficient State-Based CRDTs in Open Networks https://hal.inria.fr/hal-02303490/document https://hal.inria.fr/hal-02303490/document Peer-to-Peer Ordered Search Indexes https://0fps.net/2020/12/19/peer-to-peer-ordered-search-indexes/ https://0fps.net/2020/12/19/peer-to-peer-ordered-search-inde... (which adds useful context about the above) Sorry for just dropping these links, I should already be asleep :)

westurner 6y ago
> Merkle Search Trees: Efficient State-Based CRDTs in Open Networks https://hal.inria.fr/hal-02303490/document https://hal.inria.fr/hal-02303490/document https://scholar.google.com/scholar?cites=7160577141569533185&as_sdt=5,43&sciodt=0,43&hl=en https://scholar.google.com/scholar?cites=7160577141569533185... ... "Merkle Hash Grids Instead of Merkle Trees" (2020) https://scholar.google.com/scholar?cluster=13503894708682701593&hl=en&as_sdt=5,43&sciodt=0,43 https://scholar.google.com/scholar?cluster=13503894708682701... Browser-side "Blockchain Certificate Transparency" applications need to support at least exact key lookup by domain/SAN and then also by cert fingerprint value; but the whole CT chain with every cert issue and revocation event is impractically large in terms of disk space. https://github.com/amark/gun#history https://github.com/amark/gun#history may also be practically useful.