guilload

3 ms·

1. 1.▲
   by guilload 3y ago

   Quickwit ( https://quickwit.io/ ) | Paris, France | Onsite and remote (based in Europe) | Full-time The company is fully remote but we also have a small office in Paris. We prefer candidates based in Europe but can make excep
2. 2.▲
   by guilload 5y ago

   We indexed Common Crawl only for the purpose of this demo so this is one-time thing, we won't deal with updates.
3. 3.▲
   by guilload 5y ago

   There are more recent versions of the dataset. We used the february/march snapshot from this year and the April snapshot just came out ( https://commoncrawl.org/2021/04/april-2021-crawl-archive-now... ).
4. 4.▲
   by guilload 5y ago

   We store the URI of each shard making up the index and, optionally, partition key and value(s). Along with a few flags, we also store the shard size, creation and last modification time. This additional metadata is not required for the quer
5. 5.▲
   by guilload 5y ago

   What we store on S3 is a regular tantivy index and another tiny data structure that we call "turbo index", which makes queries faster on object storages. For this demo, the tantivy indexes are fairly large and we issue HTTP Range