3 ms·

The /cli/list endpoint is the search endpoint you're looking for. It provides FTS but I can make it clearer in the docs, thanks for the tip. As for the AI stuf

The /cli/list endpoint is the search endpoint you're looking for. It provides FTS but I can make it clearer in the docs, thanks for the tip. As for the AI stuff don't worry, none of it is touching core, it's all in an optional community plugin only for those who want it. I'm not personally a huge AI person but I have clients who are already using it and getting massive value from it, so it's worth mentioning. (They're doing some automated QA on thousands of collected captured and feeding results into spreadsheets)

rcarmo 2y ago
Thanks, I'll have a look. My use for this is very different--I want to be able to use a specific subset of my archived pages (which is mostly reference documentation) to "chat" with, providing different LLM prompts depending on subset and fetching plaintext chunks as reference info for the LLM to summarize (and point me back to the archived pages if I need more info).
- nikisweeting 2y ago
  Ok that makes sense, I think archivebox works as the first step in a pipeline there, with some other tool doing the LLM analysis and query stuff.
  - rcarmo 2y ago
    Yep. That's what I've built for myself, I just can't really get at the data inside ArchiveBox until I upgrade.
    - pbronez 2y ago
      How did you build it? I can imagine an architecture where I throw everything into ArchiveBox, then run VectorDB as a plugin with Gradio or some such as the client. https://vectordb.com/ https://vectordb.com/
      - rcarmo 2y ago
        You're overcomplicating things. You don't need a vector database, FTS works just as well for non-homogenous content.
sunshine-o 2y ago
I have been using ArchiveBox recently and love it. About search, one thing I haven't yet figured out how to do easily is to plug it to my SearXNG instance as they only seem to support Elasticsearch, Meilisearch or Solr [0] So this new plugin architecture will allow for a meilisearch plugin I guess (with relevancy ranking). - [0] https://docs.searxng.org/dev/engines/offline/search-indexer-engines.html https://docs.searxng.org/dev/engines/offline/search-indexer-...
- nikisweeting 2y ago
  Definitely doable! Search plugins are one of the first that I implemented. We already provide Sonic, ripgrep, and SQLiteFTS as plugins, so adding something like Solr should be straightforward. Check out the existing plugins to see how it's done: https://github.com/ArchiveBox/ArchiveBox/pull/1534/files?file-filters%5B%5D=.py&show-deleted-files=true&show-viewed-files=true#diff-9583cabf0acbd028077cddc613cc471ea83d18ed05a0f947ca3cc08db75e3b8d https://github.com/ArchiveBox/ArchiveBox/pull/1534/files?fil... archivebox/plugins_search/sonic/*

rcarmo 2y ago

Thanks, I'll have a look. My use for this is very different--I want to be able to use a specific subset of my archived pages (which is mostly reference documentation) to "chat" with, providing different LLM prompts depending on subset and fetching plaintext chunks as reference info for the LLM to summarize (and point me back to the archived pages if I need more info).

nikisweeting 2y ago
Ok that makes sense, I think archivebox works as the first step in a pipeline there, with some other tool doing the LLM analysis and query stuff.
- rcarmo 2y ago
  Yep. That's what I've built for myself, I just can't really get at the data inside ArchiveBox until I upgrade.
  - pbronez 2y ago
    How did you build it? I can imagine an architecture where I throw everything into ArchiveBox, then run VectorDB as a plugin with Gradio or some such as the client. https://vectordb.com/ https://vectordb.com/
    - rcarmo 2y ago
      You're overcomplicating things. You don't need a vector database, FTS works just as well for non-homogenous content.

sunshine-o 2y ago

I have been using ArchiveBox recently and love it. About search, one thing I haven't yet figured out how to do easily is to plug it to my SearXNG instance as they only seem to support Elasticsearch, Meilisearch or Solr [0] So this new plugin architecture will allow for a meilisearch plugin I guess (with relevancy ranking). - [0] https://docs.searxng.org/dev/engines/offline/search-indexer-engines.html https://docs.searxng.org/dev/engines/offline/search-indexer-...

nikisweeting 2y ago
Definitely doable! Search plugins are one of the first that I implemented. We already provide Sonic, ripgrep, and SQLiteFTS as plugins, so adding something like Solr should be straightforward. Check out the existing plugins to see how it's done: https://github.com/ArchiveBox/ArchiveBox/pull/1534/files?file-filters%5B%5D=.py&show-deleted-files=true&show-viewed-files=true#diff-9583cabf0acbd028077cddc613cc471ea83d18ed05a0f947ca3cc08db75e3b8d https://github.com/ArchiveBox/ArchiveBox/pull/1534/files?fil... archivebox/plugins_search/sonic/*