4 ms·

I trialed Aleph recently and was impressed by its progress against an ambitious goal. My impressions as a user were as follows: 1. Aleph is excellent out-of-th

I trialed Aleph recently and was impressed by its progress against an ambitious goal. My impressions as a user were as follows: 1. Aleph is excellent out-of-the-box for its – OCR, via Tesseract or Google’s Vision API – Full text search, via Elasticsearch – Browser based UI, via React 2. Aleph does a okay job but has room for improvement with – Entity extraction – Language detection where “okay” means it’s accurate enough to be useful for filtering by names, emails, languages, etc., but you’ll probably encounter occasional errors. I also noticed search latency in my deployment and would love to try the Elasticsearch tips from the HN thread last week [1]. This latency does not appear in the production deployment by the Aleph team. [1]: https://news.ycombinator.com/item?id=22396918 https://news.ycombinator.com/item?id=22396918 Again, props to the Aleph team for their success so far.

pudo 7y ago
The latency issue with ES probably comes from how we use the indexes: each type of entity (documents, emails, people, companies) ends up with its own index. 48 in total at the moment. This of course introduces lots of overhead at the beginning, but with a larger dataset and with a clustered ES it pays off at scale. We should consider re-adding single index operation mode for small deployment.
bryanrasmussen 7y ago
Given your trial would you think it useful to investigate moving a search solution of standards and regulations in PDF and text documents to aleph? That is to say is it a good enough search solution for structured and unstructured data that it would make sense to build on top of it instead of rolling your own?
- divbzero 7y ago
  Yes, definitely worth investigating if you need full text search as well as content extraction from PDFs. I found the production deployment installation [1] to be the most straightforward. [1]: https://docs.alephdata.org/developers/installation#production-deployment https://docs.alephdata.org/developers/installation#productio... Feel free to ping me if you decide to try it and have questions. In addition, the Aleph team is active on both GitHub and Slack.

pudo 7y ago

The latency issue with ES probably comes from how we use the indexes: each type of entity (documents, emails, people, companies) ends up with its own index. 48 in total at the moment. This of course introduces lots of overhead at the beginning, but with a larger dataset and with a clustered ES it pays off at scale. We should consider re-adding single index operation mode for small deployment.

bryanrasmussen 7y ago

Given your trial would you think it useful to investigate moving a search solution of standards and regulations in PDF and text documents to aleph? That is to say is it a good enough search solution for structured and unstructured data that it would make sense to build on top of it instead of rolling your own?

divbzero 7y ago
Yes, definitely worth investigating if you need full text search as well as content extraction from PDFs. I found the production deployment installation [1] to be the most straightforward. [1]: https://docs.alephdata.org/developers/installation#production-deployment https://docs.alephdata.org/developers/installation#productio... Feel free to ping me if you decide to try it and have questions. In addition, the Aleph team is active on both GitHub and Slack.