4 ms·

You need to add data to Elasticsearch for it to be useful. I wrote a search engine for Confluence using Elasticsearch- which is amazing, but you need to add com

by lykr0n 8y ago

You need to add data to Elasticsearch for it to be useful. I wrote a search engine for Confluence using Elasticsearch- which is amazing, but you need to add components to get it functional. I'm thinking a "index this site" or "Index this fileshare" and go.

runlevel1 8y ago
Any chance of open sourcing it? Confluence's search is appallingly bad, and tags barely help.
- lykr0n 8y ago
  Maybe. I've PoC'ed it, but it's really rough. The biggest issue is UI. The indexer and query engine is primitive, but it works. I've been meaning to re-write it. If I do it after work I can opensource it. Look for a Show HN Post or I'll fire off an email directly too you in about a week.
sansnomme 8y ago
Think you can write a blog post on general purpose indexing with Elasticsearch? (Also, thoughts on Mayan EDMS?)
- logon_carrier 8y ago
  Nice mention of Mayan. It is a fantastic tool and the secret weapon in our arsenal! I'm the IT manager at a legal firm, these are our use case scenarios for Mayan: - Defeat document dumps (https://en.wikipedia.org/wiki/Document_dump https://en.wikipedia.org/wiki/Document_dump). We just scan everything and use Mayan's full text search. - We added an .eml importer to upload dumped emails from Exchange servers. The OCR and the indexing sort all the emails by keywords which we then compile as evidence in cases. - We uploaded tons of historical cases, judicial opinions, laws, amendments, etc. We now have a superb, free, self-hosted, electronic legal research library. It is not documented, but we tried it and versions > 3.0 of Mayan added search operators (AND, OR, NOT). This puts it on par with dedicated full text search engines like Elasticsearch. We have been talking about adding something like a Mayan web scrapper to get PDFs from government sites automatically and bypass the FOIA requests for published files since we'll have them indexed preemptively. Final note: Mayan scales beautifully but after > 500K documents start using Postgres connection pooling or you'll kill your DB server.

by lykr0n 8y ago

runlevel1 8y ago

Any chance of open sourcing it? Confluence's search is appallingly bad, and tags barely help.

lykr0n 8y ago
Maybe. I've PoC'ed it, but it's really rough. The biggest issue is UI. The indexer and query engine is primitive, but it works. I've been meaning to re-write it. If I do it after work I can opensource it. Look for a Show HN Post or I'll fire off an email directly too you in about a week.

sansnomme 8y ago

Think you can write a blog post on general purpose indexing with Elasticsearch? (Also, thoughts on Mayan EDMS?)

logon_carrier 8y ago
Nice mention of Mayan. It is a fantastic tool and the secret weapon in our arsenal! I'm the IT manager at a legal firm, these are our use case scenarios for Mayan: - Defeat document dumps (https://en.wikipedia.org/wiki/Document_dump https://en.wikipedia.org/wiki/Document_dump). We just scan everything and use Mayan's full text search. - We added an .eml importer to upload dumped emails from Exchange servers. The OCR and the indexing sort all the emails by keywords which we then compile as evidence in cases. - We uploaded tons of historical cases, judicial opinions, laws, amendments, etc. We now have a superb, free, self-hosted, electronic legal research library. It is not documented, but we tried it and versions > 3.0 of Mayan added search operators (AND, OR, NOT). This puts it on par with dedicated full text search engines like Elasticsearch. We have been talking about adding something like a Mayan web scrapper to get PDFs from government sites automatically and bypass the FOIA requests for published files since we'll have them indexed preemptively. Final note: Mayan scales beautifully but after > 500K documents start using Postgres connection pooling or you'll kill your DB server.