3 ms·
The person who can make an amazing, open source search engine (like the Google Appliance) that can take in Exchange Servers, File Shares, and any other data sou
by lykr0n 8y ago
The person who can make an amazing, open source search engine (like the Google Appliance) that can take in Exchange Servers, File Shares, and any other data source and turn around and be able to do full text search on the data would become very rich- or very well liked.
Some of the examples here show that even in the government, information is not really centrally cataloged or searchable. One hand doesn't know about the other. Maybe some of it is malice, but I have to think part of it is just not having the ability to search Email, File Shares, Sharepoint, et cetera quickly and easily.
- marklyon 8y agoLike Relativity, Nuix or any of the existing eDiscovery processing tools?
- deleted 8y ago[deleted]
- sansnomme 8y agoElasticsearch?
- lykr0n 8y agoYou need to add data to Elasticsearch for it to be useful. I wrote a search engine for Confluence using Elasticsearch- which is amazing, but you need to add components to get it functional. I'm thinking a "index this site" or "Index this fileshare" and go.
- runlevel1 8y agoAny chance of open sourcing it? Confluence's search is appallingly bad, and tags barely help.
- lykr0n 8y agoMaybe. I've PoC'ed it, but it's really rough. The biggest issue is UI. The indexer and query engine is primitive, but it works. I've been meaning to re-write it. If I do it after work I can opensource it. Look for a Show HN Post or I'll fire off an email directly too you in about a week.
- sansnomme 8y agoThink you can write a blog post on general purpose indexing with Elasticsearch? (Also, thoughts on Mayan EDMS?)
- logon_carrier 8y agoNice mention of Mayan. It is a fantastic tool and the secret weapon in our arsenal! I'm the IT manager at a legal firm, these are our use case scenarios for Mayan: - Defeat document dumps (https://en.wikipedia.org/wiki/Document_dump https://en.wikipedia.org/wiki/Document_dump). We just scan everything and use Mayan's full text search. - We added an .eml importer to upload dumped emails from Exchange servers. The OCR and the indexing sort all the emails by keywords which we then compile as evidence in cases. - We uploaded tons of historical cases, judicial opinions, laws, amendments, etc. We now have a superb, free, self-hosted, electronic legal research library. It is not documented, but we tried it and versions > 3.0 of Mayan added search operators (AND, OR, NOT). This puts it on par with dedicated full text search engines like Elasticsearch. We have been talking about adding something like a Mayan web scrapper to get PDFs from government sites automatically and bypass the FOIA requests for published files since we'll have them indexed preemptively. Final note: Mayan scales beautifully but after > 500K documents start using Postgres connection pooling or you'll kill your DB server.
- techslave 8y agothat is a solved problem. the person who can do “AI” on the results to save all the human postprocessing hours stands to make a fortune.