IME people like it when they can submit something and move on with their life. I think a small data explorer for web can go a long way and a simple web portal t

IME people like it when they can submit something and move on with their life. I think a small data explorer for web can go a long way and a simple web portal that allows submission of zipped files. That might cover the first 90%, then you can deal with the last 10% that need additional help. For your second question about data ingest, unless you're enforcing standards on receiving data (for example, check it as people try to submit and fire back if it's not right) I don't see a way to index except by set schema. Elasticsearch would allow you to index on dynamic fields, from there you could merge data.