4 ms·
MongoDB is horrible, I get it. What do I use in this situation: 1) I need to store 100,000,000+ json files in a database 2) query the data in these json file
by crazybit 6y ago
MongoDB is horrible, I get it.
What do I use in this situation:
1) I need to store 100,000,000+ json files in a database
2) query the data in these json files
3) json files come from thousands upon thousands of different sources, each with their own drastically different "schema"
4) constantly adding more json files from constantly new sources
5) no time to figure out the schema prior to adding into the database
6) don't care if a json file is lost once in awhile
7) only 1 table, no relational tables needed
8) easy replication and sharding across servers sought after
9) don't actually require json, so long as data can be easily mapped from json to database format and back
10) can self host, no cloud only lock-in
Recommendations?
- kreetx 6y agoI don't think it's that simple (being horrible). MongoDB can be great for some specific situations, perhaps yours. It's just that it's not for many others, and you'd need to be an expert to find this out from the docs.
- gilbetron 6y agoElasticsearch? http://smnh.me/indexing-and-searching-arbitrary-json-data-using-elasticsearch/ http://smnh.me/indexing-and-searching-arbitrary-json-data-us... Depends on what your queries look like, I guess.
- inglor 6y agoJust adding that I have used elasticsearch for a use case under the above constraints several times in the past and it worked well. Ironically once because mongo was such a pain to work with I dumped the data from it into ES to get the better API, usability and Kibana.
- oblio 6y agoPostgresql with 1 table with JSON fields?