5 ms·
When you get piles of JSON data from somewhere else (e.g. an API response from a provider you subscribe to) and want to store all of it intact, query it, and th
by dheera 6y ago
When you get piles of JSON data from somewhere else (e.g. an API response from a provider you subscribe to) and want to store all of it intact, query it, and that somewhere else may insert new fields at any time in the future without telling you.
When you have multiple types of objects with differently populated fields that you want to store in a single collection and query across them in currently unknown ways, and where you need flexibility for the unknown.
- drodgers 6y agoThat sounds like a great use case for OLAP and fuzzy search solutions. Eg. Dump all of the data to S3 and search it in parallel via AWS Athena. If you want fast fuzzy search, then elasticsearch is built for this. If the data is small enough that you want B-tree indexes (with the accompanying slower writes and lower query latencies), then any SQL database with a JSON column will work.
- dheera 6y ago> Eg. Dump all of the data to S3 and search it in parallel via AWS Athena. Too proprietary. Can't be run locally, on robots, etc. Can't be transferred to other cloud providers. Also S3 is horribly slow to write or delete thousands of records at once.
- kornish 6y ago> Too proprietary. Every major cloud provider has a blob store. Most are S3-interface compatible, or can be fronted by something which is, like Minio. > Can't be run locally, on robots, etc. If the contents of your S3 keys are just line-delineated JSON files, you can easily download those files and run scripts or process them locally. > Can't be transferred to other cloud providers. Again, not true — a tool like rclone handles this case in a single command. Depends on the amount of data, but if anything it's easier to move a bunch of flat files between providers than copying a database backup around. You have to pay for egress bandwidth, though, of course. > Also S3 is horribly slow to write or delete thousands of records at once. If you're keeping one JSON record per S3 key, that is a blatant misuse of the tool and performance will be terrible. On the other hand, if you batch records into files of appropriate sizes, it's very cost-effective to store and query via Athena, Spark on EMR, etc. If you need individual record-level access as identifiable by a key, then you'd probably be better off with Redis or Memcached. (though, those will not be as good for bulk offline processing) It's all about your access patterns.
- WJW 6y ago> Also S3 is horribly slow to write or delete thousands of records at once. I'll add that S3 is only slow to delete things sequentially. You can delete or write an effectively infinite amount of items in parallel.
- dheera 6y ago> On the other hand, if you batch records into files of appropriate sizes Too much work. If you're a startup that's too much stuff to maintain. Much easier to get a hosted MongoDB and launch tomorrow. And when you only have 2 months rent in the bank and investors want demo after demo in order to give you cash, tools like MongoDB do make a difference. And yes, I've been in that situation before.
- dolni 6y agoYour ability to turn the smallest of technical hurdles into an insurmountable problem is truly impressive.
- saturn_vk 6y agoThat's what I'm using postgresql for, among other things. The only drawback I've ran into so far is that it sometimes doesn't pick the proper index for a specific query.
- mschuster91 6y ago> and that somewhere else may insert new fields at any time in the future without telling you. That's what interface contracts and API versioning is for - and new fields shouldn't be any problem at all unless you're parsing the JSON by hand.