4 ms·
I would store the original data unaltered as files. Then I would index and transform it into the lowest common denominator formats, probably something like thi
by forgotmypw17 5y ago
I would store the original data unaltered as files.
Then I would index and transform it into the lowest common denominator formats, probably something like this:
For text, plaintext with header-like fields. For images and video, the most commonly readable formats.
Then I would build index database(s) on top, probably SQLite.
And on top of that, I would make a lightweight web-based system for querying and categorizing.
This way, the data modification happens only one way, and it is easy to maintain transparency and clarity of process. You never lose your original data, so if there’s a problem, you can go back one step before the problem. And no relying on proprietary or low-support data formats.
- akshatmittal108 5y agoAgree. DynamoDB + S3 seems like a good choice from my side. S3 will store the data - whether it is images or videos. DynamoDB can store the metadata associated with which museum uploaded the file, what date, type etc. NoSQL suits the dynamic nature of content which you are storing. You can build a website on top of it to support whatever you want. Also a simple dropbox may also work if the goal is just to aggregate information and nothing else. All depends on what you want to do with the data.
- dmje 5y agoThanks, I know nothing about dynamodb so that's a good tip, will go and research!
- forgotmypw17 5y agoI personally would not use DynamoDB, because of its proprietary license and young age.
- qorrect 5y agoI like this plan except I would go for ElasticCache.