4 ms·
Ask HN: Best way to store thousands of ~10mb files via browser upload interface?
We've got decent (albeit old school) full stack in-house dev experience, so re-inventing the wheel is optional given our tailored needs.
Some solutions we've come up with:
1. Store the files as MEDIUMBLOBs in our MySQL database - but we're not sure how responsive this database would be once we hit 100TB+ of files.
2. Store the files on a standard Linux file system, and limit the MySQL database to tracking values linked to each file (but not storing the actual files themselves). The data tracked in this database would thus be limited, and therefore scale better. In said data, we'd track things such as the date the was file created, a customer's name, and around 100 other variable text fields.
What are your thoughts on the above two options?
And are there other good/better options?
Please note: we're a bunch of old guys that still tinker in LAMP, so kindly take it easy on us if you're responding with modern framework solutions. But we're open to learning new things!
Cheers from South Africa.
- epc 5y agoI’d compute a couple of checksums on the files and store those plus metadata in a database, storing the files either on multiple RAIDs under your control, or one or more geographically dispersed cloud storage systems. Whether local storage or cloud storage is best is a function of how often the files are going to be downloaded or otherwise manipulated by your service.
- herodoturtle 5y agoThanks epc. We're leaning towards cloud storage (the simplest turnkey kind), because of the administrative benefits it affords us at a price we're willing to pay. > I’d compute a couple of checksums on the files and store those plus metadata in a database Please enlighten us - what are these checksums? - and would they be similar in size to the file? In other words, are we still storing the actual file data in the database (along with its metadata)? Or did you perhaps mean "compute a couple of checksums on the files and store those [on the file system, and the metadata in a database]"? We think you meant the former because you said "plus", but just wanted to check. The fellas say hi.
- epc 5y agoChecksums: I'd want to verify the data integrity, especially if it's to be stored for a long time and retrieved in the distant future. So I'd compute at least some sort of SHA sum (I personally use SHA512 but I'm doing it on files in an archive that I rarely retrieve). Possibly a short fast CRC would suffice. I'd store the CRC/hash plus whatever metadata you need to track (account uploading, IP address, filename, etc,) in the database plus whatever pointer you need to locate it in your cloud storage service(s). I wouldn't store the files in the database unless you're going to be using something from the database service on the files (are you indexing them for search, for example?). I'd also sketch out the realistic use to find the potential breaking points. When we did the image storage system for the 1996 Olympic web site we thought we were so, so smart for how we sliced up the filenames and directory structure, only to learn that the underlying DFS we were using was doing its own very sophisticated file slicing and dicing resulting in suboptimal performance. We would have been much better off dumping all of the files into date/time directories and leaving it at that. |Or do what neximo64 suggests in https://news.ycombinator.com/item?id=27778952 https://news.ycombinator.com/item?id=27778952
- herodoturtle 5y agoThanks for the detailed insight and advice. That all makes perfect sense. We will do as you and neximo64 suggest. You've both been a great help, cheers.
- neximo64 5y agoon S3, South Africa region using a javascript library that uploads them using pre-signed urls for you and returns the final urls which you can save in your mysql database
- herodoturtle 5y agoI could write a poem about the succinct beauty of this response. Quick question: should we be concerned with the number of files per directory and / or the number of directories? Put another way, could we use your solution with a single giant directory of ~10mb files spanning roughly 100TB?
- neximo64 5y agoIt shouldn't be an issue. S3 stores the data as blobs and the directory listing is contrived to seem similar to a file system. So you can have unlimited files in there. 100 TB should also be just fine. There's also some other nice features such as ACLs, automated backups, expirations that could work as its one of AWS's more mature products. Some more info on the files per directory: https://stackoverflow.com/questions/394198/max-files-per-directory-in-s3 https://stackoverflow.com/questions/394198/max-files-per-dir...
- herodoturtle 5y agoThank you.
- deleted 5y ago[deleted]