3 ms·

Thanks epc. We're leaning towards cloud storage (the simplest turnkey kind), because of the administrative benefits it affords us at a price we're willing to p

Thanks epc. We're leaning towards cloud storage (the simplest turnkey kind), because of the administrative benefits it affords us at a price we're willing to pay. > I’d compute a couple of checksums on the files and store those plus metadata in a database Please enlighten us - what are these checksums? - and would they be similar in size to the file? In other words, are we still storing the actual file data in the database (along with its metadata)? Or did you perhaps mean "compute a couple of checksums on the files and store those [on the file system, and the metadata in a database]"? We think you meant the former because you said "plus", but just wanted to check. The fellas say hi.

epc 5y ago
Checksums: I'd want to verify the data integrity, especially if it's to be stored for a long time and retrieved in the distant future. So I'd compute at least some sort of SHA sum (I personally use SHA512 but I'm doing it on files in an archive that I rarely retrieve). Possibly a short fast CRC would suffice. I'd store the CRC/hash plus whatever metadata you need to track (account uploading, IP address, filename, etc,) in the database plus whatever pointer you need to locate it in your cloud storage service(s). I wouldn't store the files in the database unless you're going to be using something from the database service on the files (are you indexing them for search, for example?). I'd also sketch out the realistic use to find the potential breaking points. When we did the image storage system for the 1996 Olympic web site we thought we were so, so smart for how we sliced up the filenames and directory structure, only to learn that the underlying DFS we were using was doing its own very sophisticated file slicing and dicing resulting in suboptimal performance. We would have been much better off dumping all of the files into date/time directories and leaving it at that. |Or do what neximo64 suggests in https://news.ycombinator.com/item?id=27778952 https://news.ycombinator.com/item?id=27778952
- herodoturtle 5y ago
  Thanks for the detailed insight and advice. That all makes perfect sense. We will do as you and neximo64 suggest. You've both been a great help, cheers.

epc 5y ago

Checksums: I'd want to verify the data integrity, especially if it's to be stored for a long time and retrieved in the distant future. So I'd compute at least some sort of SHA sum (I personally use SHA512 but I'm doing it on files in an archive that I rarely retrieve). Possibly a short fast CRC would suffice. I'd store the CRC/hash plus whatever metadata you need to track (account uploading, IP address, filename, etc,) in the database plus whatever pointer you need to locate it in your cloud storage service(s). I wouldn't store the files in the database unless you're going to be using something from the database service on the files (are you indexing them for search, for example?). I'd also sketch out the realistic use to find the potential breaking points. When we did the image storage system for the 1996 Olympic web site we thought we were so, so smart for how we sliced up the filenames and directory structure, only to learn that the underlying DFS we were using was doing its own very sophisticated file slicing and dicing resulting in suboptimal performance. We would have been much better off dumping all of the files into date/time directories and leaving it at that. |Or do what neximo64 suggests in https://news.ycombinator.com/item?id=27778952 https://news.ycombinator.com/item?id=27778952

herodoturtle 5y ago
Thanks for the detailed insight and advice. That all makes perfect sense. We will do as you and neximo64 suggest. You've both been a great help, cheers.