3 ms·
Why not hash the image on upload and store them with names based on hash? That way if one copy already exists it will just get over-written on next user upload
by zenlikethat 10y ago
Why not hash the image on upload and store them with names based on hash? That way if one copy already exists it will just get over-written on next user upload
- aquark 10y agoSomeone would still need to handle life-timing the objects in the presence of deletes. If the storage layer does it then it is one less thing to handle at the application level. If you upload by hash then you need your own ref counting for deletes (or just don't support delete ... storage is cheap after all!)
- Spivak 10y agoWouldn't any account based system have an indirect reference counter basically for free? Images are associated with accounts so you safely delete an image when there are no accounts that own it.
- corobo 10y agoI don't trust user input and I don't trust hashing output, I definitely don't trust mixing the two! Serious answer though in that instance it comes down to already having a naming system in place that uses hashids to generate the filename. I could re-write the whole thing but it's not really a project that grasps my attention for long bursts. As it stands I've just moved the whole lot to s3 for now, even with it's larger footprint it's not much more in terms of cost - Mainly I use the image storage project as a way to test new object storage. As an example off the top of my head I'm thinking block level de-duplication would be good for storing backup tars from multiple machines. In my previous post I was more hinting at file-level dedupe but that probably wouldn't do as much.