3 ms·

Any functionality or plans for deduplication? For example if I used this for an image host and a handful of users upload the same image the image only gets stor

by corobo 10y ago

Any functionality or plans for deduplication? For example if I used this for an image host and a handful of users upload the same image the image only gets stored ${replication} times instead of ${replication}*${uploads} times. I had been using Skylable's sx server for this but unfortunately it looks like they're ceasing operations, I'm in the market for a decent replacement.

zenlikethat 10y ago
Why not hash the image on upload and store them with names based on hash? That way if one copy already exists it will just get over-written on next user upload
- aquark 10y ago
  Someone would still need to handle life-timing the objects in the presence of deletes. If the storage layer does it then it is one less thing to handle at the application level. If you upload by hash then you need your own ref counting for deletes (or just don't support delete ... storage is cheap after all!)
  - Spivak 10y ago
    Wouldn't any account based system have an indirect reference counter basically for free? Images are associated with accounts so you safely delete an image when there are no accounts that own it.
- corobo 10y ago
  I don't trust user input and I don't trust hashing output, I definitely don't trust mixing the two! Serious answer though in that instance it comes down to already having a naming system in place that uses hashids to generate the filename. I could re-write the whole thing but it's not really a project that grasps my attention for long bursts. As it stands I've just moved the whole lot to s3 for now, even with it's larger footprint it's not much more in terms of cost - Mainly I use the image storage project as a way to test new object storage. As an example off the top of my head I'm thinking block level de-duplication would be good for storing backup tars from multiple machines. In my previous post I was more hinting at file-level dedupe but that probably wouldn't do as much.
olalonde 10y ago
I wrote a "microservice" on top of S3 (tested with Minio too) and PostgreSQL that does this, among other things (e.g. image transformations, TUS uploads, web page screenshots). I would consider open sourcing it if there's enough interest.
- bedros 10y ago
  two questions, first, how does this work with amazon S3, for example, if I use django-storages with support for S3, is it just plug-n-play with minio? second, deduplication feature is great, for minio and S3 usuage. if the users decided to attach same file to multiple comments (in my case) deduplication would save me a lot.
- muminoff 10y ago
  I would be more interested in your solution as I was also thinking this kind of implementation. But, mine is more related with meta-data and video file thumbnail generating.
mocchira 10y ago
We have a plan for deduplication in version 2.2. but it should take much time (at least one year) to reach that point.

by corobo 10y ago

zenlikethat 10y ago

Why not hash the image on upload and store them with names based on hash? That way if one copy already exists it will just get over-written on next user upload

aquark 10y ago
Someone would still need to handle life-timing the objects in the presence of deletes. If the storage layer does it then it is one less thing to handle at the application level. If you upload by hash then you need your own ref counting for deletes (or just don't support delete ... storage is cheap after all!)
- Spivak 10y ago
  Wouldn't any account based system have an indirect reference counter basically for free? Images are associated with accounts so you safely delete an image when there are no accounts that own it.
corobo 10y ago
I don't trust user input and I don't trust hashing output, I definitely don't trust mixing the two! Serious answer though in that instance it comes down to already having a naming system in place that uses hashids to generate the filename. I could re-write the whole thing but it's not really a project that grasps my attention for long bursts. As it stands I've just moved the whole lot to s3 for now, even with it's larger footprint it's not much more in terms of cost - Mainly I use the image storage project as a way to test new object storage. As an example off the top of my head I'm thinking block level de-duplication would be good for storing backup tars from multiple machines. In my previous post I was more hinting at file-level dedupe but that probably wouldn't do as much.

olalonde 10y ago

I wrote a "microservice" on top of S3 (tested with Minio too) and PostgreSQL that does this, among other things (e.g. image transformations, TUS uploads, web page screenshots). I would consider open sourcing it if there's enough interest.

bedros 10y ago
two questions, first, how does this work with amazon S3, for example, if I use django-storages with support for S3, is it just plug-n-play with minio? second, deduplication feature is great, for minio and S3 usuage. if the users decided to attach same file to multiple comments (in my case) deduplication would save me a lot.
muminoff 10y ago
I would be more interested in your solution as I was also thinking this kind of implementation. But, mine is more related with meta-data and video file thumbnail generating.

mocchira 10y ago

We have a plan for deduplication in version 2.2. but it should take much time (at least one year) to reach that point.