3 ms·

It's probably deduplicated on the server though, so the millions and millions of messages with that logo likely share the same piece of disk space. Probably one

It's probably deduplicated on the server though, so the millions and millions of messages with that logo likely share the same piece of disk space. Probably one reason why free providers don't tend to offer End-to-End encryption. It prevents deduplication (and otherwise compressing redundant information).

justsomehnguy 2y ago
Nope. Back in the day Exchange offered SIS but in 2010 they ditched it. It's plainly not effective any more. Even regarding the OP' "the jpg signature logo" - it's a part of multipart in the message, not a separate file. And one more thing - you can't just turn the dedup and be dandy, now you need to check against the hashes to determine if this chunk is unique or you already have it. And with TBs of data you need TBs of hashes to check. Until you have like 99% dedup efficiency, ie 99% of your incoming data is literally the same data you already have - it doesn't worth it. https://techcommunity.microsoft.com/blog/exchange/dude-wheres-my-single-instance/595267 https://techcommunity.microsoft.com/blog/exchange/dude-where...
- Sprite_tm 2y ago
  Not sure about Microsoft solutions, but modern file systems like zfs and btrfs can do this on the filesystem level, no support from the mail server needed.
  - tored 2y ago
    NTFS and ReFS has both deduplication support (only on Windows Server).
- LegionMammal978 2y ago
  Also, I can imagine such schemes getting ditched for the same reasons of "now people can detect whether we already have a copy of the file, which is a privacy hole!" that killed cross-origin caching in browsers. Or if everything in the year 20XX gets pushed into using E2E encryption, since that's pretty much antithetical to deduplication.
palata 2y ago
> Probably one reason why free providers don't tend to offer End-to-End encryption. I would think that email providers tend to not offer E2EE just because it fundamentally isn't practical with email. Providers like Proton try to do it, but it works only if you talk to someone else on Proton (at which point you may as well do it on Signal, which has better encryption).

justsomehnguy 2y ago

Nope. Back in the day Exchange offered SIS but in 2010 they ditched it. It's plainly not effective any more. Even regarding the OP' "the jpg signature logo" - it's a part of multipart in the message, not a separate file. And one more thing - you can't just turn the dedup and be dandy, now you need to check against the hashes to determine if this chunk is unique or you already have it. And with TBs of data you need TBs of hashes to check. Until you have like 99% dedup efficiency, ie 99% of your incoming data is literally the same data you already have - it doesn't worth it. https://techcommunity.microsoft.com/blog/exchange/dude-wheres-my-single-instance/595267 https://techcommunity.microsoft.com/blog/exchange/dude-where...

Sprite_tm 2y ago
Not sure about Microsoft solutions, but modern file systems like zfs and btrfs can do this on the filesystem level, no support from the mail server needed.
- tored 2y ago
  NTFS and ReFS has both deduplication support (only on Windows Server).
LegionMammal978 2y ago
Also, I can imagine such schemes getting ditched for the same reasons of "now people can detect whether we already have a copy of the file, which is a privacy hole!" that killed cross-origin caching in browsers. Or if everything in the year 20XX gets pushed into using E2E encryption, since that's pretty much antithetical to deduplication.

palata 2y ago

> Probably one reason why free providers don't tend to offer End-to-End encryption. I would think that email providers tend to not offer E2EE just because it fundamentally isn't practical with email. Providers like Proton try to do it, but it works only if you talk to someone else on Proton (at which point you may as well do it on Signal, which has better encryption).