4 ms·

I'm curious of the 2021 measure of total disk space that Discord consumes. Servers that I'm in share images every few minutes, which must add up pretty quick.

SilverRed 5y ago
I'd imagine that the images are not part of the main database and that they are in some kind of s3 like file storage system.
latchkey 5y ago
Unique images or just copied from elsewhere?
- objektif 5y ago
  How does that matter? Do they keep track of all images on the internet?
  - jjice 5y ago
    I don't know much about image de-deuplication, but maybe they can get some sort of fingerprint/hash for an image, see if they already have it, and then serve that already existing image. I'd imagine a hash like SHA256 would be tricky because if that image was compressed an additional time at all throughout it's internet journey, then we'd get a different resulting hash, but maybe there is an effective way to fingerprint images. I have a utility on my machine (czkawka maybe?) that does really good image de-duplication with what seemed like a common algorithm (based on a quick look at the source). No idea though, just spit balling.
    - PeterCorless 5y ago
      Yes. There are ways to group images that seem to be the same. TinEye and Google image search do that. So you'd have a collection of related hashes that equal "Bob's prom photo where he looks like a goofer."
      - objektif 5y ago
        Yes definitely I have seen it work in action but you cant just tell a user "here use this smaller and more pixelated version of your image that we think is kind of similar".
        PeterCorless 5y ago
        Oof! No, you can't. :D
    - vortico 5y ago
      I'd imagine Discord uses deduplication, but I bet it doesn't save them 5% storage space.
      - kroltan 5y ago
        The plural of anecdote isn't data, but about 20% of the images I post on Discord come from Discord in the first place, cross-posting among different servers.
      - Ekaros 5y ago
        I think it might, spamming same meme images over and over is quite common in some servers. On other hand the bigger pictures might overhelm these just in size.
        vortico 5y ago
        Yeah, that's why I assumed it wouldn't help that much. People re-upload 100kB memes all the time, but the bulk would probably be 5MB phone pictures that won't typically be re-uploaded.
snak 5y ago
The Casssandra cluster mentioned (12 nodes, 1TB each) only handles text, as far as the article goes.
- jhgg 5y ago
  We're well over 12 nodes in current year :P
  - PeterCorless 5y ago
    lol. :D
  - snak 5y ago
    Oh, just noticed the article is from 2017. Is there a newer one, related/similar to this one?

SilverRed 5y ago

I'd imagine that the images are not part of the main database and that they are in some kind of s3 like file storage system.

latchkey 5y ago

Unique images or just copied from elsewhere?

objektif 5y ago
How does that matter? Do they keep track of all images on the internet?
- jjice 5y ago
  I don't know much about image de-deuplication, but maybe they can get some sort of fingerprint/hash for an image, see if they already have it, and then serve that already existing image. I'd imagine a hash like SHA256 would be tricky because if that image was compressed an additional time at all throughout it's internet journey, then we'd get a different resulting hash, but maybe there is an effective way to fingerprint images. I have a utility on my machine (czkawka maybe?) that does really good image de-duplication with what seemed like a common algorithm (based on a quick look at the source). No idea though, just spit balling.
  - PeterCorless 5y ago
    Yes. There are ways to group images that seem to be the same. TinEye and Google image search do that. So you'd have a collection of related hashes that equal "Bob's prom photo where he looks like a goofer."
    - objektif 5y ago
      Yes definitely I have seen it work in action but you cant just tell a user "here use this smaller and more pixelated version of your image that we think is kind of similar".
      - PeterCorless 5y ago
        Oof! No, you can't. :D
  - vortico 5y ago
    I'd imagine Discord uses deduplication, but I bet it doesn't save them 5% storage space.
    - kroltan 5y ago
      The plural of anecdote isn't data, but about 20% of the images I post on Discord come from Discord in the first place, cross-posting among different servers.
    - Ekaros 5y ago
      I think it might, spamming same meme images over and over is quite common in some servers. On other hand the bigger pictures might overhelm these just in size.
      - vortico 5y ago
        Yeah, that's why I assumed it wouldn't help that much. People re-upload 100kB memes all the time, but the bulk would probably be 5MB phone pictures that won't typically be re-uploaded.

snak 5y ago

The Casssandra cluster mentioned (12 nodes, 1TB each) only handles text, as far as the article goes.

jhgg 5y ago
We're well over 12 nodes in current year :P
- PeterCorless 5y ago
  lol. :D
- snak 5y ago
  Oh, just noticed the article is from 2017. Is there a newer one, related/similar to this one?