7 ms·

For comparison, Instagram sends ~200 byte base64 thumbnails in its API responses. It's a low-res JPEG with a fixed header to save a bit more space. This is also

For comparison, Instagram sends ~200 byte base64 thumbnails in its API responses. It's a low-res JPEG with a fixed header to save a bit more space. This is also used to blur sensitive images. You can get even better results with webp. It requires minimal javascript support-- just enough to build an img tag with the appropriate data url, and renders just as fast as any other tiny image. Upscaling blurs away the imperfections. https://stackoverflow.com/q/49625771 https://stackoverflow.com/q/49625771

alephnan 7y ago
Can you clarify the significance of the fixed header?
- metalliqaz 7y ago
  if the header never changes, it doesn't have to be stored or transmitted. the client-side code can simply prepend the header after getting the data.
- Scaevolus 7y ago
  JPEGs were never meant to be <1KB, so the header can be rather large-- looking at one example, the truncated JPEG data Instagram sends is 248B, and the additional fixed header information is 607B, resulting in an 848B thumbnail (the truncated data includes dimensions). That 607B of fixed header includes some compression settings that don't need to change between images: - 132B of quantization tables, saying how much to divide every coefficient by-- for subtle details the difference between "128" and "129" doesn't matter, and you can just send "2 * 64". - a 418B Huffman table, which stores the frequencies of different coefficient values after quantization-- small values are much more common than large ones, given the divisors. You can have smaller image data with properly tuned tables, but that only matters for large images. For 250B of image data, it's a waste. - Miscellaneous other details that are fixed, like the file type magic headers, the color space information, and start/end markers. You can dig in more by putting a thumbnail like https://i.imgur.com/M0EsJH1.jpg https://i.imgur.com/M0EsJH1.jpg into this site: https://cyber.meme.tips/jpdump/ https://cyber.meme.tips/jpdump/
DagAgren 7y ago
Yes, this was inspired by that, and the fact that our database engineer went "I'm not putting that much garbage in my database"! And I figured, yeah, that's still a lot of data, and it's massively misusing several different tools. Why not create something that is actually designed for the task?
- mark242 7y ago
  the fact that our database engineer went "I'm not putting that much garbage in my database"! This is really disingenuous-- a DBA should not be a gatekeeper of what gets put into a database. Storing 200 bytes of extra data per row costs you absolutely nothing. Even if you assume 100 million photos, that's 20GB of data. That's a little over $2 of EBS cost per month. Who cares.
  - streb-lo 7y ago
    > disingenuous Don't think that's the correct adjective here.
  - JoshTriplett 7y ago
    Unless you have some clever storage mechanism (such as a column-oriented database or a separate table with a join), 200 extra bytes directly in each row of a frequently used table makes that table less efficiently use disk and memory caches. Consider if the entire rest of that row takes up 100 bytes, and now it takes up 300. You now can fit a third as many rows in each disk block or memory page.
    - socceroos 7y ago
      This is so true. Fitting rows in the page was a big win for query times in my case.
    - ska 7y ago
      Hence columnar stores. But parent is right that the choice of what shouldn't be coming from your DBA; they should be more interested in how.
deleted 7y ago
[deleted]
shakna 7y ago
> It's a low-res JPEG with a fixed header to save a bit more space. This is also used to blur sensitive images. You can get even better results with webp. That is not my experience at all. On images less than 100kb, webp adds an overhead, and doesn't reduce the file size at all. A fixed header is also harder to achieve.

alephnan 7y ago

Can you clarify the significance of the fixed header?

metalliqaz 7y ago
if the header never changes, it doesn't have to be stored or transmitted. the client-side code can simply prepend the header after getting the data.
Scaevolus 7y ago
JPEGs were never meant to be <1KB, so the header can be rather large-- looking at one example, the truncated JPEG data Instagram sends is 248B, and the additional fixed header information is 607B, resulting in an 848B thumbnail (the truncated data includes dimensions). That 607B of fixed header includes some compression settings that don't need to change between images: - 132B of quantization tables, saying how much to divide every coefficient by-- for subtle details the difference between "128" and "129" doesn't matter, and you can just send "2 * 64". - a 418B Huffman table, which stores the frequencies of different coefficient values after quantization-- small values are much more common than large ones, given the divisors. You can have smaller image data with properly tuned tables, but that only matters for large images. For 250B of image data, it's a waste. - Miscellaneous other details that are fixed, like the file type magic headers, the color space information, and start/end markers. You can dig in more by putting a thumbnail like https://i.imgur.com/M0EsJH1.jpg https://i.imgur.com/M0EsJH1.jpg into this site: https://cyber.meme.tips/jpdump/ https://cyber.meme.tips/jpdump/

DagAgren 7y ago

Yes, this was inspired by that, and the fact that our database engineer went "I'm not putting that much garbage in my database"! And I figured, yeah, that's still a lot of data, and it's massively misusing several different tools. Why not create something that is actually designed for the task?

mark242 7y ago
the fact that our database engineer went "I'm not putting that much garbage in my database"! This is really disingenuous-- a DBA should not be a gatekeeper of what gets put into a database. Storing 200 bytes of extra data per row costs you absolutely nothing. Even if you assume 100 million photos, that's 20GB of data. That's a little over $2 of EBS cost per month. Who cares.
- streb-lo 7y ago
  > disingenuous Don't think that's the correct adjective here.
- JoshTriplett 7y ago
  Unless you have some clever storage mechanism (such as a column-oriented database or a separate table with a join), 200 extra bytes directly in each row of a frequently used table makes that table less efficiently use disk and memory caches. Consider if the entire rest of that row takes up 100 bytes, and now it takes up 300. You now can fit a third as many rows in each disk block or memory page.
  - socceroos 7y ago
    This is so true. Fitting rows in the page was a big win for query times in my case.
  - ska 7y ago
    Hence columnar stores. But parent is right that the choice of what shouldn't be coming from your DBA; they should be more interested in how.

deleted 7y ago

[deleted]

shakna 7y ago

> It's a low-res JPEG with a fixed header to save a bit more space. This is also used to blur sensitive images. You can get even better results with webp. That is not my experience at all. On images less than 100kb, webp adds an overhead, and doesn't reduce the file size at all. A fixed header is also harder to achieve.