3 ms·

It still surprises me how many have the intuition that loading the data will take more space than the file. Even more annoying when it is. (Compressed or bina

by taeric 4y ago

It still surprises me how many have the intuition that loading the data will take more space than the file. Even more annoying when it is. (Compressed or binary formats not withstanding.)

vlovich123 4y ago
Isn’t that almost always true unless the format is specifically designed to take the same amount on disk and in RAM? If that’s the case I think typically it means the file on disk is larger than it needs to be. That’s because you have to convert the file to some kind of in-memory representation that’s easier to work with to manipulate it and what not which requires structure that isn’t necessary at rest. Can you give an example where the file on disk is going to be larger than what it is in memory? Provided you’re not just reading it and working with it as an opaque binary blob.
- hnlmorg 4y ago
  XML would probably fall into that category. But I do agree with your point more generally speaking.
- taeric 4y ago
  Depends on the data, obviously; but record format data should be compact in memory in ways that text isn't. My favorite example is the meg's of commas and quotes that just aren't needed in memory. So, my examples are largely csv files. JSON and xml are the same, in many ways. The big curve ball will be text heavy data. But all too often textual data is categorical, such that even that should be smaller in memory. In large, this is why parquet files are a ridiculous win in space.
  - coldtea 4y ago
    My favorite example is the meg's of commas and quotes that just aren't needed in memory. Depending on the language and internal represenation, for any comma and quote not needed (which for a JSON document with, say, an object without nesting, is just 5 bytes per entry), you might get your strings doubled in size (because e.g. the language converted your fitting into 8bit utf-8 strings to its 16bit native string format), or have some huge constant boxing overhead (in say, Python), and several other fun things besides...
    - taeric 4y ago
      Right, is why I said strings can be a curve ball. But most of the data I see can avoid that fairly easily.
- nice2meetu 4y ago
  I think it comes more from the thought that human text readable format is usually more verbose than binary representations.
- taeric 4y ago
  Coming back to this, I was curious and decided to challenge my assertion some. https://taeric.github.io/challenging-my-filesize-intuition.html https://taeric.github.io/challenging-my-filesize-intuition.h... is a few examples where I was fishing around. I didn't find any where my assertion is not true. Curious if you have any easy examples to look at that do show the opposite.
deleted 4y ago
[deleted]

by taeric 4y ago

It still surprises me how many have the intuition that loading the data will take more space than the file. Even more annoying when it is. (Compressed or binary formats not withstanding.)

vlovich123 4y ago

Isn’t that almost always true unless the format is specifically designed to take the same amount on disk and in RAM? If that’s the case I think typically it means the file on disk is larger than it needs to be. That’s because you have to convert the file to some kind of in-memory representation that’s easier to work with to manipulate it and what not which requires structure that isn’t necessary at rest. Can you give an example where the file on disk is going to be larger than what it is in memory? Provided you’re not just reading it and working with it as an opaque binary blob.

hnlmorg 4y ago
XML would probably fall into that category. But I do agree with your point more generally speaking.
taeric 4y ago
Depends on the data, obviously; but record format data should be compact in memory in ways that text isn't. My favorite example is the meg's of commas and quotes that just aren't needed in memory. So, my examples are largely csv files. JSON and xml are the same, in many ways. The big curve ball will be text heavy data. But all too often textual data is categorical, such that even that should be smaller in memory. In large, this is why parquet files are a ridiculous win in space.
- coldtea 4y ago
  My favorite example is the meg's of commas and quotes that just aren't needed in memory. Depending on the language and internal represenation, for any comma and quote not needed (which for a JSON document with, say, an object without nesting, is just 5 bytes per entry), you might get your strings doubled in size (because e.g. the language converted your fitting into 8bit utf-8 strings to its 16bit native string format), or have some huge constant boxing overhead (in say, Python), and several other fun things besides...
  - taeric 4y ago
    Right, is why I said strings can be a curve ball. But most of the data I see can avoid that fairly easily.
nice2meetu 4y ago
I think it comes more from the thought that human text readable format is usually more verbose than binary representations.
taeric 4y ago
Coming back to this, I was curious and decided to challenge my assertion some. https://taeric.github.io/challenging-my-filesize-intuition.html https://taeric.github.io/challenging-my-filesize-intuition.h... is a few examples where I was fishing around. I didn't find any where my assertion is not true. Curious if you have any easy examples to look at that do show the opposite.

deleted 4y ago

[deleted]