4 ms·
Depends on the data, obviously; but record format data should be compact in memory in ways that text isn't. My favorite example is the meg's of commas and quot
by taeric 4y ago
Depends on the data, obviously; but record format data should be compact in memory in ways that text isn't. My favorite example is the meg's of commas and quotes that just aren't needed in memory.
So, my examples are largely csv files. JSON and xml are the same, in many ways.
The big curve ball will be text heavy data. But all too often textual data is categorical, such that even that should be smaller in memory.
In large, this is why parquet files are a ridiculous win in space.
- coldtea 4y agoMy favorite example is the meg's of commas and quotes that just aren't needed in memory. Depending on the language and internal represenation, for any comma and quote not needed (which for a JSON document with, say, an object without nesting, is just 5 bytes per entry), you might get your strings doubled in size (because e.g. the language converted your fitting into 8bit utf-8 strings to its 16bit native string format), or have some huge constant boxing overhead (in say, Python), and several other fun things besides...
- taeric 4y agoRight, is why I said strings can be a curve ball. But most of the data I see can avoid that fairly easily.