3 ms·
Sqlite format is smaller than the original format only because xml is super verbose, so any uncompressed binary format ends up being less than lightly zipped xm
by cm2187 3y ago
Sqlite format is smaller than the original format only because xml is super verbose, so any uncompressed binary format ends up being less than lightly zipped xml.
But sqlite files aren't small. One thing I don't understand is why they don't do string deduplication in sqlite (as in you only store a string once and every other occurence is just a pointer to that string). It seems such an obvious and easy way to reduce file size, memory consumption and therefore increase performance (less I/O). Is there a technical reason why this would not be desirable?
- Etheryte 3y agoMy first guess is that if you always store the full string you don't need to scan the database to see if you already have the same string. Essentially you choose to use more space but reduce load. Regardless of whether you do the string deduping on inserts or async later on, you have to do it at some point and the unpredictable performance overhead might be undesirable.
- cm2187 3y agoWell it should be a dictionary lookup, it should be pretty fast and predictable. And for freeing it up, it should be a good candidate for reference counting.
- The_Colonel 3y agoIf you have the same (long-ish) string repeating many times in a database, it points to a DB schema needing normalization.
- cm2187 3y agoI guess it depends on the use case. If you load a csv file into a sqlite database, normalisation isn't the first thing you do.
- kortex 3y agoThere is nonzero overhead for doing so: optimizing for duplicate strings invariably adds cost to handling unique strings. This sounds like something you could do at the schema and application level.