3 ms·

Dumping raw binary is a problem if you want your data to be readable across platforms or versions of the same code. It is also a bad idea if reading a complete

Dumping raw binary is a problem if you want your data to be readable across platforms or versions of the same code. It is also a bad idea if reading a complete structure into memory is not possible, or a poor use of time. For example, if you need to read a single part of a gigabyte data set.

StillBored 10y ago
None of this is a show stopper, especially with a little planning. AKA pad the structures a bit initially, or put version numbers in them. Even if you fail to do any of that early on, it can be bolted on with a bit of preprocessing as part of a version upgrade. Its really no different than a schema change between version numbers. Plus, if your really paranoid, you verify field size and offsets at startup time against a previously generated bit of metadata. Also, if you happen to need to pick out a tiny bit of data from some huge tree/whatever data structure then map it with appropriate madvise() and simply let the kernel page in the portions you need to access. The whole dataset doesn't need to be read to pick out a couple bytes here and there. Further if this is a common use case you probably shouldn't be using this method anyway, rather storing the index separately from the data in a manner which allows them to be accessed independently. This mechanism holds up for multiple TB sized datasets, and can be tuned by playing with the amount of physical ram in the machine vs the speed of the disk. Then when storage class memory takes off, you don't even have to rewrite your application. Finally, for a single GB dataset, doing a bulk read is likely well under a 1 second operation on most reasonable storage mechanisms available for server use. Compared with picking off a couple hundred thousand fields from an object storage mechanism this is going to seem pretty speedy.

3 ms·

Dumping raw binary is a problem if you want your data to be readable across platforms or versions of the same code. It is also a bad idea if reading a complete

by frozenport 10y ago

StillBored 10y ago
None of this is a show stopper, especially with a little planning. AKA pad the structures a bit initially, or put version numbers in them. Even if you fail to do any of that early on, it can be bolted on with a bit of preprocessing as part of a version upgrade. Its really no different than a schema change between version numbers. Plus, if your really paranoid, you verify field size and offsets at startup time against a previously generated bit of metadata. Also, if you happen to need to pick out a tiny bit of data from some huge tree/whatever data structure then map it with appropriate madvise() and simply let the kernel page in the portions you need to access. The whole dataset doesn't need to be read to pick out a couple bytes here and there. Further if this is a common use case you probably shouldn't be using this method anyway, rather storing the index separately from the data in a manner which allows them to be accessed independently. This mechanism holds up for multiple TB sized datasets, and can be tuned by playing with the amount of physical ram in the machine vs the speed of the disk. Then when storage class memory takes off, you don't even have to rewrite your application. Finally, for a single GB dataset, doing a bulk read is likely well under a 1 second operation on most reasonable storage mechanisms available for server use. Compared with picking off a couple hundred thousand fields from an object storage mechanism this is going to seem pretty speedy.