3 ms·

C++ has "built in" serialization the same way that C does.. Except its not acceptable to use anymore, even though its orders of magnitude faster than the altern

C++ has "built in" serialization the same way that C does.. Except its not acceptable to use anymore, even though its orders of magnitude faster than the alternatives.. AKA you write your structures out directly as binary data. Bonus if you align them. The limitation is that you can't have classes/structures with actual pointers in them (rather using offsets) and you have to encourage your compiler to pack/align them consistently. Bonus if you override the new/placement allocator so that you can serialize gigabytes of data (including pointers) with a simple map/unmap call. Frankly, the idea that data should be human readable is complete crap, especially for things which are going to be machine processed far more than human processed. AKA use XML for your config file, not for the gigabytes of data you send over the wire/store on disk/etc, that will never be read by a human. Its funny that HTML/etc has everyone in this strange mindset that somehow its more efficient to build human readable markup, which 99.9999% of the time is never read by a human, and has to be compressed with algorithms that further burn up tons of CPU time in order to squash the resulting data. The untold tons of carbon wasted transfering more data than necessary and/or compressing/decompressing it is insane for the possibility that someone will want to read it as it flys over the wire, rather than decoding it at the endpoint and viewing it in a debugger...

jstimpfle 10y ago
> AKA you write your structures out directly as binary data. What does that even mean? Depends on what the output should be and how much control you need, I guess. To comment on the binary vs text digression - computation overhead is probably not that much if the data is converted to a (non-portable) internal DOM representation and then typeset / rasterized anyway. And is bandwidth/storage overhead? I assume a short video quickly outweighs the costs for 100%, or in extreme cases 1000%, overhead compared to a compressed DOM representation. And the human factors are definitely there.
frozenport 10y ago
Dumping raw binary is a problem if you want your data to be readable across platforms or versions of the same code. It is also a bad idea if reading a complete structure into memory is not possible, or a poor use of time. For example, if you need to read a single part of a gigabyte data set.
- StillBored 10y ago
  None of this is a show stopper, especially with a little planning. AKA pad the structures a bit initially, or put version numbers in them. Even if you fail to do any of that early on, it can be bolted on with a bit of preprocessing as part of a version upgrade. Its really no different than a schema change between version numbers. Plus, if your really paranoid, you verify field size and offsets at startup time against a previously generated bit of metadata. Also, if you happen to need to pick out a tiny bit of data from some huge tree/whatever data structure then map it with appropriate madvise() and simply let the kernel page in the portions you need to access. The whole dataset doesn't need to be read to pick out a couple bytes here and there. Further if this is a common use case you probably shouldn't be using this method anyway, rather storing the index separately from the data in a manner which allows them to be accessed independently. This mechanism holds up for multiple TB sized datasets, and can be tuned by playing with the amount of physical ram in the machine vs the speed of the disk. Then when storage class memory takes off, you don't even have to rewrite your application. Finally, for a single GB dataset, doing a bulk read is likely well under a 1 second operation on most reasonable storage mechanisms available for server use. Compared with picking off a couple hundred thousand fields from an object storage mechanism this is going to seem pretty speedy.
fsloth 10y ago
Yes, binary data is good. It's better to explicitly serialize/deserialize it, though. That way endianness won't become an issue ever, and the programmer does not need to be concerned over runtime memory layout of the structs.
- slededit 10y ago
  At this point little endian has pretty much won. Even IBM with their POWER architecture now uses little endian for their machines targeting Linux. Its far more efficient to assume little Endian and let the weird machines do their own translations when loading the data.

jstimpfle 10y ago

> AKA you write your structures out directly as binary data. What does that even mean? Depends on what the output should be and how much control you need, I guess. To comment on the binary vs text digression - computation overhead is probably not that much if the data is converted to a (non-portable) internal DOM representation and then typeset / rasterized anyway. And is bandwidth/storage overhead? I assume a short video quickly outweighs the costs for 100%, or in extreme cases 1000%, overhead compared to a compressed DOM representation. And the human factors are definitely there.

frozenport 10y ago

Dumping raw binary is a problem if you want your data to be readable across platforms or versions of the same code. It is also a bad idea if reading a complete structure into memory is not possible, or a poor use of time. For example, if you need to read a single part of a gigabyte data set.

StillBored 10y ago
None of this is a show stopper, especially with a little planning. AKA pad the structures a bit initially, or put version numbers in them. Even if you fail to do any of that early on, it can be bolted on with a bit of preprocessing as part of a version upgrade. Its really no different than a schema change between version numbers. Plus, if your really paranoid, you verify field size and offsets at startup time against a previously generated bit of metadata. Also, if you happen to need to pick out a tiny bit of data from some huge tree/whatever data structure then map it with appropriate madvise() and simply let the kernel page in the portions you need to access. The whole dataset doesn't need to be read to pick out a couple bytes here and there. Further if this is a common use case you probably shouldn't be using this method anyway, rather storing the index separately from the data in a manner which allows them to be accessed independently. This mechanism holds up for multiple TB sized datasets, and can be tuned by playing with the amount of physical ram in the machine vs the speed of the disk. Then when storage class memory takes off, you don't even have to rewrite your application. Finally, for a single GB dataset, doing a bulk read is likely well under a 1 second operation on most reasonable storage mechanisms available for server use. Compared with picking off a couple hundred thousand fields from an object storage mechanism this is going to seem pretty speedy.

fsloth 10y ago

Yes, binary data is good. It's better to explicitly serialize/deserialize it, though. That way endianness won't become an issue ever, and the programmer does not need to be concerned over runtime memory layout of the structs.

slededit 10y ago
At this point little endian has pretty much won. Even IBM with their POWER architecture now uses little endian for their machines targeting Linux. Its far more efficient to assume little Endian and let the weird machines do their own translations when loading the data.