6 ms·

I know that it is counterintuitive, but storing doubles in binary format can take (for many use cases) much more space than storing the text. Each double, uncom

I know that it is counterintuitive, but storing doubles in binary format can take (for many use cases) much more space than storing the text. Each double, uncompressed is 8 bytes, 3.2e-3 is only 6 bytes, a 1 digit integer is 1 byte, a 5 digit integer is 3 bytes less than the double to store it... Also, when humans enter data in some specific units they want to see the same number back (and not a rounded -in binary- version of it, although very good algorithms exist to that end). Text compression of number data is extremely efficient. Having cross-platform binary formats is complex and binary formats go stale much sooner. Having humans peek at the file in text mode and debut or use simple command line tools with copied and pasted data is also useful for exploration purposes. The best of both worlds is to have a text representation coupled with one-to-one binary encoding; but as I pointed out, many times, the binary format actually takes, unintuitively, more space...

Const-me 5y ago
> much more space than storing the text In other cases it’s the opposite. Convert a model from millimeters to inches, and that 1.0/25.4 multiplier gonna cause decimal representation to be much longer than 8 digits. Mechanical engineers are doing things like that pretty often. > when humans enter data in some specific units I don’t think it’s a reasonable expectation for a human to type 100 MB, or 1GB of numbers. People do work with triangle meshes that large. These models were made by specialized hardware or software tools, not by entering numbers. It’s the same story with images: they have too many megapixels now, and people aren’t entering millions of RGB numbers. Instead, people using scanners, digital cameras, and Photoshop to deal with these volumes of data. > Text compression of number data is extremely efficient The most expensive part is not compression, it’s printing and parsing. That code is very hard to vectorize or parallelize. When software reads or writes large continuous buffer of binary data like std::vector in C++, the throughput is often limited by physical disk. On many modern computers that’s at least 100 MBytes/sec, often much faster. I think it’s borderline impossible to print or parse float numbers that fast. Especially given that in 3MF, there’s some XML producer/parser down the stack who needs to escape/unescape these strings.
- willis936 5y ago
  This is an issue that shows up often. ASCII is used because the performance cost is less than the benefit of guaranteeing that the exact value retrieved is the exact value stored. I often wonder why BCD file formats are not used.
- Shorel 5y ago
  The problem there is using inches.
  - ddingus 5y ago
    While many agree, and I for sure do, fact is manufacturing in a lot of the US is a mixed unit scenario at best. Much of Additive uses SI units, fortunately. One of the bigger players remains with Imperial units, unfortunately I do sales, service, support in this industry. One of the more hilarious queries I got recently was, "Does your machine support freedom units?" We all had a nice chuckle, and yes that shop works mixed and it's a pain point for them being a mix of older school and new school tech and people all over the map. They don't see a transition in the near future either. Too much of the work remains Imperial as does equipment that has a long service life ahead. The real problem lies in formats and software that makes assumptions about units, and or makes it all difficult to manage, or input / extract mixed unit data. I see meters used fairly often too. Not the same non integer conversion, but the other problems do remain.
- ggrrhh_ta 5y ago
  I understand all your points (I have implemented all sort of parsers, even in mmaped memory regions, jumped around text parsers to not look for '\0' or do in-place changes, etc...) and I agree with them. I think text is a good format for the "truth"; a binary one-to-one representation of that text to avoid floats parsing is the best alternative. Counterintuitive as it is, many times, the text representation takes less space (sometimes there are huge lot of little round values, for examples 0s or 1s or 2s, in some columns).
- ggrrhh_ta 5y ago
  in the inches/mm case... my choice would be, if we can support units, then write the file data as given by the user - use that data as the ultimate source of truth; use a more efficient/standard representation in the backend and have processes that test that you can re-generate the file from the user as accurately as possible with the backend data. But of course, it depends on the exact case and how much users expect to edit by hand their files.
aldanor 5y ago
You could store decimals (either power 10 or power 2) with a given precision. Essentially, when storing floats as text, you are already doing that.
Rexxar 5y ago
Maybe creating a standard textual hexadecimal-double format could be a good middle ground. "a.2aE3b" easy to include in text and to parse and maybe easy to read once used to it. hex => dec a.8 => 10.5 f.88 => 15.55 9.e => 9.875 11.8 => 17.5 a.8E1 => 168 (15.5*16) a.8E2 => 2688 (15.5*16^2)
- willis936 5y ago
  Rather than constraining to floating point, why not just use a 4-bit BCD format with a zeros place symbol and terminator symbol? That way you can express exact values efficiently. Of course it becomes unwieldy when specifying extreme values (such as 10^100), but for most reasonable values (10^20 to 10^-20) it beats both floating point and ASCII for size and speed.
okamiueru 5y ago
Part of serialization is getting to pick how you encode numbers. And, one should do so with a good match on the range or required precision. I'm not sure double values are required here or necessary. > Text compression of number data is extremely efficient Of course it isn't. It is arguably one of the worst there is. It can only ever by more efficient when representing a very low subset of values and the alternative binary representation is exceedingly poor. But those constraint would make the whole discussion kinda silly. Serializing text, is after all binary encoding data. If you follow this with compression algorithms, you'll likely not end up too far away in either case. Just take the example you gave. "A 5 digit integer is 3 bytes less than the double to store it". Yes, true. However, a 5 digit integer can also be represented by 12 bits. And you can even pack them if this is all you really care about (there is a good reason why you wouldn't want to do this, as word alignment is much more important, as you would be able to directly map data to memory allocations, but let's for argument sake say that storage was king). For each 5 digit numbers as a text file, you'll need another byte to encode the line break or delimiter. I.e. 6 bytes instead of 12 bits, or 4 times wasted space. For a million values, 5.7 MiB instead of 1.4 MiB.
- tzs 5y ago
  > It can only ever by more efficient when representing a very low subset of values and the alternative binary representation is exceedingly poor. In almost all cases in practice one is only serializing a very low subset of values. A 1 TB file, no matter how efficient the encoding, could only contain serialization of less than 0.00005% of the possible IEEE 754 double values.
  - okamiueru 5y ago
    That is of course true. My point was to highlight the kind of situations where straight serialization of text would be somewhat closer in required storage, when compared to similar serialization of number values. That serializations of such numbers would only encode a subset of the values it can represent, I would say, is besides the point. It is after all how number representations work. To reiterate my original point: It is never the case that serializing text that represents numbers is the most efficient representation. The posts here seem to either suggest or think this is the case. And this was only the size consideration. It is even worse in terms of processing. You need to do O(n) operations on the length of text data just in order to decode the values. A step that is entirely avoided with a correct binary format. Just a anecdotal example here on the computational aspect. Maybe 10 years ago or so, I used an OBJ loading library that would straight up decode 3d models represented as ASCII text. I wrote a simple de/serialization program, just 70 lines of code, to both dump and read dumped data. The load speed for a complicated scene went from 38 to 0.7 seconds. This isn't to say there aren't good use cases for text formats. However, it is always a tradeoff for human readable data, at the expense of efficiency (storage and computational). The former is much less so when using a simple compression step. Very often this tradeoff is acceptable.
phoronixrly 5y ago
I acknowledge that in a correctly designed binary format given data may in some cases be more compact than when represented as compressed XML. I ask myself though will it practically matter in the 3MF vs STL case, given 3MF uses a different approach to represent geometries that produces both more accurate and far more compact results than STL for a similarly high precision[0]? [0] https://hawkridgesys.com/blog/3mf-the-file-format-for-the-future-of-3d-printing https://hawkridgesys.com/blog/3mf-the-file-format-for-the-fu...
stkdump 5y ago
The binary stl format uses float precision. But the point is much more that parsing takes CPU cycles.

Const-me 5y ago

> much more space than storing the text In other cases it’s the opposite. Convert a model from millimeters to inches, and that 1.0/25.4 multiplier gonna cause decimal representation to be much longer than 8 digits. Mechanical engineers are doing things like that pretty often. > when humans enter data in some specific units I don’t think it’s a reasonable expectation for a human to type 100 MB, or 1GB of numbers. People do work with triangle meshes that large. These models were made by specialized hardware or software tools, not by entering numbers. It’s the same story with images: they have too many megapixels now, and people aren’t entering millions of RGB numbers. Instead, people using scanners, digital cameras, and Photoshop to deal with these volumes of data. > Text compression of number data is extremely efficient The most expensive part is not compression, it’s printing and parsing. That code is very hard to vectorize or parallelize. When software reads or writes large continuous buffer of binary data like std::vector in C++, the throughput is often limited by physical disk. On many modern computers that’s at least 100 MBytes/sec, often much faster. I think it’s borderline impossible to print or parse float numbers that fast. Especially given that in 3MF, there’s some XML producer/parser down the stack who needs to escape/unescape these strings.

willis936 5y ago
This is an issue that shows up often. ASCII is used because the performance cost is less than the benefit of guaranteeing that the exact value retrieved is the exact value stored. I often wonder why BCD file formats are not used.
Shorel 5y ago
The problem there is using inches.
- ddingus 5y ago
  While many agree, and I for sure do, fact is manufacturing in a lot of the US is a mixed unit scenario at best. Much of Additive uses SI units, fortunately. One of the bigger players remains with Imperial units, unfortunately I do sales, service, support in this industry. One of the more hilarious queries I got recently was, "Does your machine support freedom units?" We all had a nice chuckle, and yes that shop works mixed and it's a pain point for them being a mix of older school and new school tech and people all over the map. They don't see a transition in the near future either. Too much of the work remains Imperial as does equipment that has a long service life ahead. The real problem lies in formats and software that makes assumptions about units, and or makes it all difficult to manage, or input / extract mixed unit data. I see meters used fairly often too. Not the same non integer conversion, but the other problems do remain.
ggrrhh_ta 5y ago
I understand all your points (I have implemented all sort of parsers, even in mmaped memory regions, jumped around text parsers to not look for '\0' or do in-place changes, etc...) and I agree with them. I think text is a good format for the "truth"; a binary one-to-one representation of that text to avoid floats parsing is the best alternative. Counterintuitive as it is, many times, the text representation takes less space (sometimes there are huge lot of little round values, for examples 0s or 1s or 2s, in some columns).
ggrrhh_ta 5y ago
in the inches/mm case... my choice would be, if we can support units, then write the file data as given by the user - use that data as the ultimate source of truth; use a more efficient/standard representation in the backend and have processes that test that you can re-generate the file from the user as accurately as possible with the backend data. But of course, it depends on the exact case and how much users expect to edit by hand their files.

aldanor 5y ago

You could store decimals (either power 10 or power 2) with a given precision. Essentially, when storing floats as text, you are already doing that.

Rexxar 5y ago

Maybe creating a standard textual hexadecimal-double format could be a good middle ground. "a.2aE3b" easy to include in text and to parse and maybe easy to read once used to it. hex => dec a.8 => 10.5 f.88 => 15.55 9.e => 9.875 11.8 => 17.5 a.8E1 => 168 (15.5*16) a.8E2 => 2688 (15.5*16^2)

willis936 5y ago
Rather than constraining to floating point, why not just use a 4-bit BCD format with a zeros place symbol and terminator symbol? That way you can express exact values efficiently. Of course it becomes unwieldy when specifying extreme values (such as 10^100), but for most reasonable values (10^20 to 10^-20) it beats both floating point and ASCII for size and speed.

okamiueru 5y ago

Part of serialization is getting to pick how you encode numbers. And, one should do so with a good match on the range or required precision. I'm not sure double values are required here or necessary. > Text compression of number data is extremely efficient Of course it isn't. It is arguably one of the worst there is. It can only ever by more efficient when representing a very low subset of values and the alternative binary representation is exceedingly poor. But those constraint would make the whole discussion kinda silly. Serializing text, is after all binary encoding data. If you follow this with compression algorithms, you'll likely not end up too far away in either case. Just take the example you gave. "A 5 digit integer is 3 bytes less than the double to store it". Yes, true. However, a 5 digit integer can also be represented by 12 bits. And you can even pack them if this is all you really care about (there is a good reason why you wouldn't want to do this, as word alignment is much more important, as you would be able to directly map data to memory allocations, but let's for argument sake say that storage was king). For each 5 digit numbers as a text file, you'll need another byte to encode the line break or delimiter. I.e. 6 bytes instead of 12 bits, or 4 times wasted space. For a million values, 5.7 MiB instead of 1.4 MiB.

tzs 5y ago
> It can only ever by more efficient when representing a very low subset of values and the alternative binary representation is exceedingly poor. In almost all cases in practice one is only serializing a very low subset of values. A 1 TB file, no matter how efficient the encoding, could only contain serialization of less than 0.00005% of the possible IEEE 754 double values.
- okamiueru 5y ago
  That is of course true. My point was to highlight the kind of situations where straight serialization of text would be somewhat closer in required storage, when compared to similar serialization of number values. That serializations of such numbers would only encode a subset of the values it can represent, I would say, is besides the point. It is after all how number representations work. To reiterate my original point: It is never the case that serializing text that represents numbers is the most efficient representation. The posts here seem to either suggest or think this is the case. And this was only the size consideration. It is even worse in terms of processing. You need to do O(n) operations on the length of text data just in order to decode the values. A step that is entirely avoided with a correct binary format. Just a anecdotal example here on the computational aspect. Maybe 10 years ago or so, I used an OBJ loading library that would straight up decode 3d models represented as ASCII text. I wrote a simple de/serialization program, just 70 lines of code, to both dump and read dumped data. The load speed for a complicated scene went from 38 to 0.7 seconds. This isn't to say there aren't good use cases for text formats. However, it is always a tradeoff for human readable data, at the expense of efficiency (storage and computational). The former is much less so when using a simple compression step. Very often this tradeoff is acceptable.

phoronixrly 5y ago

I acknowledge that in a correctly designed binary format given data may in some cases be more compact than when represented as compressed XML. I ask myself though will it practically matter in the 3MF vs STL case, given 3MF uses a different approach to represent geometries that produces both more accurate and far more compact results than STL for a similarly high precision[0]? [0] https://hawkridgesys.com/blog/3mf-the-file-format-for-the-future-of-3d-printing https://hawkridgesys.com/blog/3mf-the-file-format-for-the-fu...

stkdump 5y ago

The binary stl format uses float precision. But the point is much more that parsing takes CPU cycles.