3 ms·
> much more space than storing the text In other cases it’s the opposite. Convert a model from millimeters to inches, and that 1.0/25.4 multiplier gonna cause
by Const-me 5y ago
> much more space than storing the text
In other cases it’s the opposite. Convert a model from millimeters to inches, and that 1.0/25.4 multiplier gonna cause decimal representation to be much longer than 8 digits. Mechanical engineers are doing things like that pretty often.
> when humans enter data in some specific units
I don’t think it’s a reasonable expectation for a human to type 100 MB, or 1GB of numbers. People do work with triangle meshes that large. These models were made by specialized hardware or software tools, not by entering numbers. It’s the same story with images: they have too many megapixels now, and people aren’t entering millions of RGB numbers. Instead, people using scanners, digital cameras, and Photoshop to deal with these volumes of data.
> Text compression of number data is extremely efficient
The most expensive part is not compression, it’s printing and parsing. That code is very hard to vectorize or parallelize.
When software reads or writes large continuous buffer of binary data like std::vector in C++, the throughput is often limited by physical disk. On many modern computers that’s at least 100 MBytes/sec, often much faster.
I think it’s borderline impossible to print or parse float numbers that fast. Especially given that in 3MF, there’s some XML producer/parser down the stack who needs to escape/unescape these strings.
- willis936 5y agoThis is an issue that shows up often. ASCII is used because the performance cost is less than the benefit of guaranteeing that the exact value retrieved is the exact value stored. I often wonder why BCD file formats are not used.
- Shorel 5y agoThe problem there is using inches.
- ddingus 5y agoWhile many agree, and I for sure do, fact is manufacturing in a lot of the US is a mixed unit scenario at best. Much of Additive uses SI units, fortunately. One of the bigger players remains with Imperial units, unfortunately I do sales, service, support in this industry. One of the more hilarious queries I got recently was, "Does your machine support freedom units?" We all had a nice chuckle, and yes that shop works mixed and it's a pain point for them being a mix of older school and new school tech and people all over the map. They don't see a transition in the near future either. Too much of the work remains Imperial as does equipment that has a long service life ahead. The real problem lies in formats and software that makes assumptions about units, and or makes it all difficult to manage, or input / extract mixed unit data. I see meters used fairly often too. Not the same non integer conversion, but the other problems do remain.
- ggrrhh_ta 5y agoI understand all your points (I have implemented all sort of parsers, even in mmaped memory regions, jumped around text parsers to not look for '\0' or do in-place changes, etc...) and I agree with them. I think text is a good format for the "truth"; a binary one-to-one representation of that text to avoid floats parsing is the best alternative. Counterintuitive as it is, many times, the text representation takes less space (sometimes there are huge lot of little round values, for examples 0s or 1s or 2s, in some columns).
- ggrrhh_ta 5y agoin the inches/mm case... my choice would be, if we can support units, then write the file data as given by the user - use that data as the ultimate source of truth; use a more efficient/standard representation in the backend and have processes that test that you can re-generate the file from the user as accurately as possible with the backend data. But of course, it depends on the exact case and how much users expect to edit by hand their files.
- ddingus 5y agoThis is best practice, IMHO. Consider the input data immutable and work from there.