6 ms·

That is of course true. My point was to highlight the kind of situations where straight serialization of text would be somewhat closer in required storage, when

That is of course true. My point was to highlight the kind of situations where straight serialization of text would be somewhat closer in required storage, when compared to similar serialization of number values. That serializations of such numbers would only encode a subset of the values it can represent, I would say, is besides the point. It is after all how number representations work. To reiterate my original point: It is never the case that serializing text that represents numbers is the most efficient representation. The posts here seem to either suggest or think this is the case. And this was only the size consideration. It is even worse in terms of processing. You need to do O(n) operations on the length of text data just in order to decode the values. A step that is entirely avoided with a correct binary format. Just a anecdotal example here on the computational aspect. Maybe 10 years ago or so, I used an OBJ loading library that would straight up decode 3d models represented as ASCII text. I wrote a simple de/serialization program, just 70 lines of code, to both dump and read dumped data. The load speed for a complicated scene went from 38 to 0.7 seconds. This isn't to say there aren't good use cases for text formats. However, it is always a tradeoff for human readable data, at the expense of efficiency (storage and computational). The former is much less so when using a simple compression step. Very often this tradeoff is acceptable.

mijoharas 5y ago
> It is never the case that serializing text that represents numbers is the most efficient representation. This is false.
- prerok 5y ago
  Can you elaborate? GP's argument is verbose with a lot of explanations and examples that I agree with. What text encoding and how would it be more efficient?
  - ggrrhh_ta 5y ago
    When the text file is a source of data rather than a reflection of the data that you want to operate on, and particularly, when a lot of that data has been introduced by humans (or based on human input) in some suitable units, it very often happens that many numbers that are generate are short, shorter than a regular double. I know it is difficult to believe, so, don't; I was not trying in my comment to say that text is better everywhere always; of course not; I myself let users have text as a reference while creating a one-to-one representation in a well known binary format for speed of access. When you really have huge amounts of data, each use of the data usually requires filtering/selection/manipulation/new memory layout/etc. that makes the original optimizations in the binary format useless and complicated to maintain, and more difficult to allow for low friction use by anyone even 10 years in the future. I consider the textual representation as the source of truth; but the binary allows me to do the filtering faster, although in many occasions it takes more space. Yes, one could choose better encodings for the binary data (the 12-bit referenced above, etc.) but the complexity of the code increases and it becomes even more obscure to the humans that have to use (sometime debug) on that data. I understand is difficult to believe. Well, do not believe. Probably my experience is very niche.
    - okamiueru 5y ago
      Seems we then spoke past each other. I did not focus on practical aspects or other such tradeoffs. Just, what is the most efficient way to represent numbers. If your argument is that for many practical applications, text is good enough. Then, I don't disagree. But, if the argument was that text is a more efficient way to store numerical data. Then, there is no need to bring beliefs into this discussion. It is just not the case. Serialisation of numbers have been solved by engineers many decades ago, and there are many to chose from based on extra information on what the numbers should represent. One such representation is actually useful as a lookup table for text. But, to suggest that breaking up a single numerical value into individual lookup table values, is somehow more efficient than storing the original value with the right choice of encoding it... I don't know. I don't care all that much. But, surely you can see my argument? The whole thing is a tautology. For the sake of silliness, if ascii encoding numbers somehow was the most efficient way of encoding numbers (it isn't, but, let's say it was) then it would only then be exactly as efficient as "choosing the best way to serialize a number". But, surely, whatevever that data was, would then be most efficiently represented by the ascii representation. So, we should store the ascii representation of the bytes that store the lookup table values, as encoded ascii values. Doesn't take more than a handful iterations before you run out of storage space available on the planet.
      - ggrrhh_ta 5y ago
        I agree with everything; for users that are experts in their domain (not computer science), if you don't use text in the frontend that they can parse with little (but very inefficient) code in their scripting language of choice, then, they will be reinventing some even more difficult/complex/yet-another way to handle text that you will have to support in the backend for performance - and in the backend I choose binary formats (although my experience, without focusing too much in any other aspects of performance than retrieval, because most uses with large amounts of data require specific filtering/reordering/new memory layout for efficient algorithms, etc.). And, counterintuitively, when you do that, and you have hundreds of files which are hundreds of megabytes where some columns are integers, and you see the size of your binary data you wonder... why is my binary data few times larger?...
  - okamiueru 5y ago
    I too am curious. Not that it really matters. The statement itself is a tautology. There is no "opinion" here. Just straight logic and math. Text serialization is a way to serialize numbers. It's just a very poor and roundabout way made for human readable text. The idea that such a roundabout way of storing numbers as other numbers should somehow make things more efficient than storing the actual numbers... Is kinda silly.
ggrrhh_ta 5y ago
In my original comment to which you replied I wrote "The best of both worlds is to have a text representation coupled with one-to-one binary encoding;" for the reasons I wrote in that comment. I am aware of all your points on ser/des of text and I fully agree with them. Also, I was constraining the use case for text to data that is either the input of humans, or generated from those inputs. For data that is gathered from devices, etc. I don't think a text representation would be of much use. Also, between a set of parameters entered by a human, and an audio or a video, there are a lot of use cases in between (all of the points in space of N-sided polygon after a human entered 4 or 5 parameters, ...., generation of statistical data from parameters or other inputs, ...). I think, as a rule of thumb, that the less close the data is to the inputs of humans, the less sense it makes to store the textual representation.
ddingus 5y ago
The computation burden on these isn't so stiff as to force optimization, just FYI. Even high count models load and process on a reasonable machine in a reasonable amount of time.

mijoharas 5y ago

> It is never the case that serializing text that represents numbers is the most efficient representation. This is false.

prerok 5y ago
Can you elaborate? GP's argument is verbose with a lot of explanations and examples that I agree with. What text encoding and how would it be more efficient?
- ggrrhh_ta 5y ago
  When the text file is a source of data rather than a reflection of the data that you want to operate on, and particularly, when a lot of that data has been introduced by humans (or based on human input) in some suitable units, it very often happens that many numbers that are generate are short, shorter than a regular double. I know it is difficult to believe, so, don't; I was not trying in my comment to say that text is better everywhere always; of course not; I myself let users have text as a reference while creating a one-to-one representation in a well known binary format for speed of access. When you really have huge amounts of data, each use of the data usually requires filtering/selection/manipulation/new memory layout/etc. that makes the original optimizations in the binary format useless and complicated to maintain, and more difficult to allow for low friction use by anyone even 10 years in the future. I consider the textual representation as the source of truth; but the binary allows me to do the filtering faster, although in many occasions it takes more space. Yes, one could choose better encodings for the binary data (the 12-bit referenced above, etc.) but the complexity of the code increases and it becomes even more obscure to the humans that have to use (sometime debug) on that data. I understand is difficult to believe. Well, do not believe. Probably my experience is very niche.
  - okamiueru 5y ago
    Seems we then spoke past each other. I did not focus on practical aspects or other such tradeoffs. Just, what is the most efficient way to represent numbers. If your argument is that for many practical applications, text is good enough. Then, I don't disagree. But, if the argument was that text is a more efficient way to store numerical data. Then, there is no need to bring beliefs into this discussion. It is just not the case. Serialisation of numbers have been solved by engineers many decades ago, and there are many to chose from based on extra information on what the numbers should represent. One such representation is actually useful as a lookup table for text. But, to suggest that breaking up a single numerical value into individual lookup table values, is somehow more efficient than storing the original value with the right choice of encoding it... I don't know. I don't care all that much. But, surely you can see my argument? The whole thing is a tautology. For the sake of silliness, if ascii encoding numbers somehow was the most efficient way of encoding numbers (it isn't, but, let's say it was) then it would only then be exactly as efficient as "choosing the best way to serialize a number". But, surely, whatevever that data was, would then be most efficiently represented by the ascii representation. So, we should store the ascii representation of the bytes that store the lookup table values, as encoded ascii values. Doesn't take more than a handful iterations before you run out of storage space available on the planet.
    - ggrrhh_ta 5y ago
      I agree with everything; for users that are experts in their domain (not computer science), if you don't use text in the frontend that they can parse with little (but very inefficient) code in their scripting language of choice, then, they will be reinventing some even more difficult/complex/yet-another way to handle text that you will have to support in the backend for performance - and in the backend I choose binary formats (although my experience, without focusing too much in any other aspects of performance than retrieval, because most uses with large amounts of data require specific filtering/reordering/new memory layout for efficient algorithms, etc.). And, counterintuitively, when you do that, and you have hundreds of files which are hundreds of megabytes where some columns are integers, and you see the size of your binary data you wonder... why is my binary data few times larger?...
- okamiueru 5y ago
  I too am curious. Not that it really matters. The statement itself is a tautology. There is no "opinion" here. Just straight logic and math. Text serialization is a way to serialize numbers. It's just a very poor and roundabout way made for human readable text. The idea that such a roundabout way of storing numbers as other numbers should somehow make things more efficient than storing the actual numbers... Is kinda silly.

ggrrhh_ta 5y ago

In my original comment to which you replied I wrote "The best of both worlds is to have a text representation coupled with one-to-one binary encoding;" for the reasons I wrote in that comment. I am aware of all your points on ser/des of text and I fully agree with them. Also, I was constraining the use case for text to data that is either the input of humans, or generated from those inputs. For data that is gathered from devices, etc. I don't think a text representation would be of much use. Also, between a set of parameters entered by a human, and an audio or a video, there are a lot of use cases in between (all of the points in space of N-sided polygon after a human entered 4 or 5 parameters, ...., generation of statistical data from parameters or other inputs, ...). I think, as a rule of thumb, that the less close the data is to the inputs of humans, the less sense it makes to store the textual representation.

ddingus 5y ago

The computation burden on these isn't so stiff as to force optimization, just FYI. Even high count models load and process on a reasonable machine in a reasonable amount of time.