5 ms·

I'd argue that there's no meaningful difference between ASCII text and UTF-8 that just happens to not use any characters higher than '~'. As long as there's no

by teach 11y ago

I'd argue that there's no meaningful difference between ASCII text and UTF-8 that just happens to not use any characters higher than '~'. As long as there's no BOM, anyway.

deleted 11y ago
[deleted]
dghf 11y ago
s/meaningful// Any ASCII text is ipso facto UTF-8 text.
- jstimpfle 11y ago
  If you consider representation, yes. If you consider semantics, no.
  - dghf 11y ago
    Can you explain? UTF-8 was deliberately designed to be "backwards compatible" (for want of a better phrase) with ASCII. Absent a BOM mark, there is no way to tell the difference between a piece of ASCII text on the one hand and, on the other, a piece of UTF-8 text that uses no code point above U+007f. How do they differ semantically?
    - jstimpfle 11y ago
      Yes. I already tried to explain with an analogy about faithfulness (which I removed because it got many downvotes). When you deal with ASCII, that's also the semantic level on which you operate. Much simpler than Unicode (I don't think there are many people who really understand Unicode). When you have a Unicode string that uses only characters that are also present in ASCII, that doesn't make it an ASCII string. It has more meaning. Btw that has nothing to do with BOM. BOM is itself not part of ASCII, and not encoded as a single byte, and it doesn't affect single-byte sequences.
      - dghf 11y ago
        But wherein does that meaning lie? If the two strings represent the same text, you can't tell the difference between them, because there is none. (I guess technically you can't tell the difference between an ASCII string and a string of random 8-bit numbers in the range 0 <= x <= 127, but if when represented as ASCII characters those numbers are meaningful -- comprehensible English, for example -- the odds are good it's the former not the latter.)
        technofire 11y ago
        I think what he's getting at is that whatever program is reading the text must be prepared to read UTF-8 and not just ASCII. Sure, the underlying data is the same, but the data type is not the same; analogously, a byte with the decimal value of 65 might be the number 65 if cast as an integer, but might be the character 'A' if cast as a char.
      - emddudley 11y ago
        Are you talking about how, when someone is limited to ASCII, he makes substitutions he wouldn't make if he were using Unicode? Such as using :-) instead of U+263A? Or are you maybe talking at a software level, where you can make some simplifications and assumptions about a string when it is guaranteed to be ASCII? Because otherwise I can't see how there could be any semantic or meaningful difference between ASCII and Unicode when the text is the same. An ASCII 'a' has exactly the same meaning as a Unicode 'Latin Small Letter A'.
        jstimpfle 11y ago
        You can look at it from an information theoretic standpoint. The information you get from the string is also all the things that it does not contain. Btw. I have no clue why I am constantly downvoted. Whoever did this, I don't think there is an incentive to downvote my two ancestor comments. It's demotivating.
        coldtea 11y ago
    - khedoros 11y ago
      Loading it from disk, there's no way to tell the difference, because the representation is the same and a selection of bytes are a selection of bytes. Say you read the text into a program. You can load the text into an AsciiString variable or a UnicodeString variable, and it's valid in either of those cases. They have the same binary representation...but they're semantically different. They each have properties that the other one doesn't (ASCII has random access, Unicode is linear access, either one could have an ASCII string appended to it, but only the Unicode one could have a Unicode string appended, etc). The difference isn't the representation; the difference is the interpretation that we attach to the representation.

by teach 11y ago

I'd argue that there's no meaningful difference between ASCII text and UTF-8 that just happens to not use any characters higher than '~'. As long as there's no BOM, anyway.

deleted 11y ago

[deleted]

dghf 11y ago

s/meaningful// Any ASCII text is ipso facto UTF-8 text.

jstimpfle 11y ago
If you consider representation, yes. If you consider semantics, no.
- dghf 11y ago
  Can you explain? UTF-8 was deliberately designed to be "backwards compatible" (for want of a better phrase) with ASCII. Absent a BOM mark, there is no way to tell the difference between a piece of ASCII text on the one hand and, on the other, a piece of UTF-8 text that uses no code point above U+007f. How do they differ semantically?
  - jstimpfle 11y ago
    Yes. I already tried to explain with an analogy about faithfulness (which I removed because it got many downvotes). When you deal with ASCII, that's also the semantic level on which you operate. Much simpler than Unicode (I don't think there are many people who really understand Unicode). When you have a Unicode string that uses only characters that are also present in ASCII, that doesn't make it an ASCII string. It has more meaning. Btw that has nothing to do with BOM. BOM is itself not part of ASCII, and not encoded as a single byte, and it doesn't affect single-byte sequences.
    - dghf 11y ago
      But wherein does that meaning lie? If the two strings represent the same text, you can't tell the difference between them, because there is none. (I guess technically you can't tell the difference between an ASCII string and a string of random 8-bit numbers in the range 0 <= x <= 127, but if when represented as ASCII characters those numbers are meaningful -- comprehensible English, for example -- the odds are good it's the former not the latter.)
      - technofire 11y ago
        I think what he's getting at is that whatever program is reading the text must be prepared to read UTF-8 and not just ASCII. Sure, the underlying data is the same, but the data type is not the same; analogously, a byte with the decimal value of 65 might be the number 65 if cast as an integer, but might be the character 'A' if cast as a char.
    - emddudley 11y ago
      Are you talking about how, when someone is limited to ASCII, he makes substitutions he wouldn't make if he were using Unicode? Such as using :-) instead of U+263A? Or are you maybe talking at a software level, where you can make some simplifications and assumptions about a string when it is guaranteed to be ASCII? Because otherwise I can't see how there could be any semantic or meaningful difference between ASCII and Unicode when the text is the same. An ASCII 'a' has exactly the same meaning as a Unicode 'Latin Small Letter A'.
      - jstimpfle 11y ago
        You can look at it from an information theoretic standpoint. The information you get from the string is also all the things that it does not contain. Btw. I have no clue why I am constantly downvoted. Whoever did this, I don't think there is an incentive to downvote my two ancestor comments. It's demotivating.
        coldtea 11y ago
  - khedoros 11y ago
    Loading it from disk, there's no way to tell the difference, because the representation is the same and a selection of bytes are a selection of bytes. Say you read the text into a program. You can load the text into an AsciiString variable or a UnicodeString variable, and it's valid in either of those cases. They have the same binary representation...but they're semantically different. They each have properties that the other one doesn't (ASCII has random access, Unicode is linear access, either one could have an ASCII string appended to it, but only the Unicode one could have a Unicode string appended, etc). The difference isn't the representation; the difference is the interpretation that we attach to the representation.