5 ms·

It's not that lowercase saves data. It's that normalization saves data. You can normalize to uppercase and get the same effect. In fact, the argument to use up

It's not that lowercase saves data. It's that normalization saves data. You can normalize to uppercase and get the same effect. In fact, the argument to use uppercase for normalization is that lowercase does not round trip. You are assuming ASCII content (or UTF-8 content that happens to only be ASCII). Throw Unicode into the mix and you get problems to deal with. Separately, after normalization, to decompress the data you are assuming every sentence must be capitalized and anything that appears to be a title or proper name must be title cased. I imagine that for some pages this isn't 100% so after decompression you will probably make have a letter be uppercase where it was originally not. Thus, your technique must be classified as a lossy algorithm.

KTibow 3y ago
>You can normalize to uppercase and get the same effect. This isn't quite true. Most HTML, JS, and user content can't be uppercased. >Thus, your technique must be classified as a lossy algorithm. I think you might be misunderstanding some parts of this. The article is just suggesting changing the case to sentence case everywhere, whether transferred or displayed. The compression comes from separate lossless algorithms.
- calibas 3y ago
  Converting everything to lowercase for the purposes of saving space when zipping is technically a lossy compression algorithm. It's an algorithm for saving space, and it doesn't recreate a perfect copy of the original data.
tmtvl 3y ago
> In fact, the argument to use uppercase for normalization is that lowercase does not round trip. I believe neither upper or lower case round trips properly, which is why Unicode invented fold case. Whenever this topic comes up I like to refer to one of my favourite talks on the topic: A Million Billion Squiggly Characters by Ricardo Signes: <https://www.youtube.com/watch?v=TmTeXcEixEg https://www.youtube.com/watch?v=TmTeXcEixEg>. Whilst nominally about Unicode handling in Perl, it's general enough that anyone using any language can learn something from it.
withinboredom 3y ago
> You can normalize to uppercase and get the same effect. No, you can't (unless you do it very carefully). Not every lower case letter can be transformed to an upper case ascii letter and back to the same lower case ascii letter. Allow me to introduce you to the letter i (in Turkey). If you naively uppercase('i') in an standard alphabet, you'll get "I" which isn't even the same letter if you are reading it in Turkey. http://www.i18nguy.com/unicode/turkish-i18n.html http://www.i18nguy.com/unicode/turkish-i18n.html
- Dylan16807 3y ago
  You're saying it's not the "same effect" but don't we have an equivalent issue with lowercasing?
  - withinboredom 3y ago
    No, because the unicode committee screwed it up and rejected fixes for it. So, it lowercases correctly, but not uppercases.
    - Dylan16807 3y ago
      Which character lowercases but not uppercases correctly? In which locales? My understanding is that I will either lowercase to i or to ı depending on locale. Just like i will uppercase to I or to İ depending on locale. In both cases, you have a problem if you want to support multiple locales. In both cases, if you set a fixed locale then it's fine.
      - zarzavat 3y ago
        Uppercase Eszett lowercases to lowercase eszett, which then uppercases to SS, which then lowercases to ss. It’s not incorrect but does show that you can get four different strings from one with just case operations.
        Dylan16807 3y ago

KTibow 3y ago

>You can normalize to uppercase and get the same effect. This isn't quite true. Most HTML, JS, and user content can't be uppercased. >Thus, your technique must be classified as a lossy algorithm. I think you might be misunderstanding some parts of this. The article is just suggesting changing the case to sentence case everywhere, whether transferred or displayed. The compression comes from separate lossless algorithms.

calibas 3y ago
Converting everything to lowercase for the purposes of saving space when zipping is technically a lossy compression algorithm. It's an algorithm for saving space, and it doesn't recreate a perfect copy of the original data.

tmtvl 3y ago

> In fact, the argument to use uppercase for normalization is that lowercase does not round trip. I believe neither upper or lower case round trips properly, which is why Unicode invented fold case. Whenever this topic comes up I like to refer to one of my favourite talks on the topic: A Million Billion Squiggly Characters by Ricardo Signes: <https://www.youtube.com/watch?v=TmTeXcEixEg https://www.youtube.com/watch?v=TmTeXcEixEg>. Whilst nominally about Unicode handling in Perl, it's general enough that anyone using any language can learn something from it.

withinboredom 3y ago

> You can normalize to uppercase and get the same effect. No, you can't (unless you do it very carefully). Not every lower case letter can be transformed to an upper case ascii letter and back to the same lower case ascii letter. Allow me to introduce you to the letter i (in Turkey). If you naively uppercase('i') in an standard alphabet, you'll get "I" which isn't even the same letter if you are reading it in Turkey. http://www.i18nguy.com/unicode/turkish-i18n.html http://www.i18nguy.com/unicode/turkish-i18n.html

Dylan16807 3y ago
You're saying it's not the "same effect" but don't we have an equivalent issue with lowercasing?
- withinboredom 3y ago
  No, because the unicode committee screwed it up and rejected fixes for it. So, it lowercases correctly, but not uppercases.
  - Dylan16807 3y ago
    Which character lowercases but not uppercases correctly? In which locales? My understanding is that I will either lowercase to i or to ı depending on locale. Just like i will uppercase to I or to İ depending on locale. In both cases, you have a problem if you want to support multiple locales. In both cases, if you set a fixed locale then it's fine.
    - zarzavat 3y ago
      Uppercase Eszett lowercases to lowercase eszett, which then uppercases to SS, which then lowercases to ss. It’s not incorrect but does show that you can get four different strings from one with just case operations.
      - Dylan16807 3y ago