4 ms·

If you give me a computer timestamp without a timezone, I can and will assume it's in UTC. It might not be, but if it's not and I process it as though it is, an

If you give me a computer timestamp without a timezone, I can and will assume it's in UTC. It might not be, but if it's not and I process it as though it is, and the sender doesn't like the results, that's on them. I'm willing to spend approximately zero effort trying to guess what nonstandard thing they're trying to send me unless they're paying me or my company a whole lot of money, in which case I'll convert it to UTC upon import and continue on from there. Same with UTF-8. Life's too short for bothering with anything else today. I'll deal with some weird janky encoding for the right price, but the first thing I'd do is convert it to UTF-8. Damned if I'm going to complicate the innards of my code with special case code paths for non-UTF-8. If there were some inherent issue with UTF-8 that made it significantly worse than some other encoding for a given task, I'd be sympathetic to that explanation and wouldn't be such a pain in the neck about this. For instance, if it were the case that it did a bad job of encoding Mandarin or Urdu or Xhosa or Persian, and the people who use those languages strongly preferred to use something else, I'd understand. However, I've never heard a viable explanation for not using UTF-8 other than legacy software support, and if you want to continue to use something ancient and weird, it's on you to adapt it to the rest of the world because they're definitely not going to adapt the world to you.

kccqzy 3y ago
> For instance, if it were the case that it did a bad job of encoding Mandarin I don't know if you picked this example on purpose, but using UTF-8 to encode Chinese is 50% larger than the old encoding (GB2312). I remember people cared about this like twenty years ago. I don't know of anyone that still cares about this encoding inefficiency. Any compression algorithm is able to remove such encoding inefficiency while using negligible CPU to decompress.
- PeterisP 3y ago
  A key aspect is that nowadays we rarely encode pure text - while other encodings are more efficient for encoding pure Mandarin, nowadays a "Mandarin document" may be an HTML or JSON or XML file where less than half of the characters are from CJK codespace, and the rest come from all the formatting overhead which is in the 7-bit ASCII range, and UTF-8 works great for such combined content.
  - tomoyoirl 3y ago
    To say nothing of mod_gzip
- kstrauser 3y ago
  That doesn't seem like the worst issue imaginable. I doubt there are too many cases where every byte counts, text uses a significant portion of the available space, and compression is unavailable or inefficient. If we were still cramming floppies full of text files destined for very slow computers, that'd be one thing. Web pages full of uncompressed text are still either so small that it's a moot point or so huge with JS, images, and fonts that the relative text size isn't that significant. Which is all to say that you're right, but I can't imagine that it's more than a theoretical nuisance outside some extremely niche cases.
  - bawolff 3y ago
    > Web pages full of uncompressed text Are basically non existent. Almost all modern web servers transparently compress html. Sending uncompressed text over the wire is extremely rare.
    - iraqmtpizza 3y ago
      They shouldn't be non-existent. Zip-then-encrypt is not secure due to information leakage. EDIT: also, it's not safe—message length is dependent on the values of the plaintext bytes, period. i'm not saying don't live dangerously, i'm just saying live dangerously knowing
      - fanf2 3y ago
        The information leakage problem occurs when compression is done in the TLS layer, because then the compression context includes both headers (with cookies) and bodies (containing potentially attacker-controlled data). But if you do compression at the HTTP layer using its Transfer-Encoding then the compression context only covers the body, which is safe.
        rocqua 3y ago
        It can still leak data if attackers can get their input reflected. I.e. I send you a word, and then I get to observe a compressed and encrypted message including my word and sensitive data. If my word matches the sensitive data, the cyphertext will be smaller. Hence I can learn things about the cipgertext. That is no longer good encryption.
- gary_0 3y ago
  For Asian languages, UTF-8 is basically the same size as any other encoding when compressed[0] (and you should be using compression if you care about space) so in practice there is no data size advantage to using non-standard encodings. [0] https://utf8everywhere.org/#asian https://utf8everywhere.org/#asian
  - neild 3y ago
    In addition, Chinese characters encode more information than English letters, so a text written in Chinese will generally consume fewer bytes than the same text in English even when using UTF-8. (Consider: Horse is five letters, but 馬 is one character. Even at three bytes per character, Chinese wins.)
    - kstrauser 3y ago
      True, but even then you wouldn’t want to store it egregiously badly.
    - Panzer04 3y ago
      Presumably that derives from the overhead of encoding an english character as a full byte? Given there's only 26 characters normally, you could fit that into 5 bits instead, which funnily enough does actually line up with the chinese character encoding (5x5 vs 1x24).
    - kps 3y ago
      Yes. It's the non-Latin alphabets that lose with either UTF-8 or UTF-16, compared with stateful ISO 2022 page switching.
hnick 3y ago
It depends on the domain. If you are writing calendar software, it is legitimate to have "floating time" i.e. your medication reminder is at 7pm every day, regardless of time zone, travel, or anything else. Unfortunately Google and many other companies have decided UTC is the only way, so this causes issues with ICS files that use that format sometimes when they are generating their helpful popups in the GMail inbox.
- kstrauser 3y ago
  Those aren’t timestamps. They’re descriptions of how to select them.
  - hnick 3y ago
    Oh yes true. Somehow I missed that important detail.
- cesarb 3y ago
  > If you are writing calendar software, it is legitimate to have "floating time" i.e. your medication reminder is at 7pm every day, regardless of time zone, travel, or anything else. If you have to take medication (for instance, an antibiotic) every 24 hours, it must be taken at the same UTC hour, even if you took a train to a town in another timezone. Keeping the same local time even when the timezone changes would be wrong for that use case.
  - kstrauser 3y ago
    There are very few drugs where that's a requirement. Your kidneys and liver aren't smart enough to metabolize anything at precisely the same rate every day anyway.
  - hnick 3y ago
    If you're there for a while, you'll need to adapt anyway since your biorhythms will too. But there are plenty of other cases like a reminder to check something after dinner, or my standard wake-up alarm in the morning. Or if I plan to travel, book lunch at a nice place for 1pm, and put it in my calendar I just want it to be 1pm wherever I go, without caring about TZ changes. Calendars, alarms, and reminders have some overlap here and floating time can be good for some cases.
LaffertyDev 3y ago
I can't help myself. The grandest of nitpicks is coming your way. I'm sorry. > If you give me a computer timestamp without a timezone, I can and will assume it's in UTC. Do you mean, give you an _offset_? `2024-04-29T14:03:06.0000-8:00` the `-8:00` is an offset. It only tells you what time this stamp occurred relative to standard time. It does not tell you anything about the region or zone itself. While I have consumed APIs that give me the timezone context as part of the response, none of them are part of the timestamp itself. The only time you should assume a timestamp is UTC is if it has the `z` at the end (assuming 8601) or is otherwise marked as UTC. Without that, you have absolutely no information about where or when the time has occurred -- it is local time. And if your software assumes a local timestamp is UTC, then I argue it is not the sender of that timestamp's problem that your software is broken. My desire to meet you at 4pm has no bearing on if the DST switchover has happened, or my government decides to change the timezone rules, or if {any other way the offset for a zone can change for future or past times}. My reminder to take my medicine at 7pm is not centered on UTC or my physical location on the planet. Its just at 7pm. Every day. If I go from New York to Paris then no, I do not want your software to tell me my medicine is actually supposed to be at Midnight. Its 7pm. But, assuming you aren't doing any future scheduling, calendar appointments, bookings, ticket sales, transportation departure, human-centric logs, or any of the other ways Local Time is incredibly useful -- ignore away.
- Spivak 3y ago
  It's always nice to see someone who actually understands time. "Convert to UTC and then throw away the time zone" only works when you need to record a specific moment in time so it's crazy how often it's recommended as the universal solution. It really isn't that hard to store (datetime, zone) and now you're not throwing away information if you ever need to do date math.
  - globular-toast 3y ago
    Yeah, I've been trying to convince people forever to store time zones with timestamps when appropriate. If you record events from around the world and don't record what time zone they happened in you can't even answer basic questions like "what proportion happened before lunch time?" People love simple rules and they will absolutely take things too far. Most developers learn "just use UTC!" and think that's the last thing they ever need to learn about time.
logrot 3y ago
Fundamentally I agree, but sadly the world isn't that simple. You usually end up with having to deal with whatever eccentric sh!t that ultimately comes from the same source as the payment for the job.
layer8 3y ago
> For instance, if it were the case that it did a bad job of encoding Mandarin Please look up the issues caused by Han unification in Unicode. It’s an important reason why the Chinese and Japanese encodings are still used in their respective territories.
- Shorel 3y ago
  That makes Unicode a failure then

kccqzy 3y ago

> For instance, if it were the case that it did a bad job of encoding Mandarin I don't know if you picked this example on purpose, but using UTF-8 to encode Chinese is 50% larger than the old encoding (GB2312). I remember people cared about this like twenty years ago. I don't know of anyone that still cares about this encoding inefficiency. Any compression algorithm is able to remove such encoding inefficiency while using negligible CPU to decompress.

PeterisP 3y ago
A key aspect is that nowadays we rarely encode pure text - while other encodings are more efficient for encoding pure Mandarin, nowadays a "Mandarin document" may be an HTML or JSON or XML file where less than half of the characters are from CJK codespace, and the rest come from all the formatting overhead which is in the 7-bit ASCII range, and UTF-8 works great for such combined content.
- tomoyoirl 3y ago
  To say nothing of mod_gzip
kstrauser 3y ago
That doesn't seem like the worst issue imaginable. I doubt there are too many cases where every byte counts, text uses a significant portion of the available space, and compression is unavailable or inefficient. If we were still cramming floppies full of text files destined for very slow computers, that'd be one thing. Web pages full of uncompressed text are still either so small that it's a moot point or so huge with JS, images, and fonts that the relative text size isn't that significant. Which is all to say that you're right, but I can't imagine that it's more than a theoretical nuisance outside some extremely niche cases.
- bawolff 3y ago
  > Web pages full of uncompressed text Are basically non existent. Almost all modern web servers transparently compress html. Sending uncompressed text over the wire is extremely rare.
  - iraqmtpizza 3y ago
    They shouldn't be non-existent. Zip-then-encrypt is not secure due to information leakage. EDIT: also, it's not safe—message length is dependent on the values of the plaintext bytes, period. i'm not saying don't live dangerously, i'm just saying live dangerously knowing
    - fanf2 3y ago
      The information leakage problem occurs when compression is done in the TLS layer, because then the compression context includes both headers (with cookies) and bodies (containing potentially attacker-controlled data). But if you do compression at the HTTP layer using its Transfer-Encoding then the compression context only covers the body, which is safe.
      - rocqua 3y ago
        It can still leak data if attackers can get their input reflected. I.e. I send you a word, and then I get to observe a compressed and encrypted message including my word and sensitive data. If my word matches the sensitive data, the cyphertext will be smaller. Hence I can learn things about the cipgertext. That is no longer good encryption.
gary_0 3y ago
For Asian languages, UTF-8 is basically the same size as any other encoding when compressed[0] (and you should be using compression if you care about space) so in practice there is no data size advantage to using non-standard encodings. [0] https://utf8everywhere.org/#asian https://utf8everywhere.org/#asian
- neild 3y ago
  In addition, Chinese characters encode more information than English letters, so a text written in Chinese will generally consume fewer bytes than the same text in English even when using UTF-8. (Consider: Horse is five letters, but 馬 is one character. Even at three bytes per character, Chinese wins.)
  - kstrauser 3y ago
    True, but even then you wouldn’t want to store it egregiously badly.
  - Panzer04 3y ago
    Presumably that derives from the overhead of encoding an english character as a full byte? Given there's only 26 characters normally, you could fit that into 5 bits instead, which funnily enough does actually line up with the chinese character encoding (5x5 vs 1x24).
  - kps 3y ago
    Yes. It's the non-Latin alphabets that lose with either UTF-8 or UTF-16, compared with stateful ISO 2022 page switching.

hnick 3y ago

It depends on the domain. If you are writing calendar software, it is legitimate to have "floating time" i.e. your medication reminder is at 7pm every day, regardless of time zone, travel, or anything else. Unfortunately Google and many other companies have decided UTC is the only way, so this causes issues with ICS files that use that format sometimes when they are generating their helpful popups in the GMail inbox.

kstrauser 3y ago
Those aren’t timestamps. They’re descriptions of how to select them.
- hnick 3y ago
  Oh yes true. Somehow I missed that important detail.
cesarb 3y ago
> If you are writing calendar software, it is legitimate to have "floating time" i.e. your medication reminder is at 7pm every day, regardless of time zone, travel, or anything else. If you have to take medication (for instance, an antibiotic) every 24 hours, it must be taken at the same UTC hour, even if you took a train to a town in another timezone. Keeping the same local time even when the timezone changes would be wrong for that use case.
- kstrauser 3y ago
  There are very few drugs where that's a requirement. Your kidneys and liver aren't smart enough to metabolize anything at precisely the same rate every day anyway.
- hnick 3y ago
  If you're there for a while, you'll need to adapt anyway since your biorhythms will too. But there are plenty of other cases like a reminder to check something after dinner, or my standard wake-up alarm in the morning. Or if I plan to travel, book lunch at a nice place for 1pm, and put it in my calendar I just want it to be 1pm wherever I go, without caring about TZ changes. Calendars, alarms, and reminders have some overlap here and floating time can be good for some cases.

LaffertyDev 3y ago

I can't help myself. The grandest of nitpicks is coming your way. I'm sorry. > If you give me a computer timestamp without a timezone, I can and will assume it's in UTC. Do you mean, give you an _offset_? `2024-04-29T14:03:06.0000-8:00` the `-8:00` is an offset. It only tells you what time this stamp occurred relative to standard time. It does not tell you anything about the region or zone itself. While I have consumed APIs that give me the timezone context as part of the response, none of them are part of the timestamp itself. The only time you should assume a timestamp is UTC is if it has the `z` at the end (assuming 8601) or is otherwise marked as UTC. Without that, you have absolutely no information about where or when the time has occurred -- it is local time. And if your software assumes a local timestamp is UTC, then I argue it is not the sender of that timestamp's problem that your software is broken. My desire to meet you at 4pm has no bearing on if the DST switchover has happened, or my government decides to change the timezone rules, or if {any other way the offset for a zone can change for future or past times}. My reminder to take my medicine at 7pm is not centered on UTC or my physical location on the planet. Its just at 7pm. Every day. If I go from New York to Paris then no, I do not want your software to tell me my medicine is actually supposed to be at Midnight. Its 7pm. But, assuming you aren't doing any future scheduling, calendar appointments, bookings, ticket sales, transportation departure, human-centric logs, or any of the other ways Local Time is incredibly useful -- ignore away.

Spivak 3y ago
It's always nice to see someone who actually understands time. "Convert to UTC and then throw away the time zone" only works when you need to record a specific moment in time so it's crazy how often it's recommended as the universal solution. It really isn't that hard to store (datetime, zone) and now you're not throwing away information if you ever need to do date math.
- globular-toast 3y ago
  Yeah, I've been trying to convince people forever to store time zones with timestamps when appropriate. If you record events from around the world and don't record what time zone they happened in you can't even answer basic questions like "what proportion happened before lunch time?" People love simple rules and they will absolutely take things too far. Most developers learn "just use UTC!" and think that's the last thing they ever need to learn about time.

logrot 3y ago

Fundamentally I agree, but sadly the world isn't that simple. You usually end up with having to deal with whatever eccentric sh!t that ultimately comes from the same source as the payment for the job.

layer8 3y ago

> For instance, if it were the case that it did a bad job of encoding Mandarin Please look up the issues caused by Han unification in Unicode. It’s an important reason why the Chinese and Japanese encodings are still used in their respective territories.

Shorel 3y ago
That makes Unicode a failure then