8 ms·

Most developers are basically powerless here, they use the standard libraries for time provided by their language, and those have to deal with what they get fro

Most developers are basically powerless here, they use the standard libraries for time provided by their language, and those have to deal with what they get from the kernel. There's nothing they can do about timestamps in the filesystem. For anyone designing new systems that store or transmit time, you should be aware of TAI64. https://cr.yp.to/libtai/tai64.html https://cr.yp.to/libtai/tai64.html Unless you are dealing with scheduling, or generating events with a particular cadence (e.g. daily, weekly, monthly) there is no reason to include UTC or time zones in your data model. They are view only concepts, with conversion to the local display format happening from TAI64 on the way out.

quickthrower2 4y ago
> Unless you are dealing with scheduling, or generating events with a particular cadence Or are concerned with spacetime :-)
- jxf 4y ago
  Aren't we all concerned with spacetime, technically?
  - FPGAhacker 4y ago
    Meh, I’m not concerned.
    - kevin_thibedeau 4y ago
      It carries a lot of weight.
    - bigiain 4y ago
      Next minute, FPGAhacker's GPS navigator directs him to drive into the local zoo's lion cage... (I don't have a reference handy, but I recall reading that if you don't allow for time dilation effects of the GPS satellite's clocks moving faster in orbit than the ground below, you lose just over 10km of position accuracy per day.)
moomoo11 4y ago
First time I hear about this, will have to read more. Is this used widely? I’ve always run my servers on UTC time and stored all dates the same.
- chronogram 4y ago
  If you're just talking about running a typical Linux server with your server applications on top, then it's 99% of the time all 64-bit and nothing to worry about. 64-bit time is going to be used on 32-bit systems as well, so if you have those systems make sure to upgrade them in a decade. Though saying this, I think you still have to manually remember to upgrade XFS filesystems so maybe there will be some things to worry about next decade.
- viraptor 4y ago
  TAI is not a timezone. For actual clock, you still want to use UTC most likely. TAI is more of a format for describing that time. You can convert from UTC to TAI by removing the 37 leap seconds we got so far. Check out https://en.wikipedia.org/wiki/International_Atomic_Time https://en.wikipedia.org/wiki/International_Atomic_Time
  - imoverclocked 4y ago
    The pedant in me is forcing me to say: The general conversion is a little more subtle than just removing 37 leap seconds. This is especially if you want to convert a date in the past since we haven't always had 37 leap seconds to account for. Also, things can get tricky right around those leap seconds when it matters.
- SPBS 4y ago
  https://dyscour.se/post/12679668746/using-tai64-for-logging https://dyscour.se/post/12679668746/using-tai64-for-logging > This means that the date is stored as a 64-bit number, the bottom 63 bits being the number of seconds since 1970-01-01 in TAI, offset by 2^62. In essence, this makes the value quite similar to time_t (which uses UTC), except for two things: > There are 63 bits to play with, not just 32 (no year-2038 problems!). > The timestamp is always monotonic, counting all leap seconds. TAI time seems to be just UTC unix timestamps but accounting for leap seconds. Since leap seconds are slated to be retired after 2035, I’m guessing TAI time and Unix time won’t be so different after that.
jeffbarr 4y ago
> Under many cosmological theories, the integers under 2^63 are adequate to cover the entire expected lifetime of the universe; in this case no extensions will be necessary. Pessimists!
- mdemare 4y ago
  Only if you use seconds, unlike the JVM which uses milliseconds.
  - yetihehe 4y ago
    This bites you in the ass in embedded devices, which then have strange behaviours every 49 days.
    - deleted 4y ago
      [deleted]
    - sgerenser 4y ago
      Experienced this first hand on an embedded device with an RTOS with a 5ms tick. The thing was locking up every 124 days (2^31 ticks). That’s long enough where it had to happen 2-3 times before someone put 2 and 2 together.
- ArnoVW 4y ago
  Indeed. As they say, eternity is very long, especially towards the end. For proof, check out this video that explains how eventually even protons will give out and evaporate. The thing that blew my mind: when things started getting very weird, and I was thinking "ok but we're roughly at the end now", I checked the indicator. It was only half-way. It is a 30 minute video. And the time speeds up logarithmically. https://www.youtube.com/watch?v=uD4izuDMUQA https://www.youtube.com/watch?v=uD4izuDMUQA
  - marcosdumay 4y ago
    AFAIK, the most accepted theory is that the Universe will allow for useful computation forever. Always get slower, but adding up to an unbounded amount of it.
13of40 4y ago
> Most developers are basically powerless here In fact, there are so many corner cases in date/time, trying to calculate anything like that yourself is probably as ill-advised as rolling your own crypto.
- bongoman37 4y ago
  Obligatory xkcd: https://xkcd.com/1883/ https://xkcd.com/1883/
Veserv 4y ago
Yes, people should use a TAI-correlated time for timestamps and events corresponding to the present or past (the future is a different story). However, they should not use TAI64 since, even though this is my first time seeing it, it is clearly a antiquated format. It is literally using a big-endian(!), 63-bit, unsigned (but biased) integer to encode time. Every single modern computer is little-endian by default and the usage of a 63-bit unsigned integer instead of a [64-bit unsigned integer] or [64-bit signed integer with reserved upper and lower values] is bizarre.
- alphazard 4y ago
  It's not helpful to criticize aspects of a system which have good reasoning behind them, without understanding and presenting that reasoning in good faith, and providing a better alternative. Endianness of formats is not a concern on modern CPUs since we have efficient mechanisms for adjusting endianness on load and store. The choice of biased 63 bit integer ensures the the same arithmetic can be performed using both a signed and unsigned 64 bit integer. The JVM comes to mind as a system without support for unsigned 64 bit integers (maybe they've fixed that by now). As mentioned the upper values are reserved for extensions, meaning the most significant bit could be set to switch to another interpretation, potentially when migrating to something more modern. So far it's unused, just evidence of good forward thinking protocol design. Additionally, big endianess allows the format to be sorted using string comparison.
  - DiggyJohnson 4y ago
    Even if I agree with the thrust of your point, I think it’s too much to claim GP is not criticizing “in good faith” just because their critique wasn’t as defended as strongly as you’d prefer. Seems like a misuse of the term, unless you think GP had ulterior motives behind their critique of a datetime standard format.
- butlerm 4y ago
  Network formats (at least the popular ones) typically transmit in "network byte order", i.e most significant byte first. There are advantages in being able to read a hex dump without manually reversing the bytes in your head for every field that is two bytes or larger. Far from antiquated I would consider the selection of big-endian formats as an indication they actually gave some thought about what they were doing, ranking debugging convenience higher than a marginal efficiency gain on some cpus. The same consideration generally applies to all binary formats intended to be portable across systems.
ajuc 4y ago
You can just change the system time :) Would be better to mock the time provider, but it's not the end of the world.
makeitdouble 4y ago
It never came to my mind that TAI64 could be something useable in “every day” programming. Thanks ! > Unless you are dealing with scheduling It felt like a niche use at first read (“I’m not building a calendar afterall”), but it’s more than half of what I’d use time manipulation methods for actually. Anything with a validity start or an expiration date for instance would fall into “scheduling”. But then timeouts and end of process estimation also fall into “scheduling” but need to not be affected by timezones etc. and be TAI64 in this case. And there must be a ton of weird cases that don’t come to my mind at the moment but would screw me when time comes.
- alphazard 4y ago
  To expand on calendar scheduling, there is still a place for TAI64. The calendar application is typically asking for "all of the events between this timestamp and that timestamp". You will have to use the Gregorian calendar, and maybe the timezone of the meeting location to figure out what those events are, and when their start and stop timestamps are. "every Monday", or "the first Thursday of the month" are all rules for generating events. It doesn't make sense to store them as UTC either. The application is going to take in some rules, and some timestamps (TAI64) and it's going to produce a list of events within those timestamps. UTC, leap seconds, the whole mess will come into play, in memory, but not in storage.
tw1984 4y ago
It may sounds convenient to just use TAI, but that does open a whole new category of issues - you get UTC and TAI time mixed, we humans are good at making mistakes, god knows how many horrible bugs, damages or even deaths could be caused by mixing those two in systems.
nicolaslem 4y ago
> There is no reason to include UTC or time zones in your data model. That is a bold claim. Not everything is about the entity viewing the data, sometimes it is about the entity producing the data. Not storing the timezone is losing some information. A few examples: - The "Date" header in emails include the timezone (or UTC offset) of the sender. It allows the recipient to know whether the email was written in the morning or the evening. - A SQL query that builds a report of restaurant orders per hour needs to normalize for time of the day in local time otherwise it is not possible to know whether people order lunch at 11 am, midday or 1 pm.
- foxhill 4y ago
  > [...] sometimes it is about the entity producing the data so should we not store that data, then? > The "Date" header in emails include the timezone (or UTC offset) of the sender [...] i don't think that's strictly true - at best it may contain the timezone configured on the local machine when the e-mail was sent (what happens when you fly?), at worst it always contains UTC anyway for privacy concerns. in either case i don't think most people know this is a field that is transmitted, and it's presumably something the sending/recieving party likely already know each other (for personal communication) > A SQL query that build a report of restaurant orders per hour needs to normalize for time of the day in local time [...] i'd posit that that data should be stored principally then, surely? fundamentally, why should the timestamp contain two data points? i wouldn't expect the database to have a single column for, say, my user id and registering user-agent string, so why would i expect it to combine a point in time with a vague location?
  - nicolaslem 4y ago
    Ultimately my comment was about entropy: a date with a timezone contains more information than a date alone. The technicalities about database columns, inaccuracies or privacy implications do not change this fact. I argued that keeping this information instead of discarding it is often useful and gave a few examples. I understand that sometimes systems can get by with only storing local dates or UTC dates but sometimes a system needs to be able to deal with both and keeping these extra bits of information is the only way.
nine_k 4y ago
The moment your data is produced in more than one time zone, you're lost without the time zone. This may happen if you have customers in more than one city, or your customers use a mobile device to access your services. Maybe your CLI utility can be timezone-oblivious, but those who develop and operate systems mentioned above, or just happen to travel, would be more happy if you handled timezones. Naive timestamps are fine as long as they do not leave a single machine, and are used to calculate durations, timeouts, etc. These do not depend on the absolute value a clock shows, but only on differences between such values.
- christophilus 4y ago
  You just have to store the user’s time zone as a preference and then convert to that time zone from UTC.
  - jsmith45 4y ago
    That is usually fine. It is generally good for system generated times representing the past. You can store in UTC and display in in any zone. It is not good for user entered historical times that they expect will remain exactly as they entered. Even if the user specifies a specific timezone, for long enough ago timestamps the TZDB does get occasional updates to past timezone data to better reflect what actually happened. If the user entered 10 AM America/New_York for a long ago date, you convert it to UTC, and then the TZDB updates historical timestamp info, based on newly uncovered evidence, now it might appear as 9 AM or 11 AM or whatever. It is also insufficient for user entered future times. If the user enters a time for an event 5 years from now in their local timezone, and the local DST rules chance, they would typically still want the event to occur at the local time, which means converting to UTC and then back won't work.
bloak 4y ago
I like the idea of wider use of TAI but there seem to be plenty of obstacles. For example: * Is it possible to set up a standards-compliant network time server that serves the time in TAI? (The older protocols, at least, seem to specify UTC.) * Is it possible to put the time in TAI in e-mail headers? (It would be a nice visible way of showing one's support for TAI, but it doesn't seem to be possible according to RFC 2822 -- https://www.rfc-editor.org/rfc/rfc2822#section-3.3 https://www.rfc-editor.org/rfc/rfc2822#section-3.3 -- though you could just put "TAI" as an "obs-zone": according to section 3.4 it would be "considered equivalent to "-0000" unless there is out-of-band information confirming their meaning")
traceroute66 4y ago
> For anyone designing new systems that store or transmit time, you should be aware of TAI64 I wouldn't hold your breath when it comes to TAI64 adoption. We live in a world where people still debate whether leap smearing is a good thing or not (and then barely implement it properly). And we live in a world where IPv6 adoption is, well, "still happening" to put it politely. Most people have not even heard of TAI64, I had heard of it, but I had forgotten about it until it was re-mentioned here about two decades since I first read up on it. The truth is in your first paragraph. Developers will use what they are given, their role its to get the coding done in the shortest reasonable timeframe. Dictating the use of a barley used form such as TAI64 is "above the pay grade" of most developers. Finally, in this increasingly cloud-first world in which we live, unless one of the big-three suddenly embrace TAI64, the whole idea of TAI64 is effectively dead in the water.

quickthrower2 4y ago

> Unless you are dealing with scheduling, or generating events with a particular cadence Or are concerned with spacetime :-)

jxf 4y ago
Aren't we all concerned with spacetime, technically?
- FPGAhacker 4y ago
  Meh, I’m not concerned.
  - kevin_thibedeau 4y ago
    It carries a lot of weight.
  - bigiain 4y ago
    Next minute, FPGAhacker's GPS navigator directs him to drive into the local zoo's lion cage... (I don't have a reference handy, but I recall reading that if you don't allow for time dilation effects of the GPS satellite's clocks moving faster in orbit than the ground below, you lose just over 10km of position accuracy per day.)

moomoo11 4y ago

First time I hear about this, will have to read more. Is this used widely? I’ve always run my servers on UTC time and stored all dates the same.

chronogram 4y ago
If you're just talking about running a typical Linux server with your server applications on top, then it's 99% of the time all 64-bit and nothing to worry about. 64-bit time is going to be used on 32-bit systems as well, so if you have those systems make sure to upgrade them in a decade. Though saying this, I think you still have to manually remember to upgrade XFS filesystems so maybe there will be some things to worry about next decade.
viraptor 4y ago
TAI is not a timezone. For actual clock, you still want to use UTC most likely. TAI is more of a format for describing that time. You can convert from UTC to TAI by removing the 37 leap seconds we got so far. Check out https://en.wikipedia.org/wiki/International_Atomic_Time https://en.wikipedia.org/wiki/International_Atomic_Time
- imoverclocked 4y ago
  The pedant in me is forcing me to say: The general conversion is a little more subtle than just removing 37 leap seconds. This is especially if you want to convert a date in the past since we haven't always had 37 leap seconds to account for. Also, things can get tricky right around those leap seconds when it matters.
SPBS 4y ago
https://dyscour.se/post/12679668746/using-tai64-for-logging https://dyscour.se/post/12679668746/using-tai64-for-logging > This means that the date is stored as a 64-bit number, the bottom 63 bits being the number of seconds since 1970-01-01 in TAI, offset by 2^62. In essence, this makes the value quite similar to time_t (which uses UTC), except for two things: > There are 63 bits to play with, not just 32 (no year-2038 problems!). > The timestamp is always monotonic, counting all leap seconds. TAI time seems to be just UTC unix timestamps but accounting for leap seconds. Since leap seconds are slated to be retired after 2035, I’m guessing TAI time and Unix time won’t be so different after that.

jeffbarr 4y ago

> Under many cosmological theories, the integers under 2^63 are adequate to cover the entire expected lifetime of the universe; in this case no extensions will be necessary. Pessimists!

mdemare 4y ago
Only if you use seconds, unlike the JVM which uses milliseconds.
- yetihehe 4y ago
  This bites you in the ass in embedded devices, which then have strange behaviours every 49 days.
  - deleted 4y ago
    [deleted]
  - sgerenser 4y ago
    Experienced this first hand on an embedded device with an RTOS with a 5ms tick. The thing was locking up every 124 days (2^31 ticks). That’s long enough where it had to happen 2-3 times before someone put 2 and 2 together.
ArnoVW 4y ago
Indeed. As they say, eternity is very long, especially towards the end. For proof, check out this video that explains how eventually even protons will give out and evaporate. The thing that blew my mind: when things started getting very weird, and I was thinking "ok but we're roughly at the end now", I checked the indicator. It was only half-way. It is a 30 minute video. And the time speeds up logarithmically. https://www.youtube.com/watch?v=uD4izuDMUQA https://www.youtube.com/watch?v=uD4izuDMUQA
- marcosdumay 4y ago
  AFAIK, the most accepted theory is that the Universe will allow for useful computation forever. Always get slower, but adding up to an unbounded amount of it.

13of40 4y ago

> Most developers are basically powerless here In fact, there are so many corner cases in date/time, trying to calculate anything like that yourself is probably as ill-advised as rolling your own crypto.

bongoman37 4y ago
Obligatory xkcd: https://xkcd.com/1883/ https://xkcd.com/1883/

Veserv 4y ago

Yes, people should use a TAI-correlated time for timestamps and events corresponding to the present or past (the future is a different story). However, they should not use TAI64 since, even though this is my first time seeing it, it is clearly a antiquated format. It is literally using a big-endian(!), 63-bit, unsigned (but biased) integer to encode time. Every single modern computer is little-endian by default and the usage of a 63-bit unsigned integer instead of a [64-bit unsigned integer] or [64-bit signed integer with reserved upper and lower values] is bizarre.

alphazard 4y ago
It's not helpful to criticize aspects of a system which have good reasoning behind them, without understanding and presenting that reasoning in good faith, and providing a better alternative. Endianness of formats is not a concern on modern CPUs since we have efficient mechanisms for adjusting endianness on load and store. The choice of biased 63 bit integer ensures the the same arithmetic can be performed using both a signed and unsigned 64 bit integer. The JVM comes to mind as a system without support for unsigned 64 bit integers (maybe they've fixed that by now). As mentioned the upper values are reserved for extensions, meaning the most significant bit could be set to switch to another interpretation, potentially when migrating to something more modern. So far it's unused, just evidence of good forward thinking protocol design. Additionally, big endianess allows the format to be sorted using string comparison.
- DiggyJohnson 4y ago
  Even if I agree with the thrust of your point, I think it’s too much to claim GP is not criticizing “in good faith” just because their critique wasn’t as defended as strongly as you’d prefer. Seems like a misuse of the term, unless you think GP had ulterior motives behind their critique of a datetime standard format.
butlerm 4y ago
Network formats (at least the popular ones) typically transmit in "network byte order", i.e most significant byte first. There are advantages in being able to read a hex dump without manually reversing the bytes in your head for every field that is two bytes or larger. Far from antiquated I would consider the selection of big-endian formats as an indication they actually gave some thought about what they were doing, ranking debugging convenience higher than a marginal efficiency gain on some cpus. The same consideration generally applies to all binary formats intended to be portable across systems.

ajuc 4y ago

You can just change the system time :) Would be better to mock the time provider, but it's not the end of the world.

makeitdouble 4y ago

It never came to my mind that TAI64 could be something useable in “every day” programming. Thanks ! > Unless you are dealing with scheduling It felt like a niche use at first read (“I’m not building a calendar afterall”), but it’s more than half of what I’d use time manipulation methods for actually. Anything with a validity start or an expiration date for instance would fall into “scheduling”. But then timeouts and end of process estimation also fall into “scheduling” but need to not be affected by timezones etc. and be TAI64 in this case. And there must be a ton of weird cases that don’t come to my mind at the moment but would screw me when time comes.

alphazard 4y ago
To expand on calendar scheduling, there is still a place for TAI64. The calendar application is typically asking for "all of the events between this timestamp and that timestamp". You will have to use the Gregorian calendar, and maybe the timezone of the meeting location to figure out what those events are, and when their start and stop timestamps are. "every Monday", or "the first Thursday of the month" are all rules for generating events. It doesn't make sense to store them as UTC either. The application is going to take in some rules, and some timestamps (TAI64) and it's going to produce a list of events within those timestamps. UTC, leap seconds, the whole mess will come into play, in memory, but not in storage.

tw1984 4y ago

It may sounds convenient to just use TAI, but that does open a whole new category of issues - you get UTC and TAI time mixed, we humans are good at making mistakes, god knows how many horrible bugs, damages or even deaths could be caused by mixing those two in systems.

nicolaslem 4y ago

> There is no reason to include UTC or time zones in your data model. That is a bold claim. Not everything is about the entity viewing the data, sometimes it is about the entity producing the data. Not storing the timezone is losing some information. A few examples: - The "Date" header in emails include the timezone (or UTC offset) of the sender. It allows the recipient to know whether the email was written in the morning or the evening. - A SQL query that builds a report of restaurant orders per hour needs to normalize for time of the day in local time otherwise it is not possible to know whether people order lunch at 11 am, midday or 1 pm.

foxhill 4y ago
> [...] sometimes it is about the entity producing the data so should we not store that data, then? > The "Date" header in emails include the timezone (or UTC offset) of the sender [...] i don't think that's strictly true - at best it may contain the timezone configured on the local machine when the e-mail was sent (what happens when you fly?), at worst it always contains UTC anyway for privacy concerns. in either case i don't think most people know this is a field that is transmitted, and it's presumably something the sending/recieving party likely already know each other (for personal communication) > A SQL query that build a report of restaurant orders per hour needs to normalize for time of the day in local time [...] i'd posit that that data should be stored principally then, surely? fundamentally, why should the timestamp contain two data points? i wouldn't expect the database to have a single column for, say, my user id and registering user-agent string, so why would i expect it to combine a point in time with a vague location?
- nicolaslem 4y ago
  Ultimately my comment was about entropy: a date with a timezone contains more information than a date alone. The technicalities about database columns, inaccuracies or privacy implications do not change this fact. I argued that keeping this information instead of discarding it is often useful and gave a few examples. I understand that sometimes systems can get by with only storing local dates or UTC dates but sometimes a system needs to be able to deal with both and keeping these extra bits of information is the only way.

nine_k 4y ago

The moment your data is produced in more than one time zone, you're lost without the time zone. This may happen if you have customers in more than one city, or your customers use a mobile device to access your services. Maybe your CLI utility can be timezone-oblivious, but those who develop and operate systems mentioned above, or just happen to travel, would be more happy if you handled timezones. Naive timestamps are fine as long as they do not leave a single machine, and are used to calculate durations, timeouts, etc. These do not depend on the absolute value a clock shows, but only on differences between such values.

christophilus 4y ago
You just have to store the user’s time zone as a preference and then convert to that time zone from UTC.
- jsmith45 4y ago
  That is usually fine. It is generally good for system generated times representing the past. You can store in UTC and display in in any zone. It is not good for user entered historical times that they expect will remain exactly as they entered. Even if the user specifies a specific timezone, for long enough ago timestamps the TZDB does get occasional updates to past timezone data to better reflect what actually happened. If the user entered 10 AM America/New_York for a long ago date, you convert it to UTC, and then the TZDB updates historical timestamp info, based on newly uncovered evidence, now it might appear as 9 AM or 11 AM or whatever. It is also insufficient for user entered future times. If the user enters a time for an event 5 years from now in their local timezone, and the local DST rules chance, they would typically still want the event to occur at the local time, which means converting to UTC and then back won't work.

bloak 4y ago

I like the idea of wider use of TAI but there seem to be plenty of obstacles. For example: * Is it possible to set up a standards-compliant network time server that serves the time in TAI? (The older protocols, at least, seem to specify UTC.) * Is it possible to put the time in TAI in e-mail headers? (It would be a nice visible way of showing one's support for TAI, but it doesn't seem to be possible according to RFC 2822 -- https://www.rfc-editor.org/rfc/rfc2822#section-3.3 https://www.rfc-editor.org/rfc/rfc2822#section-3.3 -- though you could just put "TAI" as an "obs-zone": according to section 3.4 it would be "considered equivalent to "-0000" unless there is out-of-band information confirming their meaning")

traceroute66 4y ago

> For anyone designing new systems that store or transmit time, you should be aware of TAI64 I wouldn't hold your breath when it comes to TAI64 adoption. We live in a world where people still debate whether leap smearing is a good thing or not (and then barely implement it properly). And we live in a world where IPv6 adoption is, well, "still happening" to put it politely. Most people have not even heard of TAI64, I had heard of it, but I had forgotten about it until it was re-mentioned here about two decades since I first read up on it. The truth is in your first paragraph. Developers will use what they are given, their role its to get the coding done in the shortest reasonable timeframe. Dictating the use of a barley used form such as TAI64 is "above the pay grade" of most developers. Finally, in this increasingly cloud-first world in which we live, unless one of the big-three suddenly embrace TAI64, the whole idea of TAI64 is effectively dead in the water.