6 ms·

I think Arrow still leaves too much confusion around how to handle timezones. That's why I wrote DMC(https://github.com/rhettg/dmc https://github.com/rhettg/dmc

by rhettg 12y ago

I think Arrow still leaves too much confusion around how to handle timezones. That's why I wrote DMC(https://github.com/rhettg/dmc https://github.com/rhettg/dmc). Not to be confused with Delorean (https://github.com/myusuf3/delorean https://github.com/myusuf3/delorean) which is a lot like Arrow.

vbit 12y ago
I did not know about DMC and am very happy to see this as I've often desired a simple but proper API for time. I always wondered why so many date+time libraries complicate matters so much. I strongly agree with the principles you chose - specificaly - all time storage and math should be done in UTC (the 'unicode' of time). Timezones only come in when you display or parse times (the 'encodings' of time). I hope more and more libraries apply this idea. It makes the API a lot simpler and error retardant. Good job!
- csirac2 12y ago
  I strongly agree with the principles you chose - specificaly - all time storage and math should be done in UTC I've had this argument many times before. Throwing away TZ is in fact throwing away important contextual information. Having worked on a few data cleaning projects to re-purpose old scientific datasets, I can tell you that the quality of these data would have suffered markedly, and many corrections would not have been possible if the TZ/offsets had been discarded, Eg: "The geolocation for these records cross a TZ boundary here but the UTC offset doesn't reflect that until here so that explains the 1hr gap in the record where the user realized her mistake half-way into the new TZ so we should add an hour to these 860 records" Yes, it would be better if raw data was recorded in UTC in the first place but when working with such data the ability to make decisions and inferences from the TZ is very useful.
  - vbit 12y ago
    > Yes, it would be better if raw data was recorded in UTC in the first place but when working with such data the ability to make decisions and inferences from the TZ is very useful. I maintain that time and location are orthogonal. If the application needs both, store both - properly in separate fields. That's better than shoving the granular location in the form of a timezone in the time field.
    - csirac2 12y ago
      Yes, from a data modeler's perspective it is definitely orthogonal. From an evidence-based perspective though, a full "chain of custody" has to be maintained (as far as alterations go), providence of information can be very important. If you care about reproducibility, how raw data has been manipulated from the original, you'd better have some record of it. For date-times almost all public datasets standardize on ISO8601. Any datetimes captured by persons or processes which aren't working in UTC from the very beginning will have the TZ offset in it. Normalizing to UTC is up to the data consumer to do as they please, but some get very upset if you try to say that an event was recorded as happening at 03:00 if the person or process actually recorded 13:00+10:00. It may seem subtle and boring and pointless, but the fact is that you may be seen to be changing the facts.
    - lomnakkus 12y ago
      The issue here seems to be two different definitions of time. There's "physical" time, i.e. a continuous line and there's human/calendar time which is different and is most definitely tied to a location. For an example, see http://en.wikipedia.org/wiki/February_30#Swedish_calendar http://en.wikipedia.org/wiki/February_30#Swedish_calendar . (No wonder that time zones complicate things enormously.)
    - angersock 12y ago
      Amen to this. We've got a complete clusterfuck in our legacy code because somebody decided that UTC was not the right answer, nor was timezones quite, and so somehow made the heinous mistake of creating a "local UTC" and separately storing the TZ. The mind boggles. I'm still straightening that mess out. People, computers care not for your "calendar time"--usually, you want physical time and can fix the rest at the view level.
    - gpvos 12y ago
      No. It is in fact best to store the local time and the location (i.e., not "+01:00", but for example "Europe/Amsterdam"). Not UTC with or without time zone. For example, if you have an appointment in the future, and the time zone or daylight savings time rules change, this is the only way that the appointment can still be at the correct moment. And time zone rules do change. And this way you can still represent absolute UTC time if you wish. With another representation, you cannot represent human-society time, which is often what you actually want.
- fanf2 12y ago
  Doing all storage and computation in UTC does not work if you are dealing with events in the future that need a fixed local time even if time zone offsets change. http://fanf.livejournal.com/104586.html http://fanf.livejournal.com/104586.html
  - techdragon 12y ago
    As a collector of edge cases and other 'valuable weirdness', I'm at a loss to picture such a use case short of some obscure archival process with interoperability constraints... can you elaborate to clarify when this may be a problem? I was fairly sure ISO8601 with correct timezone information for every item, was completely sufficient for all my time needs, so if theres something it fails at, id like to know more.
stuaxo 12y ago
Cool! Have you looked at the moment.js manual ? They have a lot of really useful examples (I believe Arrow was inspired by Moment).
pjkundert 12y ago
Indeed; since timezones have the same abbreviations for multiple different timezones, it is difficult to collect a set of abbreviations that are consistent across a data set. I wrote cpppo (https://github.com/pjkundert/cpppo.git https://github.com/pjkundert/cpppo.git) to work with industrial data communications in various timezones. The cpppo.history module assures fast and consistent handling (and serialization/deserialization) of time series data across a selected set of consistent timezone abbreviations. If speed and consistency is an issue it may be of interest.
plunckett 12y ago
I think dmc leaves too much to be implemented. Why dismiss a full project on PyPi for a half-written, untested one?
- rhettg 12y ago
  You're right. DMC is really just an experiment. I don't mean to disparage Arrow, it does improve the interfaces around datetime, I was just disappointed it didn't attempt to prevent these common timezone related bugs. I bring up dmc for discussion, because I started it right after finding arrow. Building a full wrapper like this hadn't occurred to me until I saw arrow do it.

by rhettg 12y ago

vbit 12y ago

I did not know about DMC and am very happy to see this as I've often desired a simple but proper API for time. I always wondered why so many date+time libraries complicate matters so much. I strongly agree with the principles you chose - specificaly - all time storage and math should be done in UTC (the 'unicode' of time). Timezones only come in when you display or parse times (the 'encodings' of time). I hope more and more libraries apply this idea. It makes the API a lot simpler and error retardant. Good job!

csirac2 12y ago
I strongly agree with the principles you chose - specificaly - all time storage and math should be done in UTC I've had this argument many times before. Throwing away TZ is in fact throwing away important contextual information. Having worked on a few data cleaning projects to re-purpose old scientific datasets, I can tell you that the quality of these data would have suffered markedly, and many corrections would not have been possible if the TZ/offsets had been discarded, Eg: "The geolocation for these records cross a TZ boundary here but the UTC offset doesn't reflect that until here so that explains the 1hr gap in the record where the user realized her mistake half-way into the new TZ so we should add an hour to these 860 records" Yes, it would be better if raw data was recorded in UTC in the first place but when working with such data the ability to make decisions and inferences from the TZ is very useful.
- vbit 12y ago
  > Yes, it would be better if raw data was recorded in UTC in the first place but when working with such data the ability to make decisions and inferences from the TZ is very useful. I maintain that time and location are orthogonal. If the application needs both, store both - properly in separate fields. That's better than shoving the granular location in the form of a timezone in the time field.
  - csirac2 12y ago
    Yes, from a data modeler's perspective it is definitely orthogonal. From an evidence-based perspective though, a full "chain of custody" has to be maintained (as far as alterations go), providence of information can be very important. If you care about reproducibility, how raw data has been manipulated from the original, you'd better have some record of it. For date-times almost all public datasets standardize on ISO8601. Any datetimes captured by persons or processes which aren't working in UTC from the very beginning will have the TZ offset in it. Normalizing to UTC is up to the data consumer to do as they please, but some get very upset if you try to say that an event was recorded as happening at 03:00 if the person or process actually recorded 13:00+10:00. It may seem subtle and boring and pointless, but the fact is that you may be seen to be changing the facts.
  - lomnakkus 12y ago
    The issue here seems to be two different definitions of time. There's "physical" time, i.e. a continuous line and there's human/calendar time which is different and is most definitely tied to a location. For an example, see http://en.wikipedia.org/wiki/February_30#Swedish_calendar http://en.wikipedia.org/wiki/February_30#Swedish_calendar . (No wonder that time zones complicate things enormously.)
  - angersock 12y ago
    Amen to this. We've got a complete clusterfuck in our legacy code because somebody decided that UTC was not the right answer, nor was timezones quite, and so somehow made the heinous mistake of creating a "local UTC" and separately storing the TZ. The mind boggles. I'm still straightening that mess out. People, computers care not for your "calendar time"--usually, you want physical time and can fix the rest at the view level.
  - gpvos 12y ago
    No. It is in fact best to store the local time and the location (i.e., not "+01:00", but for example "Europe/Amsterdam"). Not UTC with or without time zone. For example, if you have an appointment in the future, and the time zone or daylight savings time rules change, this is the only way that the appointment can still be at the correct moment. And time zone rules do change. And this way you can still represent absolute UTC time if you wish. With another representation, you cannot represent human-society time, which is often what you actually want.
fanf2 12y ago
Doing all storage and computation in UTC does not work if you are dealing with events in the future that need a fixed local time even if time zone offsets change. http://fanf.livejournal.com/104586.html http://fanf.livejournal.com/104586.html
- techdragon 12y ago
  As a collector of edge cases and other 'valuable weirdness', I'm at a loss to picture such a use case short of some obscure archival process with interoperability constraints... can you elaborate to clarify when this may be a problem? I was fairly sure ISO8601 with correct timezone information for every item, was completely sufficient for all my time needs, so if theres something it fails at, id like to know more.

stuaxo 12y ago

Cool! Have you looked at the moment.js manual ? They have a lot of really useful examples (I believe Arrow was inspired by Moment).

pjkundert 12y ago

Indeed; since timezones have the same abbreviations for multiple different timezones, it is difficult to collect a set of abbreviations that are consistent across a data set. I wrote cpppo (https://github.com/pjkundert/cpppo.git https://github.com/pjkundert/cpppo.git) to work with industrial data communications in various timezones. The cpppo.history module assures fast and consistent handling (and serialization/deserialization) of time series data across a selected set of consistent timezone abbreviations. If speed and consistency is an issue it may be of interest.

plunckett 12y ago

I think dmc leaves too much to be implemented. Why dismiss a full project on PyPi for a half-written, untested one?

rhettg 12y ago
You're right. DMC is really just an experiment. I don't mean to disparage Arrow, it does improve the interfaces around datetime, I was just disappointed it didn't attempt to prevent these common timezone related bugs. I bring up dmc for discussion, because I started it right after finding arrow. Building a full wrapper like this hadn't occurred to me until I saw arrow do it.