7 ms·

What’s naive and dangerous about Python’s Datetime objects?

by ggolu2 6y ago

alanfranz 6y ago
Your question implies that you don't know about the nuisances of the datetime library :-) (see https://docs.python.org/3/library/datetime.html https://docs.python.org/3/library/datetime.html it's the first paragraph!) Python datetime objects, by design, can be naive or timezone-aware. Timezone-aware datetime objects are OK; they identify a certain instant in time. Naive datetime objects are Python-only abstractions (AFAIK) that don't identify anything in the real world; they're highly error prone, because there's no "right" way to use them. They sort of work properly only if used in a very limited scope - e.g. your own code only, for small sections - but they're risky because they're not a different type (with respect to tz-aware), and it's hard to tell what any code accepting a datetime does if passed a naive object. Some libraries like java.time DO have a similar concept (e.g. LocalTime, LocalDate) but they keep it well separate from the "real" concept (e.g. Instant or Date in Java) so you can't use them accidentally. Example: you pass a naive datetime object to any library which must translate it to an instant, like an ISO string with a well defined timezone. What does the library do? Throw an exception? Associate an arbitrary timezone (e.g. UTC)? Associate the local, current timezone? There's no "correct" behaviour.
- aucontraire 6y ago
  I agree that python datetime objects are problematic, but for the opposite reason. It is tzinfo that is the sneaky disaster, the plain datetimes are fine. Transparent timezone awareness always fail, unless you are 100% certain that a tzaware datetime object will remain uncoverted from the very top to the very bottom of the stack and all the way up again no matter who is reading and what they are doing. For longterm minimization of pain, bugs and effort, you convert datetimes to UTC as early as possible and take them back to some localized version as late as possible (in the frontend, for a webapp, so that the backend never needs to know there is such a thing as timezones (except for separate validation and correction routines, since timezone definitions always end up being incorrect to some degree when you use them at scale)). If the localization of the datetime is an essential aspect (such as the departure time of a ship leaving port), you store a UTC value together with a record of the location. Only at the latest possible moment of processing, should you do a lookup on the location data to make a local time. Obviously, there will be exceptions to this rule. If you batch process billions of timestamps under a tight deadline and must do calculations in local time, it might make sense to have the values persisted localized.
  - alanfranz 6y ago
    > I agree that python datetime objects are problematic, but for the opposite reason. It is tzinfo that is the sneaky disaster, the plain datetimes are fine. Why naive datetimes should be fine? How are they fine? What do they represent? > For longterm minimization of pain, bugs and effort, you convert datetimes to UTC This COULD work if naive objects had an IMPLIED UTC in their contract - e.g. naive objects are declared as ALWAYS UTC. Your argument fails as soon as you pass a naive datetime object somewhere in a library/framework and it gets accepted, and/or you try serializing it without augmenting with a TZ. As I said, naive datetime only works if you control 100% of their usage. No libraries, no external points of contact. And, the reason because tz-aware objects sometimes fail for the opposite reason (e.g. libraries assume naive objects) is a fault of the API design (they're not distinct types), but the problem lies in the existence of the naive version, not vice-versa. For the records: in the backend I always use tz-aware datetime objects with a fixed UTC timezone. That's the best way IMHO not to get crazy with time problems in Python. So, your points about datetime handling are all valid and correct (timezone is mostly a "UI problem" and should not leak into the backend) but they don't prove your "naive works better" argument.
    - icebraining 6y ago
      > Why naive datetimes should be fine? How are they fine? What do they represent? They represent the date/time wherever the user is (location-independent). If I want to take a pill every Monday and Thursday at 10am, I don't want to get a notification at 5am just because I moved from the UK to NY.
      - aucontraire 6y ago
        Or maybe you do want to take the pill at 5am, since your are only there for a few days and it is critical that you maintain an exact 24 hour interval between doses. As an assembly worker in the timestamp-wrapper-class factory, I am not in a position to try being clever about it. :-)
      - alanfranz 6y ago
        This is LocalDate/LocalTime in java.time/joda.time parlance. But it's a different beast; in fact, you're talking about a repeated action. But if I tell you that on "March 23rd, 2020, 9:50:01am" I did something, what does that mean to you? When did it happen? That's a naive datetime. It's got its place: but the idea that the API and the usage should be similar to a precise representation of time, as if it the two were interchangeable is... dangerous, and it's the source of a lot of problems with dt in the Python world.
- pydry 6y ago
  >What does the library do? Throw an exception? Associate an arbitrary timezone (e.g. UTC)? Associate the local, current timezone? Naive datetime is what datetime.utcnow() returns. UTC is essentially a "default" timezone. I've always thought it made most sense in a library to assume it's UTC.
  - alanfranz 6y ago
    Your assumption is just a guess. Take a look at the python docs I linked just above here: "A naive object does not contain enough information to unambiguously locate itself relative to other date/time objects. Whether a naive object represents Coordinated Universal Time (UTC), local time, or time in some other timezone is purely up to the program"
    - pydry 6y ago
      I'd consider any datetimes that have an implicit timezone that isn't or might not be UTC a bug in any system. This isn't restricted to python. Servers that spit out logs with timestamps, for instance, should be spitting out UTC. It makes sense to build systems that deal with timezones at the very edges (and sometimes not even then) and use UTC for everything else. It's simpler that way.
      - jayjader 6y ago
        Unfortunately, in Python, whenever you try to render a datetime "aware" by using the provided conversion method (`astimezone()`) it will assume the naive datetime is in local time zone, not UTC. The datetime module provides `timezone.utc` to be used whenever you want to have datetimes _in utc_, but it needs to be explicitly used by the programmer.
  - gimboland 6y ago
    https://blog.ganssle.io/articles/2019/11/utcnow.html https://blog.ganssle.io/articles/2019/11/utcnow.html
kissgyorgy 6y ago
Basically everything.

What’s naive and dangerous about Python’s Datetime objects?

by ggolu2 6y ago

What’s naive and dangerous about Python’s Datetime objects?

Your question implies that you don't know about the nuisances of the datetime library :-) (see https://docs.python.org/3/library/datetime.html https://docs.python.org/3/library/datetime.html it's the first paragraph!) Python datetime objects, by design, can be naive or timezone-aware. Timezone-aware datetime objects are OK; they identify a certain instant in time. Naive datetime objects are Python-only abstractions (AFAIK) that don't identify anything in the real world; they're highly error prone, because there's no "right" way to use them. They sort of work properly only if used in a very limited scope - e.g. your own code only, for small sections - but they're risky because they're not a different type (with respect to tz-aware), and it's hard to tell what any code accepting a datetime does if passed a naive object. Some libraries like java.time DO have a similar concept (e.g. LocalTime, LocalDate) but they keep it well separate from the "real" concept (e.g. Instant or Date in Java) so you can't use them accidentally. Example: you pass a naive datetime object to any library which must translate it to an instant, like an ISO string with a well defined timezone. What does the library do? Throw an exception? Associate an arbitrary timezone (e.g. UTC)? Associate the local, current timezone? There's no "correct" behaviour.

aucontraire 6y ago
I agree that python datetime objects are problematic, but for the opposite reason. It is tzinfo that is the sneaky disaster, the plain datetimes are fine. Transparent timezone awareness always fail, unless you are 100% certain that a tzaware datetime object will remain uncoverted from the very top to the very bottom of the stack and all the way up again no matter who is reading and what they are doing. For longterm minimization of pain, bugs and effort, you convert datetimes to UTC as early as possible and take them back to some localized version as late as possible (in the frontend, for a webapp, so that the backend never needs to know there is such a thing as timezones (except for separate validation and correction routines, since timezone definitions always end up being incorrect to some degree when you use them at scale)). If the localization of the datetime is an essential aspect (such as the departure time of a ship leaving port), you store a UTC value together with a record of the location. Only at the latest possible moment of processing, should you do a lookup on the location data to make a local time. Obviously, there will be exceptions to this rule. If you batch process billions of timestamps under a tight deadline and must do calculations in local time, it might make sense to have the values persisted localized.
- alanfranz 6y ago
  > I agree that python datetime objects are problematic, but for the opposite reason. It is tzinfo that is the sneaky disaster, the plain datetimes are fine. Why naive datetimes should be fine? How are they fine? What do they represent? > For longterm minimization of pain, bugs and effort, you convert datetimes to UTC This COULD work if naive objects had an IMPLIED UTC in their contract - e.g. naive objects are declared as ALWAYS UTC. Your argument fails as soon as you pass a naive datetime object somewhere in a library/framework and it gets accepted, and/or you try serializing it without augmenting with a TZ. As I said, naive datetime only works if you control 100% of their usage. No libraries, no external points of contact. And, the reason because tz-aware objects sometimes fail for the opposite reason (e.g. libraries assume naive objects) is a fault of the API design (they're not distinct types), but the problem lies in the existence of the naive version, not vice-versa. For the records: in the backend I always use tz-aware datetime objects with a fixed UTC timezone. That's the best way IMHO not to get crazy with time problems in Python. So, your points about datetime handling are all valid and correct (timezone is mostly a "UI problem" and should not leak into the backend) but they don't prove your "naive works better" argument.
  - icebraining 6y ago
    > Why naive datetimes should be fine? How are they fine? What do they represent? They represent the date/time wherever the user is (location-independent). If I want to take a pill every Monday and Thursday at 10am, I don't want to get a notification at 5am just because I moved from the UK to NY.
    - aucontraire 6y ago
      Or maybe you do want to take the pill at 5am, since your are only there for a few days and it is critical that you maintain an exact 24 hour interval between doses. As an assembly worker in the timestamp-wrapper-class factory, I am not in a position to try being clever about it. :-)
    - alanfranz 6y ago
      This is LocalDate/LocalTime in java.time/joda.time parlance. But it's a different beast; in fact, you're talking about a repeated action. But if I tell you that on "March 23rd, 2020, 9:50:01am" I did something, what does that mean to you? When did it happen? That's a naive datetime. It's got its place: but the idea that the API and the usage should be similar to a precise representation of time, as if it the two were interchangeable is... dangerous, and it's the source of a lot of problems with dt in the Python world.
pydry 6y ago
>What does the library do? Throw an exception? Associate an arbitrary timezone (e.g. UTC)? Associate the local, current timezone? Naive datetime is what datetime.utcnow() returns. UTC is essentially a "default" timezone. I've always thought it made most sense in a library to assume it's UTC.
- alanfranz 6y ago
  Your assumption is just a guess. Take a look at the python docs I linked just above here: "A naive object does not contain enough information to unambiguously locate itself relative to other date/time objects. Whether a naive object represents Coordinated Universal Time (UTC), local time, or time in some other timezone is purely up to the program"
  - pydry 6y ago
    I'd consider any datetimes that have an implicit timezone that isn't or might not be UTC a bug in any system. This isn't restricted to python. Servers that spit out logs with timestamps, for instance, should be spitting out UTC. It makes sense to build systems that deal with timezones at the very edges (and sometimes not even then) and use UTC for everything else. It's simpler that way.
    - jayjader 6y ago
      Unfortunately, in Python, whenever you try to render a datetime "aware" by using the provided conversion method (`astimezone()`) it will assume the naive datetime is in local time zone, not UTC. The datetime module provides `timezone.utc` to be used whenever you want to have datetimes _in utc_, but it needs to be explicitly used by the programmer.
- gimboland 6y ago
  https://blog.ganssle.io/articles/2019/11/utcnow.html https://blog.ganssle.io/articles/2019/11/utcnow.html

kissgyorgy 6y ago

Basically everything.