13 ms·
Stop using utcnow and utcfromtimestamp
- omnicognate 4y agoStrongly implied but not explicitly stated unless I missed it: the actual surprising behaviour here is that utcnow returns a naive datetime, not a timezone aware one, despite the appropriate timezone to use being obvious.
- meitham 4y agoThanks for pointing this, I wish this was stated more clearly.
- laboratorymice 4y agostrictly speaking UTC is not a timezone
- omnicognate 4y agoTechnically correct (best kind of correct) but irrelevant here. "utc" (lowercase) is a timezone in the sense that it is a named instance of python's `timezone` class provided by the standard library, and the obvious choice if utcnow() were to return a timezone-aware datetime.
- pganssle 4y agoI did not realize that I never stated that in the original article. I've updated it to be a bit more explicit. Thanks!
- mannykannot 4y agoI was about to argue for your initial version! I would say that utcnow() is marginally acceptable (though only on backwards-compatibility grounds), but only if the conversion from a naive time to aware time raises an error if done without an explicit timezone specification. It is the combination keeping utcnow() as-was (or, more generally, the concept of naive time at all), while introducing a default for the conversion, that turns it into a subtle trap. Unfortunately, now that the default has been introduced, there seems no way to get to get out of this situation while preserving backwards-compatibility. The lesson here, perhaps, is to not make it easier to use that which should have been deprecated. Thanks for your clear and to-the-point article!
- deleted 4y ago[deleted]
- grawlinson 4y agoThis kind of stuff really needs to be made explicitly clear in any/all API documentation as it inevitably leads to underlying issues later down the line. It's amazing how something as "simple" as official time/date libraries still have footguns like these. I'm not bashing anyone that's involved as I realize we're all human and prone to mistakes, I just wish we could all do better.
- oefrha 4y agoBoth datetime.utcnow() and datetime.utcfromtimestamp() have red box warnings in the documentation. People copying code from StackOverflow won’t see them, of course. I should mention that the warnings are relatively recent, added in 3.8. https://docs.python.org/3/library/datetime.html#datetime.datetime.utcnow https://docs.python.org/3/library/datetime.html#datetime.dat...
- quickthrower2 4y agoWhen dealing with any date-time library in any platform I find it best to reread the documentation almost every time I use it.
- 9wzYQbTYsAIc 4y agoI’d second that recommendation.
- radicalbyte 4y agoSame here - it's why I like working with languages which feature IntelliSense. The documentation is right there in your IDE as you're working.
- quickthrower2 4y agoTrue but often you need something to explain at a bit more of an overview level. Intelisense helps when you know the object and want to explore the methods.
- eusto 4y ago
- calderwoodra 4y agoWhat libraries are folks using for dates/datetimes these days in python? I've been using arrow but I believe it also has some issues and doesn't work well with pandas/etc.
- alibarber 4y agopendulum has been our go-to for a couple of years now, I have been quite happy with it.
- the__alchemist 4y agoDatetime. Arrow makes the unfortunate conflation of Date, Time, and Datetime types, which causes all sorts of subtle errors when working with Dates that don't have times attached, and vice versa. The best Datetime lib I've seen in any language is Chrono in Rust. Maybe there's a way to port it to Python via FFI.
- kortex 4y agoDepends what version python you are on and what work you are doing. Our application deals with user calendars and scheduling across multiple calendar providers (google, microsoft, zoom, you name it), so I deal with the full brunt of timezone complexity all the time. I use the stdlib datetime.datetime with pytz, with a few in-house helper functions, and it's frankly kinda awful across the board but we deal with it. Pendulum is really nice, and I want to use it as my daily driver, but I unfortunately found some timezone bugs [1] in v2 that make me hesitant to trust it for what we are doing in production (it screws up DST when doing `.utcoffset()`). If you aren't constantly dealing with data timezones from all over the world, it's probably fine, and it's the only one that the least surprising thing at each step. Pytz has the whole .localize footgun. The std datetime lib has horrible ergonomics and can't even cope with IANA timezones until the zoneinfo module was introduced in 3.9. Every single day it bugs me that the classes `datetime` and `time` are lowercase, and there is no consensus on datetime.datetime vs from datetime import datetime. I tried using arrow and while I applaud them for moving away from the horrible datetime class, that makes it pretty much incompatible with any other time library, including things like pydantic and orjson. I don't have much experience with dateutil. 1 - https://github.com/sdispater/pendulum/issues/655 https://github.com/sdispater/pendulum/issues/655 Seems to have started in 2.x but is fixed in the 3.0alpha. You have to be specifically using utcoffset, which in general is probably not something you should do and use the timezone types to do conversions, but it was enough to give me pause
- nly 4y agoAm I the only one who would expect the behavior in the top example? I don't want my equality operator doing timezone conversions. If one type has no timezone attached and one does, then you probably shouldn't be able to compare them at all. Likewise, for a type that has a timezone attached, if the timezone differs between a and b, then I want a == b to return false, even if it represents the same instant in time. Equality can be a tricky thing. Named functions are the way to go when two different developers might intuit different behaviors.
- stavros 4y ago> Likewise, for a type that has a timezone attached, if the timezone differs between a and b, then I want a == b to return false, even if it represents the same instant in time. Really? So if you want to compare that two times are really the same time for two different timezones, you'd want to convert them to UTC first? What's the use case for this? Seems like the less-used way to me.
- nly 4y agoYou don't need to convert them to UTC, just one to the timezone of the other, but yes. "a.is_same_instant(b)" is a lot more clear to me than "a == b", especially in a dynamic language where it might not be obvious what the types involved are.
- EdwardDiego 4y agoIn a very globalised world, where your developers can be working in a multitude of countries, I've found that the only sane thing to do is a) run all your boxes with a TZ of Zulu and b) the only date format allowed in logs is RFC-3339. Sure, you can format datetimes in localised formats for end users, but standardising on UTC+0 and RFC-3339 for programming and maintenance purposes just prevents so much confusion and hassle. Not that I've ever had to spend a day trying to troubleshoot a bug that was alerted by our monitoring system at 2020-11-09T09:11:05Z (as it was defaulting to UTC), but the relevant logs were timestamped at 10:11:05 09/11/2020 (I only wished they'd used the normal German period separators in the date to give me a clue) because of course your German colleagues want their timestamps in German format and defaulting to CET... When you have colleagues in Europe, Oceania, and distributed from coast to coast in the US, doing everything in UTC means everyone only has to do one mental conversion from their local timezone to UTC, as opposed to trying to remember if they're in LA or New York, or if they're in Mountain Time, if they're in the parts of the US that are Mountain Time but don't do daylight savings.
- andreareina 4y agoI have no end of complaints about cloudwatch but them automatically parsing datetimes and presenting them in my browser's timezone is a godsend.
- EdwardDiego 4y agoI'm okay with that when the tool makes it clear it's done so, and it's easy to switch to the original TZ. (I always set CW back to UTC, for consistency)
- Dunedan 4y agoCloudWatch at least lets you choose in which timezone to display datetimes. Other AWS services don't let you choose and don't even show in which timezone they display datetimes, which is somewhat infuriating to me, when I'm in a hurry anyway and suddenly have to wonder which timezone the shown dates are in. I also remember that AWS used to display some datetimes in Pacific Standard Time, which is completely useless to somebody outside the US, however I haven't seen that in a while anymore. Similar to the global language switch I wish the AWS Management Console would allow setting in which timezone to display datetimes globally as well.
- BiteCode_dev 4y agoYes, the datetime api once timezone is involved is quite bad in python. This is why for anything that uses timezone, I use pendulum: https://pendulum.eustace.io/ https://pendulum.eustace.io/ It's compatible with the datetime api, but it has sane default, nice tools to convert between timezone, some cool date adjustment stuff, and can humanize time in several languages. Basically, datetime has the same problem as text vs raw bytes in python 2.7, except it has never been fixed.
- ur-whale 4y agoTime APIs in Python are the worst when it comes to time zones. Inconsistent through various python version, poorly documented and utterly confusing to use, as demonstrated by this article.
- munch117 4y agoActually it's sticking with backwards compatibility that is causing most of the problems. It's not hard to come up with an improved library design. But it is extremely hard to improve it while remaining compatible with the billions of lines of Python code out there.
- raverbashing 4y agoAgree. utcnow is awful. It doesn't do what it says on the tin. And just the naive/aware crap is almost like the unicode conversion previously. Ugh
- iainmerrick 4y agoTimezones aren’t really the problem, implicit locales are the problem. All this stuff would be a lot easier if timezones always had to be stated explicitly.
- munch117 4y agoThe real problem is that datetime.datetime.now() returns a naïve datetime. If only it returned something that contained the UTC offset in which is was created, then all of this would be much easier to deal with. Conversions to timestamps would just work, and comparisons would just work. Note that I said "UTC offset", not "timezone". datetime doesn't deal in UTC offsets, it deals in timezones. And timezones are a zillion times more complicated, which is why no one want to deal with them on the critical path, and no one has ever proposed having now() return an object with .tzinfo set. Logical though it may be.
- cesarb 4y ago> All this stuff would be a lot easier if timezones always had to be stated explicitly. On Java, you can use the forbidden-apis build plugin (https://github.com/policeman-tools/forbidden-apis https://github.com/policeman-tools/forbidden-apis) to fail the build whenever a timezone or locale or charset is not specified explicitly (it forbids the methods from the Java API which use an implicit timezone/locale/charset). I don't know whether there's something similar for Python; it might be harder because Python is much more dynamic (though it might be possible to use monkeypatching to warn whenever the bad methods are used).
- iainmerrick 4y agoYeah, I think this is something where static typechecking helps a lot.
- snthd 4y ago(2019) A follow up from April 2022: https://blog.ganssle.io/articles/2022/04/naive-local-datetimes.html https://blog.ganssle.io/articles/2022/04/naive-local-datetim...
- snthd 4y agoPylint issue #6234 "Warn usage of datetime.utcnow()" https://github.com/PyCQA/pylint/issues/6234 https://github.com/PyCQA/pylint/issues/6234
- greatgib 4y agoThis article does not make any sense. You don't get a timestamp from nowhere without knowing the kind of time it references: local or UTC or fixed to anything like client time. For python the important thing to understand is the difference between tz aware and naive datetime objects. But after you can do depending on your needs. For example, if you ensure to convert your inputs to UTC and output UTC also, you can easily just work with utcnow by only dealing with UTC internally. You would have to do that to store datetime in MySQL for example.
- Dunedan 4y ago> You don't get a timestamp from nowhere without knowing the kind of time it references: local or UTC or fixed to anything like client time. Imagine you have a software which deals with CSV files sent by clients. Clients are international corporations. What's the timezone the datetimes in the CSV files belongs to?
- greatgib 4y agoSomething has generated this file no? Timestamp has nothing to do with the problem. If you get dates without timezone or any info with your CSV, you can't deduce anything. That has nothing to do with the api used in python to decode it...
- stall84 4y agoi love articles like this
- zzzeek 4y agoI would say the real bug is this, that is, in the astimezone() method: https://docs.python.org/3/library/datetime.html#datetime.datetime.astimezone https://docs.python.org/3/library/datetime.html#datetime.dat... "If self is naive, it is presumed to represent time in the system timezone." presumed is kind of a bad word in Python. I would never write an API today that "presumed" something that can be lots of other things and I would not allow any tz conversion on a naive datetime without requiring the existing known timezone be passed.
- pganssle 4y agoAs mentioned in the article, this was a deliberate change in Python 3. If you read this article: https://blog.ganssle.io/articles/2022/04/naive-local-datetimes.html https://blog.ganssle.io/articles/2022/04/naive-local-datetim... you will see why this actually makes quite a bit of sense, given the design constraints the authors were working with.
- zzzeek 4y agoI tried to read it, will try again later but I wasn't really getting it. the idea would be, .astimezone() is bad, and should be something equivalent to .convert_between_timezones(from, to), where there's some convenient way for "from" to indicate "the system time zone", but you still have to be explicit that you consider this arbitrary number to be in a particular time zone. not following why the "this is the system timezone" must be hardcoded to be an invisible assumption, as opposed to something somewhat explicit. I mean this is literally not far off from a simple namechange of ".astimezone()".
- teaearlgraycold 4y agoThis makes me realize being a developer in England is a huge footgun.
- unity1001 4y agoHow about using unix time and converting to whatever timezone when needed...
- oblvious-earth 4y agoOkay but my most common case for datetime.utcnow() is that I want the time in UTC for an external system that doesn't understand timezones, you really want me to do datetime.now(tz=timezone.utc).replace(tz=None)?
- Aramgutang 4y agoI was familiar with the issue described, but it was useful to learn the shortcut of passing a timezone to `now()`, since I used to rely on `utcnow().replace()`, which was needlessly verbose. # what I used to do: >>> datetime.utcnow().replace(tzinfo=timezone.utc) # what I'll be doing from now on: >>> datetime.now(timezone.utc)