10 ms·
Pytz: The Fastest Footgun in the West (2018)
- stan_kirdey 3y agoI've read the title and thought "oh this seems to be a weird name for a footgun. It would totally make sense if it was about timezones". Then I clicked the link.
- iknownothow 3y agoI know some of the HN crowd are gonna wag their finger at me but the moment I have to deal with timestamps in Python, even for the most simplest of cases, I parachute in pandas. `pd.Timestamp` (and its friends like `pd.Timedelta`) in my opinion are the most powerful timestamp toolkit available anywhere. To me it's very often easily worth the extra 50-150MB of install size that just works. I've used pandas for around 7 years now and it was inconsistent and buggy at the beginning. But these days, it's practically flawless. Also I'm aware that pd.Timestamp is built on top of many awesome libraries.
- robertlagrant 3y agoI'm not a Pandas person - is it possible to install the Pandas timestamp functionality as a smaller library, or do you need to install the whole shebang?
- iknownothow 3y agoI had investigated if this was possible to do but I couldn't find a way to do it myself. It's possible that pd.Timestamp might be intricately tied interally to vectorization optimizations using numpy or others. Therefore separating out just the timestamp stuff from pandas would be a lot of work.
- dagw 3y agoPython is in many ways the anti-JavaScript when it comes to packaging. Where JavaScript will have a installable package that contains a single trivial function, Python packages will bundle hundreds of functions, all vaguely related to an area, with no easy way of just installing what you actually need.
- noitpmeder 3y agoMy experience has been exactly the opposite, with two exceptions: pandas and numpy. Generally every other library works in terms of standard python types where possible. Very few other places will you see things like a re-implementation of a time library.
- dagw 3y agowith two exceptions ...and scipy and all the scikit libraries and whole bunch of others. 'Accidentally' pulling in hundreds of MBs of libraries you never use for a 100 line data processing script is par for the course in python land.
- nerdponx 3y agoMy experience is that Pendulum does a good job of handling date/time data as a drop-in replacement for the standard library classes.
- FObersteiner 3y ago> pd.Timestamp is built on top of many awesome libraries ... pytz for example ;-)
- orthoxerox 3y agoYes, and? Doesn't mean it inherits all the usability flaws of pytz.
- FObersteiner 3y agoNo, I think pandas actually abstracts away pytz's footguns, so that's a good thing.
- marcosdumay 3y agoAs long as it hides the footguns, you are good to go. Pytz is not technically flawed.
- pganssle 3y agoWell, pytz actually is technically flawed at the moment because it cannot read TZif Version 2+ files, so it breaks for dates >2038 (and it would break if you tried to point it at system data generated with zic's "slim" mode), see this: https://github.com/stub42/pytz/issues/48 https://github.com/stub42/pytz/issues/48 But you are right that pandas' use of pytz is benign and as far as I know doesn't suffer the same usability issues.
- irjustin 3y agoPendulum is quite good[0]. Relatively intuitive interface. When I left Rails, this is one area I really missed. Time.use_zone("Singapore") { (Time.zone.now - 3.days).beginning_of_week } This readable line in datetime utils or even pendulum, is such a pain. [0] https://github.com/sdispater/pendulum https://github.com/sdispater/pendulum
- ttymck 3y agoHow does the rails example work? It must be using a global/shared state?
- irjustin 3y agoSorry I'm a bit confused by the question could you elaborate?
- wussboy 3y agoI've been using pendulum heavily in production for over a year now and it has made my life very much simpler. It's a great library.
- jstx1 3y agoHandling timestamps well was one of the motivations for building pandas in the first place (Wes McKinney started pandas while he was working at an investment management company).
- arcanemachiner 3y agoDjango has similar functionality: from django.utils import timezone now = timezone.now() later = timezone.now() + timezone.timedelta(minutes=1)
- deleted 3y ago[deleted]
- noirscape 3y agoThis article is correct, although slightly outdated - Python has been shipping its own timezone library called zoneinfo to solve this specific issue[0]. On the various *nix systems it just uses your systems timezone data, but it will fall back on the tzdata pypi package it that's not available (for example on Windows you need it if I'm not mistaken.) [0]: https://docs.python.org/3/library/zoneinfo.html https://docs.python.org/3/library/zoneinfo.html
- jwilk 3y ago> New in version 3.9.
- lexicality 3y agoAvailable to all in a backport: https://pypi.org/project/backports.zoneinfo/ https://pypi.org/project/backports.zoneinfo/
- 8organicbits 3y agoBut 3.9 is no longer new. If you're still on 3.8 you're running out of time to upgrade as official support is ending next year. The project has also been backported to 3.6+, so you can still use it if you're stuck on a legacy/unsupported version. See https://www.python.org/downloads/ https://www.python.org/downloads/ and https://pypi.org/project/backports.zoneinfo/ https://pypi.org/project/backports.zoneinfo/
- mark-r 3y agoI have no choice, because 3.9 isn't supported on Windows 7. I've always wondered what feature of Windows was so important that they had to drop compatibility with older versions, it's intentionally a very stable interface.
- maleldil 3y agoWindows 7 support ended on 14 Jan 2020. It makes sense not to officially support it. Have you tried building 3.9 from source to see if there's a particular API that's unsupported?
- vesinisa 3y agoInteresting. I got to thinking why a negative fold (e.g. -1) is not used to indicate local times that do not exist due to standard time to DST transition (moving clocks forward in spring). It is in fact discussed by the PEP: https://peps.python.org/pep-0495/#strict-invalid-time-checking https://peps.python.org/pep-0495/#strict-invalid-time-checki... The PEP provides instructions for detecting invalid times (that work as long as you use the standard library zoneinfo database.) In short - for invalid times, the UTC offset will be different for different fold versions (1/0) but with converse order than for times that exist doubly: # Non-existent wall time (std -> DST): >>> print(datetime(2023, 3, 26, 1, 30, tzinfo=ZoneInfo('Europe/London'), fold=0).utcoffset()) 0:00:00 >>> print(datetime(2023, 3, 26, 1, 30, tzinfo=ZoneInfo('Europe/London'), fold=1).utcoffset()) 1:00:00 # Ambigious wall time (DST -> std): >>> print(datetime(2023, 10, 29, 1, 30, tzinfo=ZoneInfo('Europe/London'), fold=0).utcoffset()) 1:00:00 >>> print(datetime(2023, 10, 29, 1, 30, tzinfo=ZoneInfo('Europe/London'), fold=1).utcoffset()) 0:00:00
- karmicthreat 3y agoPytz throwing an exception on ambiguous datetimes was the first footgun I discovered with it.
- masklinn 3y agoThat is the opposite of a foot gun.
- quickthrower2 3y agoA foot nerf gun
- sumtechguy 3y agoThat depends if you got it into production or not first :)
- bhargav 3y ago> The title of this post claims that pytz is the fastest footgun in the west, by which I mean that pytz is a quite well-optimized library, and historically it has been faster than dateutil.tz Wat. The author has no idea what a foot gun is. Pytz is a must because using the standard library and timezone manipulations often leads to bugs. There are nuances around “naive” date times where one can easily just add a fake timezone value, not realizing that the time didn’t actually get localized to the timezone. There are other libraries such as Arrow and Pendulum worth checking out but pytz is probably enough.
- roelschroeven 3y agoDid you read the article? The author explains exactly why they think Pytz is not enough, and why they think dateutil.tz (not the standard library) is a better alternative.
- pganssle 3y agoTo be clear, when I wrote this article, I had not yet created the `zoneinfo` module in the standard library. All the recommendations for `dateutil.tz` apply to `zoneinfo` as well, except that `zoneinfo` is also faster than `pytz`.
- randlet 3y agoThank you so much for your work on zoneinfo. I spend my days working on a legacy Django app and can't wait to switch from pytz to zoneinfo. In my experience, timezones are always a struggle so your efforts to make them easier to deal with is hugely appreciated.
- Joeboy 3y ago> Wat. The author has no idea what a foot gun is. In the article, "fastest" is italicized. The sentence is about whether the footgun is fast, not whether the footgun is a footgun.
- Twirrim 3y agoEh? They're pointing out it's an extremely well optimised and fast library, and then pointing out a number of ways in which it's a footgun because it doesn't behave as you'd expect, e.g. NYC = pytz.timezone('America/New_York') dt = datetime(2018, 2, 14, 12, tzinfo=NYC) print(dt) # 2018-02-14 12:00:00-04:56 How is that not a footgun?
- pmayrgundter 3y agopython, the footgun factory
- benreesman 3y agoTangential, but man time zones are just really hard. Some theoretically “simple” rules seem to routinely violate human intuition, and the almanac actually changes based on laws and stuff. Keeping everything in Zulu ms or ns and rendering into timezones as late as possible is a useful guideline as far as it carries one, but sometimes it’s not enough, this is one of a zillion reasons why e.g. calendaring is just really hard. I’ve long since mentally tagged “timezone handling”, in any language, as a slow zone: let’s stop and really think this through.
- Cthulhu_ 3y ago> Zulu ms or ns Is that timestamps as in unix timestamps? That won't work with birthdates for people more than 53 years old for example; a colleague mentioned a weird issue for someone born in 1938, but that was because the timezone in that location at that time was something like GMT+1:23, then the Germans came around and standardized it to +1:20, then later it was normalized to +1:00. tl;dr, time and date can only be accurate if time, date and location are known. Actually, some places have moved to different timezones over time as well, so you can't just say "this time has a timezone of GMT", you have to ask "what is the timezone of this location at this time/date"
- Aachen 3y agoTo get to/from that local time representation, you can still store it as a timestamp (to know when it actually happened) and use a timezone database that tells you what the people living there in that moment labeled that time as, right?
- IggleSniggle 3y agoYou still have the fundamental issue of storing it correctly in the first place. Do most languages time libraries account for localized temporal shifts of this nature by default? It does seem true that you could query an authoritative source, but on entry: If I say the event happened 60 years ago exactly, and it’s 1pm, what do I mean by “exactly?” Do I mean the day 60 years ago, at 1pm? Or 60 years ago, taking into account timezone shifts? Do I, the person entering this information, even know or care about this distinction? If you enter an event as 60 years ago, would your average joe expect that the time would shift by 23 minutes, or would they think there’s something wrong with the software? What about timezone differences that vary depending not only on geography, but also on the ethnicity of the people associated with the event?
- vrglvrglvrgl 3y ago[dead]
- JediWing 3y agoWorking with date/times in python made me miss Joda-Time/Java time libraries. For my money it's the most well designed date/time API I've worked with.
- civilized 3y agoIs something like this why you can only book a ticket on Lufthansa with your correct birth date if you set your computer's time zone to German time?
- Sebb767 3y ago> Published Mon 19 March 2018 (updated Wed 01 April 2020) This needs a (2018) (or (2020)) @dang
- mark-r 3y agoIt's a bit unfair to complain that pytz doesn't use the "standard" interface, when that standard interface didn't exist when it was created. The necessary calls for syncing the time zone with the date on datetime creation were a relatively recent addition.
- spott 3y agoI don't think it is unfair. If you are looking to solve a timezone problem right now, then you care about what the standard interface is right now... not "when the library was created".
- mark-r 3y agoThe documentation is very clear on how to use it to solve your timezone problem. The trouble comes when people expect it to "just work" without understanding how it works.
- roelschroeven 3y agoUntil now I never took a close look at how Python deals with timezones. Now I did, and it turns out the way Python handles arithmetic on timezone-aware datetimes is not what I expected. An example to illustrate: from zoneinfo import ZoneInfo from datetime import datetime, timedelta, timezone localtz = ZoneInfo('Europe/Brussels') utctz = timezone.utc a_local = datetime(2023, 3, 26, 1, 59, 59, tzinfo=localtz) # right before change to DST b_local = a_local + timedelta(seconds=2) # 2 seconds later; should be right after change to DST c_local = a_local + timedelta(hours=2) # 2 hours later a_utc = a_local.astimezone(utctz) b_utc = b_local.astimezone(utctz) c_utc = c_local.astimezone(utctz) print('datetimes:') print(f'(A) {a_local} (utc: {a_local.astimezone(utctz)})') print(f'(B) {b_local} (utc: {b_local.astimezone(utctz)})') print(f'(C) {c_local} (utc: {c_local.astimezone(utctz)})') print('deltas:') print(f'(B-A) local: {b_local-a_local} utc: {b_utc-a_utc}') print(f'(C-A) local: {c_local-a_local} utc: {c_utc-a_utc}') This prints: datetimes: (A) 2023-03-26 01:59:59+01:00 (utc: 2023-03-26 00:59:59+00:00) (B) 2023-03-26 02:00:01+01:00 (utc: 2023-03-26 01:00:01+00:00) (C) 2023-03-26 03:59:59+02:00 (utc: 2023-03-26 01:59:59+00:00) deltas: (B-A) local: 0:00:02 utc: 0:00:02 (C-A) local: 2:00:00 utc: 1:00:00 For (B), I would expect 2023-03-26 03:00:01+02:00: that is the point in time 2 seconds after (A). The datetime Python produced doesn't even exist (because of the start of DST, time jumped from 2:00 to 3:00). For (C), I would expect 2023-03-26 04:59:59+02:00: that is the point in time 2 hours after (A). As you can see by the times in UTC, (C) is only 1 hour later then (A) instead of 2. Both feel really wrong. The article compares pointer arithmetic using Python's time zone model vs pytz's time zone model, and concludes that pytz is worse because you have to normalize the result for the time zone to be adjusted. But the pytz result is correct (according to my expectations, at least), while Python's standard approach does adjust the time zone but without properly adjusting the time (and hence is incorrect, according to my expectations). It seems the author only looked at the time zone, without checking the time itself. I can work around it by converting to UTC and back, fortunately. I had a quick look at Pendulum and it seems it has functions to do arithmetic properly.
- NAHWheatCracker 3y agoI've run into this at work and I handle by always add timedeltas in UTC or just design everything to work in UTC. Unfortunately, some people dislike the former because it's 3 lines of code (convert to UTC, add delta, convert to local). I've had the same data scientist reject 3 MRs fixing this sort of issue in his code. At least it only causes a once per year error or a blip that no one notices. Always using UTC doesn't work always depending on requirements. The worst was interacting with an API that required precisely every hour of the following day to be sent without good documentation.
- smarx007 3y agoFor a second, I thought Jack Ganssle [1] got himself a hipster domain and started to do Python. [1]: http://www.ganssle.com/tem-back.htm http://www.ganssle.com/tem-back.htm
- nodemaker 3y agoLol I was caught by this footgun a while back. Use arrow - there I saved you a ton of time.
- hoosieree 3y agoI see what you did there.
- zzzeek 3y agoI was going to complain that Python for all of its "batteries included" lacks a simple facility in the standard library to "please give me a timezone for EDT", but apparently one has been added in 3.9: https://docs.python.org/3/library/zoneinfo.html#module-zoneinfo https://docs.python.org/3/library/zoneinfo.html#module-zonei...
- larkost 3y agoThe real problem that still is not fixed is that `datetime.date.now()` and others do not do the right thing and encode the system timezone. Rather they record a time without timezone. I have seen this cause so many problems because people are just not aware.
- pganssle 3y agoActually, naïve times are considered aware in the system time zone. There's no good way to create a `tzinfo` object that behaves like "the system time zone" and doesn't have unfortunate consequences. See this article: https://blog.ganssle.io/articles/2022/04/naive-local-datetimes.html https://blog.ganssle.io/articles/2022/04/naive-local-datetim... It's not an ideal situation, but given path dependence I don't think it can be improved much from here.
- Too 3y agoEven worse is that utcnow() returns timezone unaware dates, when it could have easily returned dates with timezone set to utc. Leading to accidentally mix of local and utc times.
- dr_kiszonka 3y agoWhat are your favorite datetime related Python packages? (not necessarily for tz handling) I like dateparser for certain use cases: https://dateparser.readthedocs.io/en/latest/ https://dateparser.readthedocs.io/en/latest/
- aeyes 3y agoarrow
- dr_kiszonka 3y agoThis arrow[0], correct? Initially, I thought you meant Apache Arrow and was confused. https://arrow.readthedocs.io/en/latest/ https://arrow.readthedocs.io/en/latest/ (looks good!)
- aeyes 3y agoYes, exactly!
- Too 3y agoFreezegun for unit testing. Can stop and step time predictably. Any code using datetime functions will get the time controlled by the test, with no additional abstractions or other boilerplate needed.