9 ms·
Python datetime pitfalls, and what libraries are (not) doing about it
- WesolyKubeczek 3y agoI wish such an overview also existed for Go. Yes I could do it but I’m lazy.
- smitty1e 3y agoI don't think any of the DST complaints are legitimate. If doing work involved with timestamps, not keeping everything in GMT is questionable. GMT is the UTF-8 of timestamps: the low common-denominator rules. The datetime library is mechanism. DST is policy. Policy is wildly variable and your system library would be unmaintainable if it had to chase policy. Just say no.
- lifthrasiir 3y agoAs long as you can't predict the future, some future date since a specific cut-off can't be reliably converted to UTC (not GMT, which does observe DST) anyway.
- graemep 3y agoGMT does not have DST. It is only slightly different from UTC. That said, you should really use UTC. I think you are confusing it with British Summer Time which is GMT +1
- pasc1878 3y agoBeing a pedant here. What is being confused is GMT/BST/(BDST) with the time in a location Greenwich. Currently the time in London is BST between end March to end of October and GMT the rest of the year. Historically the dates have changed and also had period where it was BST in the winter and BDST (GNT+2) in the summer, and also years with BST all year round. There are still people who suggest BST all year round (I think if Scotland when independent it could happen) Also now UK is not in the EU it is more free to change things.
- jasode 3y ago>That said, you should really use UTC. No, that "store UTC everywhere" is often repeated but it's incorrect advice when applied to future human-constructed datetimes such as appointments or social events. Future "human-interpreted wall-clock" datetimes cannot be unambiguously stored as future UTC values with perfect roundtrip fidelity. I've tried to explain the difference in previous comment: https://news.ycombinator.com/item?id=19534843 https://news.ycombinator.com/item?id=19534843
- Glyptodon 3y agoThe number of times very smart people seem to think having to alter a pile of UTC times because their associated region has changed their DST behavior is better than having local zoneless and just changing TZDB always weirds me out.
- graemep 3y agoThat is a good point, but fortunately a rare case. The other problem is that when you are using human inputs such as appointments you may not know which is appropriate. If I arrange a meeting with someone in another timezone, and there is such a change, how can a system know whether I want the UTC time, my timezone, or the other person's timezone. I am not sure there is a good way to do this in many cases. As you say some culturally defined times are definitely tied to the local timezone, but a lot of things will not be. There is a loss of information, but that information may not be useful on many cases. I think I agree with you in that there is no universal solution. However, store UTC is probably good enough a lot of the time. In general, timezone changes would be announced well ahead of time so UTC would be fine except for times fixed far ahead. I have actually lived through timezone changes that were decided on in emergency circumstances (Sri Lanka faced with a severe electricity shortage) and the result was chaos. People genuinely did not know whether an appointment would take place in "old time" or "new time"! No system devised earlier could have coped with that because intent was no unambiguous.
- Hackbraten 3y ago> how can a system know whether I want the UTC time, my timezone, or the other person's timezone A decent system will allow the user to make that choice on a case-by-case basis but assume the common case as a default. I've seen apps pull that off successfully, e.g. iCal.
- praash 3y agoA true observation, especially since many European countries (or just some citizens) wish to get rid of DST. However, ±1 hour is not much of an error when talking about a time delay of at least a year.
- autarch 3y ago> However, ±1 hour is not much of an error when talking about a time delay of at least a year. It is if this a doctor's appointment or a meeting with a friend or ... you get my point.
- macintux 3y agoI missed most of my last chance to see my cousin play college basketball because I trusted her college’s website, and it was off by an hour due to timezone confusion.
- praash 3y agoThis is the only sane policy. Timezone conversion must be delayed to the very last moment before displaying time to users. If I needed to define a daily alarm in a local wall clock time, I'd happily deal with a zoneless dateless hour-minute tuple and determine the alarm time by combining a date and a time using a timezone. Right after I'd convert that and handle the rest in UTC.
- pasc1878 3y agoYou need to know more than that. Is the appoint meant 2pm in London or in New York. So for forward times you need a location as well as the date and time. Yes when it happens you store in UTC. In both cases you can display to user with local timezone. (Although if the London user is going top fly to NY they will still want to see the NY time and not their local time)
- autarch 3y ago> This is the only sane policy. Timezone conversion must be delayed to the very last moment before displaying time to users. No, this only works for datetimes in the _past_. For future datetimes you often must store the datetime in the user's (IANA) timezone. Otherwise your application could be off. And your app could be off by more than one hour. In 2011 Samoa moved across the international date line. See https://www.theguardian.com/world/2011/dec/30/samoa-loses-day-date-line https://www.theguardian.com/world/2011/dec/30/samoa-loses-da....
- Spivak 3y agoThank god someone in the thread understands the problem. Everyone saying "just make everything UTC and convert" is both throwing away information and making it impossible to do future calculations. "Schedule a 1-1 with my boss every two weeks at 1 PM" If you take my local time zone, convert it to UTC which puts my meeting at say 9AM UTC, schedule every two weeks at 9AM UTC and convert back to my local time zone it will be wrong when I hit change DST. But if I instead store the (datetime, zone) pair when I say "1PM fourteen calendar days from now in zone" it will always be correct. The UTC "trick" only works when your problem domain is representing specific moments in time.
- Kwpolska 3y agoDST and local time are relevant in calculations when humans are involved. A user in Japan shouldn't be told it's still Sunday just because it's still Sunday somewhere in western Europe. If I set up a recurring meeting every Monday at 10:00, it shouldn't suddenly shift to 09:00 or 11:00 just because of a DST switchover.
- kroolik 3y agoWhat if the meeting is between people in EU and USA, where DST transitions occur at different moments?
- rightbyte 3y agoHmm ... if a problem is hard enough maybe just don't try to solve it? Treat the meeting time as a string, rather than a time, since it is ambiguous anyway what the organizer meant, and let the user figure out what the string means.
- airstrike 3y agoTell those people not to meet at 2am on the night DST changes
- demondemidi 3y agoAll the meetings are hosed for about two weeks. This is a known issue for anyone working with time zones that don’t change together.
- Kwpolska 3y agoCalendars like Outlook let you specify a timezone for the event. If it's set to Europe/Warsaw, the US goes out of sync for a few weeks. If America/New_York, the EU does. This still isn't a problem for a meeting between LA and NY, for example (unless the meeting is in the middle of the night).
- demondemidi 3y ago
- Glyptodon 3y agoI tend to prefer: - Things happen/happened at UTC times - Things will happen at zoneless time + IANA DB label. Though there are some edges. Obviously it's possible to pre-apply and convert to UTC, which is okay, but whether that makes sense to do in practice varies some. Future UTC time of often worthless for understanding human centric dates and times.
- Kwpolska 3y agoFor past events, knowing the local time might be useful context. I'd store local time with UTC offset, or with IANA time zone name (which might change the UTC meaning if there's some change affecting past dates in the IANA database).
- tremon 3y agoIf doing work involved with timestamps Of course, any argument becomes much easier if you're allowed to just discard half the purpose of the thing you're arguing about. The Python datetime library doesn't just exist to record timestamps. From the actual documentation: > The datetime module supplies classes for manipulating dates and times. > [..] the focus of the implementation is on efficient attribute extraction for output formatting and manipulation. Datetime manipulation is part of the explicit focus of the module. So why would you argue that we shouldn't expect said manipulations to be DST-aware?
- TrackerFF 3y agoISO 8601 is the only way to go. With that said, it's hard to control what systems others use. I frequently work with datasets that have 4-5 different timestamp formats. One of the most annoying bugs I've worked on, was in a table where almost everything followed a DDMMYYYY format - sans a 2-3 month period where the dates were flipped to MMDDYYYY
- Thorrez 3y agoDoes ISO 8601 fix any of the problems mentioned in the article? It doesn't really seem relevant to me.
- sitkack 3y ago> doesn't really seem relevant to me Isn't this overly 'you' centric? Wouldn't this require access to your inner monologue? Doesn't seem like a referentially invariant supporting argument, to _me_.
- falserum 3y agoCharitable interpretation of previous comment would be “I think it does not seem relevant to problems specified in the article”. Iso 8601 is about date formatting, while article is about a bit orthogonal to that. (But I agree/hope that ISO 8601 is the future that is less ambiguous than current state)
- sitkack 3y ago> I think it does not seem relevant to problems specified in the article Is a low effort way to make someone form a rebuttal in response to no argument. It is pseudo intellectual way to sound smart while offering nothing in return. As if that persons opinion has weight because “they don’t see the relevance”, I know lots of people that “don’t see relevance”. Is our responsibility to educate their feigned confusion?
- 3y ago
- frumiousirc 3y agoBest code strategy here is twofold. 1. Maximize the code context where date/time is represented as a simple count of Unix epoch seconds. 2. Never pass calendar/clock representations through API methods. 3. Always pass a timezone to methods that require it. 4. Counting, like time, is hard. The only contexts I see where calendar/clock representation are needed are: - Accept calendar/clock info from the user or external data sources. - Present calendar/clock info to the user or external data sources. - Internal "calendar aware" operations (find the Unix epoch second count "one month from now"). These all require a timezone to be specified. They may still be a mess internally but their operation is unambiguous. They may use imperfect data types like datetime but the issues of these data times are confined to their internal code context.
- pepoluan 3y agoThe problem with Unix Epoch is that it began in 1970. There are applications where you need dates before 1970 and that starts to get hairy.
- magicalhippo 3y agoSigned integers exist. What is hairy about using them?
- zokier 3y agoBigger problem of 1970 epoch is that the 1970-1972 period was confusing time for UTC; having the epoch at 1972 would be arguably much cleaner Fun excerpt from Wikipedia: > As an intermediate step at the end of 1971, there was a final irregular jump of exactly 0.107758 TAI seconds, making the total of all the small time steps and frequency shifts in UTC or TAI during 1958–1971 exactly ten seconds, so that 1 January 1972 00:00:00 UTC was 1 January 1972 00:00:10 TAI exactly, and a whole number of seconds thereafter. At the same time, the tick rate of UTC was changed to exactly match TAI. I'll leave it as exercise to the reader to think of all the implications of whatever timesteps and tickrate shifts that happened pre-1972.
- zokier 3y ago> 1. Maximize the code context where date/time is represented as a simple count of Unix epoch seconds. Either your values will be different from UNIX timestamp (and you need to write your own conversion functions), or they are not simple count of seconds and will have ambiguities. Neither is not great situation.
- guitarnick 3y agoThe author provides this example to illustrate inconsistent handling of non-existent datetimes: # This time doesn't exist on this date d = datetime(2023, 3, 26, 2, 30, tzinfo=paris) # No timestamp exists, so it just makes one up t = d.timestamp() datetime.fromtimestamp(t) == d # False Criticisms: 1) The example would fail just as well for any datetime with given tzinfo. Because fromtimestamp returns native datetimes. 2) The timestamp isn't just "made up". Its behaviour is clearly documented in PEP 495, as linked by the author [0]. In this case it consistently corresponds to datetime(2023, 3, 26, 3, 30, tzinfo=paris). [0] https://peps.python.org/pep-0495/ https://peps.python.org/pep-0495/ Finally if we disallow creating non-existent datetimes in the proposed library, how do we represent the 2am in "clock changes forwards at 2am"? Use 3am? There are tradeoffs.
- falserum 3y ago> how do we represent the 2am in "clock changes forwards at 2am". If we are willing to adjust a tiny bit, there are options: Option1: “01:59 is followed by 03:00”. Option2: “clock changes forwards at 2am GMT+2” (GMT+2 is offset before the hour jump)
- ariebovenberg 3y agoAuthor of the blog post here— 1) You're absolutely right, I'll fix this mistake in the example. 2) "Made up" was perhaps stirring the pot too much ;), but the fact remains that a timestamp is created when one essentially doesn't exist. That this is meticulously documented in PEP495 doesn't change this fact. Note that PEP495 also explicitly describes some of the other pitfalls. Documenting a 'suprise' is not the same as eliminating it. > Finally if we disallow creating non-existent datetimes in the proposed library, how do we represent the 2am in "clock changes forwards at 2am"? Use 3am? There are tradeoffs. Indeed, there are always tradeoffs. Often, only time can tell what was the 'least bad' decision. It is telling though that modern datetime libraries (Noda Time, Chrono, Temporal) in other languages choose to not represent these 'missing' local times.
- dwattttt 3y ago> Documenting a 'surprise' is not the same as eliminating it. I'm going to have to remember that one
- amluto 3y ago> # The local system timezone > LocalDateTime, Here’s my feedback: one should essentially never use “local” time. I would even argue that the existence of systemwide local time is a historical mistake. Why would you ever want to use the “local” timezone of the system on which you are running? If the user is literally sitting in front of a monitor plugged in to the machine, it might be appropriate (but it should be a per-user setting). But if the user is using X forwarding or ssh or a remote desktop, you have no idea what timezone the user is in. (And this was common in the mainframe era!) If you’re servicing a request over the network, you don’t want local time. The only sort of legitimate use for “local” time I can think of is compatibility with libc and it’s horrible time API. You can’t even tell what local time means without getenv, which is a footgun. So maybe LocalDateTime should be called LibcCompatLocalDateTime.
- jncfhnb 3y agoImagine having the audacity to run software on the machine in front of you
- ahartmetz 3y agoYou shouldn't do that, it's bad for some business models.
- cm2187 3y agoThen how do you express that something must happen at a date in the future at 4pm local time? If you use UTC based on today's expectation of daylight savings, they may have changed by the time this date happens.
- amluto 3y agoYou start by figuring out what you’re even trying to do. For example, I read your comment three times, and I don’t know what you mean by “4 pm local time”. I don’t even have what I would consider to be a credible guess.
- 3y ago
- falserum 3y agoAcross all languages, what are the best libraries, with least amount if pitfalls? (java.time seems good, anything even better elsewhere?)
- assbuttbuttass 3y agoI really like Go's "time" package. The only pitfall I can think of is that it doesn't really handle overflow at all. https://pkg.go.dev/time https://pkg.go.dev/time
- rad_gruchalski 3y agoAnd date time parsing sucks with its predefined mask elements.
- pgwhalen 3y agoGo's time package seems like one of the bad ones, since it makes the classic mistake of combining zoned and naive times into the same type (arguably it doesn't support naive times at all).
- zokier 3y agojava.time doesn't handle leap seconds very well :( jshell> ChronoUnit.SECONDS.between(Instant.parse("1974-01-01T00:00:00Z"), Instant.parse("2024-01-01T00:00:00Z")) % 86400 $22 ==> 0 astropy gets you the correct answer: >>> from astropy.time import Time >>> (Time('2024-01-01T00:00:00Z') - Time('1974-01-01T00:00:00Z')).sec % 86400 24.0 of course astropy is bit specialized and not really something I'd use as general purpose date/time library. But this is one good litmus test for datetimes, feel free to try it with your favorites.
- zokier 3y agoI tested this with the library from TFA, and it fails too :( >>> from whenever import UTCDateTime >>> (UTCDateTime(2024, 1, 1) - UTCDateTime(1974, 1, 1)).total_seconds() % 86400 0.0
- cutler 3y agoWhy doesn't someone just port Luxon from Javascript which seems to have solved most of these problems. Why are we still reinventing the wheel in 2024?
- darkwater 3y agoHow do you solve "give me the epoch for 2:30AM" the night daylight changes?
- assbuttbuttass 3y agofrom whenever import ( # For the "UTC everywhere" case UTCDateTime, # Simple localization sans DST OffsetDateTime, # Full-featured IANA timezones ZonedDateTime, # The local system timezone LocalDateTime, # Detached from any timezones NaiveDateTime, ) Why so many different types? Doesn't ZonedDateTime cover all of the other use cases?
- jcranmer 3y agoNo. First, there's a fundamental difference between naïve datetimes and timezone-aware datetimes. Try to combine them in one representation, and you are guaranteed to end up with a broken datetime library that spawns endless blog posts attacking it. Second, there's a more subtle distinction between a datetime which has a known timezone and one which merely has a known instantaneous offset. A timezone is essentially a naïve date time -> tzoffset, albeit one whose values for future dates tends to be surprisingly unrobust. Among the offset-only date times, UTC is a very common case that deserves its own alias. Finally, separating the local-tz datetime from other tz-aware datetimes can be very useful, for if the user changes their local timezone, then a local-tz datetime usually wants to switch the timezone it is using as well.
- assbuttbuttass 3y agoIMO, any program using naive timestamps is probably doing something wrong. Same with fixed offsets (except UTC obviously). For local time zone, how often does the local time zone change, and you want to modify all timestamps in the system? That actually seems like a huge pitfall
- Hackbraten 3y ago> IMO, any program using naive timestamps is probably doing something wrong. Here's an example where you'd legitimately want to process and store a naive (wall clock) time object: https://news.ycombinator.com/item?id=39419845 https://news.ycombinator.com/item?id=39419845
- demondemidi 3y agoExcellent research. Clearly explains the problem, shows how the industry addresses it, and provides a solution. You’re hired! And I never want to write time code ever again. I wonder how many DST bugs I have in the wild after 40 years of coding…
- rhelz 3y agoReally, there is no good technical solution for this, because in part, it is not even a technical problem. The laws about timezones, etc, very from country to country, from year to year etc. And some parts of the world are claimed by two or more different countries, each one of which has an opinion about what time it should be there. One might say "well lets skip all this political b.s. and do everything in UTC" but really, viewing time as running according to a single universal clock is a relic of Newtonian Physics. There is no single, global answer to what time it is. And we are getting to the point where special relativity is kicking in. High-frequency trading is already in the 50-microsecond range. Not so far off from the relativistic time dilations experienced even in flying in a jet airplane: flying around the world eastward results in about a 180 ns slowdown, flying around the world westward give about an 80 ns speedup. Even concepts like simultaneity become relative at the speeds which we will soon be computing at. Imagine two corporate raiders flying in their private jets, biding on the same block of controlling stock: who bought it first would be different depending upon which frame of reference you were in. If you have to write programs for when time really matters (as I did writing stock-trading software) my advice is to just accept the realities, and glory in the complexities. Being able to write software which can keep track of time, and obey all the rules and regulations of various exchanges and countries, is a competitive advantage for you.
- make3 3y agoI think they could improve the quality of life of the users a lot without addressing the "high frequency trading inside a jetplane flying against the rotation of the earth" edge case lol (not to diminish having an accurate understanding of time ofc, GPS is the main example of an application, etc)
- heavenlyblue 3y ago> Imagine two corporate raiders flying in their private jets, biding on the same block of controlling stock: who bought it first would be different depending upon which frame of reference you were in. Market maker decides that, not them.
- svilen_dobrev 3y agowhat special font/characters is this article using? in that scorecard table i got all same-looking boxes. yes/no Y/N T/F x/o would have been much more readable. And also in all bullet points also see boxes..
- ariebovenberg 3y agoAuthor of the post here— They are emoji. I've added plain text to the table now, so it is readable without them. edit: I've used a workaround to display emoji as images now
- deleted 3y ago[deleted]
- the__alchemist 3y agoI will state this briefly as a tangent. This article mentions third party libraries; the top are Arrow and Pendulum. These are a wellspring of subtle bugs due to a decision they made that I suspect is based off the Javascript library Moment: They conflate Dates, Times, and Datetimes, as a single type. If you are using their Arrow etc type to represent a Date or Time, the resulting code will be fragile, and fail in surprising ways. Additionally, if my function signature accepts a time or date, you have no way of validating the parameter, and can do things like ask for "seconds" on a date. The moment/arrow/pendulum community advocates that the concept of a date or time individually is meaningless; I disagree. It is application specific, and there are applications that call for any of the 3 variants. Python's builtin `datetime` library has flaws as pointed out in the article, but is safer than Arrow or Moment. I think Rust's Chrono lib is the best datetime library I've seen, in terms of not surprising you with errors.
- ariebovenberg 3y agoAuthor of the article here— I actually did start out basically porting Chrono to Python, but decided against it for two reasons: 1. While I really liked handling local/UTC conversions with the "LocalResult" enum in Rust, it translated poorly to Python. Somewhere along the line you'd need to transition from Rust's "errors as values" to Python's exception-oriented world. I actually wish Python would go Rust's way, but I'm not going to swim against the current here... 2. Most modern libraries in other languages choose to disambiguate local datetimes. I felt it was safer to go the road more traveled here.
- timmaxw 3y agoWell done. It's horrifying that the Python standard library "+/-" operators don't handle DST correctly.
- aimor 3y agoI don't really understand why OffsetDateTime doesn't allow adding time deltas (reason provided is that it might have it's offset change relative to UTC). Is the offset supposed to be relative to UTC or is it supposed to be relative to ANY TimeZone (including those with time changes)? I thought it was relative to UTC, but even the other way I just don't understand why that's a problem. I might be making some assumptions about the behavior I expect.
- ariebovenberg 3y agoAuthor here. "Offset" is indeed relative to UTC. The problem with allowing adding timedeltas is that you give users the impression they are doing valid arithmetic, while they may not be. Example: You receive a timestamp of an event in Paris: 2024-03-31 01:00:00+01:00. If you allow addition, you may be tempted to think you can "just add three hours" and get 04:00:00+01:00. However, because Paris has a DST transition, you'd actually want to have 05:00:00+02:00 In the end, the choice is between "users know what they're doing, just let them" and "users need to be prevented from making common mistakes". I choose the latter. Note that also NodaTime doesn't support arithmetic on their OffsetDateTime, for the same reason.
- aimor 3y agoThank you for answering, I understand the dilemma now and why you chose to force users to explicitly convert to UTC. When I see the example my thinking is: Well, if you want a time offset relative to Europe/Paris then define your time zone that way instead. Of course I haven't seen other libraries give that option and maybe for good reasons.
- ariebovenberg 3y agoYeah—the perfect solution doesn't exist. The problem is that fixed-offset datetimes are very common since RFC3339 and ISO8061 don't support timezone names. The result is that you often end up with fixed-offset datetimes that should instead be linked to a IANA tz ID.
- magicalhippo 3y ago