7 ms·
Bug story: Sorting by timestamp
- senderista 3y agoTimestamp resolution doesn't just depend on the storage granularity of the timestamp type; it also depends on the actual accuracy of whatever system call is used to populate the timestamp.
- adam-p 3y agoThat's a good point. I'll add a note to the post.
- dotancohen 3y agoAnother common reason that time goes backwards is DST. The amount of DST-related bugs that I've fixed over the years amazes me, because every single developer has moved clocks back and forth twice a year for their entire lives, bar a few years in the beginning. And even this fine article mentions the time-has-gone-back possibility yet ignores DST.
- WirelessGigabit 3y agoOh the joys of living in Arizona! Not having to deal with confusion twice per year! Not having to deal with these sort issues! Or so I thought. All calendar invites I own are set in Arizona time, so for everybody else they shift. Queue the rescheduling requests. For the ones I don't own it actually is better for me as they all shift an hour later.
- aljarry 3y ago> time goes backwards is DST Only if you deal with timestamps that don't contain timezone information. With TIMESTAMPTZ in postgres it's transparent, you don't have to do anything specifically to manage DST.
- realaleris149 3y agoPostgres does not store the actual time zone information, it just stores it in UTC. Is a bit counterintuitive but this is how it works [1]. The "with time zone" part is just for parsing and displaying back. When is displayed the value is converted from UTC to the local time in the current timezone which can result in some very interesting discussions :). [1] https://www.postgresql.org/docs/current/datatype-datetime.html#DATATYPE-DATETIME-INPUT-TIME-STAMPS https://www.postgresql.org/docs/current/datatype-datetime.ht...
- aljarry 3y agoTIL :) thanks, I misunderstood that part.
- loloquwowndueo 3y agoEvery single US developer maybe. In Mexico DST started in 1994 so developers who started before that (hello!) did need some adjustment and operating system support was not a given. There are also countries where DST is not observed or differs from the US yet developers from those countries might be working remotely for a US company, with US developers or needing to accommodate a significant US clientele.
- dotancohen 3y agoI'm not in the US.
- gwbas1c 3y agoDST is common throughout the world: The days that it starts and stops vary, both by locale, and by year. The operating system keeps track of every locale's DST dates, all the way as long as DST has been a thing. When governments change the date, via law changes, the change usually gets passed along in an OS update.
- deleted 3y ago[deleted]
- o11c 3y agoNote that Windows has major bugs here - from memory, it only keeps track of the 2 most recent DST rules, so historical timestamps will be wrong after the rules change twice.
- pritambaral 3y agoReporting from India here. No DST ever. I still know what DST is and how it works, but I've never had to change clocks due to DST.
- mook 3y agoLooking at the Wikipedia article, it looks like it's not that popular in Asia and Africa. Lots of places seems to have observed it at some point in the past, though a lot of them seem to have been last century. https://en.m.wikipedia.org/wiki/Daylight_saving_time_by_country https://en.m.wikipedia.org/wiki/Daylight_saving_time_by_coun...
- gwbas1c 3y ago> The amount of DST-related bugs that I've fixed over the years amazes me > yet ignores DST Uhm, you do know that you're supposed to use a neutral timezone (Such as UTC) internally in your code / schema, and convert to local time in the UI? The bugs that I've hit generally have to do with using local time accidentally when UTC is expected. > because every single developer has moved clocks back and forth twice a year for their entire lives WTF? Every single developer knows that their database schema should be in UTC and thus immune to DST issues.
- e28eta 3y agoIt sounds like you’re ignoring the fact that sometimes programs want to do things based on the user’s current time, and there _are_ reasons for developers to need to be aware of DST and handle edge cases correctly. I inherited some code that was sending push notifications to users with a daily summary. I don’t remember why it was tripping over DST, since the notifications were at a “reasonable” hour (9am?) and afaik the time change happens during the hours most people are asleep, but the ruby API being used had errors for “there are zero / two times that match the time you asked for”. I had to scratch my head for a little on why we were seeing both the zero and two errors in September / October, despite being intellectually well aware that the southern hemisphere seasons are the opposite of the northern.
- takinola 3y agoBut using UTC solves that problem, no? The time only changes in DST but not in UTC. If the code checks the time against UST, it will never trip up. Anyways, the only real solution for managing time is to use a library like Luxon so you can stop thinking about it. Time is like cryptographic encryption - Don't roll your own solution if there is a battle tested library available.
- recursive 3y agoIt does not. I've got a specification that all inputs and outputs are in local time. Sometimes the timezone isn't known yet. Or at all.
- _kst_ 3y agoThat should only matter if times are stored and sorted as local time. Storing time as UTC (e.g. seconds since 1970) and computing local time from that and the current time zone should avoid DST-related bugs.
- andreareina 3y agoUntil you deal with the future. If I have a recurring 2 pm meeting, I'd be a little put out if some of those suddenly became 3 pm because of a DST change.
- jonathanlydall 3y ago> because every single developer has moved clocks back and forth twice a year for their entire lives Where I live, South Africa, we have a single timezone which never has DST. It is very easy for a developer here to be oblivious of time zones/DST as in practice it will never bite them in the local only market. Well, I lie, it’s only almost never, a common smell of incorrect time handling is when the deployment instructions have a note to ensure the server is set to the correct time zone. Tangental pet peeve of mine, in the early 2000s I received an abuse report with quoted logs with the timestamps being qualified as merely EST (or some other US time zone), which I found super annoying as I had to look up what the offset for such time zone was and then manually apply conversions before I could do any investigating of my own.
- WirelessGigabit 3y agoSidenote on showing (or not showing) timestamps. Consider a page with 10 rows, and in each row you show the relative date. 2 issues with that: 1) If I see a whole page and I see 1 week ago the range is 7 days. If I see 1 year ago the range is 365 days. That is too much for most of the things. 2) If I am on a page without visual indication of sort order and I'm on a page that shows 10 entries with '1 year ago' I have no clue about the sort order. I hate relative dates.
- deleted 3y ago[deleted]
- Terr_ 3y agoTo be pedantic, that's not a problem of relatives times as much as the data being lossy vague approximations. Still, "1 year, 7 months, 2 days, 5 hours, 3 minutes ago" isn't always ideal either, not even when it's done in a lexically sortable way. Ultimately it boils down to a choice which doesn't match problem the user has, and in different circumstances someone might want relative or absolute.
- WirelessGigabit 3y agoIt can be solved by putting in an absolute date. I still don't understand what kind of problem relative dates solved.
- Terr_ 3y agoThey work when relative-times are part of the question or mental model the user has when approaching the system. Then the user doesn't have to mentally cross-convert between relative measures and absolute timestamps, which is less error-prone and annoying.
- aljarry 3y agoInteresting note on the NOW() (or CURRENT_TIMESTAMP), they are equivalent to transaction_timestamp(), which means - start time of the current transaction. So, if you'd insert multiple items in a single transaction, all of them would end up with the same value in the "created" column.
- jupp0r 3y agoI found it to be a generally useful rule to never "ORDER BY created" but instead "ORDER BY created,id" instead to achieve stable sorting. I recently added some indices to a few tables to speed up a complicated query with lots of subqueries and joins and ran into many unit test failures because usage of the new indices changed the order in which items with the same "created" values were returned.
- bufferoverflow 3y agoIf there's an increasing ID, just sort by ID.
- zoky 3y agoWon’t work if your ID is a UUID. Also, this is more generally applicable to any date, not just created.
- keep_reading 3y agoUUIDv7 is sortable by time
- zoky 3y agoAnd I’m sure that will be incredibly useful in 20 years when UUIDv7 has entirely supplanted UUIDv4 in all legacy systems (and you only need to sort by date created), but for now let’s just go with the best practice for the foreseeable future and sort in a way that not only ensures consistency but future-proofs us indefinitely.
- keep_reading 3y agoThen don't use UUIDs, use snowflakes / flakes
- adam-p 3y agoNice. That's a good general rule to follow.
- vivzkestrel 3y agoCREATE INDEX IF NOT EXISTS feed_items_pubdate_id ON public.feed_items USING btree (pubdate DESC NULLS FIRST, id DESC NULLS FIRST) TABLESPACE pg_default; create an index on both the pubdate (timestamptz column with potential duplicates like your post mentions) and a uuid primary key column (id in my case) problem solved
- adam-p 3y agoThere are a couple of shortcomings that I think I see: 1. If the clock moves backwards (and an "older" record gets created), it doesn't help. 2. If a duplicate timestamp is created with an id that sorts earlier than one of the other dups (and a user has already synced to the first dup). In both of those cases, the user won't receive the new record. (Yes, neither is probable, but neither is impossible.)
- neonsunset 3y agoThis title is instant PTSD flashback - at one place I worked there was a system that would order events by their timestamps and multiple downstream systems would rely on that. One day, I fixed an issue in the message producer that was causing the routine to take unreasonable amount of time and resources so that latency went from 1s to ~50ms. Three hours later P1 is raised and the entire architecture had to be refactored. I still have the screenshot of the latency graph saved somewhere :)