3 ms·
"Funny" story about a binary 8601 implementation. I was architecting a true-realtime telemetry pipeline for a AAA videogame studio (later used by a dozen or so
by Darkphibre 6y ago
"Funny" story about a binary 8601 implementation.
I was architecting a true-realtime telemetry pipeline for a AAA videogame studio (later used by a dozen or so franchises for all titles by the publisher). It had a requirement for subsecond client notification of per-user aggregated statistics, including complicated event interactions (such as ensuring the grenade that you threw that killed someone, would only be attributed to you if a prior event hadn't already occured), which resulted in sequentiality guarantees. But with that, we could get rid of windowed event processing (and those inherent latencies), and instead treat them as a true stream of events. Keep in mind we could support up to 400ms one-way latency, so we only had 200ms for processing/routing in the client and in the cloud. I measured client code in μs.
Annnyways. The principal in our org insisted that time not be be transmitted as an epoch. I argued for weeks, but he was insistent... "No magical offsets to arbitrary dates. ISO8601. Do it." He came from and worked most heavily with the web services group. Apparently some point in the past, services had picked the wrong epoch and there was confusion (I still can't understand how that wouldn't be caught right away, there's only a few standard epochs that are quite different in resulting time)? The events in my design were bit-packed using a competing protocol to Protocol Buffers... there's no way I was doubling the payloads just to store a timestamp as a string.
So I read and re-read ISO8601. And it dawned on me: ISO-8601 never specifies the encoding. It specifies the order in which information is conveyed, and if in text, the delimiters to use. One example clue is in the definition of basic format, and the call-out that most of the document leverages plain text to communicate their ideas.
basic format
format of a date and time representation or date and time format representation comprising the minimum number of time elements necessary for the accuracy required. The basic format should be avoided in plain text.
I ended up proposing the most terrible hack I've had to live with. It involved bit-packed 64-bit integer that stored: 12 bits for year, 4 bits for month, 5 bits for date, 5 bits for hours, 6 bits for minutes, 6 bits for seconds, and the remaining bits for an integer that stores the sub-second fraction. We lost a few orders of magnitude resolution on that end due to the inefficiencies of the earlier packing, but... it worked. And it was in the right order. And there were no "magic" epochs to dissuade the principal engineer.
Most importantly, it sailed through the review process.
Still makes my skin crawl a decade later. And I secretly suspect you technically have to have a T between date and time to be ISO8601 compliant, it's been a long while since I read the spec. But there you go.
- colejohnson66 6y agoBut... if you’re sending the timestamp down to sub-second precision, the receiving computer has to compare that number to their own time (which is based on an epoch). So, an epoch is used either way. It sounds like a convoluted solution to a problem that doesn’t need to exist.
- Darkphibre 6y agoHmm? This timestamp is known to be local-time, used for measuring durations-between-events on a single session, and not used for cross-session reporting. Time would be compared from one event to the next. Cross-session reporting (such as calculating DAUs or MAUs) reports on the timestamp from the ingestion service, which is applied at the batch level before a batch of events is placed on the event hub for routing (events are batched and transmitted every 16ms off the client). Typically you can get away with treating everything that happened in the same simulation frame as having happened instantaneously... though we use a monotonically incrementing guaranteed-unique sequence identifier for processing using a high-water-mark (one stream is at-least-once sequentially-guaranteed with retries, while the other stream is lower priority and at-most-once sequentially-guarantied without retries). No amount of ISO spec will help with client clocks being completely arbitrary. :D The ingestion service clocks are NTP synced to datacenter-hosted atomic clocks, but we still see a few-second skew here and there. I've even done research into how often they drift backwards in time (but that doesn't really happen, since NTP will just slew the system time by delaying, or adjust the clock massively on startup before the game launches).
- opk 6y agoYour format will have had the advantage of inheriting the ISO8601 scheme for representing leap seconds (60 in the seconds component). With an epoch, you have to either pretend leap seconds don't exist (typical on Unix), know when they all were before you can print a time or accept the drift (common with GPS units).