7 ms·
I got the sense while crawling data from their API that the engineering quality is poor at Parler. Dates were represented as strings in "YYYYMMDD" format (so to
by formalsystems 6y ago
I got the sense while crawling data from their API that the engineering quality is poor at Parler. Dates were represented as strings in "YYYYMMDD" format (so today would be "20210113053923") instead of UNIX timestamps, certain fields were duplicated for no reason (e.g. every object would have an identical "id" and "_id" key), counts of impressions/comments/etc would be the display strings rather than raw numbers (so "2k" or "5m"), and various moderation flags were in place like a boolean "sensitive" which was always false, even for posts that had been downvoted significantly.
- paranoidrobot 6y agoOf all the things to criticise Parler's tech folks over, using ISO8601 (minus the non-digit characters) shouldn't be one.
- jhanschoo 6y agoIDK the issue OP saw with using ISO over UNIX timestamps, but one reason why you might want accuracy down to the second for dates is with providing accurate relative time/date across timezones.
- CameronNemo 6y agoIs ISO8601 without punctuation still ISO8601? Most log parsers I have seen would not pick up the Parker format. ex gr https://docs.python.org/3/library/datetime.html#datetime.date.fromisoformat https://docs.python.org/3/library/datetime.html#datetime.dat... https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns#L68 https://github.com/elastic/logstash/blob/v1.4.2/patterns/gro...
- Earwig 6y agoYes... kind of. Per https://en.wikipedia.org/wiki/ISO_8601 https://en.wikipedia.org/wiki/ISO_8601, there is a "basic format" without separators and an "extended format" that includes them for readability. However, a T is still required to separate the date and time in the most recent version of the standard.
- Groxx 6y agoISO 8601 is pretty absurd when you actually read it. `2021-W02` and `--01-14` are valid, as is `--1013` (quick! guess what that means! and beware that `-1013` is valid too!) Please, everyone, use a single format at all times in your systems. I don't really care what it is, though I'm fond of `2021-01-14T06:28:08Z` because it's unambiguous. But don't just say "use ISO 8601", it's far too vague and you'll inevitably have variations.
- gerikson 6y agoWithout having read the spec... * `2021-W02` means the second (ISO) week of 2021. Perfectly valid and used in a lot of planning. * `--01-14` - I'm assuming this is a recurring date: every 14 Jan for every year * `--1013` - at 1PM every 10th of the month? Guessing here I believe ISO 8601 is a ISO codification of a DIN standard, and based on other standards processes I'm guessing some German manufacturing companies were the only ones who bothered showing up, so their internal software practices were encoded into the spec because no-one else cared..
- vianneychevalie 6y agoThat is such a common problem when standardizing, I've started to force my clients to have at least one person of each entity in project teams. Often the biggest entity will end up accidentally forcing their practices, sometimes sub-optimal, to entire organizations, simply by having the manpower to show up to meetings.
- gerikson 6y agoEdit `--1013` is 13 Oct in any year: https://en.wikipedia.org/wiki/ISO_8601#Truncated_representations https://en.wikipedia.org/wiki/ISO_8601#Truncated_representat... (`--01-14` is Jan 14 in any year, the last dash is "optional"). The "duration" (`P`) and "repetition" (`R`) syntax is also pretty wild.
- rakoo 6y agoIndeed, what you really want to say is "use RFC3339" (https://www.ietf.org/rfc/rfc3339.txt https://www.ietf.org/rfc/rfc3339.txt)
- johnrob 6y agoOne big advantage of using string representations of dates is avoiding misunderstood timezone calculations that may or may not occur at various layers of the backend stack. The downside of course is storage space.
- userbinator 6y agoDates were represented as strings in "YYYYMMDD" format (so today would be "20210113053923") instead of UNIX timestamps Such a representation naturally avoids the Y2K38 problem, and could go beyond Y10K. It's traditional in Windows and DOS (neither of which have the Y2K38 problem) to store timestamps as a structure of fields. The other things you noted I agree with, however.
- 3np 6y agoWell, assuming they're storing the strings as ASCII, that's 98 bits - the y2k38 problem is for 32 bit integers, so a 64 bit integer would be way, way more than needed for human needs for foreseeable generations.
- Quarrelsome 6y agoYou can also instantly read them which makes troubleshooting easier. I mean sure, if your shit is too slow maybe switch to less text in release mode but YAGNI.
- CoolGuySteve 6y agoIf they're using a javascript 53bit int representation for the seconds (or an int64_t cast down to a javascript big int) then it's a Y142711K problem, by which point the Imperium of Mankind will hopefully have settled on a more robust format.
- freeone3000 6y agoThat's how we ended up with the 2038 problem!
- wmil 6y agoThe tech-priests will have lost the ability to fix it.
- toyg 6y agoI expect Slaanesh and friends will manage to sabotage that somehow.
- 6y ago
- latch 6y agoI think most JSON libraries encode dates in something that's closer to what Parler is doing than when you think is correct (e.g, using ISO 8601 or something) I could see the argument for representing impressions as a string (especially if it's updated asynchronously and denormalized like that). The major downside is localization.
- uxp100 6y agoI think the display strings thing is because exact number of impressions etc is slightly sensitive information. The whole site was "gamed" from the start, but providing exact vote counts makes it easier for other people to game. I guess. Don't really know, but I do believe that the numbers given by reddit, for example, are exact, but fake. Fuzzed a bit. HN also hides some of this, or behaves misleadingly, your downvotes don't always count, I think.
- formalsystems 6y agoThey would display numbers less than 1000 as-is, and only start adding the "k" and "m" prefix after the 4-digit and 7-digit threshold was crossed.
- bkilrain 6y agoBut how could they maintain an accurate count? Maybe they were just persisting the user-friendly format alongside the actual count...
- bonestamp2 6y agoThe endpoint of the API is probably just rounding the accurate number and returning a friendly number... or it's all bullshit anyway.
- deleted 6y ago[deleted]
- kevindurb 6y agoIf I remember correctly mongo stores the id in “_id” and has a getter for “id” so maybe they just iterated all the keys of the model when they stringified their output
- kijin 6y agoElasticsearch, too. In either case, it looks like they're just piping raw backend responses to the API endpoint without removing unnecessary fields.
- developer2 6y agoYep, that's an indication of Elasticsearch being used (and not transforming documents to a standard representation that strips such fields).
- ashtonkem 6y agoIt seems like they basically just exposed a lot of data directly, as apparently most of their APIs didn’t enforce any authentication or hide records that had been soft deleted. Apparently the records were strictly sequential, which I don’t believe is true for Mongo which IIRC includes the node ID in part of it.