20 ms·

Co-founder (Conrad) here. SQLite does indeed have a narrow set of types. Timestamps in Postgres are converted to text in SQLite in a format that is compatible w

by ochiba 3y ago

Co-founder (Conrad) here. SQLite does indeed have a narrow set of types. Timestamps in Postgres are converted to text in SQLite in a format that is compatible with ISO8601 and SQLite's functions. Type conversions are documented here: https://docs.powersync.com/usage/sync-rules/types https://docs.powersync.com/usage/sync-rules/types

zlg_codes 3y ago
I see you're using ISO8601 and including timezone data, which seems best for generalizing. Somewhat related: if you were storing UNIX timestamps that only need the resolution of a day and will always be UTC, would you use an integer for the UNIX time or just a shortened ISO8601 without H:M:S data? I've been trying to decide on this for a feature in a tool I'm making. In either case, SQLite has the ability to convert back and forth, so it'd really be the exported JSON containing the data whose readability would be affected more than anything.
- matharmin 3y ago
  Personally I prefer the shortened ISO8601 format (YYYY-MM-DD). While it's not the most efficient, efficiency doesn't often matter on that level. It's unambiguous, can be parsed by practically any library, and easily human-readable. I'd avoid using unix timestamps for day-resolution, since it's very easy to make a mistake with the timezone when parsing, which you only notice when changing between a positive and negative timezone. Julian day is another option, but support for it is not quite as universal. If efficiency is important, that's the format I'd use.
  - vlovich123 3y ago
    What’s the ambiguity issue with Unix time stamps? Unix time stamps are expressed as a UTC offset always so I’m not sure how time zones come into play / what parsing might involved.
    - zlg_codes 3y ago
      I think he's referring to behavior that some software has when it encounters raw UNIX timestamps without TZ information. I recall seeing a change like that in Python's datetime module as of late, so that even if UNIX timestamps are UTC by default, it's more explicit. In the case of my project, the goal is to have UTC-only timestamps/dates, and since the resolution I'm using it for is a calendar day, time zone doesn't matter. I will still need to be explicit about it in my code when converting. Or maybe edge cases like historical timezone changes, leap seconds?
      - vlovich123 3y ago
        By definition unix timestamps are UTC. Python's datetime module has a lot of footguns because naive timestamps can represent UTC, local time, or even a non-standard application-specific time. Python's aware timestamps just carry the tzinfo information along with the naive timestamp so that you can get to UTC from the abstract naive timestamp. It's a really overdesigned footgun of Python's datetime module more than anything, but again not unix timestamps which are always an offset from 1970 midnight UTC.
        zlg_codes 3y ago
        Cool, thanks for the clarification!
    - matharmin 3y ago
      You can do it in a consistent way, for example always store the timestamp at 00:00 UTC. But that's not obvious just by looking at the values, which is why it can be ambiguous. And when parsing those values, you could do for example (JS) `new Date(ts*1000).getDay()`. Of course that's not correct - you need to use the UTC methods. But it's easy to miss, and may pass all your tests until you switch the timezone. Those aren't massive issues - you just have to be careful with the parsing and serialization, and you need to be careful in any case. It just explains my personal preference.
      - 3y ago

by ochiba 3y ago

zlg_codes 3y ago

I see you're using ISO8601 and including timezone data, which seems best for generalizing. Somewhat related: if you were storing UNIX timestamps that only need the resolution of a day and will always be UTC, would you use an integer for the UNIX time or just a shortened ISO8601 without H:M:S data? I've been trying to decide on this for a feature in a tool I'm making. In either case, SQLite has the ability to convert back and forth, so it'd really be the exported JSON containing the data whose readability would be affected more than anything.

matharmin 3y ago
Personally I prefer the shortened ISO8601 format (YYYY-MM-DD). While it's not the most efficient, efficiency doesn't often matter on that level. It's unambiguous, can be parsed by practically any library, and easily human-readable. I'd avoid using unix timestamps for day-resolution, since it's very easy to make a mistake with the timezone when parsing, which you only notice when changing between a positive and negative timezone. Julian day is another option, but support for it is not quite as universal. If efficiency is important, that's the format I'd use.
- vlovich123 3y ago
  What’s the ambiguity issue with Unix time stamps? Unix time stamps are expressed as a UTC offset always so I’m not sure how time zones come into play / what parsing might involved.
  - zlg_codes 3y ago
    I think he's referring to behavior that some software has when it encounters raw UNIX timestamps without TZ information. I recall seeing a change like that in Python's datetime module as of late, so that even if UNIX timestamps are UTC by default, it's more explicit. In the case of my project, the goal is to have UTC-only timestamps/dates, and since the resolution I'm using it for is a calendar day, time zone doesn't matter. I will still need to be explicit about it in my code when converting. Or maybe edge cases like historical timezone changes, leap seconds?
    - vlovich123 3y ago
      By definition unix timestamps are UTC. Python's datetime module has a lot of footguns because naive timestamps can represent UTC, local time, or even a non-standard application-specific time. Python's aware timestamps just carry the tzinfo information along with the naive timestamp so that you can get to UTC from the abstract naive timestamp. It's a really overdesigned footgun of Python's datetime module more than anything, but again not unix timestamps which are always an offset from 1970 midnight UTC.
      - zlg_codes 3y ago
        Cool, thanks for the clarification!
  - matharmin 3y ago
    You can do it in a consistent way, for example always store the timestamp at 00:00 UTC. But that's not obvious just by looking at the values, which is why it can be ambiguous. And when parsing those values, you could do for example (JS) `new Date(ts*1000).getDay()`. Of course that's not correct - you need to use the UTC methods. But it's easy to miss, and may pass all your tests until you switch the timezone. Those aren't massive issues - you just have to be careful with the parsing and serialization, and you need to be careful in any case. It just explains my personal preference.
    - 3y ago