6 ms·

I have a confusion about ID structure/format: The ID is composed of two parts: a millisecond time and a sequence number. The number after the dot is the

I have a confusion about ID structure/format: The ID is composed of two parts: a millisecond time and a sequence number. The number after the dot is the sequence number, and is used in order to distinguish entries added in the same millisecond. Does this mean for example that 1506872463535.11 comes after 1506872463535.2 (because 11 > 2)? If so that means treating these as decimals (which will be easy to do inadvertently) will yield the wrong order (as would sorting them lexicographically). If so it seems like something other than a decimal point would be a better separator (colon perhaps).

ihattendorf 9y ago
What about regions that treat the comma as a decimal separator? I agree with the other commenter that this is no different than using periods in IPv4 addresses.
- anyfoo 9y ago
  As a member of such a region, I suspect that those regions a well aware that the prevalent notation throughout programming uses '.' as a decimal separator. IPv4 addresses usually have 4 components and do not have an inherently fractional unit as their first component.
  - mijamo 9y ago
    Disagree. '.' is used for version numbers everywhere, even when it's only 2 components (ex: Wordpress versioning, Django etc.) and it seems pretty clear that 1.11 > 1.2 there. Once the specification is stated clearly as it is right now it is not a problem. Another symbol could be used (for instance #) but I really don't see a need for that.
- IanCal 9y ago
  > What about regions that treat the comma as a decimal separator? A good reason not to change it from a period to a comma, not really relevant to whether to change it from a period to something else. > I agree with the other commenter that this is no different than using periods in IPv4 addresses. IP addresses have no useful concept of "before" or "after", whereas these do.
sterex 9y ago
Perhaps decoupling the <$.#> as (# x 0.01) would solve the problem right? Not sure if that's what they do, but it isn't very complex.
antirez 9y ago
Yes actually maybe it's a good idea to change the point with something else. Thanks for the hint.
- ryanschneider 9y ago
  Supporting `MAXSIZE` as well as `MAXLEN` on `XADD` would also handle a nice Kafka feature (the ability to define your log size in either number of messages or size on disk). Something like: `XADD MAXSIZE ~ 2147484000 * foo bar` to cap the stream at 2GB + 1 node.
  - dvirsky 9y ago
    And if ids are timestamps, maybe we can define it as MAXTIME as well.
- rakoo 9y ago
  Why keep the separation at all ? Are clients expected to be able to query for a given timestamp precisely ? Because then you get all the problems with clock synchronization, especially given that the Streams' clock is monotonic and I'd expect clients' clock to not be
  - unkown-unknowns 9y ago
    It might be useful to be able to query by server time regardless of whether your client clock is in sync. You retrieve some set of data and the next time you can ask the server to give you everything newer than x, where x was the highest time stamp you got from the server previously.
    - rakoo 9y ago
      Yes exactly, you want to ask what is newer than x, where x is the last event you're aware of, but you don't really care about the date and time in that case. If you just store the last id given by redis Streams naively then you don't even care that they're timestamps; at that point my question is, why even bother with the distinction. Just ask for everything after x and be done with it.
ecesena 9y ago
I don't think you can ever think to a total order between events. In your example. My understanding is that the 2 events in your example happened in parallel (by redis definition of time granularity), and there's no correct ordering between the two. What I want to say is that even if you change to a ":" you'd get wrong results. I think antirez is saying that, by serializing the id in this way, you can also get timeseries at the ms precision for free. Edit: nvm, antirez just replied :)
- deleted 9y ago
  [deleted]
twic 9y ago
The dot doesn't make that a decimal, any more than it makes IP addresses or version numbers decimals. As for treating them as decimals inadvertently, well, hopefully client libraries will expose IDs as pairs of integers, not as strings. If users convert them into strings and then back into meaningless pseudo-decimals, well, great, we'll have an entertaining post about someone's outage to read.
- rjeli 9y ago
  But, it has milliseconds on the left. Before I read the explanation, I immediately thought it was a floating point timestamp. IP addresses have no reasonable decimal interpretation.
- hueving 9y ago
  This is a terrible attitude to have when responding to an obvious potential UX confusion, particularly when it will only come up in edge cases (>10 per millisecond).
  - maerF0x0 9y ago
    10 per milli as an edge case is domain specific.
- anyfoo 9y ago
  It's not much of a problem for (v4) IPs, because they almost always consist of 4 numbers separated by a dot, making them immediately distinguishable from decimal numbers. If two-component IPs were common (they are sometimes seen in CIDR notation, but not often), the dot would have been an unfortunate separator choice as well. For versions with only two components, I would argue that the dot can be confusing already. Why use a separator that has the potential of confusion when there are several other choices with less potential?
- xapata 9y ago
  Ah yes, like trap answers on a multiple choice exam. I suppose the Zen of that design would be: "There should be more than one obvious way to do it, but only one correct way."
archit3cture 9y ago
In the XADD example, you have XADD mystream * sensor-id 1234 temperature 10.5 to let the server choose an id. It could also be interesting to have XADD mystream 1506872463535.* sensor-id 1234 temperature 10.5 to be able to add an element in a specific msec bucket

ihattendorf 9y ago

What about regions that treat the comma as a decimal separator? I agree with the other commenter that this is no different than using periods in IPv4 addresses.

anyfoo 9y ago
As a member of such a region, I suspect that those regions a well aware that the prevalent notation throughout programming uses '.' as a decimal separator. IPv4 addresses usually have 4 components and do not have an inherently fractional unit as their first component.
- mijamo 9y ago
  Disagree. '.' is used for version numbers everywhere, even when it's only 2 components (ex: Wordpress versioning, Django etc.) and it seems pretty clear that 1.11 > 1.2 there. Once the specification is stated clearly as it is right now it is not a problem. Another symbol could be used (for instance #) but I really don't see a need for that.
IanCal 9y ago
> What about regions that treat the comma as a decimal separator? A good reason not to change it from a period to a comma, not really relevant to whether to change it from a period to something else. > I agree with the other commenter that this is no different than using periods in IPv4 addresses. IP addresses have no useful concept of "before" or "after", whereas these do.

sterex 9y ago

Perhaps decoupling the <$.#> as (# x 0.01) would solve the problem right? Not sure if that's what they do, but it isn't very complex.

antirez 9y ago

Yes actually maybe it's a good idea to change the point with something else. Thanks for the hint.

ryanschneider 9y ago
Supporting `MAXSIZE` as well as `MAXLEN` on `XADD` would also handle a nice Kafka feature (the ability to define your log size in either number of messages or size on disk). Something like: `XADD MAXSIZE ~ 2147484000 * foo bar` to cap the stream at 2GB + 1 node.
- dvirsky 9y ago
  And if ids are timestamps, maybe we can define it as MAXTIME as well.
rakoo 9y ago
Why keep the separation at all ? Are clients expected to be able to query for a given timestamp precisely ? Because then you get all the problems with clock synchronization, especially given that the Streams' clock is monotonic and I'd expect clients' clock to not be
- unkown-unknowns 9y ago
  It might be useful to be able to query by server time regardless of whether your client clock is in sync. You retrieve some set of data and the next time you can ask the server to give you everything newer than x, where x was the highest time stamp you got from the server previously.
  - rakoo 9y ago
    Yes exactly, you want to ask what is newer than x, where x is the last event you're aware of, but you don't really care about the date and time in that case. If you just store the last id given by redis Streams naively then you don't even care that they're timestamps; at that point my question is, why even bother with the distinction. Just ask for everything after x and be done with it.

ecesena 9y ago

I don't think you can ever think to a total order between events. In your example. My understanding is that the 2 events in your example happened in parallel (by redis definition of time granularity), and there's no correct ordering between the two. What I want to say is that even if you change to a ":" you'd get wrong results. I think antirez is saying that, by serializing the id in this way, you can also get timeseries at the ms precision for free. Edit: nvm, antirez just replied :)

deleted 9y ago
[deleted]

twic 9y ago

The dot doesn't make that a decimal, any more than it makes IP addresses or version numbers decimals. As for treating them as decimals inadvertently, well, hopefully client libraries will expose IDs as pairs of integers, not as strings. If users convert them into strings and then back into meaningless pseudo-decimals, well, great, we'll have an entertaining post about someone's outage to read.

rjeli 9y ago
But, it has milliseconds on the left. Before I read the explanation, I immediately thought it was a floating point timestamp. IP addresses have no reasonable decimal interpretation.
hueving 9y ago
This is a terrible attitude to have when responding to an obvious potential UX confusion, particularly when it will only come up in edge cases (>10 per millisecond).
- maerF0x0 9y ago
  10 per milli as an edge case is domain specific.
anyfoo 9y ago
It's not much of a problem for (v4) IPs, because they almost always consist of 4 numbers separated by a dot, making them immediately distinguishable from decimal numbers. If two-component IPs were common (they are sometimes seen in CIDR notation, but not often), the dot would have been an unfortunate separator choice as well. For versions with only two components, I would argue that the dot can be confusing already. Why use a separator that has the potential of confusion when there are several other choices with less potential?
xapata 9y ago
Ah yes, like trap answers on a multiple choice exam. I suppose the Zen of that design would be: "There should be more than one obvious way to do it, but only one correct way."

archit3cture 9y ago

In the XADD example, you have XADD mystream * sensor-id 1234 temperature 10.5 to let the server choose an id. It could also be interesting to have XADD mystream 1506872463535.* sensor-id 1234 temperature 10.5 to be able to add an element in a specific msec bucket