7 ms·

Yes actually maybe it's a good idea to change the point with something else. Thanks for the hint.

ryanschneider 9y ago
Supporting `MAXSIZE` as well as `MAXLEN` on `XADD` would also handle a nice Kafka feature (the ability to define your log size in either number of messages or size on disk). Something like: `XADD MAXSIZE ~ 2147484000 * foo bar` to cap the stream at 2GB + 1 node.
- dvirsky 9y ago
  And if ids are timestamps, maybe we can define it as MAXTIME as well.
rakoo 9y ago
Why keep the separation at all ? Are clients expected to be able to query for a given timestamp precisely ? Because then you get all the problems with clock synchronization, especially given that the Streams' clock is monotonic and I'd expect clients' clock to not be
- unkown-unknowns 9y ago
  It might be useful to be able to query by server time regardless of whether your client clock is in sync. You retrieve some set of data and the next time you can ask the server to give you everything newer than x, where x was the highest time stamp you got from the server previously.
  - rakoo 9y ago
    Yes exactly, you want to ask what is newer than x, where x is the last event you're aware of, but you don't really care about the date and time in that case. If you just store the last id given by redis Streams naively then you don't even care that they're timestamps; at that point my question is, why even bother with the distinction. Just ask for everything after x and be done with it.
    - lsiebert 9y ago
      Redis also has TIME to get the current server time with milliseconds and the unix time stamp. I'm reasonably sure that's what's being used to get the first part of the ID anyway.
      - rakoo 9y ago
        Ah, I didn't know that. Although the post says that the timestamp of an id might also be the timestamp of the last message, since the clock can go backwards, so in the worst case a client might get some duplicate messages.
agumonkey 9y ago
What were or would be your prefered alternative syntax ? `:` ?
- antirez 9y ago
  Not sure... : looks ok actually, even _ or - could make some sense. The # is a bit too heavy on the eyes :-)
  - lsiebert 9y ago
    I think a mostly vertical symbol is better, ":" or "|" is preferably to "_" or "-".
    - anyfoo 9y ago
      Why is that?
      - agumonkey 9y ago
        verticality convey difference in kind of information better IMO
    - simonhamp 9y ago
      I agree. Perhaps a forward-slash (/) to denote subordination, e.g. 1507035873/11
  - femto113 9y ago
    I'll probably eventually hate myself for even bringing this up but I can't help but notice the similarity between this ID structure and Version 1 (aka timestamp) UUIDs. While I wouldn't go as far as recommending that you fully adopt that form, it might be worth considering if you could make these IDs compatible with UUIDs by defining a canonical transform. The critical differences are: - UUIDs use a different epoch (15 Oct 1582 vs 1 Jan 1970) - UUIDs count 100 ns blocks instead of ms - UUIDs include a 6 byte "node id" - UUIDs allow only up to 15 bits of "sequence" I think that last one is the biggest deal, since as currently specced redis allows 64 bits of sequence, which is obviously much bigger than 15. The options I see are either up the time resolution used by redis, encode some of redis's sequence bits into the UUID's time bits, or just live with it as a limitation--in practice 2^15 is a lot of messages to get in a single millisecond (though in cases of clocks jumping back might not be too much). You'd also need to come up with some thing for the node id, perhaps the first 6 bytes of a cluster node ID or similar.
unkown-unknowns 9y ago
I vote that you use a dash instead.
- zedpm 9y ago
  It turns out that's what it will be [0], as dash retains the ability to easily copy the whole identifier in most terminals. [0]: https://github.com/antirez/redis/commit/1189d90d749c84e9842400509345e581197d7383 https://github.com/antirez/redis/commit/1189d90d749c84e98424...
pookeh 9y ago
You can also have a look at the technique used here to create collision-free sequential unique IDs across a cluster, even if it is just for inspiration: https://www.npmjs.com/package/cuid https://www.npmjs.com/package/cuid Example: c - h72gsb32 - 0000 - udoc - l363eofy The groups, in order, are: 1. 'c' - identifies this as a cuid, and allows you to use it in html entity ids. The fixed value helps keep the ids sequential. 2. Timestamp 3. Counter - a single process might generate the same random string. The weaker the pseudo-random source, the higher the probability. That problem gets worse as processors get faster. The counter will roll over if the value gets too big. 4. Client fingerprint. For example, the first two chars are extracted from the process.pid. The next two chars are extracted from the hostname. 5. Pseudo random (Math.random())

Supporting `MAXSIZE` as well as `MAXLEN` on `XADD` would also handle a nice Kafka feature (the ability to define your log size in either number of messages or size on disk). Something like: `XADD MAXSIZE ~ 2147484000 * foo bar` to cap the stream at 2GB + 1 node.

dvirsky 9y ago
And if ids are timestamps, maybe we can define it as MAXTIME as well.

rakoo 9y ago

Why keep the separation at all ? Are clients expected to be able to query for a given timestamp precisely ? Because then you get all the problems with clock synchronization, especially given that the Streams' clock is monotonic and I'd expect clients' clock to not be

unkown-unknowns 9y ago
It might be useful to be able to query by server time regardless of whether your client clock is in sync. You retrieve some set of data and the next time you can ask the server to give you everything newer than x, where x was the highest time stamp you got from the server previously.
- rakoo 9y ago
  Yes exactly, you want to ask what is newer than x, where x is the last event you're aware of, but you don't really care about the date and time in that case. If you just store the last id given by redis Streams naively then you don't even care that they're timestamps; at that point my question is, why even bother with the distinction. Just ask for everything after x and be done with it.
  - lsiebert 9y ago
    Redis also has TIME to get the current server time with milliseconds and the unix time stamp. I'm reasonably sure that's what's being used to get the first part of the ID anyway.
    - rakoo 9y ago
      Ah, I didn't know that. Although the post says that the timestamp of an id might also be the timestamp of the last message, since the clock can go backwards, so in the worst case a client might get some duplicate messages.

agumonkey 9y ago

What were or would be your prefered alternative syntax ? `:` ?

antirez 9y ago
Not sure... : looks ok actually, even _ or - could make some sense. The # is a bit too heavy on the eyes :-)
- lsiebert 9y ago
  I think a mostly vertical symbol is better, ":" or "|" is preferably to "_" or "-".
  - anyfoo 9y ago
    Why is that?
    - agumonkey 9y ago
      verticality convey difference in kind of information better IMO
  - simonhamp 9y ago
    I agree. Perhaps a forward-slash (/) to denote subordination, e.g. 1507035873/11
- femto113 9y ago
  I'll probably eventually hate myself for even bringing this up but I can't help but notice the similarity between this ID structure and Version 1 (aka timestamp) UUIDs. While I wouldn't go as far as recommending that you fully adopt that form, it might be worth considering if you could make these IDs compatible with UUIDs by defining a canonical transform. The critical differences are: - UUIDs use a different epoch (15 Oct 1582 vs 1 Jan 1970) - UUIDs count 100 ns blocks instead of ms - UUIDs include a 6 byte "node id" - UUIDs allow only up to 15 bits of "sequence" I think that last one is the biggest deal, since as currently specced redis allows 64 bits of sequence, which is obviously much bigger than 15. The options I see are either up the time resolution used by redis, encode some of redis's sequence bits into the UUID's time bits, or just live with it as a limitation--in practice 2^15 is a lot of messages to get in a single millisecond (though in cases of clocks jumping back might not be too much). You'd also need to come up with some thing for the node id, perhaps the first 6 bytes of a cluster node ID or similar.

unkown-unknowns 9y ago

I vote that you use a dash instead.

zedpm 9y ago
It turns out that's what it will be [0], as dash retains the ability to easily copy the whole identifier in most terminals. [0]: https://github.com/antirez/redis/commit/1189d90d749c84e9842400509345e581197d7383 https://github.com/antirez/redis/commit/1189d90d749c84e98424...

pookeh 9y ago

You can also have a look at the technique used here to create collision-free sequential unique IDs across a cluster, even if it is just for inspiration: https://www.npmjs.com/package/cuid https://www.npmjs.com/package/cuid Example: c - h72gsb32 - 0000 - udoc - l363eofy The groups, in order, are: 1. 'c' - identifies this as a cuid, and allows you to use it in html entity ids. The fixed value helps keep the ids sequential. 2. Timestamp 3. Counter - a single process might generate the same random string. The weaker the pseudo-random source, the higher the probability. That problem gets worse as processors get faster. The counter will roll over if the value gets too big. 4. Client fingerprint. For example, the first two chars are extracted from the process.pid. The next two chars are extracted from the hostname. 5. Pseudo random (Math.random())