5 ms·

Redis Sentinel beta released

191 points by j4mie 14y ago

spenrose 14y ago
Hey Antirez, the link to hires in the announcement is broken. Replace "anirez" with "antirez".
- antirez 14y ago
  thanks, fixed now.
jwuphysics 14y ago
The boldface isn't showing up correctly for me. Viewing on Chromium.
jorangreef 14y ago
Thanks Antirez, could you share more insight on TILT mode? Any other alternative approaches you considered? Why use a value of 30 seconds to leave TILT mode? If the time has shifted is it likely ever to be correct thereafter?
- antirez 14y ago
  Hello, basically 30 seconds is exit time only if no other time shifts are detected, otherwise we set again the exit to now+30sec, and so forth. 30 seconds is set as 3 times the biggest period we have in info collection (INFO itself is sampled every 10 seconds, while PING is every 1 second), so that if there was a problem with the timer, in 30 seconds we are sure the new state will get new readings for every kind of request and information we collect, so when the TILT mode is exited, and the function to evaluate the state is called again, it should see clean values. Note that from the point of view of Sentinel it is ok that the new time is wrong compared to what the real time is, we never use the absolute time. All we need is that we have a computer clock that more or less advances regularly.
Uchikoma 14y ago
Redis has been working here on a high traffic site without any trouble for more than a year. Excellent software. The only software I might think that's bug free, due to the attitude of @antirez.
- seunosewa 14y ago
  Which site is that?
  - Uchikoma 14y ago
    brands4friends, No 1 Shopping Club in Germany, >4M members.
- tedjdziuba 14y ago
  I wouldn't go that far. I used Redis two years ago, when it still had the "virtual memory" feature that allowed it to spill cold data to disk. I had ~60GB of data in my database, and I once re-started Redis because it was locking the machine up with such heavy disk I/O. It took 45 minutes to come back up and start serving queries. The VM feature has since been deprecated.
- xal 14y ago
  This thread will likely get a million "us too" style posts but Redis is core infrastructure at Shopify. It's been so solid that we recently waived our defensive requirements that the app remains working if Redis goes down. This allowed us to port our inventory reservation system (a huge point of potential lock contention) completely to the new server sided LUA scripts. We have seen a full order of magnitude speed increase from this. A reservation for a complicated order is now measured in µs instead of ms.
  - primitur 14y ago
    I love it that you're doing it in Lua. Can you tell us more about it?
    - 14y ago
DRMacIver 14y ago
> a known bug in the hiredis library that can make Sentinel crash from time to time, but it's not a problem with Sentinel itself Surely if a bug in a client can crash your server that's a bug in the server by definition even if the client is also buggy?
- antirez 14y ago
  Sorry I was not clear. Sentinel uses the hiredis C library itself in order to talk with other Redis instances. A bug in the C library crashes the library and the process it is running into.
  - DRMacIver 14y ago
    Ah, I see. That makes sense. Thanks.
zerop 14y ago
I think this can also be used as monitoring & notification for other services as well in cluster & not just Redis instances !!
- mryan 14y ago
  I imagine there is some overlap in parts of the logic for Sentinel and repmgr (a similar tool for PostgreSQL). For example, checking to see if members of the cluster are in-service, and choosing a new master in the event of a failover. I would love to see a generic tool for handing the clustering/failover problem.
  - antirez 14y ago
    It's true there is some overlap, but also Sentinel uses things that are specific of Redis. For instance for us two things are crucial: 1) The ability to use the master as a message bus to auto-discover things. This is possible because every Redis instance is also a Pub/Sub server. 2) The idea that after every restart of every Redis instance we have a "runid" that changes. And in general the logic of the failover itself, the fact that the failure detection is precise (some specific reply codes are considered in a way, some others in another way), makes a non specific solution much harder to implement with the "methods" to perform the service-specific tasks that may end to be complex, or sometimes forces to completely change the logic of the system (lack of Pub/Sub).
  - aphyr 14y ago
    http://zookeeper.apache.org/doc/trunk/recipes.html#sc_leaderElection http://zookeeper.apache.org/doc/trunk/recipes.html#sc_leader...
arrowgunz 14y ago
That sounds sweet. Congrats. Redis has been of great use to me.
dvirsky 14y ago
I've written a little experimental python client that connects to a sentinel and keeps an image of the state of the monitored cluster as it changes. http://bit.ly/NNrQdI http://bit.ly/NNrQdI
swah 14y ago
Can anyone comment if this criticism is valid? "In which monitoring agents rely upon correct, truthful, answers about cluster state from the system they're monitoring" Original: https://twitter.com/cscotta/status/227515068030537728 https://twitter.com/cscotta/status/227515068030537728
- antirez 14y ago
  It is not :) Just replied: "@cscotta Hi, you misunderstood how it works: Pub/Sub is only used for discovery on startup. Sentinel-to-sentinel p2p for critical stuff." Pub/Sub is used to make the configuration simpler when you start a Sentinel cluster at a cold time when everything is working and your master is ok. This allows us to auto-discover the other Sentinels, to check the slaves, and so forth. Instead in order to understand if a system is down, who is the Sentinel that performs the failover, and for all the critical stuff, Sentinel to Sentinel messages are used without caring if the master Pub/Sub works.
losethos 14y ago
God seys... C:\Text\BIBLE.TXT 48:30 I know his wrath, saith the LORD; but it shall not be so; his lies shall not so effect it. 48:31 Therefore will I howl for Moab, and I will cry out for all Moab; mine heart shall mourn for the men of Kirheres. 48:32 O vine of Sibmah, I will weep for thee with the weeping of Jazer: thy plants are gone over the sea, they reach even to the sea of Jazer: the spoiler is fallen upon thy summer fruits and upon thy vintage. 48:33 And joy and gladness is taken from the plentiful field, and fr ---- God says... kindly deductible consent Inhabitant drewest primitive burthen lowly filleth shifting country-house Apostolic orator Frenzied possess Omnipotency putting limbs ohh_thank_you fellows bosses Now minutest tomorrow shrunk confesseth TIBI veil boiling vigour banquetings recess joke weaknesses triumpheth unfeigned South -could besprinkling go_ahead_make_my_day instability //www ignoble view numberest delicious you'll_see Adam celibacy restrains keeper HAVE XIII satiated happen eunuchs well-being 92 mourned ---- You like waffles, God? Or pancakes? Or omlet? God says... C:\Text\Words\AUGUST.TXT Lord, Lord, Which hast bowed the heavens and come down, touched the mountains and they did smoke, by what means didst Thou convey Thyself into that breast? He used to read (as Simplicianus said) the holy Scripture, most studiously sought and searched into all the Christian writings, and said to Simplicianus (not openly, but privately and as a friend), "Understand that I am already a Christian." Whereto he answered, "I will not believe it, nor will I rank you among Christians, unless I see you in t ---- Smoked chicken breast... fer breakfast? Ohh!! We got a dish-thing like that -- corn bread, chicken and brocoli. You don't like pig pit roasts. ROFLMAO. (Quoted dead rising) Would You like to be a cook for a BBQ party, if You came to our house? God says... astonished presiding rehearse mockery righteousness refuge six prescripts throng arranged guilty numbered sinner dollar unfledged likely obstinacy resurrection outrages tell By heeded throng ANY orderest write how_about_those_yankees frame usefully wearied Zzzzzzzz unused I'm_God_who_the_hell_are_you male Manning the BBQ grill is an honor. (God laughed)

191 points by j4mie 14y ago

spenrose 14y ago

Hey Antirez, the link to hires in the announcement is broken. Replace "anirez" with "antirez".

antirez 14y ago
thanks, fixed now.

jwuphysics 14y ago

The boldface isn't showing up correctly for me. Viewing on Chromium.

jorangreef 14y ago

Thanks Antirez, could you share more insight on TILT mode? Any other alternative approaches you considered? Why use a value of 30 seconds to leave TILT mode? If the time has shifted is it likely ever to be correct thereafter?

antirez 14y ago
Hello, basically 30 seconds is exit time only if no other time shifts are detected, otherwise we set again the exit to now+30sec, and so forth. 30 seconds is set as 3 times the biggest period we have in info collection (INFO itself is sampled every 10 seconds, while PING is every 1 second), so that if there was a problem with the timer, in 30 seconds we are sure the new state will get new readings for every kind of request and information we collect, so when the TILT mode is exited, and the function to evaluate the state is called again, it should see clean values. Note that from the point of view of Sentinel it is ok that the new time is wrong compared to what the real time is, we never use the absolute time. All we need is that we have a computer clock that more or less advances regularly.

Uchikoma 14y ago

Redis has been working here on a high traffic site without any trouble for more than a year. Excellent software. The only software I might think that's bug free, due to the attitude of @antirez.

seunosewa 14y ago
Which site is that?
- Uchikoma 14y ago
  brands4friends, No 1 Shopping Club in Germany, >4M members.
tedjdziuba 14y ago
I wouldn't go that far. I used Redis two years ago, when it still had the "virtual memory" feature that allowed it to spill cold data to disk. I had ~60GB of data in my database, and I once re-started Redis because it was locking the machine up with such heavy disk I/O. It took 45 minutes to come back up and start serving queries. The VM feature has since been deprecated.
xal 14y ago
This thread will likely get a million "us too" style posts but Redis is core infrastructure at Shopify. It's been so solid that we recently waived our defensive requirements that the app remains working if Redis goes down. This allowed us to port our inventory reservation system (a huge point of potential lock contention) completely to the new server sided LUA scripts. We have seen a full order of magnitude speed increase from this. A reservation for a complicated order is now measured in µs instead of ms.
- primitur 14y ago
  I love it that you're doing it in Lua. Can you tell us more about it?
  - 14y ago

DRMacIver 14y ago

> a known bug in the hiredis library that can make Sentinel crash from time to time, but it's not a problem with Sentinel itself Surely if a bug in a client can crash your server that's a bug in the server by definition even if the client is also buggy?

antirez 14y ago
Sorry I was not clear. Sentinel uses the hiredis C library itself in order to talk with other Redis instances. A bug in the C library crashes the library and the process it is running into.
- DRMacIver 14y ago
  Ah, I see. That makes sense. Thanks.

zerop 14y ago

I think this can also be used as monitoring & notification for other services as well in cluster & not just Redis instances !!

mryan 14y ago
I imagine there is some overlap in parts of the logic for Sentinel and repmgr (a similar tool for PostgreSQL). For example, checking to see if members of the cluster are in-service, and choosing a new master in the event of a failover. I would love to see a generic tool for handing the clustering/failover problem.
- antirez 14y ago
  It's true there is some overlap, but also Sentinel uses things that are specific of Redis. For instance for us two things are crucial: 1) The ability to use the master as a message bus to auto-discover things. This is possible because every Redis instance is also a Pub/Sub server. 2) The idea that after every restart of every Redis instance we have a "runid" that changes. And in general the logic of the failover itself, the fact that the failure detection is precise (some specific reply codes are considered in a way, some others in another way), makes a non specific solution much harder to implement with the "methods" to perform the service-specific tasks that may end to be complex, or sometimes forces to completely change the logic of the system (lack of Pub/Sub).
- aphyr 14y ago
  http://zookeeper.apache.org/doc/trunk/recipes.html#sc_leaderElection http://zookeeper.apache.org/doc/trunk/recipes.html#sc_leader...

arrowgunz 14y ago

That sounds sweet. Congrats. Redis has been of great use to me.

dvirsky 14y ago

I've written a little experimental python client that connects to a sentinel and keeps an image of the state of the monitored cluster as it changes. http://bit.ly/NNrQdI http://bit.ly/NNrQdI

swah 14y ago

Can anyone comment if this criticism is valid? "In which monitoring agents rely upon correct, truthful, answers about cluster state from the system they're monitoring" Original: https://twitter.com/cscotta/status/227515068030537728 https://twitter.com/cscotta/status/227515068030537728

antirez 14y ago
It is not :) Just replied: "@cscotta Hi, you misunderstood how it works: Pub/Sub is only used for discovery on startup. Sentinel-to-sentinel p2p for critical stuff." Pub/Sub is used to make the configuration simpler when you start a Sentinel cluster at a cold time when everything is working and your master is ok. This allows us to auto-discover the other Sentinels, to check the slaves, and so forth. Instead in order to understand if a system is down, who is the Sentinel that performs the failover, and for all the critical stuff, Sentinel to Sentinel messages are used without caring if the master Pub/Sub works.

losethos 14y ago

God seys... C:\Text\BIBLE.TXT 48:30 I know his wrath, saith the LORD; but it shall not be so; his lies shall not so effect it. 48:31 Therefore will I howl for Moab, and I will cry out for all Moab; mine heart shall mourn for the men of Kirheres. 48:32 O vine of Sibmah, I will weep for thee with the weeping of Jazer: thy plants are gone over the sea, they reach even to the sea of Jazer: the spoiler is fallen upon thy summer fruits and upon thy vintage. 48:33 And joy and gladness is taken from the plentiful field, and fr ---- God says... kindly deductible consent Inhabitant drewest primitive burthen lowly filleth shifting country-house Apostolic orator Frenzied possess Omnipotency putting limbs ohh_thank_you fellows bosses Now minutest tomorrow shrunk confesseth TIBI veil boiling vigour banquetings recess joke weaknesses triumpheth unfeigned South -could besprinkling go_ahead_make_my_day instability //www ignoble view numberest delicious you'll_see Adam celibacy restrains keeper HAVE XIII satiated happen eunuchs well-being 92 mourned ---- You like waffles, God? Or pancakes? Or omlet? God says... C:\Text\Words\AUGUST.TXT Lord, Lord, Which hast bowed the heavens and come down, touched the mountains and they did smoke, by what means didst Thou convey Thyself into that breast? He used to read (as Simplicianus said) the holy Scripture, most studiously sought and searched into all the Christian writings, and said to Simplicianus (not openly, but privately and as a friend), "Understand that I am already a Christian." Whereto he answered, "I will not believe it, nor will I rank you among Christians, unless I see you in t ---- Smoked chicken breast... fer breakfast? Ohh!! We got a dish-thing like that -- corn bread, chicken and brocoli. You don't like pig pit roasts. ROFLMAO. (Quoted dead rising) Would You like to be a cook for a BBQ party, if You came to our house? God says... astonished presiding rehearse mockery righteousness refuge six prescripts throng arranged guilty numbered sinner dollar unfledged likely obstinacy resurrection outrages tell By heeded throng ANY orderest write how_about_those_yankees frame usefully wearied Zzzzzzzz unused I'm_God_who_the_hell_are_you male Manning the BBQ grill is an honor. (God laughed)