3 ms·

It's hard to see running a critical service over an SSH tunnel as a long-term solution. Even with monit and autossh guarding the connection, SSH isn't exactly

It's hard to see running a critical service over an SSH tunnel as a long-term solution. Even with monit and autossh guarding the connection, SSH isn't exactly rock-solid when used this way. If the redis protocol is so compressible, maybe this behavior should be integrated as a protocol option like [1]? [1] http://dev.mysql.com/doc/refman/5.5/en/replication-options-slave.html#sysvar_slave_compressed_protocol http://dev.mysql.com/doc/refman/5.5/en/replication-options-s...

antirez 14y ago
A lot of redundancy can be eliminated, and we have good past experience (and results) with this approach because we do it in the RDB format used for storage. For the replication link it's easy to imagine a simple way to encode command names as operation IDs, the same for prefixed length stuff. So now you have: *3 $3 SET $3 foo $3 bar But this may become like: <set-opcode><2><3>foo<3>bar where the opcode is 1 byte, and the rest of the prefixed lengths are variable depending on the argument size, but most of the times 1 or two bytes. This of course is much more compact. However it's not bad to have the current format to be exactly like the AOF, and the client protocol itself, so everything is the same currently... and is future/backward compatible without issues. But well performances always have some price.
solso 14y ago
that's a good point... as mentioned in the post we use ssh tunnels and have experienced quite a few glitches, but in this case it has been running without a hiccup for 3 weeks (proof of nothing of course). I suspect (guess) that not having an idle connection help the tunnel to not randomly drop.
- semiquaver 14y ago
  Can you talk more about what the glitches were like? We're a few months away from production with our first product that uses redis, so this is very interesting. Have you simulated a network partition? How does redis handle the reconnect?
  - ironchef 14y ago
    Depending on length of time the network is "out" there's a avariety of things that could occur. In general, the master would kick off a bgsave and the slave would re-bootstrap once it received said snapshot.
    - solso 14y ago
      that's totally correct. In the case of a redis replication, the ssh tunnel failing would be quite costly, since the whole database will have to be send again since the slave would do a SYNC op. And that can be multiple GB in our case. But so far, we haven't experience it.
  - solso 14y ago
    The glitches with the ssh tunnel were not for the redis replication setup. This has been running fine for 3 weeks already. However we also use ssh tunnels for our continuous integration (jenkins) and the irc bots that we use for development (http://3scale.github.com/2012/06/29/irc-driven-development-part-2/ http://3scale.github.com/2012/06/29/irc-driven-development-p...). In this setup we have the autossh going awol every one or two weeks. But as the blog post mentions, the ssh tunnel runs between our HQ (fiber) and Amazon, not very reliable. Furthermore, there is a lot of idle periods, which seems to trigger most of the issues. We cannot give more specifics since we forcefully just restart the daemon with a monit/munin combo.
    - alexfoo 14y ago
      Firewalls (which may be beyond your control and that you may not even be aware of) between you and the other end often drop connections that have been idle for a while (anything from 5 minutes to a couple of hours), especially if the firewall is busy tracking tens of thousands of active connections. The outbound firewall of our company drops my outbound ssh connections if they are idle for just 5 minutes. Adding:- Host * ServerAliveInterval 60 in my ~/.ssh/config file prevented most of the drops (the rest are explained by proper network outages or my DSL connection at home flapping.) A corresponding ClientAliveInterval setting in the sshd_config file also helps mask the problem [EDIT] if I ssh home from a machine at work that doesn't share my home directory ssh config file.
pepve 14y ago
It's hard to see running a critical service over an SSH tunnel as a long-term solution. Since I'm tempted to set this up myself (not for Redis though), what is the alternative?
- __alexs 14y ago
  vtund http://vtun.sourceforge.net/ http://vtun.sourceforge.net/ I've only used it for tunnelling some traffic out of my home network to a VPS but it's been rock solid for me over several years of frequent use.
  - newhouseb 14y ago
    Has anyone used this for something like the OP's problem in production?
- jevinskie 14y ago
  OpenVPN advertises LZO compression but it seems to be poorly documented (just an --comp-lzo argument in the manpage). Is OpenVPN considered "enterprise" enough for this sort of application, even disregarding the compression support?
  - fomojola 14y ago
    OpenVPN actually works very well with compression: used it for 2 years to synchronize databases between 2 sites in India and one on the US and the compression cut the sync time nearly in half. It was pretty much fire-and-forget: I think I only looked at it twice in the 2 years after I set it up, both times to replace certs. I'd use that over an ssh tunnel, if only because you wouldn't need the autossh tricks.

antirez 14y ago

A lot of redundancy can be eliminated, and we have good past experience (and results) with this approach because we do it in the RDB format used for storage. For the replication link it's easy to imagine a simple way to encode command names as operation IDs, the same for prefixed length stuff. So now you have: *3 $3 SET $3 foo $3 bar But this may become like: <set-opcode><2><3>foo<3>bar where the opcode is 1 byte, and the rest of the prefixed lengths are variable depending on the argument size, but most of the times 1 or two bytes. This of course is much more compact. However it's not bad to have the current format to be exactly like the AOF, and the client protocol itself, so everything is the same currently... and is future/backward compatible without issues. But well performances always have some price.

solso 14y ago

that's a good point... as mentioned in the post we use ssh tunnels and have experienced quite a few glitches, but in this case it has been running without a hiccup for 3 weeks (proof of nothing of course). I suspect (guess) that not having an idle connection help the tunnel to not randomly drop.

semiquaver 14y ago
Can you talk more about what the glitches were like? We're a few months away from production with our first product that uses redis, so this is very interesting. Have you simulated a network partition? How does redis handle the reconnect?
- ironchef 14y ago
  Depending on length of time the network is "out" there's a avariety of things that could occur. In general, the master would kick off a bgsave and the slave would re-bootstrap once it received said snapshot.
  - solso 14y ago
    that's totally correct. In the case of a redis replication, the ssh tunnel failing would be quite costly, since the whole database will have to be send again since the slave would do a SYNC op. And that can be multiple GB in our case. But so far, we haven't experience it.
- solso 14y ago
  The glitches with the ssh tunnel were not for the redis replication setup. This has been running fine for 3 weeks already. However we also use ssh tunnels for our continuous integration (jenkins) and the irc bots that we use for development (http://3scale.github.com/2012/06/29/irc-driven-development-part-2/ http://3scale.github.com/2012/06/29/irc-driven-development-p...). In this setup we have the autossh going awol every one or two weeks. But as the blog post mentions, the ssh tunnel runs between our HQ (fiber) and Amazon, not very reliable. Furthermore, there is a lot of idle periods, which seems to trigger most of the issues. We cannot give more specifics since we forcefully just restart the daemon with a monit/munin combo.
  - alexfoo 14y ago
    Firewalls (which may be beyond your control and that you may not even be aware of) between you and the other end often drop connections that have been idle for a while (anything from 5 minutes to a couple of hours), especially if the firewall is busy tracking tens of thousands of active connections. The outbound firewall of our company drops my outbound ssh connections if they are idle for just 5 minutes. Adding:- Host * ServerAliveInterval 60 in my ~/.ssh/config file prevented most of the drops (the rest are explained by proper network outages or my DSL connection at home flapping.) A corresponding ClientAliveInterval setting in the sshd_config file also helps mask the problem [EDIT] if I ssh home from a machine at work that doesn't share my home directory ssh config file.

pepve 14y ago

It's hard to see running a critical service over an SSH tunnel as a long-term solution. Since I'm tempted to set this up myself (not for Redis though), what is the alternative?

__alexs 14y ago
vtund http://vtun.sourceforge.net/ http://vtun.sourceforge.net/ I've only used it for tunnelling some traffic out of my home network to a VPS but it's been rock solid for me over several years of frequent use.
- newhouseb 14y ago
  Has anyone used this for something like the OP's problem in production?
jevinskie 14y ago
OpenVPN advertises LZO compression but it seems to be poorly documented (just an --comp-lzo argument in the manpage). Is OpenVPN considered "enterprise" enough for this sort of application, even disregarding the compression support?
- fomojola 14y ago
  OpenVPN actually works very well with compression: used it for 2 years to synchronize databases between 2 sites in India and one on the US and the compression cut the sync time nearly in half. It was pretty much fire-and-forget: I think I only looked at it twice in the 2 years after I set it up, both times to replace certs. I'd use that over an ssh tunnel, if only because you wouldn't need the autossh tricks.