5 ms·
How Heroku is using Redis in distributed Erlang systems
- tptacek 15y agoThis is a great deck, but I'd love to see one on how we can use Redis at Heroku; it looks like the options are (a) Redis To Go, which is unreasonably expensive, and (b) an insecure connection to an m.large at EC2.
- rb2k_ 15y agoInsecure? Why not use ssh port forwarding?
- espeed 15y agoCommunications are secure among EC2 instances within us-east, which is where Heroku is.
- bluesnowmonkey 15y agoI think the issue is that of you open up your EC2 instance's security group to your Heroku app, you're opening it up to any Heroku app.
- rednaught 15y agoSecure meaning there is a private back-end network or there is some form of tunneling/encryption end-points available and enabled by default for instances? Here's the current AWS security guide that I was able to find: http://awsmedia.s3.amazonaws.com/pdf/AWS_Security_Whitepaper.pdf http://awsmedia.s3.amazonaws.com/pdf/AWS_Security_Whitepaper...
- espeed 15y agoRead the section entitled "Packet sniffing by other tenants" - the hypervisor won't permit you to put an interface into promiscuous mode.
- rednaught 15y agoThank you. So the Xen dom0s are using IP Routing and not bridging. Do we know the details of Heroku's LXC configurations for dynos on their Amazon Domu instances? http://devcenter.heroku.com/articles/dyno-isolation http://devcenter.heroku.com/articles/dyno-isolation "On the Heroku dyno manifold, each and every process has full isolation from all other processes, including those of the underlying operating system and other processes from the same application. This provides complete protection from either of the circumstances above, and many others." Based on that, do we presume they are also using routes and not bridges?
- tptacek 15y agoI'm unwilling to trust IP access control for anything in 2011. If I could do SSH port forwarding, that'd probably be my first choice. I have a AES-128-CTR+HMAC-SHA256 encrypted sleeve for the Redis protocol with an attendant Redis::Connection driver, but this is exactly the kind of thing I'm always telling people not to do.
- espeed 15y agoIt's not ip access control - it's hypervisor control.
- tptacek 15y agoI'm a lot more worried about allowing arbitrary EC2 hosts to connect to my Redis port than I am about people sniffing the traffic.
- imbriaco 15y agoThis motivated me to play around with Net::SSH port forwarding this afternoon and evening. What I ended up with was a tiny patch that allows you to forward a local UNIX domain socket to a remote TCP port over an SSH connection. It's a fun hack if nothing else: https://gist.github.com/1144486 https://gist.github.com/1144486 https://github.com/imbriaco/net-ssh/commit/aa453511f546cd524d2495f55429b2d168efd7d1 https://github.com/imbriaco/net-ssh/commit/aa453511f546cd524...
- imbriaco 15y agoClosing the loop and replying to myself, I got carried away and wound up with a full Sinatra demo app that does this: https://github.com/imbriaco/heroku-redis-sshtunnel https://github.com/imbriaco/heroku-redis-sshtunnel
- deleted 15y ago[deleted]
- nivertech 15y ago- Redis is a DSL for persistent data structures and a network server, which exposes it. - ETS (Erlang Term Store) is an optimized native in-memory lookup table [1]. - DETS (Disk ETS) is the same as ETS but with disk persistency. There is some limitations on the size of database. - Mnesia is a distributed/replicated STM built on top of ETS/DETS. You can think about ETS/DETS as a lightweight Redis without network interface. ETS/DETS support much less data structures, than Redis, i.e. only sets, ordered sets, bags, duplicate bags. Redis supports strings, lists, sets, sorted sets, hashes, etc. Redis is much more featured, than ETS/Mnesia. Both have different scalability profiles. Using Mnesia in distributed cluster effectively places a limit on the number of nodes in cluster (tens). Clustering support for Redis is not yet released. Most production installatons using Master-Slave replication and/or sharding. On raw single node request/s metric Redis is faster than Mnesia. Redis is single-threaded, while Mnesia can utilize SMP and clusters. I would like to see Redis stripped of TCP network interface and integrated in the same way as ETS in Erlang. [1] http://erldocs.com/R14B03/stdlib/ets.html?i=0&search=ets:new#new/2 http://erldocs.com/R14B03/stdlib/ets.html?i=0&search=ets...
- charlesju 15y agoA little bit off topic, how did this individual create the presentation in HTML5?