8 ms·
AWS Case Study: Parse (YC S11)
- gfodor 14y agoI realize the diagram is likely heavily over-simplified, but what's the reason for the nginx layer? It doesn't look like you guys are serving assets through those boxes, so it seems like you have an unnecessary extra hop from the ELB to the app servers. My guess is this is either nginx caching, a holdover from the pre-aws architecture or a measure to keep the system decoupled from ELB?
- trustfundbaby 14y ago> I realize the diagram is likely heavily over-simplified, but what's the reason for the nginx layer probably to load balance between unicorn instances in the particular amazon EC2 instance
- tcwc 14y agoI can't speak for Parse, but I've come up with something similar in the past. Nginx/HAproxy as a combo is far more flexible than the ELB alone, you might want to use it for rate limiting, better load balancing algorithms, better logging, tweaking headers, handling errors, or controlling buffer sizes for example.
- pjscott 14y agoOr for preventing DoS attacks from slow client connections. Unicorn is not designed to be exposed to the outside world; it needs a reverse proxy that does request buffering, like nginx (which the Unicorn docs recommend).
- tcwc 14y agoTrue, though the ELB also does some simple buffering of HTTP requests.
- spimmy 14y agoWe use nginx for serving static files and doing some trivial top-level routing. Haproxy is for load-balancing among families of unicorn app servers.
- trustfundbaby 14y agoWhy both redis and memcached?
- manuelflara 14y agoI can't speak for Parse, but I use both of these technologies in one site. Memcached is, well, for any kind data that you want to cache: results of SQL queries, blocks of the site (HTML), etc. Redis, on the other hand, it's a final data storage on its own (like MySQL, because it's also persistent), but it's faster than MySQL for certain things. For example, for things like "List of profiles I've visited / have visited me recently on Facebook". This seems like a very simple MySQL table and query, but once it becomes big it starts to get slow. On the other hand, you can have one key per user on Redis, like "<id_user>_visit_history" which is a sorted set of <time, id_user>. You could do the same with Memcached (although it doesn't support the same data types), but then it wouldn't be persistent.
- trustfundbaby 14y agoRight ... thats my point. redis is as fast if not faster than memcached for caching exactly the same things you talked about (results of sql queries, blocks of the site etc) so why not just use redis for everything and eliminate memcached.
- shykes 14y agoHere's one possible answer, for one specific use case: http://engineering.pinterest.com/posts/2012/memcache-games/ http://engineering.pinterest.com/posts/2012/memcache-games/
- spimmy 14y agoWe use redis because we use resque extensively for the push notifications architecture. Memcache is superior to redis for simple caching in front of data sources.
- AznHisoka 14y ago
- deleted 14y ago[deleted]
- tszming 14y agoWhy Squid behind the cloud code servers? Are you moving from MongoDB to Cassandra?
- lolz 14y agoCan you tell us more about RAID10 comment? Are you doing RAID0 now instead for disk volumes? What would happen if there's another EBS outage that would take out a lot of your MongoDBs. Would you just recreate them from the very latest S3 backups?
- jedberg 14y agoI'm really humbled to see the architecture that we came up with at reddit with help from Heroku and JustinTV, which I then taught to Amazon and AWS, is still the one AWS is highlighting today (and presumably teaching their customers). It's an amazing feeling to see your work live on beyond your involvement. * They flew me up to Seattle to teach Amazon retail how to move to AWS back when retail had just started that process. Amusingly, the guy mainly responsible for that move now works for Pinterest (a company that "does cloud right").
- voltagex_ 14y agoWas it a strange experience to teach a company how to use its own product?
- thibaut_barrere 14y agoThis happens fairly often: customers will use your product in unexpected ways and sometimes for much better output than you originally thought.
- jedberg 14y agoI was surprised when they asked, that's for sure. It was a little odd because I felt like there is no way I could teach these people anything they didn't already know, but I got really good questions and feedback so at the end I didn't feel nearly as weird about it.