4 ms·

Twemproxy, a Redis/memcached proxy from Twitter with auto-sharding

90 points by mjackson 14y ago

jaytaylor 14y ago
Direct link to github repository: https://github.com/twitter/twemproxy https://github.com/twitter/twemproxy As a sidenote, look at the amount of shenanigans of complexity and redirects in the "github repository link" contained in the article: http://links.services.disqus.com/api/click? format=go& key=cfdfcf52dffd0a702a61bad27507376d& loc=http%3A%2F%2Fantirez.com%2Fnews%2F44& subId=804356& v=1& libid=1354646989332& out=https%3A%2F%2Fgithub.com%2Ftwitter%2Ftwemproxy& ref=http%3A%2F%2Fnews.ycombinator.com%2Fnews& title=Twemproxy%2C%20a%20Redis%20proxy%20from%20Twitter%20-%20Antirez%20weblog& txt=https%3A%2F%2Fgithub.com%2Ftwitter%2Ftwemproxy& jsonp=vglnk_jsonp_13546470034491 HOLY MOLY!
- janerik 14y ago
  I see only one link to the github repo and it's a direct link.
stock_toaster 14y ago
Been using twemproxy with memcache for a while now. It has been great. Damn solid, quality stuff.
mjackson 14y ago
Related: The director of Twitter's Application Services group, Raffi Krikorian, gave an interesting talk about the architecture of Twitter's real-time delivery service at QCon in October. In the talk he discusses in pretty good detail the ways in which Twitter uses Redis to cache copies of users' timelines as tweets are made. Here's a link to a video of the talk: http://www.infoq.com/presentations/Real-Time-Delivery-Twitter http://www.infoq.com/presentations/Real-Time-Delivery-Twitte...
- famo 14y ago
  Very good presentation, thanks for the link
druiid 14y ago
Well, that's cool and I'm happy to see a library/tool like this being released. I do have an honest question though, in that is this giving much besides what moxi currently is able to do? I use that in a relatively large-scale memcached production environment and have been pretty happy with it. Guess perhaps I'll have to do some digging between the two.
- thezilch 14y ago
  For memcached, it doesn't look you'd get much in terms of features -- you'd have to benchmark for speed of the library. Worse, it would appear you would lose Multi-get Escalation, which could be substantial for a lot of workloads.
blantonl 14y ago
For a second here, I was thinking this was an open source implementation of a decentralized Redis/Memcached store that caches and proxies API queries to the Twitter API - thus possibly presenting an approach to aggregating Twitter API calls and reducing API limit problems for consumers. Sort of a "de-duplication" process for Twitter API calls. But, it's not.

90 points by mjackson 14y ago

Direct link to github repository: https://github.com/twitter/twemproxy https://github.com/twitter/twemproxy As a sidenote, look at the amount of shenanigans of complexity and redirects in the "github repository link" contained in the article: http://links.services.disqus.com/api/click? format=go& key=cfdfcf52dffd0a702a61bad27507376d& loc=http%3A%2F%2Fantirez.com%2Fnews%2F44& subId=804356& v=1& libid=1354646989332& out=https%3A%2F%2Fgithub.com%2Ftwitter%2Ftwemproxy& ref=http%3A%2F%2Fnews.ycombinator.com%2Fnews& title=Twemproxy%2C%20a%20Redis%20proxy%20from%20Twitter%20-%20Antirez%20weblog& txt=https%3A%2F%2Fgithub.com%2Ftwitter%2Ftwemproxy& jsonp=vglnk_jsonp_13546470034491 HOLY MOLY!

janerik 14y ago
I see only one link to the github repo and it's a direct link.

stock_toaster 14y ago

Been using twemproxy with memcache for a while now. It has been great. Damn solid, quality stuff.

mjackson 14y ago

Related: The director of Twitter's Application Services group, Raffi Krikorian, gave an interesting talk about the architecture of Twitter's real-time delivery service at QCon in October. In the talk he discusses in pretty good detail the ways in which Twitter uses Redis to cache copies of users' timelines as tweets are made. Here's a link to a video of the talk: http://www.infoq.com/presentations/Real-Time-Delivery-Twitter http://www.infoq.com/presentations/Real-Time-Delivery-Twitte...

famo 14y ago
Very good presentation, thanks for the link

druiid 14y ago

Well, that's cool and I'm happy to see a library/tool like this being released. I do have an honest question though, in that is this giving much besides what moxi currently is able to do? I use that in a relatively large-scale memcached production environment and have been pretty happy with it. Guess perhaps I'll have to do some digging between the two.

thezilch 14y ago
For memcached, it doesn't look you'd get much in terms of features -- you'd have to benchmark for speed of the library. Worse, it would appear you would lose Multi-get Escalation, which could be substantial for a lot of workloads.

blantonl 14y ago

For a second here, I was thinking this was an open source implementation of a decentralized Redis/Memcached store that caches and proxies API queries to the Twitter API - thus possibly presenting an approach to aggregating Twitter API calls and reducing API limit problems for consumers. Sort of a "de-duplication" process for Twitter API calls. But, it's not.