3 ms·

It seems that the only way to scale a site like this is to build decoupled services maybe using a messaging queue (AMQP, etc..). How can I learn more about thi

It seems that the only way to scale a site like this is to build decoupled services maybe using a messaging queue (AMQP, etc..). How can I learn more about this kind of systems? Also is there a open source application[s] built in a distributed way that I could learn from?

runjake 15y ago
I hate to point out the obvious answer, but it's a good introduction with links to several implementations: http://en.wikipedia.org/wiki/AMQP http://en.wikipedia.org/wiki/AMQP
- famoreira 15y ago
  I meant some open source applications that are built in a distributed way (hence using something like AMQP in a publish/subscribe fashion)
  - rabidsnail 15y ago
    I don't really know of any. Nobody writes these sort of distributed applications from scratch. They're almost always evolved from monolithic database-backed applications. And the process of evolution is half development half ops. I doubt that tumblr as it exists today could be deployed from scratch, even if you had all of the source code of all of the systems.
    - rabidsnail 15y ago
      However, if you want to go truly distributed (much more so than any website I know of) there are open source projects you can learn from: http://freenetproject.org/ http://freenetproject.org/ http://amule.org http://amule.org (or http://mldonkey.sourceforge.net/Main_Page http://mldonkey.sourceforge.net/Main_Page if you're into functional programming) http://www.torproject.org http://www.torproject.org (especially the hidden services stuff) And you probably want to read this: http://pmg.csail.mit.edu/papers/osdi99.pdf http://pmg.csail.mit.edu/papers/osdi99.pdf
encoderer 15y ago
Exactly. If you don't know what Thrift is, check it out. In addition to solving the problem space using simple, focused applications (in the classical Unix style), it solves other practical issues. Like PHP having shitty clients for most modern data stores (Cassandra, Redis, hell, even Memcached) compared to what you get in, say, Python, or a JVM language.
stock_toaster 15y ago
mongrel2[1] (uses Ømq[2]) maybe? [1]: http://mongrel2.org/ http://mongrel2.org/ [2]: http://www.zeromq.org/ http://www.zeromq.org/
gizzlon 15y ago
Gearman, briefly mentioned in the article, can probably help you as well. Let's you do cross language, cross platform (a)synchronous function calls . Looks cool, never used it.. http://gearman.org/ http://gearman.org/ http://gearman.org/index.php#how_does_gearman_work http://gearman.org/index.php#how_does_gearman_work

runjake 15y ago

I hate to point out the obvious answer, but it's a good introduction with links to several implementations: http://en.wikipedia.org/wiki/AMQP http://en.wikipedia.org/wiki/AMQP

famoreira 15y ago
I meant some open source applications that are built in a distributed way (hence using something like AMQP in a publish/subscribe fashion)
- rabidsnail 15y ago
  I don't really know of any. Nobody writes these sort of distributed applications from scratch. They're almost always evolved from monolithic database-backed applications. And the process of evolution is half development half ops. I doubt that tumblr as it exists today could be deployed from scratch, even if you had all of the source code of all of the systems.
  - rabidsnail 15y ago
    However, if you want to go truly distributed (much more so than any website I know of) there are open source projects you can learn from: http://freenetproject.org/ http://freenetproject.org/ http://amule.org http://amule.org (or http://mldonkey.sourceforge.net/Main_Page http://mldonkey.sourceforge.net/Main_Page if you're into functional programming) http://www.torproject.org http://www.torproject.org (especially the hidden services stuff) And you probably want to read this: http://pmg.csail.mit.edu/papers/osdi99.pdf http://pmg.csail.mit.edu/papers/osdi99.pdf

encoderer 15y ago

Exactly. If you don't know what Thrift is, check it out. In addition to solving the problem space using simple, focused applications (in the classical Unix style), it solves other practical issues. Like PHP having shitty clients for most modern data stores (Cassandra, Redis, hell, even Memcached) compared to what you get in, say, Python, or a JVM language.

stock_toaster 15y ago

mongrel2[1] (uses Ømq[2]) maybe? [1]: http://mongrel2.org/ http://mongrel2.org/ [2]: http://www.zeromq.org/ http://www.zeromq.org/

gizzlon 15y ago

Gearman, briefly mentioned in the article, can probably help you as well. Let's you do cross language, cross platform (a)synchronous function calls . Looks cool, never used it.. http://gearman.org/ http://gearman.org/ http://gearman.org/index.php#how_does_gearman_work http://gearman.org/index.php#how_does_gearman_work