6 ms·

How We Made GitHub Fast: A detailed look at GitHub's new architecture

214 points by mojombo 17y ago

wallflower 17y ago
> For our data serialization and RPC protocol we are using BERT and BERT-RPC. You haven’t heard of them before because they’re brand new. I invented them because I was not satisfied with any of the available options that I evaluated, and I wanted to experiment with an idea that I’ve had for a while. > As much as I want to like Thrift, I just can’t. I find the entire concept behind IDLs and code generation abhorrent. No (CORBA) skeletons in his closet. Most importantly, his non-requirement: > No need to encode binary data http://github.com/blog/531-introducing-bert-and-bert-rpc http://github.com/blog/531-introducing-bert-and-bert-rpc
- hassy 17y ago
  BERT supports binary data natively, you don't need to encode it like with JSON-RPC.
  - wallflower 17y ago
    Thanks for the clarification
- timf 17y ago
  I wonder if they evaluated AMQP...
  - vidarh 17y ago
    His first requirement is "extreme simplicity". That alone would disqualify AMQP. Just seeing the size of the AMQP spec makes me want to cry. I'm sure it's the right choice for some things, but most apps that need RPC or message passing don't need anything remotely as complex.
    - timf 17y ago
      Rather than the spec (for message broker implementers, etc) I'd care more about the programming interface, toolchains, and features/performance. AMQP can be simple to use. And you can certainly send "simple" messages with it, they just get there reliably. The author asks us not to scream "NIH" so I will stop there - just did not see it as a potential tech examined and rejected in the BERT doc.
      - jacobolus 17y ago
        Building a simpler protocol (BERT-RPC or something with similar semantics) on top of a more complex one (AMQP, designed for complex pub-sub use cases) is sort-of backwards. It means you need to duplicate functionality that belongs at the lower layer, and drag along all kinds of extra baggage. Instead, a message queue should ideally be built on top of a simpler RPC/message-passing protocol. One thing I dislike about AMQP and similar protocols is that they put too many aspects of the communication into one layer, making it harder to re-use portions of them for other purposes. I don't know if you've ever looked at the AMQP spec, but it's kind of a monster (a 280 page PDF file).
        timf 17y ago
        > duplicate functionality that belongs at the lower layer, and drag along all kinds of extra baggage I'm just talking about sending messages not any kind of new protocol on top, for example http://code.google.com/p/py-amqplib/source/browse/demo/demo_send.py http://code.google.com/p/py-amqplib/source/browse/demo/demo_... There's of course some overhead vs. a simple BERT-RPC to set up a broker (you can set up simple ones fast...), but that should be weighed against the gains you get. And note that my curiosity/confusion also comes about because we are talking about a technology with a reliable software stack and money behind it (for example, RabbitMQ written in Erlang) vs. building a protocol from scratch. The situation is different if you are looking at equally mature technologies.
      - vidarh 17y ago
        The problem is that this breaks down the moment your needs aren't met by one of the existing message brokers, and when that happens it really sucks to be tied to a massive protocol. A reliable message broker is easy to write when your needs are simple. You can write one in a few hundred lines of code. If your needs are simple enough that they can be met by a simple protocol that can be implemented that easily, picking one means you know it's trivial to yank out your broker when it doesn't fit your needs, or you may find it easier to just write one targeting your specific needs from the outset. As I said, I'm sure AMQP has it's uses. But personally the places where I've used message brokers the needs have been simple enough that it was a toss up whether it would be more work to write a custom broker vs. configuring and working around things that wasn't a perfect fit for us with existing brokers. My first Ruby project was a message broker, actually, and it was pretty much done in a day.
nuclear_eclipse 17y ago
Off Topic: I miss your giant RSS icon... :(
dylanz 17y ago
Awesome post Tom. Smaller deployments are pretty straight-forward, and complex deployments are still pretty cookie-cutter. This is a breath of fresh air, as it uncovers a lot of the pain points you were faced with, and the not-so-common solutions. Thank you very much for sharing!
timf 17y ago
> We have patched our SSH daemon to perform public key lookups from our MySQL database That seems strange, that could be a PAM module at the least. If you patch SSHd then you are burdened with keeping up with changes, etc. There's even a module for direct mysql: http://sourceforge.net/projects/pam-mysql/ http://sourceforge.net/projects/pam-mysql/
- antonovka 17y ago
  It's a phenomenally bizarre way of solving the problem. Not only do you have to maintain patches to sshd (which increases maintenance costs and makes deploying security updates considerably more time consuming), but you also risk introducing security bugs in sshd itself. There are already plenty of external solutions to this problem that do not involve patching vendor-shipped software, and writing your own is not difficult. Here's one -- http://code.google.com/p/splatd/ http://code.google.com/p/splatd/ Will synchronize public keys from LDAP to local hosts, automatically create home directories, delete user's home directories when their accounts are deactivated after a grace period, etc -- and you don't have to patch sshd.
- noste 17y ago
  I don't think you can use PAM for authentication if you want to use public key authentication (see auth2-pubkey.c in Portable OpenSSH).
  - timf 17y ago
    Thanks, sorry for the confusion. So what they needed but did not have is an authorization (not authentication) callout after the daemon has verified the remote user's identity (vs. the built in 'callout' of looking at a user's authorized_keys file).
    - noste 17y ago
      Hmm, I think this part is still about authentication as sshd cannot authenticate the user without the keys. According to the article, GitHub does the authorization in their Gerve script.
omouse 17y ago
No diagram? :(
zikzikzik 17y ago
Why do you use DRBD instead of the built-in mysql replication?
rabbitmq 17y ago
Hi, alexis here from RabbitMQ. Yes, we implement AMQP. We also provide support for other useful things like STOMP and HTTP Pubsubhubbub. We implement these other protocols as well as AMQP because some times people don't need to use the full and awesome power of the AMQP model. AMQP is initially hard to grok. I think the main reason for this is that AMQP combines three things: Queues, Pubsub, and Messaging. These are Not The Same. Queues manage data in flight as state, Pubsub routes data to consumers, and Messaging frames it. So yes, as someone pointed out above, it would be nice to use some but not all of this from time to time. We are working on ways to make that super easy - please get in touch if you can help. Another thing that people find hard is figuring out when to use message hub technology, and when to use a database as a hub. Using a database to queue and manage subscriptions to data streams is generally Not A Good Idea. Here’s a presentation I did which attempts to articulate some of the issues with this: http://www.rabbitmq.com/resources/RabbitMQ_Oxford_Geek_Night.pdf http://www.rabbitmq.com/resources/RabbitMQ_Oxford_Geek_Night... So, for someone using AMQP or any other Pubsub tech for the first time, there can be a 'huh, where do I start' element. But as some commenters point out, if you look at the client libraries it may be easier to get started. We've actually lost count of how many clients there are, so take your pick. List of clients: http://delicious.com/alexisrichardson/rabbitmq+client http://delicious.com/alexisrichardson/rabbitmq+client Getting started: http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ (Python centric) and http://www.infoq.com/articles/AMQP-RabbitMQ http://www.infoq.com/articles/AMQP-RabbitMQ (Ruby centric) To the commenter who said the AMQP spec is 300 pages long. You may have a better time if you look at AMQP 0-91 which is much shorter than that at 40 pages mostly covering edge cases that you can ignore. The nub of AMQP can be communicated in under a page. BERT and BERT-RPC look cool. But - re the comments above - I would not see BERT-RPC as an ‘alternative’ to AMQP though. The GitHub blog post talks about PB and Thrift and JSON-RPC, all of which have been integrated with RabbitMQ. If you want to do RPC, there is no ‘one true system’ yet. Typically we have found that different people get value from different RPC metaphors in different languages. Maybe BERT-RPC will get more traction than the others - it definitely looks interesting. I hope this is all useful or at least of passing interest. Here are some more links that may be worth a glance: General background: http://www.rabbitmq.com/how.html http://www.rabbitmq.com/how.html AMQP and XMPP: http://www.igvita.com/2009/10/08/advanced-messaging-routing-with-amqp/ http://www.igvita.com/2009/10/08/advanced-messaging-routing-... Feel free to contact us directly at info at rabbitmq dot com. Cheers, alexis
brown9-2 17y ago
Ironically, github.com is now down: GitHub is Temporarily Offline. Either we're getting more requests right now than we can handle or you found a page that took too long to render.

214 points by mojombo 17y ago

> For our data serialization and RPC protocol we are using BERT and BERT-RPC. You haven’t heard of them before because they’re brand new. I invented them because I was not satisfied with any of the available options that I evaluated, and I wanted to experiment with an idea that I’ve had for a while. > As much as I want to like Thrift, I just can’t. I find the entire concept behind IDLs and code generation abhorrent. No (CORBA) skeletons in his closet. Most importantly, his non-requirement: > No need to encode binary data http://github.com/blog/531-introducing-bert-and-bert-rpc http://github.com/blog/531-introducing-bert-and-bert-rpc

hassy 17y ago
BERT supports binary data natively, you don't need to encode it like with JSON-RPC.
- wallflower 17y ago
  Thanks for the clarification
timf 17y ago
I wonder if they evaluated AMQP...
- vidarh 17y ago
  His first requirement is "extreme simplicity". That alone would disqualify AMQP. Just seeing the size of the AMQP spec makes me want to cry. I'm sure it's the right choice for some things, but most apps that need RPC or message passing don't need anything remotely as complex.
  - timf 17y ago
    Rather than the spec (for message broker implementers, etc) I'd care more about the programming interface, toolchains, and features/performance. AMQP can be simple to use. And you can certainly send "simple" messages with it, they just get there reliably. The author asks us not to scream "NIH" so I will stop there - just did not see it as a potential tech examined and rejected in the BERT doc.
    - jacobolus 17y ago
      Building a simpler protocol (BERT-RPC or something with similar semantics) on top of a more complex one (AMQP, designed for complex pub-sub use cases) is sort-of backwards. It means you need to duplicate functionality that belongs at the lower layer, and drag along all kinds of extra baggage. Instead, a message queue should ideally be built on top of a simpler RPC/message-passing protocol. One thing I dislike about AMQP and similar protocols is that they put too many aspects of the communication into one layer, making it harder to re-use portions of them for other purposes. I don't know if you've ever looked at the AMQP spec, but it's kind of a monster (a 280 page PDF file).
      - timf 17y ago
        > duplicate functionality that belongs at the lower layer, and drag along all kinds of extra baggage I'm just talking about sending messages not any kind of new protocol on top, for example http://code.google.com/p/py-amqplib/source/browse/demo/demo_send.py http://code.google.com/p/py-amqplib/source/browse/demo/demo_... There's of course some overhead vs. a simple BERT-RPC to set up a broker (you can set up simple ones fast...), but that should be weighed against the gains you get. And note that my curiosity/confusion also comes about because we are talking about a technology with a reliable software stack and money behind it (for example, RabbitMQ written in Erlang) vs. building a protocol from scratch. The situation is different if you are looking at equally mature technologies.
    - vidarh 17y ago
      The problem is that this breaks down the moment your needs aren't met by one of the existing message brokers, and when that happens it really sucks to be tied to a massive protocol. A reliable message broker is easy to write when your needs are simple. You can write one in a few hundred lines of code. If your needs are simple enough that they can be met by a simple protocol that can be implemented that easily, picking one means you know it's trivial to yank out your broker when it doesn't fit your needs, or you may find it easier to just write one targeting your specific needs from the outset. As I said, I'm sure AMQP has it's uses. But personally the places where I've used message brokers the needs have been simple enough that it was a toss up whether it would be more work to write a custom broker vs. configuring and working around things that wasn't a perfect fit for us with existing brokers. My first Ruby project was a message broker, actually, and it was pretty much done in a day.

nuclear_eclipse 17y ago

Off Topic: I miss your giant RSS icon... :(

dylanz 17y ago

Awesome post Tom. Smaller deployments are pretty straight-forward, and complex deployments are still pretty cookie-cutter. This is a breath of fresh air, as it uncovers a lot of the pain points you were faced with, and the not-so-common solutions. Thank you very much for sharing!

timf 17y ago

> We have patched our SSH daemon to perform public key lookups from our MySQL database That seems strange, that could be a PAM module at the least. If you patch SSHd then you are burdened with keeping up with changes, etc. There's even a module for direct mysql: http://sourceforge.net/projects/pam-mysql/ http://sourceforge.net/projects/pam-mysql/

antonovka 17y ago
It's a phenomenally bizarre way of solving the problem. Not only do you have to maintain patches to sshd (which increases maintenance costs and makes deploying security updates considerably more time consuming), but you also risk introducing security bugs in sshd itself. There are already plenty of external solutions to this problem that do not involve patching vendor-shipped software, and writing your own is not difficult. Here's one -- http://code.google.com/p/splatd/ http://code.google.com/p/splatd/ Will synchronize public keys from LDAP to local hosts, automatically create home directories, delete user's home directories when their accounts are deactivated after a grace period, etc -- and you don't have to patch sshd.
noste 17y ago
I don't think you can use PAM for authentication if you want to use public key authentication (see auth2-pubkey.c in Portable OpenSSH).
- timf 17y ago
  Thanks, sorry for the confusion. So what they needed but did not have is an authorization (not authentication) callout after the daemon has verified the remote user's identity (vs. the built in 'callout' of looking at a user's authorized_keys file).
  - noste 17y ago
    Hmm, I think this part is still about authentication as sshd cannot authenticate the user without the keys. According to the article, GitHub does the authorization in their Gerve script.

omouse 17y ago

No diagram? :(

zikzikzik 17y ago

Why do you use DRBD instead of the built-in mysql replication?

rabbitmq 17y ago

Hi, alexis here from RabbitMQ. Yes, we implement AMQP. We also provide support for other useful things like STOMP and HTTP Pubsubhubbub. We implement these other protocols as well as AMQP because some times people don't need to use the full and awesome power of the AMQP model. AMQP is initially hard to grok. I think the main reason for this is that AMQP combines three things: Queues, Pubsub, and Messaging. These are Not The Same. Queues manage data in flight as state, Pubsub routes data to consumers, and Messaging frames it. So yes, as someone pointed out above, it would be nice to use some but not all of this from time to time. We are working on ways to make that super easy - please get in touch if you can help. Another thing that people find hard is figuring out when to use message hub technology, and when to use a database as a hub. Using a database to queue and manage subscriptions to data streams is generally Not A Good Idea. Here’s a presentation I did which attempts to articulate some of the issues with this: http://www.rabbitmq.com/resources/RabbitMQ_Oxford_Geek_Night.pdf http://www.rabbitmq.com/resources/RabbitMQ_Oxford_Geek_Night... So, for someone using AMQP or any other Pubsub tech for the first time, there can be a 'huh, where do I start' element. But as some commenters point out, if you look at the client libraries it may be easier to get started. We've actually lost count of how many clients there are, so take your pick. List of clients: http://delicious.com/alexisrichardson/rabbitmq+client http://delicious.com/alexisrichardson/rabbitmq+client Getting started: http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ (Python centric) and http://www.infoq.com/articles/AMQP-RabbitMQ http://www.infoq.com/articles/AMQP-RabbitMQ (Ruby centric) To the commenter who said the AMQP spec is 300 pages long. You may have a better time if you look at AMQP 0-91 which is much shorter than that at 40 pages mostly covering edge cases that you can ignore. The nub of AMQP can be communicated in under a page. BERT and BERT-RPC look cool. But - re the comments above - I would not see BERT-RPC as an ‘alternative’ to AMQP though. The GitHub blog post talks about PB and Thrift and JSON-RPC, all of which have been integrated with RabbitMQ. If you want to do RPC, there is no ‘one true system’ yet. Typically we have found that different people get value from different RPC metaphors in different languages. Maybe BERT-RPC will get more traction than the others - it definitely looks interesting. I hope this is all useful or at least of passing interest. Here are some more links that may be worth a glance: General background: http://www.rabbitmq.com/how.html http://www.rabbitmq.com/how.html AMQP and XMPP: http://www.igvita.com/2009/10/08/advanced-messaging-routing-with-amqp/ http://www.igvita.com/2009/10/08/advanced-messaging-routing-... Feel free to contact us directly at info at rabbitmq dot com. Cheers, alexis

brown9-2 17y ago

Ironically, github.com is now down: GitHub is Temporarily Offline. Either we're getting more requests right now than we can handle or you found a page that took too long to render.