9 ms·

Google Doorman: Global Distributed Client Side Rate Limiting

73 points by ryancox 10y ago

dedalus 10y ago
It should have been called "Velvet Rope" :-)
- cm3 10y ago
  Care to you explain the joke for the culturally unaware?
  - CaveTech 10y ago
    When waiting to get into a club or party, you wait behind such a rope until the bouncer allows you to enter (or doesn't).
    - deleted 10y ago
      [deleted]
mixonic 10y ago
Hopefully there is a Googler reading who can answer this: I'm curious why the README.md says > Note: This is not an official Google product. However the copyright is Google, and you must sign their CLA. That seems pretty official to me? Or is there some other implication to "official" beyond ownership?
- dekhn 10y ago
  It works like this. Google has an "open source office". When you open source code at Google, it's either a product, or it isn't. Even if it's not a product (which has its own extra set of launch procedures), it's possible to open source it. It's still a Google-owned thing, but not a "product". Google ownership does not necessarily mean it's a product, but it still requires the copyright stamp and the CLA. If it's easier, just drop the word "official". It's not a Google product- IE, a saleable or free service that google provides to customers- but rather just a thing Google put out in the world, subject to the rules that Google applies.
- yegle 10y ago
  This is a project whose author is a Googler thus Google has the copyright and all the CLA stuff. However Google is not directly involved in the project, in the sense of having employees solely work on the project and have an OKR on this project, or roadmap whatsoever. Think about this: someone work for Google wrote a software and Google is kind enough to open source it. Disclaimer: I work for Google but my interpretation could be wrong.
  - brunoqc 10y ago
    Does Google also own code that you write in your spare time at home?
    - batbomb 10y ago
      Usually (close to) universal answer is something like is: "If you've used any employing entity resources, even incidentally, your employing entity owns it, especially if the project is directly related to the business" Some places will attempt to assert more ownership and some will not.
    - rixed 10y ago
      They pretend to, as would many companies if asked. I don't know how that would hold in court though. They also, for instance, pretend it's illegal to discuss your salary while the opposite seams to be true, so...
- wereHamster 10y ago
  That is because Google claims copyright on everything you create while you work there, even if you create it in your garage in your free time. You baked some cookies for your daughters birthday party? Sorry, can't distribute them for free, consumers of those must sign a CLA (Consumer License Agreement) and Google retains copyright on your recipe.
  - kuschku 10y ago
    Question: Does this also apply to Google employees at Google Hamburg, and, if yes, how far? Because, as far as I know, German law directly prohibits such contracts.
    - schwa571 10y ago
      I can't answer your question, because I don't know the answer. However, I would recommend against relying on wereHamster's apparent mis-information... unless Google's policies are different in the jurisdiction that wereHamster is familiar with, in which case I apologize.
      - kuschku 10y ago
        I think wereHamster’s text was satire, but the idea was repeated by many others: Anything related to your job you do is owned by Google. You can’t contribute to ejabberd when you work on Hangouts.
        wereHamster 10y ago
        What does it tell us if people don't recognise that as satire? Is it so close to reality to think that Google could claim rights on our cookie recipes? And if you really have to ask if such a clause applies to you, you haven't properly read your contract and/or you don't know your rights. People, please educate yourselves, don't let large corporations, even if they claim not to be evil, violate your rights.
        10y ago
wslh 10y ago
Offtopic but related: I want to like GRPC but I think Google is not working hard to make it quickly usable. Just try to perform a pip install in Visual Studio and obviously... it will not work. This is not the first time it happens, try to compile V8 (NodeJS is easy) or Chrome in Windows and it will be difficult even following the step by step instructions.
- secure 10y ago
  I have the opposite experience: The Go version of gRPC works like a charm, following the official instructions.
harryf 10y ago
So do I understand this right: Doorman could be used to rate-limit access to a website by integrating it with the sites proxy server, the result being individual clients would only get a certain number of requests per minute based on available server side resources?
- tonfa 10y ago
  I think it's more meant for internal RPC within a cluster, to e.g. avoid overloading a service.
  - stingraycharles 10y ago
    This is the correct answer, since this exact problem is a major issue when dealing with distributed system failovers. Here are a few examples of AWS service disruptions due to exactly this problem, where they were unable to recover the system because nodes kept failing over under load: https://aws.amazon.com/message/5467D2/ https://aws.amazon.com/message/5467D2/ https://aws.amazon.com/message/2329B7/ https://aws.amazon.com/message/2329B7/ http://aws.amazon.com/message/65648/ http://aws.amazon.com/message/65648/ An example quote: "When this network connectivity issue occurred, a large number of EBS nodes in a single EBS cluster lost connection to their replicas. When the incorrect traffic shift was rolled back and network connectivity was restored, these nodes rapidly began searching the EBS cluster for available server space where they could re-mirror data. Once again, in a normally functioning cluster, this occurs in milliseconds. In this case, because the issue affected such a large number of volumes concurrently, the free capacity of the EBS cluster was quickly exhausted, leaving many of the nodes “stuck” in a loop, continuously searching the cluster for free space. This quickly led to a “re-mirroring storm,” where a large number of volumes were effectively “stuck” while the nodes searched the cluster for the storage space it needed for its new replica. At this point, about 13% of the volumes in the affected Availability Zone were in this “stuck” state." So these things are very hard, can occur in totally unexpected situations, and I'm not at all surprised that a company like Google comes out with something like this.
nickpsecurity 10y ago
I know it expects cooperative behavior. However, I wonder if the protocol design could be used in a setup where an embedded firewall did the rate limiting and mediated traffic from the possibly-malicious host. Boeing already has rate-limiting in their embedded firewall but cheap or OSS project could use a OSS rate limiter prebuilt to save time. As in, is this protocol inherently cooperative or could an implementation have checks/controls added?
rixed 10y ago
Rate limiting looks outdated. Suppose you can spawn new servers (and turn down unused ones) fast enough to react to demand, and have a good LB that sends the traffic where you have capacity, then you do not need to limit anything until you reach full capacity, in which case you probably want to degrade (by query class and/or client class) rather than limit anyway. Rate limiting is an inefficient way to distribute a service, that makes sense only if you preallocate your resources and have queries with predictable cost. Let's use this technique only as a bug-prevention tool not for resource economy, as organised scarcity is likely as inefficient for the data center as it is for the distribution of goods :)
- greenleafjacob 10y ago
  If you degrade by client or query class then in the case of just one class of query or one class of client you would have an unfair distribution, because one client could consume many resources and others would be limited to whatever's available on the margins. I would like to see an adaptive system that, when resources are scarce, pushes towards a more equitable distribution.

73 points by ryancox 10y ago

dedalus 10y ago

It should have been called "Velvet Rope" :-)

cm3 10y ago
Care to you explain the joke for the culturally unaware?
- CaveTech 10y ago
  When waiting to get into a club or party, you wait behind such a rope until the bouncer allows you to enter (or doesn't).
  - deleted 10y ago
    [deleted]

mixonic 10y ago

Hopefully there is a Googler reading who can answer this: I'm curious why the README.md says > Note: This is not an official Google product. However the copyright is Google, and you must sign their CLA. That seems pretty official to me? Or is there some other implication to "official" beyond ownership?

dekhn 10y ago
It works like this. Google has an "open source office". When you open source code at Google, it's either a product, or it isn't. Even if it's not a product (which has its own extra set of launch procedures), it's possible to open source it. It's still a Google-owned thing, but not a "product". Google ownership does not necessarily mean it's a product, but it still requires the copyright stamp and the CLA. If it's easier, just drop the word "official". It's not a Google product- IE, a saleable or free service that google provides to customers- but rather just a thing Google put out in the world, subject to the rules that Google applies.
yegle 10y ago
This is a project whose author is a Googler thus Google has the copyright and all the CLA stuff. However Google is not directly involved in the project, in the sense of having employees solely work on the project and have an OKR on this project, or roadmap whatsoever. Think about this: someone work for Google wrote a software and Google is kind enough to open source it. Disclaimer: I work for Google but my interpretation could be wrong.
- brunoqc 10y ago
  Does Google also own code that you write in your spare time at home?
  - batbomb 10y ago
    Usually (close to) universal answer is something like is: "If you've used any employing entity resources, even incidentally, your employing entity owns it, especially if the project is directly related to the business" Some places will attempt to assert more ownership and some will not.
  - rixed 10y ago
    They pretend to, as would many companies if asked. I don't know how that would hold in court though. They also, for instance, pretend it's illegal to discuss your salary while the opposite seams to be true, so...
wereHamster 10y ago
That is because Google claims copyright on everything you create while you work there, even if you create it in your garage in your free time. You baked some cookies for your daughters birthday party? Sorry, can't distribute them for free, consumers of those must sign a CLA (Consumer License Agreement) and Google retains copyright on your recipe.
- kuschku 10y ago
  Question: Does this also apply to Google employees at Google Hamburg, and, if yes, how far? Because, as far as I know, German law directly prohibits such contracts.
  - schwa571 10y ago
    I can't answer your question, because I don't know the answer. However, I would recommend against relying on wereHamster's apparent mis-information... unless Google's policies are different in the jurisdiction that wereHamster is familiar with, in which case I apologize.
    - kuschku 10y ago
      I think wereHamster’s text was satire, but the idea was repeated by many others: Anything related to your job you do is owned by Google. You can’t contribute to ejabberd when you work on Hangouts.
      - wereHamster 10y ago
        What does it tell us if people don't recognise that as satire? Is it so close to reality to think that Google could claim rights on our cookie recipes? And if you really have to ask if such a clause applies to you, you haven't properly read your contract and/or you don't know your rights. People, please educate yourselves, don't let large corporations, even if they claim not to be evil, violate your rights.
        10y ago

wslh 10y ago

Offtopic but related: I want to like GRPC but I think Google is not working hard to make it quickly usable. Just try to perform a pip install in Visual Studio and obviously... it will not work. This is not the first time it happens, try to compile V8 (NodeJS is easy) or Chrome in Windows and it will be difficult even following the step by step instructions.

secure 10y ago
I have the opposite experience: The Go version of gRPC works like a charm, following the official instructions.

harryf 10y ago

So do I understand this right: Doorman could be used to rate-limit access to a website by integrating it with the sites proxy server, the result being individual clients would only get a certain number of requests per minute based on available server side resources?

tonfa 10y ago
I think it's more meant for internal RPC within a cluster, to e.g. avoid overloading a service.
- stingraycharles 10y ago
  This is the correct answer, since this exact problem is a major issue when dealing with distributed system failovers. Here are a few examples of AWS service disruptions due to exactly this problem, where they were unable to recover the system because nodes kept failing over under load: https://aws.amazon.com/message/5467D2/ https://aws.amazon.com/message/5467D2/ https://aws.amazon.com/message/2329B7/ https://aws.amazon.com/message/2329B7/ http://aws.amazon.com/message/65648/ http://aws.amazon.com/message/65648/ An example quote: "When this network connectivity issue occurred, a large number of EBS nodes in a single EBS cluster lost connection to their replicas. When the incorrect traffic shift was rolled back and network connectivity was restored, these nodes rapidly began searching the EBS cluster for available server space where they could re-mirror data. Once again, in a normally functioning cluster, this occurs in milliseconds. In this case, because the issue affected such a large number of volumes concurrently, the free capacity of the EBS cluster was quickly exhausted, leaving many of the nodes “stuck” in a loop, continuously searching the cluster for free space. This quickly led to a “re-mirroring storm,” where a large number of volumes were effectively “stuck” while the nodes searched the cluster for the storage space it needed for its new replica. At this point, about 13% of the volumes in the affected Availability Zone were in this “stuck” state." So these things are very hard, can occur in totally unexpected situations, and I'm not at all surprised that a company like Google comes out with something like this.

nickpsecurity 10y ago

I know it expects cooperative behavior. However, I wonder if the protocol design could be used in a setup where an embedded firewall did the rate limiting and mediated traffic from the possibly-malicious host. Boeing already has rate-limiting in their embedded firewall but cheap or OSS project could use a OSS rate limiter prebuilt to save time. As in, is this protocol inherently cooperative or could an implementation have checks/controls added?

rixed 10y ago

Rate limiting looks outdated. Suppose you can spawn new servers (and turn down unused ones) fast enough to react to demand, and have a good LB that sends the traffic where you have capacity, then you do not need to limit anything until you reach full capacity, in which case you probably want to degrade (by query class and/or client class) rather than limit anyway. Rate limiting is an inefficient way to distribute a service, that makes sense only if you preallocate your resources and have queries with predictable cost. Let's use this technique only as a bug-prevention tool not for resource economy, as organised scarcity is likely as inefficient for the data center as it is for the distribution of goods :)

greenleafjacob 10y ago
If you degrade by client or query class then in the case of just one class of query or one class of client you would have an unfair distribution, because one client could consume many resources and others would be limited to whatever's available on the margins. I would like to see an adaptive system that, when resources are scarce, pushes towards a more equitable distribution.