5 ms·

How did it even make sense in your head to compare an OLTP workload to a chat service? The proxy servers are there to terminate the large number of persistent

How did it even make sense in your head to compare an OLTP workload to a chat service? The proxy servers are there to terminate the large number of persistent connections. Of course it's possible to do it using less servers, but given the Hipchat guys are smart (disclosure - I'm an Atlassian and know the internals) I would give them the benefit of the doubt rather than engaging in armchair architecture.

taspeotis 13y ago
The point was to ask whether the 60 in 60 messages per second is wrong by comparison.
- mtrimpe 13y ago
  The frontend needs to keep open thousands of persistent connections; 99.9% of which will time out with receiving a single message. Practically speaking the number of frontend servers is related to the number of active listening clients and not messages. Only 60 messages per second does seem very small from a business perspective however.
  - taspeotis 13y ago
    If you take the frontend servers out of the equation it's still 52 servers for 60 messages per second.
    - msmithstubbs 13y ago
      My guess: the 60 messages per second is an average. Monday morning in the US timezones might see several multiples of that. Weekends could be a lot lower.
      - powdahound 13y ago
        It's definitely very bursty. On weekends and holidays things are much more quiet. During peak load we'll be in the hundreds/sec. Also keep in mind that chat messages don't actually make up the majority of the traffic we serve; it's presence information (away, idle, available), people connecting/disconnecting, etc. (I'm one of the HipChat co-founders.)
    - hox 13y ago
      And? The article makes no mention of what each server is responsible for, how much redundancy is built in (this is AWS where nodes die on a whim), whether all of these servers are used for production or if there is a spread across dev and staging environments, etc. When all of these factors are considered, 52 servers isn't a huge number.
- tveita 13y ago
  Looks about right from their graph: https://s3.amazonaws.com/uploads.hipchat.com/10804/368466/10joz8sztfw4dfc/HipChat1B.jpg https://s3.amazonaws.com/uploads.hipchat.com/10804/368466/10... That traffic is going to be highly bursty, though.

taspeotis 13y ago

The point was to ask whether the 60 in 60 messages per second is wrong by comparison.

mtrimpe 13y ago
The frontend needs to keep open thousands of persistent connections; 99.9% of which will time out with receiving a single message. Practically speaking the number of frontend servers is related to the number of active listening clients and not messages. Only 60 messages per second does seem very small from a business perspective however.
- taspeotis 13y ago
  If you take the frontend servers out of the equation it's still 52 servers for 60 messages per second.
  - msmithstubbs 13y ago
    My guess: the 60 messages per second is an average. Monday morning in the US timezones might see several multiples of that. Weekends could be a lot lower.
    - powdahound 13y ago
      It's definitely very bursty. On weekends and holidays things are much more quiet. During peak load we'll be in the hundreds/sec. Also keep in mind that chat messages don't actually make up the majority of the traffic we serve; it's presence information (away, idle, available), people connecting/disconnecting, etc. (I'm one of the HipChat co-founders.)
  - hox 13y ago
    And? The article makes no mention of what each server is responsible for, how much redundancy is built in (this is AWS where nodes die on a whim), whether all of these servers are used for production or if there is a spread across dev and staging environments, etc. When all of these factors are considered, 52 servers isn't a huge number.
tveita 13y ago
Looks about right from their graph: https://s3.amazonaws.com/uploads.hipchat.com/10804/368466/10joz8sztfw4dfc/HipChat1B.jpg https://s3.amazonaws.com/uploads.hipchat.com/10804/368466/10... That traffic is going to be highly bursty, though.