5 ms·

Scaling HipChat Using ElasticSearch And Redis

53 points by pscsbs 13y ago

taspeotis 13y ago
Stats * 60 messages per second ... * 26 front end proxy serves. Double that in backend app servers. 78 servers for 60 messages per second? We have one MSSQL box at work here that is regularly doing 2,500 queries/sec and that's an OLTP workload so easily more than 60 writes/sec. serves makes me think that someone didn't proof-read the article...
- georgebarnett 13y ago
  How did it even make sense in your head to compare an OLTP workload to a chat service? The proxy servers are there to terminate the large number of persistent connections. Of course it's possible to do it using less servers, but given the Hipchat guys are smart (disclosure - I'm an Atlassian and know the internals) I would give them the benefit of the doubt rather than engaging in armchair architecture.
  - taspeotis 13y ago
    The point was to ask whether the 60 in 60 messages per second is wrong by comparison.
    - mtrimpe 13y ago
      The frontend needs to keep open thousands of persistent connections; 99.9% of which will time out with receiving a single message. Practically speaking the number of frontend servers is related to the number of active listening clients and not messages. Only 60 messages per second does seem very small from a business perspective however.
      - taspeotis 13y ago
        If you take the frontend servers out of the equation it's still 52 servers for 60 messages per second.
        msmithstubbs 13y ago
        My guess: the 60 messages per second is an average. Monday morning in the US timezones might see several multiples of that. Weekends could be a lot lower.
        powdahound 13y ago
        It's definitely very bursty. On weekends and holidays things are much more quiet. During peak load we'll be in the hundreds/sec. Also keep in mind that chat messages don't actually make up the majority of the traffic we serve; it's presence information (away, idle, available), people connecting/disconnecting, etc. (I'm one of the HipChat co-founders.)
        hox 13y ago
        And? The article makes no mention of what each server is responsible for, how much redundancy is built in (this is AWS where nodes die on a whim), whether all of these servers are used for production or if there is a spread across dev and staging environments, etc. When all of these factors are considered, 52 servers isn't a huge number.
    - tveita 13y ago
      Looks about right from their graph: https://s3.amazonaws.com/uploads.hipchat.com/10804/368466/10joz8sztfw4dfc/HipChat1B.jpg https://s3.amazonaws.com/uploads.hipchat.com/10804/368466/10... That traffic is going to be highly bursty, though.
- Kiro 13y ago
  May I ask what kind of company you work at where you have that kind of queries/s?
  - taspeotis 13y ago
    Australian childcare software provider. I imagine an outsider wouldn't equate "childcare software" with "database intensive" but the online reporting childcare services have to do is quite involved [1]. Their online reporting to the government is a separate concern to day-to-day tasks in their childcare service. So childcare software has to do more than just interface with the online reporting interface. [1] http://docs.education.gov.au/system/files/doc/other/service_provider_software_technical_specifications_v1.50_0.pdf http://docs.education.gov.au/system/files/doc/other/service_... (The attendance information of children is the most complicated part, see page 160 for what an attendance record looks like.)
    - Jamsii 13y ago
      Is it hubworks?
      - taspeotis 13y ago
        No. I don't know what RDBMS HubWorks use but to my knowledge they run their software with AWS and I think there's better fits to AWS than MSSQL.
- GamblersFallacy 13y ago
  On my shit dev box, localhost Elasticsearch is indexing (AKA writing) @ approx 2,500 msg/sec.
  - est 13y ago
    Can you still process that many while holding like 2M connections open?
byroot 13y ago
> .5 terabytes of search data. 500 gibabytes isn't cool enough ?

53 points by pscsbs 13y ago

Stats * 60 messages per second ... * 26 front end proxy serves. Double that in backend app servers. 78 servers for 60 messages per second? We have one MSSQL box at work here that is regularly doing 2,500 queries/sec and that's an OLTP workload so easily more than 60 writes/sec. serves makes me think that someone didn't proof-read the article...

georgebarnett 13y ago
How did it even make sense in your head to compare an OLTP workload to a chat service? The proxy servers are there to terminate the large number of persistent connections. Of course it's possible to do it using less servers, but given the Hipchat guys are smart (disclosure - I'm an Atlassian and know the internals) I would give them the benefit of the doubt rather than engaging in armchair architecture.
- taspeotis 13y ago
  The point was to ask whether the 60 in 60 messages per second is wrong by comparison.
  - mtrimpe 13y ago
    The frontend needs to keep open thousands of persistent connections; 99.9% of which will time out with receiving a single message. Practically speaking the number of frontend servers is related to the number of active listening clients and not messages. Only 60 messages per second does seem very small from a business perspective however.
    - taspeotis 13y ago
      If you take the frontend servers out of the equation it's still 52 servers for 60 messages per second.
      - msmithstubbs 13y ago
        My guess: the 60 messages per second is an average. Monday morning in the US timezones might see several multiples of that. Weekends could be a lot lower.
        powdahound 13y ago
        It's definitely very bursty. On weekends and holidays things are much more quiet. During peak load we'll be in the hundreds/sec. Also keep in mind that chat messages don't actually make up the majority of the traffic we serve; it's presence information (away, idle, available), people connecting/disconnecting, etc. (I'm one of the HipChat co-founders.)
      - hox 13y ago
        And? The article makes no mention of what each server is responsible for, how much redundancy is built in (this is AWS where nodes die on a whim), whether all of these servers are used for production or if there is a spread across dev and staging environments, etc. When all of these factors are considered, 52 servers isn't a huge number.
  - tveita 13y ago
    Looks about right from their graph: https://s3.amazonaws.com/uploads.hipchat.com/10804/368466/10joz8sztfw4dfc/HipChat1B.jpg https://s3.amazonaws.com/uploads.hipchat.com/10804/368466/10... That traffic is going to be highly bursty, though.
Kiro 13y ago
May I ask what kind of company you work at where you have that kind of queries/s?
- taspeotis 13y ago
  Australian childcare software provider. I imagine an outsider wouldn't equate "childcare software" with "database intensive" but the online reporting childcare services have to do is quite involved [1]. Their online reporting to the government is a separate concern to day-to-day tasks in their childcare service. So childcare software has to do more than just interface with the online reporting interface. [1] http://docs.education.gov.au/system/files/doc/other/service_provider_software_technical_specifications_v1.50_0.pdf http://docs.education.gov.au/system/files/doc/other/service_... (The attendance information of children is the most complicated part, see page 160 for what an attendance record looks like.)
  - Jamsii 13y ago
    Is it hubworks?
    - taspeotis 13y ago
      No. I don't know what RDBMS HubWorks use but to my knowledge they run their software with AWS and I think there's better fits to AWS than MSSQL.
GamblersFallacy 13y ago
On my shit dev box, localhost Elasticsearch is indexing (AKA writing) @ approx 2,500 msg/sec.
- est 13y ago
  Can you still process that many while holding like 2M connections open?

byroot 13y ago

> .5 terabytes of search data. 500 gibabytes isn't cool enough ?