3 ms·
Hi antcar, You need to split your system into following: 1. metadata database (i.e. users, groups/chatrooms, etc.) 2. hot chat history (recent message
by nivertech 11y ago
Hi antcar,
You need to split your system into following:
1. metadata database (i.e. users, groups/chatrooms, etc.)
2. hot chat history (recent messages)
3. cold chat history (archived messages)
4. A messaging queue in front of cold chat DB.
Fo the metadata any fast K/V store with rich data structures or basic secondary indices will work, You can even use Redis if you use something like RedisLabs with persistence.
For the hot DB, you can use Redis (cluster or sharded), Aerospike, ScyllaDB or ElasticSearch.
For the cold DB, you can use Riak, Cassandra/ScyllaDB, DynamoDB, Aerospike.
You can even use the same database for both paths, but it need to be optimized for mostly write workloads.
15M msg/day ~ 200 msg/sec - any K/V database will do.
For the messaging queue, you can use RabbitMQ or for high t-put Kafka (or any managed cloud alternatives such as AWS Kinesis, Google Cloud Pub/Sub, Azure Events Hub, etc.)
My email is in my profile:
https://news.ycombinator.com/user?id=nivertech https://news.ycombinator.com/user?id=nivertech
Hope this helps.
- iurisilvio 11y agoI'm not related to OP, but thanks for this answer! It is not easy to find a clear answer like that for a complex problem. The first time I had to build something to scale at day one, it was a long research about databases, cluster/shard, architecture and everything else you described. It worked, but it'd been easier with an answer like this one!