4 ms·
First of all, 50GB per day is easy. Now, maybe contrary to what they say below, do the following: * Don't use queues. Use logs, such as Apache Kafka for exampl
by haddr 11y ago
First of all, 50GB per day is easy. Now, maybe contrary to what they say below, do the following:
* Don't use queues. Use logs, such as Apache Kafka for example. It is unlikely to lose any data, and in case of some failure, the log with transactions is still there for some time. Also Kafka guarantees order of messages, which might be important (or not).
* Understand what is the nature of data and what are the queries that are made later. This is crucial for properly modeling the storage system.
* Be careful with the noSQL cool-aid. If mature databases, such as postgreSQL can't handle the load, choose some NoSQL, but be careful. I would suggest HBase, but your mileage may vary.
* NoSQL DBs typically limits queries that you might issue, so the modelling part is very important.
* Don't index data that you don't need to query later.
* If your schema is relational, consider de-normalization steps. Sometimes it is better to replicate some data, than to keep relational schema and make huge joins across tables.
* Don't use MongoDB
I hope it helps!
- OldSchoolPro 11y agoInteresting points. I'm wondering about MongoDB. Have you tried it recently? It really improves with every release. We have a 20TB Mongo database and it's fast and problem free. Also regarding the queues. In my personal experience a single RabbitMQ server with good hardware can easily handle 50GB a day.
- stuartaxelowen 11y agoCan confirm, don't use mongo. We're stuck with it now, and I really wish we had used Postgres for biz data and Cassandra for the high volume non-relational data. They're just materialized views on-top of your event stream anyway.
- late2part 11y agoHere here ; +1. Kafka = good. Postgres can do a TB; if you plan bigger than that consider Cassandra. Also consider protobuf instead of json.gz.
- stuartaxelowen 11y agoOr... Arrow?! :O JK it's too early :(
- faizshah 11y agoI agree but if their problem is that they want a scalable solution for processing a growing amount of data cheaply and robustly and potentially doing more complex work in the future (it's a growing business). This is basically the tagline of the current batch of hadoop ecosystem projects. Investing in working with the hadoop ecosystem is a much better long term solution (if this is the case).
- stuartaxelowen 11y agoIMO investing in a hadoop system constrains you - make your data accessible via S3, Kafka, and most importantly, a good data model, and you can attach the right data processing tool when a problem that requires it comes along.
- faizshah 11y agoI agree that a good data model is key, in fact the way you structure your data is probably the most important thing. But the reason that the hadoop ecosystem is so good is because it's a growing ecosystem with a lot of momentum and a focus on processing large and growing amounts of data at scale and in a distributed/parallel fashion. If this is what you need the ecosystem is perfect.
- haddr 11y agoIf at some point of time you use hbase, using hadoop on top of that becomes a straightforward task. Both integrate very well with each other. But I think even without hadoop you can do. If the processing is relatively light (e.g. calculating some aggregates, etc) you can do it in near real time just connecting with another consumer to kafka topic.
- PerfectElement 11y agoCan you expand on the MongoDB comment? I'm using it for my SaaS and don't plan on getting anywhere near 50gb per day, but I'm curious to know about its limitations.