3 ms·
Well, millions of rows of data is tiny these days: if you have 100_000 IOT devices, each generate on average 10 telemetry points a second, that's already 1M wri
by dikei 5y ago
Well, millions of rows of data is tiny these days: if you have 100_000 IOT devices, each generate on average 10 telemetry points a second, that's already 1M writes per second. Storing these telemetry for a week and you already have 600 billions rows.
Of course that's a oversimplification, but the data generated today is mind-boggling large compared to the 90s.
- da39a3ee 5y ago> but the data generated today is mind-boggling large compared to the 90s. For what percentage of companies would you say your statement is true?
- nine_k 5y agoIf you have 1M writes per second, you have a data collection problem, no matter the technology. Doing it on a single node is not a great idea just because it'd be a huge SPOF. In most cases, you have colossally fewer than 1M writes per second, and you usually distribute them over several boxes. the IoT avalanche of writes in particular is naturally sharded, that is, all these writes can occur independently, and then you can replicate statistics into an analytical RDBMS. It's a large write load that needs to be transactionally ordered, like a stream of purchases in a large online store, that can be a hard(er) problem for an RDBMS.
- dikei 5y agoYup, I agree with you. Most businesses can work fine with traditional RDBMS, even on TB scale if you have a large box with lots of RAM and SSDs. My example was just one use-case where one should consider using a more unconventional DBs like Cassandra or HBase.
- smallerfish 5y ago:s/these days/at your particular job/g