3 ms·
> Second, typical commercial IoT data sources are often on the order of 100TB-1PB per day Interesting. Do you have a source on that?
by icambron 13y ago
> Second, typical commercial IoT data sources are often on the order of 100TB-1PB per day
Interesting. Do you have a source on that?
- jandrewrogers 13y agoThese are the kinds of applications and workloads I have been working on for almost a decade now, spanning many industries, so it is my area of domain expertise. The above numbers are a pretty typical range for the myriad companies and organizations I work with. A small system or pilot project might be 10 TB per day. For some government systems, 1PB per day is a small fraction. People are just beginning to take advantage of these data sources but table stakes is being able to continuously ingest and index many millions of complex spatial relationships every second, which by itself is something no popular Big Data platform supports. The (fun!) computer science challenge of it is that the systems have to be unbelievably scalable, but you can't use hash partitioning or range partitioning, and many things you were taught about database engine design turn out to be completely wrong in this context. It is an area ripe for innovation and growth.
- icambron 13y agoI was hoping you could give a source not because I was questioning the veracity of your statement, but because I wanted to learn more about it, specifically what kind of data that would be and what you do with it. I bet there are a lot of interesting challenges in that space.
- jhugg 13y agoAssume Nest has 10 million users, which is ballpark-ish. 100TB is 10megs per user per day. That sounds high unless my Nest is sending audio clips to Google. I'm sure there are some IoT apps sending 1PB/day, but there are plenty that don't.