10 ms·
This article has a large gap in the story: it ignores sensor data sources, which are both the highest velocity and highest volume data models by multiple orders
by jandrewrogers 6y ago
This article has a large gap in the story: it ignores sensor data sources, which are both the highest velocity and highest volume data models by multiple orders of magnitude. They have become ubiquitous in diverse, medium-sized industrial enterprises and it has turned them into some of the largest customers of cloud providers due to the data intensity. Organizations routinely spend $100M/year to deal with this data, and the workloads are literally growing exponentially. Almost no one provides tooling and platforms that address it. (This is not idle speculation, I’ve run just about every platform you can name through lab tests in anger. They are uniformly inadequate for these data models, everyone relies on bespoke platforms designed by specialists if they can afford the tariff.)
If you add real-time sensor data sources to the mix, the rest of the architecture model kind of falls apart. Requirements upstream have cascading effects on architecture downstream. The deficiencies are both technical and economic.
First, you need a single ordinary server (like EC2) to be able to ingest, transform, and store about 10M events per second continuously, while making that data fully online for basic queries. You can’t afford the latency overhead and systems cost of these being separate systems. You need this efficiency because the raw source may be 1B events per second; even at that rate, you’ll need a fantastic cluster architecture. Most of the open source platforms tap out at 100k events per second per server for these kinds of mixed workloads and no one can afford to run 20k+ servers because the software architecture is throughput limited (never mind the cluster management aspects at that scale).
Second, storage cost and data motion are the primary culprits that make these data models uneconomical. Open source tends to be profligate in these dimensions, and when you routinely operate on endless petabytes of data, it makes the entire enterprise problematic. To be fair, this is not to blame open source platforms per se, they were never designed for workloads where storage and latency costs were critical for viability. It can be done, but it was never a priority and you would design the software very differently if it was.
I will make a prediction. When software that can address sensor data models becomes a platform instead of bespoke, it will eat the lunch of a lot of adjacent data platforms that aren’t targeted at sensor data for a simple reason: the extreme operational efficiency of data infrastructure required to handle sensor data models applies just as much to any other data model, there simply hasn’t been an existential economic incentive to build it for those other data models. I've seen this happen several times; someone pays for bespoke sensor data infrastructure and realizes they can adapt it to run their large-scale web analytics (or whatever) many times faster and at a fraction of the infrastructure cost, even though it wasn't designed for it. And it works.
- moab 6y agoThanks for this great comment. What kind of workloads are people trying to run on sensor data that arrives at such a high velocity? Time series analysis? Anomaly detection? I wish I had a better idea of what kind of specific problems users you've run into are trying to solve, which fail on the existing software stack.
- stingraycharles 6y agoNot OP, but I work for QuasarDB and we deal with a lot of customers in this sector. It’s typically a mix of everything, but predictive maintenance, anomaly detection and failure analysis are the most common. For example, there is one process that does trend analysis and tries to “predict” acceptable boundaries of a certain sensor’s measurements, and this is then compared in real-time with the actual sensor readings. If things fail for some reason, a technical engineer will dive into the data with dashboards (think: Grafana), zoom in, compare the readings with other sensors, etc. The sheer volume of the data makes it fairly painful. Downsampling does happen, but only after a few weeks. This means that you still need enough storage capacity to deal with the full stream of data in real-time.
- jandrewrogers 6y agoThe data models for any non-trivial sensor analysis are intrinsically spatiotemporal -- every measurement or event happens at a place and time. Spatial relationships are central to the proposition of analytically reconstructing the dynamics of the physical world from disparate entities and sensors. The objective is to sample enough pixels and their relationships to sketch an accurate picture of reality as it happens. For example, a car is trying to understand its relationship to every relevant static and dynamic entity in its environment that affects its ability to operate safely. There is no business that does not benefit from having a model of reality that converges on ground truth in real-time, if you can take advantage of it. Most of the analysis that is done usually falls under one of two categories. First, inferring (you can rarely measure it directly) when something has changed in the real world that is relevant to your business so that you can adapt to it immediately -- the applies to everything from autonomous driving to agricultural supply chains. Second, detecting anomalies -- the unknown unknowns -- so that risks can be managed when the real world appears to not conform to the models upon which you base decisions. A third category is support of industrial automation, which benefits immensely from high-resolution multimodal sensor data models, though this is largely a cost reduction measure. These categories are hand-wavy but in practice, boring industrial companies have concrete metrics they are trying to achieve or risks they are trying to manage in the most efficient way possible.