2 ms·
It is quite possible to build a unified storage and execution kernel that will allow you to simultaneously and seamlessly blend (1) streaming ingest path proces
by jandrewrogers 12y ago
It is quite possible to build a unified storage and execution kernel that will allow you to simultaneously and seamlessly blend (1) streaming ingest path processing, (2) online indexing/storage to disk at wire speed, and (3) fast online query processing that immediately reflects both storage and ingest path. Saturating a 10 GbE connection with this kind of workload on an ordinary server is pretty simple if the system is designed correctly. There is nothing technical that prevents processing 10 GbE streaming data concurrent with storing that data to disk and running queries across all of that data fast enough to saturate the outbound network. I have designed and implemented kernels that do exactly this.
When everything runs at wire speed and all data is always fully online, there is no meaningful distinction between "batch" and "streaming".
That said, the reason you generally do not see anything in open source that does it is that it requires a more sophisticated internal design and architecture than you typically see in any of the open source databases or big data platforms. Even a system designed to only run on a single machine (as opposed to a parallel cluster) is probably 50kLoC of dense, complex, low-level C++ just to get a basic kernel off the ground.
The main reasons you do not see a lot of startups doing this: The design of these types of storage/execution engines is rare knowledge. You have to design and implement most of your own algorithms and data structures -- few things can be farmed out to template libraries or system call uses. Relatively few developers are sufficiently skilled in C++ to successfully implement these kinds of systems with these kinds of performance envelopes. The code base for an MVP, excluding the expansive test infrastructure, is pretty huge, so you need a lot of man-hours with the above skills and expertise.
The high initial cost of building these types of systems and the difficulty of finding the necessary talent make them unattractive to investors as startups. You often have to spend $10-20M before you are even going to know if there is traction. That is a pricy bet.
- lsh123 12y agoEverything above is a great summary of the CEP startup I was part of 7-8 years ago. It took us 3+ years to get the product off the ground to the stage where we started to see real sales. We've had to build custom code to handle networking, messaging, scheduling, storage, etc. to get to the required performance numbers.