2 ms·
I think there is some ambiguity of terminology around 'real time'. By which I mean that aggregation implies a temporal window and batching data and processing t
by brudgers 10y ago
I think there is some ambiguity of terminology around 'real time'. By which I mean that aggregation implies a temporal window and batching data and processing the data as a batch whereas really 'real time' implies right now and a single datum and stream processing. The historical concepts are OLAP [online analytic processing] and OLTP [online transaction processing], respectively. More recent thinking is that batch is a special case of stream (or vice versa depending on the task).
Architecturally, it is common to use pub/sub. Basically to ingest all the input streams into a single log type data structure and allow services to make calls against an API.
Boiled down to the simplest thing that might work:
1. Write each event in the input streams as text to a log file [OLTP]
2. Grep [maybe using tail] to find the interesting bits. [OLAP]
Where I am going is that the way a specific business is likely to solve this class of problems tends toward integrating several tools rather than something off the shelf because of issues like data integrity [i.e. it's bad for a bank to drop data on the floor, it's ok for the comments section of a blog].
The second reason that integrating tools makes sense is that issues like data access restrictions are more robustly handled at the operating system and network levels [avoids duplicating a core OS functionality and specific security/access configurations] I'd mention that maintaining data silos is often more habit than necessity and providing full access to the data or only ingesting data appropriate for full access probably simplifies the design.
Anyway, to me this looks more like software development than swiping a credit card if the set of features is non-negotiable...although, I suppose the consultant route might open up that possibility.
Good luck.