5 ms·
Ask HN: How to Design an analytical system
Hi,
I'm currently trying to improve a system we use at my company to generate reports based on events data we gather. Our system ingests a lot of events coming from different entities every day and at night it generates reports for each of theses entities. There are several challenges in this generation:
- standard data that can be counted or summed
- maps data that group by values with some keys
- Aggregation functions such as mean, median... that must be run last after all the processing is done
- Cross dependencies on 2 differents kinds of events (that aren't usually processed together and require 2 db requests)
- General scale of the task that requires everything to be page/cursor based
I'm wondering if anyone here already designed such a system, and if so how did you approach it?
I think it should be a fairly common task among tech companies but I've yet to find articles stating good practices and things to watch for.