3 ms·
I'm a fan of Kafka for streaming systems, so I'd recommend to fetch data via api, add it to kafka and then consume from kafka to monitor the price + fire alerts
by ecesena 5y ago
I'm a fan of Kafka for streaming systems, so I'd recommend to fetch data via api, add it to kafka and then consume from kafka to monitor the price + fire alerts (assuming you don't really need a real-time system with very low latency).
What language you use is up to you/your team. If you like python, I'd look into Airflow or Celery as frameworks to build your workflows.
For java/scala I'd take a look at Google Dataflow or Flink or Kafka Streams, depending on how heavy the "monitor share prices" part is.
I think the major issue you're going to encounter is not building a system that works, but building a process that allows your team to upgrade the system, test new features, debug, etc.
Dataflow, for example, lets you run more or less the same code both on a stream of data and on batches of historical data.