3 ms·

Great to see Laravel in this discussion! Beautiful framework. A couple of questions: 1. Why did you go with Python for data processing instead of PHP? 2. Why

• timvdalen 9y ago
  I agree! Lumen is my go-to framework for small web-based things right now, picking things from Laravel is so powerful. 1) At the time, Python (and with it pandas/numpy) seemed to be a good fit for the data we were working with (and the analysis was developed as a separate project from the main system that calls it). If we were to do it over right now (and we well might in the coming months) we would probably move this into PHP as well, which makes writing workers much easier. 2) Cost-wise, SQS makes a lot of sense - the system is highly distributed and cut up into parts (the description in my first comment is a bit simplified) so adding redis to the mix means adding a whole new server. For SQS, we're just paying per-message (and we don't have a _lot_ of messages, so it's cheap). Furthermore, I've built an algorithm to scale up the background workers based on the queue size of SQS. This runs in Lambda, which makes it super easy to access SQS properties versus connecting to a remote redis server. Basically, the algorithm rents new EC2 instances whenever the load is high and terminates them whenever the load is low.