5 ms·

We [1] use a Laravel back-end for the main system, hosted on autoscaling EC2 behind a load balancer. Our front-end web application is built on Angular, running

We [1] use a Laravel back-end for the main system, hosted on autoscaling EC2 behind a load balancer. Our front-end web application is built on Angular, running on S3 behind CloudFront. Since we process a lot of data, we use background jobs (with SQS as a broker) to perform our analyses. The actual data processing and wrangling is done in Python. We have released some parts of our systems as open source projects [2], [3], [4]. [1]: https://odysseyattribution.co/ https://odysseyattribution.co/ [2]: https://github.com/code-orange/telemachy https://github.com/code-orange/telemachy [3]: https://github.com/code-orange/jot https://github.com/code-orange/jot [3]: https://github.com/code-orange/statuspage https://github.com/code-orange/statuspage

bgdkbtv 9y ago
Great to see Laravel in this discussion! Beautiful framework. A couple of questions: 1. Why did you go with Python for data processing instead of PHP? 2. Why not use Redis for queue/background jobs processing?
- timvdalen 9y ago
  I agree! Lumen is my go-to framework for small web-based things right now, picking things from Laravel is so powerful. 1) At the time, Python (and with it pandas/numpy) seemed to be a good fit for the data we were working with (and the analysis was developed as a separate project from the main system that calls it). If we were to do it over right now (and we well might in the coming months) we would probably move this into PHP as well, which makes writing workers much easier. 2) Cost-wise, SQS makes a lot of sense - the system is highly distributed and cut up into parts (the description in my first comment is a bit simplified) so adding redis to the mix means adding a whole new server. For SQS, we're just paying per-message (and we don't have a _lot_ of messages, so it's cheap). Furthermore, I've built an algorithm to scale up the background workers based on the queue size of SQS. This runs in Lambda, which makes it super easy to access SQS properties versus connecting to a remote redis server. Basically, the algorithm rents new EC2 instances whenever the load is high and terminates them whenever the load is low.

bgdkbtv 9y ago

Great to see Laravel in this discussion! Beautiful framework. A couple of questions: 1. Why did you go with Python for data processing instead of PHP? 2. Why not use Redis for queue/background jobs processing?

timvdalen 9y ago
I agree! Lumen is my go-to framework for small web-based things right now, picking things from Laravel is so powerful. 1) At the time, Python (and with it pandas/numpy) seemed to be a good fit for the data we were working with (and the analysis was developed as a separate project from the main system that calls it). If we were to do it over right now (and we well might in the coming months) we would probably move this into PHP as well, which makes writing workers much easier. 2) Cost-wise, SQS makes a lot of sense - the system is highly distributed and cut up into parts (the description in my first comment is a bit simplified) so adding redis to the mix means adding a whole new server. For SQS, we're just paying per-message (and we don't have a _lot_ of messages, so it's cheap). Furthermore, I've built an algorithm to scale up the background workers based on the queue size of SQS. This runs in Lambda, which makes it super easy to access SQS properties versus connecting to a remote redis server. Basically, the algorithm rents new EC2 instances whenever the load is high and terminates them whenever the load is low.