4 ms·

Tx for the (fast) reply. The problem I have with SQS and SNS (and Celery) is you can not just throw tasks into it and eventually the system based on some hints

• scarface74 8y ago
  You’re always “locked into your infrastructure” whether it be your database, your choice of tooling or whatever. No matter how often the bushy tailed developer writes facades, uses the repository pattern, etc. you rarely change your infrastructure. The cost savings is usually not worth the risk of regressions and migration costs. There is a real cost today in employee time and maintenance costs in maintaining your own infrastructure. Often those costs are invisible to the organization since most of the employees are salaried and willing to work more than 40-45 hours a week (I’m not) The problem I have with SQS and SNS (and Celery) is you can not just throw tasks into it and eventually the system based on some hints scale up / down the workers. Of course you can rely on Lambdas but then you are locked with amazon (not need to mention that you can not control how much the lambdas will cost you). Well two things: Lambdas aren’t some magical thing that requires a lot of changes to your code. All lambda requires is one function added to your code base that takes a JSON Event and a lambda context. The only thing that your lambda handler should be doing is deserializing the event into your domain object and calling your business layer - the same thing that your regular entry point should be doing. I have a C# solution that has three modules (assemblies). One has all of the AWS dependencies with the lambda entry point, one is a regular .Net executable with TopShelf integration to create a Windows service and the third is the actual business logic. The lambda project takes the SQSEvent gets the message body, deserislizes it and sends it to the assembly with the business logic The second, runs as a Windows service reads from the queue, deserializes the message and sends the object to the same assembly with the business logic. When I push the code, AWS spins up a Linux Docker container using Code Build that builds both the Linux based Lambda and the Windows executable. There is no “lock-in” to lambda. We deploy the Windows service for QA testing. Also, reinventing the wheel allows to stay in control. Using RabbitMQ and to some extent Celery or Kafka locks you up without much control since it's foreign code base with alien language. We as software developers get paid to produce solutions that add business value and that allow the business to focus software development where it has a competitive advantage. You don’t add business value by reinventing the wheel. Besides that, no developer wants to come into a shop where all of the cross cutting concerns like logging, queue management, database access, etc are all some bespoke system where the architect thought they were a special snowflake. I would much rather go onto the Internet where if I have an issue, I can probably find someone else who had that same issue than trying to find the original creator of CustomQueueManager who may not be at the company any more. In the case of AWS, I have an “easy button”. After I have gone through all of the obvious steps and something is still wonky with a managed service where I am using their SDK, I can just take advantages of our business support plan, open a ticket and start a chat. They will not rest until they figure out the issue. As far as scaling out without using lambda. That’s easy. Just setup two alarms - one for when your queue is under a certain size and one when your queue is over a certain size and use the alarms to trigger autoscaling within an autoscaling group.
• Izkata 8y ago
  Celery does have autoscaling, though perhaps more limited than what you're thinking of: https://celery.readthedocs.io/en/latest/userguide/workers.html#autoscaling https://celery.readthedocs.io/en/latest/userguide/workers.ht...