In a past life, we were reading from twitter firehose and the offshore implemented a solution that spawns a large spark cluster to read from twitter, dumps it t

In a past life, we were reading from twitter firehose and the offshore implemented a solution that spawns a large spark cluster to read from twitter, dumps it to kinesis, then has a cloud function that writes it to dynamo, then merged it into redshift. It worked poorly, took months to build, and lost data. I implemented a python script that read from twitter firehose and wrote directly to redshift. It was about 50 lines long and did the same thing, and I deployed it to two existing prod servers that were not network bound. There was a few more cases like the on-shore database identity reconciliation being inefficient, so the company wanted to leverage cloud auto scaling. Well, if you have an exponential time algorithm and you put it into the cloud, it will still be exponential time and cloud auto-scaling will be extremely expensive, and also introduce latency between distributed system components, and indeed it was. The offshore teams initial logic was only that, cloud is auto scaling so can accommodate any performance the business logic needed. Sometimes people just view tech/cloud as lego blocks to glue together, based on the product sheet marketing, and do not have enough basic understanding of computer science to understand performance, or understand what a simple solution would look like