4 ms·
Why limit parallelism to 4x? Spin up a ton of lambdas and get it done in 10 seconds. You're also forgetting that 50ms time to load each file from cloud storage
by squeaky-clean 4y ago
Why limit parallelism to 4x? Spin up a ton of lambdas and get it done in 10 seconds.
You're also forgetting that 50ms time to load each file from cloud storage. It's still 4 hours and 2.5 minutes with your native code compared to 8 hours with Python. Suddenly not such a massive improvement.
Even then I don't see the issue with it taking 4 hours to do a million of them. Do you need to do a million per hour? Is it even likely they'll need to do a million of them total?
Do bank statements even come in frequently enough to do a realtime dashboard with? I get my bank statement every 30 days.
How much longer does it take to develop the native version? How much longer does it take to modify when a bug is found or a bank changes their statement layout? How much more do you have to pay a native code engineer compared to a Python dev and how easy are they going to be to replace eventually?
- blindhippo 4y agoOne dimension to consider here is cost of compute. Going from 8 hours to 4 hours is a 50% reduction in the time to compute and we're assuming this is occurring on the same relative hardware/instance size. At scale, that could translate into hundreds of thousands of dollars in savings. But your points are relevant - as with anything related to development, "it depends" rules the day. There isn't a clear cut "x is objectively better than y" in general.