Your choice of "big data" vs. python scripts sounds just like the classic trade-off of scope creep vs. good enough. IMO the answer is almost always "good enoug

Your choice of "big data" vs. python scripts sounds just like the classic trade-off of scope creep vs. good enough. IMO the answer is almost always "good enough". This has been expressed in countless tropes/principles from many wise people, like KISS (Keep It Simple Stupid), YAGNI (You Ain't Gonna Need It), "pre-optimization is the root of all evil", etc. If you go the YAGNI route, and when your lack of scale comes back to bite you (a happy problem to have), you'll have hard data about what exactly needs to be scaled, and you'll build a much better system. Otherwise, you'll dig deeper into the pre-optimization rabbit-hole of hypotheticals, and in that case, it's turtles all the way down (to use another trope).