> I just pulled out a python script, and combine that with Postgres and within an afternoon I had completed the project (albeit not production ready) There are

> I just pulled out a python script, and combine that with Postgres and within an afternoon I had completed the project (albeit not production ready) There are two problems here - one is that people prototype their architecture using massively over-engineered systems and the second is that a rough prototype makes it way into production. So, as a Hadoop perf engineer, I deal with both issues - "We have a Kafka stream hooked up to a Storm pipeline and it is always breaking and we can't debug ... what is this shit?" or "Postgres is stuck on BYTEA vaccuum loops and you can fix it with Hadoop, right?". There are significant advantages in prototyping with an easy to debug architecture, until all the business requirements stabilize. Sometimes the right answer is to use Postgres better (specifically, table inheritance for deletion of old data + indexes instead of delete from with <=), sometimes the right answer is a big data system designed for cold data storage & deep scans. This all comes back to the "We made a plan, now it is a death march" problem - the iterative failure process of building a prototype and having it actually fail is very important, but most people feel like they'll get fired if their work fails in the real world. Over-engineering is usually career insulation and somewhat defense against an architecture council.