Another problem with "BigData": hiring and the tendency of the ecosystem to "sustain" itself (like any system). As a company hires traditional BigData Architect

Another problem with "BigData": hiring and the tendency of the ecosystem to "sustain" itself (like any system). As a company hires traditional BigData Architects, Developers, Data Scientists, Engineers, etc. it will naturally have a tendency to choose the traditional BigData technology and solutions like BigQuery, Spark, storing everything in HDFS, etc. A trick I saw is companies hiring experienced jack-of-all-trades back-end engineers into Data teams. A lot of things get migrated from Spark to Postgres, from Kafka to REST API calls, and keep working fine and become generally more responsive. I'm on the same page as the author here: traditional BigData tech has its place and its uses, but before choosing it companies (CTOs, architects) should carefully consider if it is necessary, especially considering the cost of it and the risk of locking themselves down in a very specialized domain.