This is great advice. I'd add that the culture around data engineering projects tend to be very different. I've seen companies that treat data projects as if

This is great advice. I'd add that the culture around data engineering projects tend to be very different. I've seen companies that treat data projects as if they were this great unknown projects where the developers could get away with using bad or no patterns and not follow patterns that other applications in the company use. Technologies like Spark have made more common and easier to develop big data applications and implement design patterns that regular engineers can understand and follow. Couple a great data engineer that with great data scientist using tools like Spark, R, H2O, Alluxio, Parquet, etc. and companies can truly exploit their large sets of data effectively. The problem is DevOps and bridging the gap between a scientist's environment and a production environment and keeping both as flexible and testable as possible. We started a company to bootstrap companies into this culture by providing DevOps services and UIs which simplify the deployment of Kubernets, Spark, Druid, H2O, etc. clusters. We also provide tools and services for simplifying and automating ETL pipelines with which models can be trained. If you are interested in finding out more about these services contact us at: miguel@zero-x.co.