3 ms·
> New languages like Rust/Ocaml/Nim.. if yes then which? Completely irrelevant. DE is SQL, Python, sometimes Scala/Java. Get really good at SQL. Learn relatio
by slotrans 4y ago
> New languages like Rust/Ocaml/Nim.. if yes then which?
Completely irrelevant. DE is SQL, Python, sometimes Scala/Java.
Get really good at SQL. Learn relational fundamentals. Learn row- and column-store internals. Understand why databases are they way the are. Familiarize yourself with the differences between systems and why you need to care.
Get familiar with at least minimal Hadoop shit. Set up Spark from scratch on your machine, mess it up, do it again. Grapple with how absurdly complex it is.
Understand system-level architecture. Analytics systems tend to organize into ingest -> transform -> serving (BI tools etc)... why? There are good reasons! Non-analytics data systems have different patterns, but you will see echoes of this one.
Above all, spend time understanding the data itself. DE isn't just machinery. Semantics matter. What does the data "know"? What can it never know? Why? What mistakes do we make at the modeling step (often skipped) that result in major data shortcomings, or even permanently corrupt data that can never be salvaged? Talk to people building dashboards. Talk to data end-users. Build some reports end-to-end: understand the business problem, define the metrics, collect the data, compute the metrics, present them. The ultimate teacher.
(who am I? no one, just been doing data engineering since 2005, long before it was even a term in the industry)