3 ms·
Reading Kleppmann's book and finding projects that are trying to implement some of the concepts there is a good idea, regardless the language and tool. I'm jus
by rodrigodlu 4y ago
Reading Kleppmann's book and finding projects that are trying to implement some of the concepts there is a good idea, regardless the language and tool.
I'm just doing Python/Spark/AWS related tools. Most of my time is trying to break the bureaucratic ice between multiple layers from insane devops requirements to missing documentation on several "hidden requirements" (the ones that you find during the development process). So not much different than a lead role.
It's definitely different from a pure dev experience because it's expected that you lead changes inside the organization to make the pipelines work consistently. Without that (just plumbing) you can rely on "regular" backend devs.
This is obviously a high level data engineering perspective, not a low level like DB hacking or hyper optimizing existing pipelines and data transformations.
- thewhitetulip 4y agoI know Pyspark but not that much. I could try to be an advanced developer in Pyspark. And yes the lead stuff, I agree. But I am not currently in a Lead position yet
- beckingz 4y agoThe amount of people management and organizational change needed to do data warehousing in an organization is amazing. Good luck when you get to the point where time is better spent on process change.