4 ms·
I'm a data engineer for most of my day right now, and a lot of it is done with ruby/python/shell scripts into postgres DBs. What learning path should I go down
by zeku 6y ago
I'm a data engineer for most of my day right now, and a lot of it is done with ruby/python/shell scripts into postgres DBs.
What learning path should I go down? I'm a solo actor at work with a lot of agency to decide my workflows.
I see myself building small to medium size data collections over the next year or two at my job.
Can someone point me to some learning?
I have a CS degree etc. and my title in software engineer etc. etc.
End users of my data usually like their data as a CSV that is then read using R or Python. However there is also a use case where I will build an app to view my data in a simple way.
All of this is completely doable with my current knowledge/workflow but I can't help feel like I do a data engineering job with very different tools than i see "data engineers" speaking about online.
- langitbiru 6y agoA couple of days ago, there was a thread about "How to become a data engineer in 2021": https://news.ycombinator.com/item?id=25728198 https://news.ycombinator.com/item?id=25728198
- zeku 6y agothanks!
- ageofwant 6y agoAdopt a good Workflow tool, like apache airflow. Easiest is to rent a service from AWS https://aws.amazon.com/managed-workflows-for-apache-airflow/ https://aws.amazon.com/managed-workflows-for-apache-airflow/
- zeku 6y agothanks!
- mywittyname 6y agoSeconding this, Apache Airflow is awesome. I can't believe how much time is saves during development.