17 ms·

Ask HN: Upskilling as a Data Engineer

• travoltaj 4y ago
  I'm in the same position with the same question. Personally, I'm planning to learn about the internal implementation of databas(es), starting with the book Designing Data Intensive Applications. This is so that I learn about the current ways data is stored
  • thewhitetulip 4y ago
    I was planning to build an sqlite alternative.. haha I even started writing one in Python a few months ago. Have written the file storage part insert/update/delete works. Nothing else works though
• banglaman 4y ago
  If you’re at a consumer company learning JavaScript would make you very useful for building event data collection systems.
• rodrigodlu 4y ago
  Reading Kleppmann's book and finding projects that are trying to implement some of the concepts there is a good idea, regardless the language and tool. I'm just doing Python/Spark/AWS related tools. Most of my time is trying to break the bureaucratic ice between multiple layers from insane devops requirements to missing documentation on several "hidden requirements" (the ones that you find during the development process). So not much different than a lead role. It's definitely different from a pure dev experience because it's expected that you lead changes inside the organization to make the pipelines work consistently. Without that (just plumbing) you can rely on "regular" backend devs. This is obviously a high level data engineering perspective, not a low level like DB hacking or hyper optimizing existing pipelines and data transformations.
  • thewhitetulip 4y ago
    I know Pyspark but not that much. I could try to be an advanced developer in Pyspark. And yes the lead stuff, I agree. But I am not currently in a Lead position yet
  • beckingz 4y ago
    The amount of people management and organizational change needed to do data warehousing in an organization is amazing. Good luck when you get to the point where time is better spent on process change.
• boringds 4y ago
  FWIW I've found a lot of value in learning Typescript this year and interacting with the AWS CDK to build infrastructure as code. Upskilling my AWS knowledge also massively paid off, especially networking which I never truly understood before.
• yawaramin 4y ago
  Just fyi, OCaml is about as old as Java (circa 1995) and its direct anscestors (Caml and ML) are about as old as C (1970s to '80s). Not exactly new ;-)
  • agent281 4y ago
    Well, Python came out in 1991 so comparatively OCaml is the new hotness! ;)
• harlanji 4y ago
  I found myself specialized with data eng + platform skills in Clojure + Kafka + K8s + Node and it's a liability if you want to work outside of Big Tech, because few can afford to use tools like that. Node's relatively pretty expensive these days, believe I've used it a lot and bought into JS everywhere. But it's a really hard ecosystem compared to Python. Might be an out there take, but being able to develop a shoe string web app that can be maintained by a solopreneur might be a good skill. Should translate to rapid prototyping concepts for big corporate managers as well. I'd argue Python could eat PHP's old niche in these regards, because it's way easier to get rolling from scratch with Flask and Pip than PHP was as recently as 2016.
  • NeutralForest 4y ago
    Idk about the PHP thing. PHP8 is pretty great and Laravel is pretty awesome tbf.
• ed_elliott_asc 4y ago
  First think about your target company type/industry - the majority of corps will be using either azure, gcp, or aws - learn one (or more) of those and how they deploy using code etc. Then think about that tools they use, in azure it will likely be ADF, Databricks, maybe Synapse. Languages, python, sql, python, python, sql, some scala, more python, more sql, then get a general understanding of a few different languages (c, typescript, c#, Java, kotlin). I’ve never seen a data engineering role asking for Rust/OCaml/Nim - I’m not saying they don’t exist but I’ve not seen them and I’ve rarely seen a data engineering role not asking for either python or scala.
• swyx 4y ago
  the way I think about upskilling is it is essentially a high stakes bet on megatrends in the industry - you want to be early + right as that is how you get paid the most. take your pick from https://airbyte.com/blog/becoming-a-data-engineer-2023 https://airbyte.com/blog/becoming-a-data-engineer-2023 : - Language: SQL and Python Continue to Dominate, With Rust Looking Promising - Abstraction: Custom ETL Is Replaced by Data Connectors - Latency: Better Tools Available In The Streaming Domain - Architecture: The Lakehouse and the Semantic Layer Combine Strengths - Trust: Data Quality and Observability Become an Essential - Usability: The New Focus Is on Data Products and Data Contracts - Openness: When It Comes to Data Tools, Open Source Is the Answer - Standards: Database Open Standard Simplified by DuckDB - Roles: Data Practitioners Expand Their Knowledge into Data Engineering - Collaboration: DataOps Reduces Data Silos and Improves Teamwork - Adoption: Data Engineering Is Key Regardless of Industry or Business Size - Foundations: The Data Engineering Lifecycle is Evergreen (not my article but i work at airbyte, this was the result of our 2023 trends review)
  • richiebful1 4y ago
    Is anyone familiar with any Rust language data libraries for data engineering? My experience with Rust is it's too low level and verbose to be productive at reshaping and cleaning data.
    • jmartin2683 4y ago
      We use rust for data processing, etl and to serve apis directly from massive parquet collections using Polars. What would you like to know?
      • Kinrany 4y ago
        What are your requirements, and how do you store and load the data? Are you doing anything to select only the relevant parts of the parquet files?
        • jmartin2683 4y ago
• sammyd56 4y ago
  What is your goal? For short-term career growth, $YOUR_COMPANY's current preferred ETL tool will have the biggest ROI. Focus on design patterns: while APIs will come and go, the concepts, as you rightly say, are transferrable. If you're looking to land a new role: the market says dbt, databricks and snowflake are pretty strong bets. If it's personal interest, or a high-risk, high-reward long term play, take your pick from any of the new hotness!
  • thewhitetulip 4y ago
    Goals are either Upskilling or self learning Any guidelines about design patterns? I'm ok learning something for the sake of it. Tired of wfh and sitting in my house. I learnt Go a few years ago just for the sake of it. I was planning to learn compiler construction and create a toy language or create an sqlite alternative for fun
  • thenaturalist 4y ago
    I'd second the notion that the question is far too open. I'd add that dbt, databricks and snowflake are pretty strong bets still, but you have to acknowledge that they're becoming mainstream with an ever accelerating pace as the companies behind them churn out upskilling courses, meetups and acquire an ever larger share of the market. If you like to be a specialist, going deep into either of those still holds career value. If you're taking a more generalist view of where things are headed, the best prediction I heard someone say to set themselves apart is for Data Engineers to optimize for operationalizing data. Focusing much more on reverse ETL, becoming knowledgeable in building data web apps. The no-code or low-code movement around data apps will make the barrier of entry to set something up nonexistent, and I see how that will drive demand. Pairing (big) data query/ frontend performance and web apps is another beast though. For all my initial scepticism, I see the Data Mesh concept picking up pace in the years to come. It's vendor independent, couples well with Team Topologies and effective, decoupled, functional SWE teams. There still will be a big need for standards and conventions set by a small enabling core DE team, as of now, the knowledge gap between the baseline DE and your average SWE or Product Owner is just way too big in my experience. Last but not least, I'd throw data lake out there. Apache Iceberg is getting a lot of attention and rightfully so. TCO of a query engine on top of files is so much better than any DWH and any org being able to optimize compute on data for it's current need will be able to save massively while the "convenience" gap steadily closes. Again, pretty generic but there's much to learn around Athena, Trino and the like. I'm personally not a fan of learning a new language except maybe for Rust. There is an ever increasing stack of standard "low-code" tools for the typical ETL schlick, and Python won't go anywhere. Again, potential to differentiate will be low and ever lower in many contexts outside of proper big data. This is only me though and this view is highly context dependent, so YMMV of course.
• Dowwie 4y ago
  We are 5 days from the new year, so in 2023, a data engineer ought to take a serious look at Elixir. I manage Elixir systems for data engineering related work, so I can attest to its use in this domain within a production environment. I also have used Rust for comparable data-engineering related systems. Elixir is a phenomenal language for data engineering. I've written message processing pipelines in Rust and didn't get anywhere near the level of design considerations that Broadway / Genstage have. Some day, there may be robust open source offerings in Rust as there are in Elixir, but the ecosystem hasn't reached this state, yet. Rust asyncio is also in a minimum-viable-product condition, lacking the sweet structured concurrency that Erlang-OTP solved long ago and Elixir benefits by. Data pipeline processing in Elixir, utilizing all available cores exposed to the runtime, is straightforward and manageable. Telemetry patterns have been standardized across Elixir ecosystem. There are background worker processing libraries like Oban that help with audit trail/transparency. Smart, helpful developer communities. Elixir is not going to beat Rust on performance. CPU-bound work is going to take orders of magnitude longer to complete with Elixir than Rust. You could extend Elixir with Rust in CPU-intensive situations using what is known as NIFs but you'll need to become familiar with the tradeoffs associated with using Rust NIFs. Writing in Rust involves waiting for compilation. When a developer waits for compilation, they switch to something else and lose focus. You can use partial compilation for local development, and that speed things up. You also need to have a very modern workstation for development, preferably an M1 laptop or 16-core Ryzen, with at least 32GB ram and SSD. Elixir, however, has quick compile times as it doesn't do anywhere near the level of work that Rust compiler does. There is a tradeoff for that, though. Elixir is a dynamic language and consequently has all the problems that dynamic languages do that are automatically solved by a strongly-typed compiled language such as Rust. You also discover problems at runtime in Elixir that often would be caught by the Rust compiler. One final mention is Elixir livebooks. Elixir has thrown down the gauntlet with livebooks. Coupling adhoc data scientist workflows in notebooks with robust data engineering backends makes a lot of sense. There are high-performance livebook dataframes backed by Rust. Elixir backed by Rust is beating Python backed by Rust from developer experience down.
  • thewhitetulip 4y ago
    Have you seen anyone use Elixir? I'm hearing the language for the first time!
• cheri9 4y ago
  [dead]
• skrtskrt 4y ago
  Generalized data architecture, understanding the roles and challenges of turning data lakes into data warehouse, etc. including the user personas that use each and their needs and limitations. Different types of databases and data storage types and what they excel at. How to save money on storage. Considerations of idempotency, replay-ability, rollback-ability. You say all workflow engines are the same but even just reading the Pachyderm docs will give an idea of modern data engineering best practices - data versioning and data lineage, incremental computation, etc. Temporal also has a very cool, modern approach (distributed, robust, event-driven) to generalized workflow management (non big data specific)- if you’re used to stuff like Airflow, Temporal is a jump 10 years forward.
• jmartin2683 4y ago
  Rust! Polars/Datafusion are taking over, and for good reason.
  • Kinrany 4y ago
    How are you using Datafusion? And overall, very interested in hearing about your experience. What does the whole stack/pipeline look like?
    • jmartin2683 4y ago
      We've used both DataFusion and Polars for very similar use cases, though recently we've really begun to coalesce around Polars and most further development will use it, barring any complication that would require something else (DataFusion included). Basically what we do amounts to bringing in a ton of json, pulling objects out of that into jsonl, applying a bunch of business logic/transformation etc and storing that as parquet on s3, then querying that directly using a thin HTTP API wrapper around Polars. Everything is written in Rust. Experience thus far has been great. Initially all of the primary steps (json -> jsonl) were done in Python, but this proved an order of magnitude or two too slow. Rust (with a lot of optimizations... explicit SIMD etc) allowed for huge improvements. As far as querying the data, the biggest surprise thus far has been how performant Polars can be with thoughtful application. We're able to do a lot more with a lot less than initially expected. The cost savings for some of the ETL-type stuff vs doing it with a Spark cluster have also been very significant.
      • Kinrany 4y ago
        Thank you! I was planning to try using DataFusion to query large parquet files without retrieving them from s3 with https://github.com/datafusion-contrib/datafusion-objectstore-s3 https://github.com/datafusion-contrib/datafusion-objectstore.... I guess I should expect that that won't be necessary :)
        • jmartin2683 4y ago
          Datafusion has better support for object_store api (you can just register the store and pass it s3 uris). With polars, you have to have the file on the local file system (which I’ve found to be a huge performance win, cacheing them anyway).
• morelandjs 4y ago
  The skill I value most in a DE is the ability to build effective model deployments and project layouts that are SIMPLE. If I were to switch functions and find some protected time, I’d go off into the woods and build an example deployment compatible with company infra that is as light weight as possible. Then evangelize my team.
• gigatexal 4y ago
  At my workplace once you’re decent with Terraform, a cloud ecosystem or two, SQL and Python the next best thing to do to advance would be to hone soft skills and become a better communicator and manager of people. If you can unlock teams to accomplish things, set vision such that stuff gets done and on time that’s how you become truly valuable.
• khaledh 4y ago
  The most important DE skill you should learn is how to fix data problems when they happen. Data problems include: duplicate rows/events, upstream data errors that need to be corrected, error in your ETL logic, etc. You need to build tools to reason about the graph of data dependencies, i.e. downstream/upstream dependencies of datasets, and create a plan for repair. Datasets need to repaired in layers, starting from the ones closest to the source and working downstream to the leaves. Some datasets are self-healing or not (i.e. they're rebuilt from scratch when their inputs change), so you just let those rebuild. Incremental datasets are the worst to repair because they don't restate history and you have to fork them from the point when a problem happened and rebuild from that point onwards. Without tools to help you repair data, sooner or later you'll run into a problem that will take you days (even weeks) to fix, and your stakeholders will be breathing down your neck.
  • jrib 4y ago
    You're implying it here, but I want to state it even more explicitly: create systems that allow you and your team to know when data problems arise! These can be processes or programmatic but if nothing is looking for problems, then they often go unnoticed until they cause an issue big enough that can't be ignored.
  • hnthrowaway0315 4y ago
    This is going to be a bigger problems when pipelines span across multiple teams within multiple organizations. The only way is to 1) consolidate pipeline design into fewer teams, 2) build monitoring tools to monitor upstream tasks, 3) build gate keeper tools to double check outputs of tasks owned by my team.
• beckingz 4y ago
  What type of Data Engineer do you want to be? We're starting to see an unbundling of data engineering into analytics Engineering (BI developer on steroids), ML engineering (AutoML is good enough that if you can do good feature engineering and pipelines the marginal value of adding a data scientist is not worth the cost, and data platform engineering (K8s. K8s everywhere).
  • thewhitetulip 4y ago
    I am sorry, I did not understand Analytics engineering. do you mean that the BI developer will be expected to do DE stuff? it was how my prev org rolled!!
    • edmundsauto 4y ago
      Analytics engineers are towards the end of data flow - they built data marts and dashboards, often using code.
    • panda888888 4y ago
      Yes, look up the company called dbt to learn more. (I don't work there or anything, it's just a popular tool and their website explains it well.)
      • beckingz 4y ago
        Correct. The idea is to have a role that sits right between data engineering and data analysis, ideally with a breadth that overlaps both.
    • dataminded 4y ago
      Analytics Engineering are folks that transform data in an ELT model. The most popular tool is DBT. I think the work will continue as companies embrace the idea of not writing custom ETL code anymore. The tools are super immature though. DBT is a huge improvement over the past but far from a final solution to the problem.
• tacosbane 4y ago
  Learn SQL in depth, not just select and join. I'm tired of writing SQL for DE.
• Eumenes 4y ago
  Data engineers who can run ML/AI infrastructure get paid good money. If you have kubernetes/docker/SRE experience, even more.
• soumyadeb 4y ago
  As cliche as it may sound, getting close to the "business value" of data might be a good investment. Learning and building use cases, like a product recommendation pipeline for marketing OR a churn model for support to helping marketing ops migrate from GA (GA4) to an in-house analytics stack etc would help bring the data teams to the (much deserved) front row seat in the organization.
  • panda888888 4y ago
    To be honest, that sounds like analytics/BI or data science, not data engineering. Are you suggesting OP pivot their career into one of those roles?
    • soumyadeb 4y ago
      Well, most practical ML algorithms (liner/logistic regressions etc) should be in the wheelhouse of most engineers. I read a comment somewhere "Its much easier to teach data-science to an data-engineer than vice versa" :) I think, if we data-engineers can step up the conversation from frameworks/databases to end-to-end use case, we will get more respect (and everything that comes with it like budget) from the rest of the org.
    • mateo411 4y ago
      I think a good data engineer should incorporate some basic data analysis into their toolkit. This is what analytics/BI developers do all day everyday. Data Engineers need to do this whenever they create a new dataset. It's not hard to pick up this skillset, and this sort of analysis doesn't take a lot of time, and it will make you a really good Data Engineer.
  • codeisawesome 4y ago
    What’s a good authoritative source that tracks use-cases across industries for data? I often find resources that are either too shallow and high-level (literally one-line) or a super deep dive on a singular use case.
• PaulHoule 4y ago
  Like the ETL tools, computer languages are basically the same. It's (usually) more important to learn how to do something new with a programming language than it is to learn how to do all your old things in a new programming language. (The exception to that is that a professional programmer is frequently a maintenance programmer or problem solver and it may well be you have to learn a bit of language X in a hurry to solve a specific problem in a system written in language X.)
• lightbendover 4y ago
  If you're at a huge company, you can up-level as a DE within that company in many ways: (1) understand the hundreds of common ubiquitous datasets in great detail (what they are, their business purpose, how to access them, permissions, team ownership, etc..), (2) moving to a control plane team that works on infrastructure to better understand, (3) experiment with the broad array of ways to solve the same problem (e.g. think through how to optimize for cost instead of maintainability or implementation speed vs. scalability). #3 especially is important because you really level up when you recognize the right pattern to fit the problem you're trying to solve and have direct experience to draw from in solutioning.
• deleted 4y ago
  [deleted]
• phoehne 4y ago
  Math. A surprising number of people in 'data science' lack good understanding of basic statistics, leading to poor data quality or incorrect conclusions from the data.
  • panda888888 4y ago
    Disagree, at least in this instance. Math isn't a bad thing to learn but it's not helpful for data engineering. Data engineering is focused on data integration (data pipelines), not on any type of analysis or data science.
    • phoehne 4y ago
      FYI - I worked as a post-sales consultant for a NoSQL vendor that did a lot of search. This is what seemed the sweet spot for data ingest. ETL was part of what we did, along with search and other applications. I would never have called myself a “data engineer,” but I got to see how customers managed data at a practical level. So take the advice as thou wilt. If all you are talking about is being a programmer that works with ETL/databases, and no analysis of the data, then probably Spring Batch is a good foundation. Java connects to just about everything, performs well, and has good observability. Maybe something Apache Camel/Talend for building ingest/enrichment pipelines. For example, cleaning up addresses, redacting data, etc. They have a lot of off the shelf tools for getting something in quick, but also are easily extended for weird corner cases. Some knowledge of RDF/Ontologies for provenance issues/simple inference. This was especially helpful if we were using a service to disambiguate references to something like Washington, where it could mean George Washington, Washington State, or Washington DC. Python, but the downside of python is it gets its performance from essentially stitching together native libraries in a friendlier way. However, like Java, Python can basically talk to anything. Although I built a python library to interact with our DB for ETL purposes, Java was my go-to.g How to do it in the cloud is something I would also recommend. For a lot of places it makes sense to store large data sets on the cloud, but pricing can kill you. Understanding how to navigate both the options and the pricing and how your workload affects pricing is key. This includes things like if it’s cheaper to create a function to apply some transform during load or just run a batch update later. Also, some of the options today are hybrid and that can also impact pricing when you’re dealing with a lot of data. As a practical example, we had one very, very, very large customer that went from 60 bare-metal on-prem servers on their really sub-par storage array to about a dozen very large server on AWS, which had much better storage performance. For reasons that had nothing to do with engineer, but had to do with CapEx and budgeting, they weren’t going to upgrade their sub-par array. I’m not the first person in the discussion to suggest this, but I strongly encourage you to develop an understanding of vocabulary of your users/customers. For example, in a previous life we had people that worked just with life sciences customers. I got a taste of it when I worked on a POC to ingest PDF submissions for devices at the FDA, extract information from tables and diagrams, enrich them, make them searchable, and track changes between submissions. I had the mechanical end of doing the work. I was shielded by a life-science literate team and was just a ‘hired gun.’
• panda888888 4y ago
  I would focus on theory first, then tools. I recommend the following, in order: 1) Data modeling: Fully read Kimball's book, The Data Warehouse Toolkit, 3rd edition. It's outdated and boring but it's an excellent foundation, so don't skip it. 2) Data modeling: After #1 above, spend at least 30 minutes learning about Inmon's approach, the ER Model, 3NF, etc. You can find some decent YouTube videos. You don't need to learn this deeply, but you should understand how it's different than Kimball's approach. 3) Data warehousing & data lakes: Read the academic paper titled "Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics." For an academic paper, it's surprisingly approachable. 4) If you don't know this already, do 15 minutes of googling to understand ETL vs ELT and the pros and cons of each approach. 5) Putting it all together: Once you've done the things above, then read Kleppman's book, Designing Data-Intensive Applications. 6) Focus on the future: Spend a little time learning about streaming architectures, including batch/microbatch/streaming and pros and cons for each. You may also want to learn about some of the popular tools, but you can easily google those.
• slotrans 4y ago
  > New languages like Rust/Ocaml/Nim.. if yes then which? Completely irrelevant. DE is SQL, Python, sometimes Scala/Java. Get really good at SQL. Learn relational fundamentals. Learn row- and column-store internals. Understand why databases are they way the are. Familiarize yourself with the differences between systems and why you need to care. Get familiar with at least minimal Hadoop shit. Set up Spark from scratch on your machine, mess it up, do it again. Grapple with how absurdly complex it is. Understand system-level architecture. Analytics systems tend to organize into ingest -> transform -> serving (BI tools etc)... why? There are good reasons! Non-analytics data systems have different patterns, but you will see echoes of this one. Above all, spend time understanding the data itself. DE isn't just machinery. Semantics matter. What does the data "know"? What can it never know? Why? What mistakes do we make at the modeling step (often skipped) that result in major data shortcomings, or even permanently corrupt data that can never be salvaged? Talk to people building dashboards. Talk to data end-users. Build some reports end-to-end: understand the business problem, define the metrics, collect the data, compute the metrics, present them. The ultimate teacher. (who am I? no one, just been doing data engineering since 2005, long before it was even a term in the industry)