6 ms·
(source for everything following: I recently hired entry-level data engineers) The experience required differs dramatically between [semi]structured transactio
by josephmosby 6y ago
(source for everything following: I recently hired entry-level data engineers)
The experience required differs dramatically between [semi]structured transactional data moving into data warehouses versus highly unstructured data that the data engineer has to do a lot of munging on.
If you're working in an environment where the data is mostly structured, you will be primarily working in SQL. A LOT of SQL. You'll also need to know a lot about a particular database stack and how to squeeze it. In this scenario, you're probably going to be thinking a lot about job-scheduling workflows, query optimization, data quality. It is a very operations-heavy workflow. There are a lot of tools available to help make this process easier.
If you're working in a highly unstructured data environment, you're going to be munging a lot of this data yourself. The "operations" focus is still useful, but at the entry level data engineer, you're going to be spending a lot more time thinking about writing parsers and basic jobs. If you're focusing your practice time on writing scripts that move data in Structure A in Place X to Structure B in Place Y, you're setting yourself up for success.
I agree with a few other commentators here that Hadoop/Spark isn't being used a lot in their production environments - but - there are a lot of useful concepts in Hadoop/Spark that are helpful for data engineers to be familiar with. While you might not be using those tools on a day-to-day basis, chances are your hiring manager used them when she was in your position and it will give you an opportunity you know a few tools at a deeper level.
- llbeansandrice 6y ago> I agree with a few other commentators here that Hadoop/Spark isn't being used a lot in their production environments I guess I'm the odd-man out because that's all I've used for this kind of work. Spark, Hive, Hadoop, Scala, Kafka, etc.
- teddyuk 6y agoI'm also the odd one out, so many enterprises moving to spark on databricks.
- josephmosby 6y agoI should have specified more thoroughly. I am not seeing Spark being chosen for new data eng roll-outs. It is still very prevalent in existing environments because it still works well. (used at $lastjob myself) However - I am still seeing a lot of Spark for machine-learning work by data scientists. Distributed ML feels like it is getting split into a different toolkit than distributed DE.
- llbeansandrice 6y agoI guess it depends on what jobs you're looking for. There's a lot of exiting companies/teams (like mine) looking to hire people but we're on the "old stack" using Kafka, Scala, Spark, etc. We don't do any ML stuff but I'm on the pipeline side of it. The data scientists down the line tend to use Hive/SparkSQL/Athena for a lot of work but I'm much less involved with that. Not all jobs are new pasture and I think that's forgotten very frequently.
- markus_zhang 6y agoI'd love to do some Kafka, Scala and Spark. What kind of exp are you looking for?
- dominotw 6y agoAgree 100% with this comment, Old stack: Hadoop, spark, hive, hdfs. New stack: kafka/kinesis, fivetran/stitch/singer, airflow/dagster, dbt/dataform, snowflake/redshift
- disgruntledphd2 6y agoHuh, what replaces Spark in those lists? For my money, its the best distributed ML system out there, so I'd be interested to know what new hotness I'm missing.
- dominotw 6y ago> best distributed ML system out there I was comparing it for "traditional" data engineering stack that used spark for data munging, transformations ect. I don't have much insight into ML systems or how spark fits there. Not all data teams are building 'ml systems' though. Parent comment wasn't referring to any 'ml systems', not sure why that would be automatically inferred when someone mentions data stack .
- disgruntledphd2 6y agoYeah, I suppose. I kinda think that distributed SQL is a mostly commoditised space, and wondered what replaced Spark for distributed training. For context, I'm a DS who's spent far too much time not being able to run useful models because of hardware limitations, and a Spark cluster is incredibly good for that. Additionally, I'd argue in favour of Spark even for ETL, as the ability to write (and test!) complicated SQL queries in R, Python and Scala was super, super transformative. We don't really use Spark at my current place, and every time I write Snowflake (which is great, to be fair), I'm reminded of the inherent limitations of SQL and how wonderful Spark SQL was. I'm weird though, to be fair.
- sails 6y agoSnowflake I suppose for the average ML use case. Not for your high-performance ML, but for your average data scientist, maybe? Edit: I may be wrong[1], would be curious to know what users who've used Spark AND Snowflake would add to the conversation. [1] https://www.snowflake.com/blog/snowflake-and-spark-part-1-why-spark/ https://www.snowflake.com/blog/snowflake-and-spark-part-1-wh...
- alexpetralia 6y agoGreat points. It depends on where the business is at, the scale of their data, how processed their data is, and the timeliness/accuracy requirements of that data.
- Mauricebranagh 6y agoIt does rather depends what sort of data I bet a data engineer at CERN or JPL has quiet a different set of required skills to say Google or a company playing at data science because its the next big thing. I should imagine at CERN etc knowing which end of soldering iron gets hot might still be required in some cases. I recall back in the mumble extracting data from b&w film shot with a high speed camera, by projecting it on to graph paper taped to the wall and manualy marking the position of the "object"
- pjmlp 6y agoWhen I was there, almost 20 years ago, it was all about C++, Python and Fortran, with GUIs being done in Java Swing. I bet it is still mostly the same, just using Web GUIs nowadays.
- valenterry 6y agoWe yet have to wait for the proper sweet spot: a language that allows SQL-like handling without the restrictions of SQL. As many advantages as SQL has, in many cases it gets into the way. The closer you move to moving data (instead of doing analysis), the more it becomes annoying. On the other hand, current languages (such as python) lack support when it comes to data transformations. Even Scala, which is one of the better languages for this, has severe drawbacks compared to SQL. Hopefully better type-systems will help us out in the long term, in particular those with dependent types or similar power to describe data relations.
- andkenneth 6y agoWhat's your opinion of LINQ in C#? It's been a while since I've used it but to me it seems like one of the most powerful ways to manipulate data inside of a language.
- valenterry 6y agoI haven't used LINQ but I have a good idea about how it works. It is certainly great to write concrete SQL-like transformations (be it based on a database, a list, ...). Where it lacks is abstraction. To make that more concrete, let me ask: can you write LINQ that takes an arbitrary structure and selects every numeric field and calculate the sum over it? And if that is not possible, it should not compile. I.e. can you define a function "numericsSum(...)" and call it with "List(User(salary: Number, deductions: Number, name: String))" and have it calcuate the sum (ignoring the name-field) but having it fail to compile when calling with "List(User(name: String, date_of_birth: Date))"? Another example: is it possible to create your own special "join" function, that joins to data structures if they have exactly one common field (which you don't have to specify)? In both examples, the LINQ compiler must be able to inspect the structure of a type (such as User) at compile time (not runtime) and do pretty arbitrary calculations. Most languages don't support that and I think even LINQ only works with concrete types in that sense. Which, by the way, is already better than what most languages offer - don't get me wrong here. But it is not as powerful as what SQL "compilers" offer - however those are then limited to SQL only, lacking the ability to do general purpose programming.