4 ms·
No, it hasn't. The term has always gone in a half-dozen directions at once, and ranged anything from * an idiot making PPT decks for business presentations ba
by blagie 3y ago
No, it hasn't.
The term has always gone in a half-dozen directions at once, and ranged anything from
* an idiot making PPT decks for business presentations based on sales data; to
* a statistician with very sophisticated mathematical background but minimal programming skills doing things in R or State; to
* a person with a random degree making random dashboard in Tableau; to
* a person with sophisticate background in software engineering, data engineering, and related fields who can kind of do math
* an expert in machine learning (of various calibers)
* a physicist using their quantitative skills to munge data
... and so on. That's been confusing people since the title came out. It depends on the industry, and there's a dozen overlapping titles too, some with well-defined meanings and some varying from company to company (business analyst, data engineering, etc.).
- 7thaccount 3y agoThis is so true. Outside of maybe FAANG companies, a lot of places have wildly different expectations for that role. While one company may refer to the guy doing simple PPTs as a business analyst, others might call that a data analyst or a data scientist or something else. The pay probably mostly reflects the truth though outside of exceptions from office politics.
- gigatexal 3y agoRelatedly - and I’m a lead data engineer at my current $JOB — I’ve yet to find a definition of what a data engineer is/does that I find easy to share with people. Of course I have flippant ones (YAML dev with a bit of Python) but nothing more than: Database Admins who learned Python and now care about more of the data lifecycle than the data that resides in the DBs they managed.
- kordlessagain 3y agoAs a data engineer do you find it your job to transform and clean data? How much AI stuff do you implement that does data transformations?
- gigatexal 3y agoThat’s a good question. I think LLMs will have a place in the connector space. It would be really cool if they could dynamically handle changes in the source (the api changed and added some new data new columns etc). But right now — at least I — don’t trust AI to do much of anything in terms of ingestion. When data is extracted from the source it’s got to be as close to a 1:1 of the source as possible. Any errors introduced will have a snowball effect down the line. For data cleaning we do tend to write the same sort of things over and over. And that’s where I think things could improve. Though what makes a data engineer special in my mind is that they get to know the nuances of data in detail. They get familiar with the columns and their meanings to the business and the expected volume and all sorts of things. And when you get that deeply involved with the data you clearly see where things are jarringly and almost like a vet to a sick animal you write data cleaning things because you care about the data that much.
- zeristor 3y agoAs a Data Engineer I find my job is about finding managing, and deleting PII data.
- gigatexal 3y agoHah. If tou have found a systematic way of doing this please share.
- blagie 3y agoReach out to me in six months, and I should have one up on github if I'm lucky :) (I've had a design in the works for years, and finally should have time and budget to implement it. Probably not helpful for legacy systems, though.)
- gigatexal 3y agoI hope to see this on HN’s show HN! But let me put something in the calendar ;-) There’s no email or other such thing in your bio. :(
- jimbokun 3y agoA joke I read shortly after the term Data Scientist was introduced: Data Scientist - a statistics major living in San Francisco
- collyw 3y agoLets face it job titles are a lot of bullshit. I was a "programmer". I call myself a "software engineer". I probably do better SQL than many data engineers / scientists, which is getting annoying as I am shoehorned into roles where I plug API's together rather than deal with SQL. But Data engineer roles, always want a load of stuff I have never needed to deal with.