28 ms·

We don't need data scientists, we need data engineers

• avs733 6y ago
  I teach engineers for a living. I struggle to see how this is not just a straw man argument based on colloquial usage of terms. It is just inferences drawn based on job ads that are rarely written by people doing the job and instead are effectively human-as-seo-optimized so the best candidates can find the job they hopefully fit for and not be too confused to apply for it.
  • hn_throwaway_99 6y ago
    It's not a straw man, I've seen it clear as day in several companies. When it comes to data science, it's "garbage in, garbage out". I've seen companies do lots of "data science" with a bunch of data scientists skilled in python and jupyter notebooks, only to discover a ton of work was useless because the incoming event data was tagged incorrectly due to a bug. The actual process of collecting, aggregating, cleaning and verifying data is a hugely important skill, and not one I've really seen typical data scientists possess.
    • avs733 6y ago
      I have experienced the same thing...but I just don't think it has anything to do with whether the positions are labeled data scientist or data engineer. And I would warn you from my experience teaching statistics to undergraduate engineers...they are not going to be much better. Regularly get 'hey we have this data what test can we run?' 'what are you trying to show?' 'we don't care we just need to run a statistical test' conversations.
      • hn_throwaway_99 6y ago
        To be clear, I totally agree with you. I wasn't just arguing for changing labels, I was arguing that there is one set of "engineering" focused skills (e.g. building data pipelines, data warehouses, tagging events, etc.) and a different set of analysis skills (e.g. machine learning, statistical tests, etc.) and you shouldn't over-index on the latter without having enough of the former.
    • finnthehuman 6y ago
      >The actual process of collecting, aggregating, cleaning and verifying data is a hugely important skill, and not one I've really seen typical data scientists possess. Then they are not scientists. They have a label "scientist" but lack of rigor of actual science. I don't see why changing the label to "engineer" would suddenly make them have rigor.
      • avs733 6y ago
        Right?! This is sort of the meta failure of the argument. They are arguing that people's data skillsets are wrong. To make that argument they are analyzing based on the wrong variable in a data set.
      • deleted 6y ago
        [deleted]
    • bonniemuffin 6y ago
      I suspect this may actually be an issue of school vs real world rather than scientist vs engineer. Data in the classroom setting is pristine and beautiful; data in the real world is messy and buggy. You have to get burned by buggy data a few times (or maybe a bunch of times) in the real world to learn to look for bad data smells -- I don't think schools effectively teach this kind of intuition, regardless of whether the students are training as data engineers or data scientists. If data scientists are spending more time in school getting advanced degrees, they're not getting as much exposure to buggy data, whereas data engineers with a BS and a few years of industry experience would already have built up this skill.
      • avs733 6y ago
        >Data in the classroom setting is pristine and beautiful; data in the real world is messy and buggy. I got to take over our department's undergraduate statistics course a few years back. The first change I made was all homework, tests, and projects used real data set. I intentionally have them collect bad data (they don't know its bad before hand). First day of class we collect data using the board game operation...I give basic instructions and then halfway through ask everyone to stop and agree on how they are entering data for the variable of 'success or failure' of the surgery. Oops... In my experience teaching the course, the reason the students (engineers) find statistical reasoning hard is: * They have never been given anything 'broken', everything is curated to avoid things not working. The result is they think data has inherent meaning. A right answer. * Their entire learning experience has been stripped of context and the need to make decisions with information. They can give me a p value but are terrified (not unable, just unwilling) to interpret it or give it meaning. * They have never encountered the concept of variability...everything is presented as systems with exact inputs and outputs. When I work with postdocs, I sometimes (less frequently) encounter many of the same challenges. Data is treated as sacred and external and inherent. It's wild to me.
  • jariel 6y ago
    So I think that the delineation between the scientist working with the content, and the Engineers who actually provide the mechanics for it is very fair. If there is a question mark here - it's really how much value are we deriving from all of these data people? Where is all the ML that's changing our lives? Search, Alexa and TikTok, I can see it. In the future obviously vision systems for autonomous cars etc.. But I'm really wary about the heavily decreasing marginal returns after that. It will surely change the world, but I think in specific areas. Most of the entire field seems like an optimization on something rather than anything new. Washing Machines feed up immense amount of labour and toil. Alexa telling me the weather is not.
    • avs733 6y ago
      Most engineering and science jobs aren't a binary as much as they are a spectrum. If the article is trying to make a point about skill development and diversification, I'm totally on board. Bifurcating the roles instead is going to be less effective. To the value point...my sense has been we are seeing the Webcommerce 1.0 bubble Machine Learning edition. Lots of uses of it, not all of them have value. I am excited for where we will be in 10 or 15 years, but I suspect the difference will be huge. If you put me to a guess, I would say better data handling practices and ethics will likely be the linchpins of value creation vs. using tools for the sake of tools.
    • serjester 6y ago
      I used to work at a legacy automaker and you’d be shocked at how much ML has changed certain areas of the business. It used to take an entire department to sort warranty claims and it’s now mostly automated. Aluminum part defects are now spotted automatically on the plant floor. Don’t even get me started with telematics data. Most software isn’t consumer facing but just because you don’t see it doesn’t mean it’s not changing things around you. ML tends to be overhyped but your assessment is too pessimistic.
      • mateo411 6y ago
        I wouldn't think of a system that would automate the processing of warranty claims as ML. That's mostly applying the policy/rules to each claim. However, finding defects in aluminum parts that involves using computer vision, would absolutely be a ML solution.
  • antipaul 6y ago
    The article is so true, my latest mantra at work is “engineering is more important than data science”. Everyone is buzzing about the latter, and few even realize what is the former.
    • avs733 6y ago
      eh...I think this can be analogized to what we already see in code... You need architecture, you need backends, you need a front end, you need product design...all with data. Why are computer scientists computer scientists not engineers? Why is computer science about the code side? Why did computer engineering end up being more on the hardware end of the spectrum? Words, especially newly coined terms are pointers to meaning. That meaning is socially mediated, it is not inherent. You're saying this (adn I think the author is too) because there is a need for this group of people to look beyond titles to skillsets, and the existing titles carry linguistic baggage of the difference between science and engineering that has existed for decades.
• AndrewKemendo 6y ago
  Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effectively 3. Build ETL pipelines and data management systems 4. Manage enterprise data sharing and resiliency etc... What ends up happening is you have a bunch of Data Scientists just handed a pg_dump or flat file from some ops team. That is typically missing data or poorly formatted and they spend 90% of their time cleaning it up then running some basic regression with numpy or whatever. Need better understanding of the data lifecycle by organizations and investment in instrumentation and data management.
  • mgh2 6y ago
    Who will do the proper cleaning then?
    • alex_anglin 6y ago
      The aspiration that GP was getting to was that less cleaning is required as a result of better data engineering, I believe.
      • AndrewKemendo 6y ago
        Correct. If you build your instrumentation correctly, then you don't really need to do any "cleaning." Doesn't mean you might not need to do transformation for different uses but ideally wouldn't need to, for example change data types like turning a bool into an int.
        • dpayonk 6y ago
          If we only had to use 1st party data, that might be easier. But then again, if you’re building your product incrementally, you’re still going to have instrumentation holes that you may or may not be able to partially backfill.
• amelius 6y ago
  In other words: we need plumbers.
  • digitalsushi 6y ago
    Or, if the data is food, we have chefs making incredible plates, but we need wait staff to get it to the people who want to eat it. I would love to be on that wait staff; as an infradev I feel like the process is very close to me but I am struggling to break into it.
    • spacemanmatt 6y ago
      I would put data engineering on the supply side of the chef: This would be ingredients, delivery scheduling, and pre-prep functions. That sort of thing.
  • gregw2 6y ago
    As a data engineer I've made the same joke. But the statement is also a bit like saying you can use plumbers to design and build a chemical refinement plant which also just moves chemicals from point A to point B. Or you can design a citywide sewer system with a bunch of plumbers. There are many cleansing, refining, orchestration, dependency, data quality, governance and optimization problems to be solved and a wide variety of tools that have for whatever reason never grown into higher-level open source frameworks and are thus reimplemented in various forms in many places. Data engineering (somewhat like software engineering actually) doesn't require much if any of the math and physics I took in engineering courses in college, but it does require rigorous systems thinking about how to design and build structures that withstand adverse conditions that are thought patterns common to other engineering practices, so I don't think it's a totally crazy title for the role.
• zeku 6y ago
  I'm a data engineer for most of my day right now, and a lot of it is done with ruby/python/shell scripts into postgres DBs. What learning path should I go down? I'm a solo actor at work with a lot of agency to decide my workflows. I see myself building small to medium size data collections over the next year or two at my job. Can someone point me to some learning? I have a CS degree etc. and my title in software engineer etc. etc. End users of my data usually like their data as a CSV that is then read using R or Python. However there is also a use case where I will build an app to view my data in a simple way. All of this is completely doable with my current knowledge/workflow but I can't help feel like I do a data engineering job with very different tools than i see "data engineers" speaking about online.
  • langitbiru 6y ago
    A couple of days ago, there was a thread about "How to become a data engineer in 2021": https://news.ycombinator.com/item?id=25728198 https://news.ycombinator.com/item?id=25728198
    • zeku 6y ago
      thanks!
  • ageofwant 6y ago
    Adopt a good Workflow tool, like apache airflow. Easiest is to rent a service from AWS https://aws.amazon.com/managed-workflows-for-apache-airflow/ https://aws.amazon.com/managed-workflows-for-apache-airflow/
    • zeku 6y ago
      thanks!
    • mywittyname 6y ago
      Seconding this, Apache Airflow is awesome. I can't believe how much time is saves during development.
• mywittyname 6y ago
  I always felt that tech-focused data scientists should also be required to know how process data end-to-end; at minimum, from a SQL database to deployed model, but knowing how to collect & clean data is important too. It seems like the industry is trying fill the gap that was created by a glut of people without math/cs backgrounds going into 5-week data science courses who then need hand-holding when they get real jobs. Data science & engineering should be treated as a single collection of skill-sets. Lacking ETL experience is a major deficit, considering how prevalent that kind of work is. This might just be my personal biases coming through. I consider myself a "full-stack" data scientist & engineer. But because data scientists who can work on the backends are rare, I always end up doing the plumbing while other people do the fun analysis work. I think companies that are data "science" heavy are going to be at huge disadvantage soon. Tools like Rekognition and Google AI APIs are making the model training & deployment aspect almost trivial. At some point, the only real work involved in this space will be the data "engineering."
  • superbcarrot 6y ago
    > Data science & engineering should be treated as a single collection of skill-sets. This can be tough because there could be a lot in that skill set. You can't realistically expect someone to have solid knowledge of statistics including specialising in the sub-field and type of algorithms that your product needs, and also be able to write good code and act as a developer, and also have solid knowledge of all the tools for data streaming/processing/ETL. There is a point at which you're just stretching yourself too thin if you try to do all of these at once. Of course, stuff like knowing how to interact with a database or employing good software development practices should be a very basic prerequisite and some scientists certainly shift things too far in the other direction and use their academic knowledge as an excuse to write poor code and not learn new tools. I guess what I'm trying to say is that they are distinct skills but you still need all of them to some extent and striking the correct balance in one's skillset is really difficult.
    • mywittyname 6y ago
      These are all skills taught in standard computer science programs. Granted, some are electives, like high-level stats. But even back in 2010, data science electives were available to fill the gaps. I took three DS&E classes in college with projects that were end-to-end platforms, where you'd have to collect, clean, and analyze the data, then build, test, and deploy models from it. I would certainly hope that college courses are even more comprehensive after 10 years and an explosion in interest for the field. Also, much like being a full stack developer, a full stack data engineer doesn't need to know everything at a master level. But that you can at least handle tasks at most points in the chain.
• basseq 6y ago
  The problem that I've seen is often "data scientists" are expected to be the equivalent of full-stack engineers (or maybe more accurately: one-man CTO shops)—to understand data architecture, understand business architecture, ensure data quality, build data into product, build dashboards, derive insights, posit hypotheses, set strategy, and drive business value. Thus many "data scientists" are juiced-up report-builders who can't analyze their way out of a paper bag.
  • nerdponx 6y ago
    This is uncharitable. In my experience, this is true: > "data scientists" are expected to be the equivalent of full-stack engineers (or maybe more accurately: one-man CTO shops)—to understand data architecture, understand business architecture, ensure data quality, build data into product, build dashboards, derive insights, posit hypotheses, set strategy, and drive business value. But this is not: > Thus many "data scientists" are juiced-up report-builders who can't analyze their way out of a paper bag. Rather, the data scientists are trained in only two of the requirements you mentioned: derive insights, posit hypotheses. The rest is all self-study and on-the-job experience. This means that we are putting unrealistic expectations on data scientist and/or their training is insufficient, not that data scientists are somehow morons.
    • basseq 6y ago
      Now it's my turn to claim uncharitibility. Indeed, my context here is that people who wear the data scientist title come from multiple backgrounds, and are often asked to wear too many hats. They are non morons—they may be darn good report-builders, but haven't been trained in insights, for instance. If you're reacting to my word choice in that last sentence, know that I am frustrated with people who claim to be data scientists but can't derive insight. (And we can argue about "many".) But that's not a broad denouncement against all data scientists, either.
  • kevin_thibedeau 6y ago
    It's the new CIS.
• superbcarrot 6y ago
  The overall point that there is demand for data engineering skills seems valid but in reality you don't have to pick between data scientists and data engineers, it's not one or the other. I don't know if the argument is set up this way just to get more clicks or to encourage arguments but it would have been better to just focus on the overall state of the market and the demand for certain skills.
• C4stor 6y ago
  I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), and I feel like it gave me such an incredible foundation for future roles : - Actually big data (so, not something you could grep...) will trigger your code in every possible way. You quickly learn that with trillions of input, the probabily to reach a bug is either 0% or 100%. In turn, you quickly learn to write good tests. - You will learn distributed processing at a macro level, which in turn enlighten your thinking at a micro level. For example, even though the order of magnitudes are different, hitting data over network versus on disk is very much like hitting data on disk versus in cache. Except that when the difference ends up being in hours or days, you become much more sensible to that, so it's good training for your thoughts. - Data engineering is full of product decisions. What's often called data "cleaning" is in fact one of the import product decisions made in a company, and a data engineer will be consistently exposed to his company product, which I think makes for great personal development - Data engineering is fascinating. In adtech for example, logs of where ads are displayed are an unfiltered window on the rest of humanity, for the better or the worse. But it definitely expands your views on what the "average" person actually does on its computer (spoiler : it's mainly watching porn...), and challenges quite a bit what you might think is "normal" - You'll be plumbing technologies from all over the web, which might or might not be good news for you. So yeah, data engineering is great ! It's not harder than other specialties for developers, but imo, it's one of the fun ones !
  • secondcoming 6y ago
    Indeed, adtech is a great place to work for anyone interesting in working with data. And yes, people working in adtech hate, and block, ads too.
  • alexpetralia 6y ago
    The other thing I'd emphasize here is dealing with "state". Data is effectively state. As application engineers build increasingly "stateless" code (e.g. pure functions, serverless deployments, etc), that state gets pushed elsewhere. Someone has to manage the queues, file versions/locations, logs, databases, configurations and so on. That is all "data". State management is a tricky problem even in a single-threaded application. It's doubly so in distributed systems, where state can be inconsistent between all the moving pieces. This is the source of endless data integrity issues. I think data engineering is a great way to get some exposure to all of this.
• coding123 6y ago
  A couple of us inherited a machine learning project a while back. The code was horrible. Riddled with copy pasta (nearly half of the entire thing was copy paste and no code reuse). We basically refactored everything, standardized input and output file names. We put up a small Flask service to allow outside services hit it easily and wrapped it up in a Docker container so it was ultimately easy to deploy. Yes it was all the plumbing. However we also looked at the code, and the ML strategies, and while there was "some" level of competence, it was nothing more than word2vec add and divide. Totally horrible for actually finding key phrases that matter to the subject we're matching. So we started tackling that too with LSTM but our time got cut short and shifted off to another area. So not only was the "scientist" they hired completely crappy at the engineering, they weren't really helpful in the ML either. This is obviously of lesser value to the topic at hand, and more about making sure you hire good people I think.
  • mlthoughts2018 6y ago
    I am curious what your take is on things like this article: https://managingml.substack.com/p/the-myth-that-machine-learning-engineers https://managingml.substack.com/p/the-myth-that-machine-lear... It has been my experience too. Basically, ML / DS engineers are thrown under the bus for being poor general software engineers, but in practice it’s totally the opposite.
    • nerdponx 6y ago
      The problem is that ML engineers are not the people who wrote GP's garbage code. Data scientists wrote it, and I know at least a few of my very intelligent, high-functioning data scientist colleagues who are alarmingly, astoundingly bad programmers.
    • coding123 6y ago
      For me it's just the one experience since I haven't had any other interactions with an ML / DS person since.
  • wpietri 6y ago
    The phrase "If you can't dazzle them with brilliance, baffle them with bullshit" comes to mind.
• jimsparkman 6y ago
  Keeping in mind DE can mean different things at different companies, I spend a lot of time working on infrastructural components to just get at data reliably. Working in a product company with disparate generators of data, I’m often building out network connectivity (VPC peering, VPNs, etc.), subnets, ACLs, firewalls and load balancers across our visualization tools, managing job flows, controlling AWS costs, building read replicas for production databases, yadda yadda. There might be a ton of hoops to jump through before I can even start to process data and it’s the type of work that wouldn’t make sense to hand off to my DS counterparts.
• ziml77 6y ago
  Having to deal with data scientists, I absolutely agree. The thing that I've seen that lands in the "lab" vs production distinction is that these people expect their data to be pristine. They flip out when the world isn't as perfect as their models want. Leads to me as just a normal software developer having to do the data analysis and figure out how to clean it up. I also end up having to be the one to talk to data vendors to understand their data feeds and essentially translate that for the data scientists. Having to sit in the middle is annoying for me and suboptimal for the business.
  • superbcarrot 6y ago
    I think that this is more of a problem with the specific people that you have worked with and it isn't inherent to the role of a data scientist.
    • ct0 6y ago
      Doesn't sound like a modern Data Scientist, sounds more like a statistician with 30+ years of experience.
    • sidlls 6y ago
      It’s becoming more inherent, especially as the field is populated with people who have no experience with the “science” part. That is, with the very real and ubiquitous problem of collecting and cleaning data to make it fit for scientific study. Even theoretical physicists, for example, participate in and rely on empirical data collection, and understand deeply how messy and fraught with error it is. I don’t see the same appreciation or consideration in general in the field of data “science.”
      • superbcarrot 6y ago
        > with people who have no experience with the “science” part It's interesting that you put it that way because a lot of the other complaints in this thread are that the people who expect their data to be ready for use are exactly the people with science experience but without the relevant technical background.
• slt2021 6y ago
  Full Stack Data Scientist (data janitor + data engineer + ML engineer + ML Ops + Business Analyst) is the future
  • alexpetralia 6y ago
    These are incredibly disparate skill sets. Of course anyone would want to hire someone like this, and far more would claim to possess such a broad skill set, but in practice it is extremely rare. You'd need someone with excellent communication skills (presentation, memo writing, teamwork), project management skills (identifying & overcoming workflow bottlenecks), professional skills (timely responses, political savvy), technical skills (application programming, advanced databases, advanced machine learning, Excel modeling) and finally some business domain knowledge. This is an uncommon intersection of skills.
    • nerdponx 6y ago
      > You'd need someone with excellent communication skills (presentation, memo writing, teamwork), project management skills (identifying & overcoming workflow bottlenecks), professional skills (timely responses, political savvy), technical skills (application programming, advanced databases, advanced machine learning, Excel modeling) and finally some business domain knowledge. This is pretty much the bare minimum requirement for any data scientist job I've ever interviewed for or held.
      • alexpetralia 6y ago
        In my experience, software engineers do not make good business analysts (and data/machine learning engineering is a subset of software engineering). Most business analysts cannot program. However, it's likely that our experiences simply diverge here.
        • nerdponx 6y ago
          I'm talking specifically about data science, not business analyst or software engineer.
• deleted 6y ago
  [deleted]
  • nerdponx 6y ago
    This hits me at a personal level. This is how I imagine programmers must have felt in the 80s and 90s.
• glitchc 6y ago
  Good data is hard. Anyone conducting research in the physical sciences knows this firsthand. It takes painstaking effort to conduct carefully controlled experiments and collect a batch of good data that could then be used for analysis. The promise of ML has always been to churn out good results from not-so-good data. If I now need to sanitize my data carefully, what's the advantage?
  • claytonjy 6y ago
    What makes you think that was the promise? I would say the promise has been to replace code with data, and to build things with data that would be practically impossible to build with code. "Garbage in, garbage out" has been the mantra for a long time. Sure, there are tools and techniques to deal with not-so-good data, but those are add-ons, not a core part of the value proposition of ML.
• Wonnk13 6y ago
  What's the career trajectory for data engineers? I enjoy the pipeline building and business stakeholder interfacing, but I'm not sure I want to be a SWE a decade from now...
  • claytonjy 6y ago
    How is that different from a SWE? I see DE as a specialized SWE; tons of overlap, but DEs focus on different tools and concepts than other SWEs.
• tristor 6y ago
  Agreed. In my decently long career the types of data problems I've seen be most impactful on the business are not head-in-the-clouds ML issues, but more mundane yet more far-reaching: 1. Appropriately identifying what data needs to be captured from a product to correctly operationalize it. 2. Understanding and modeling data structures in internal applications to identify and tune backend data storage mechanisms (including DBMS). Inclusive in this is helping the application development team pick the correct structure and implement it correctly. 3. Validating implementation of instrumentation within the application so that data cleaning isn't necessary and that telemetry can be appropriately reported on. Building said reports. 4. Doing ETL and taking care of out of band data management to link disparate systems within the business to help build holistic views of the business overall. 5. Be a safeguard against the over-collection of data, because data engineers understand that data isn't an asset, it's a liability that increase costs and risks as a business or product scales, and when there's not a specific need that can be articulated clearly for that data, collecting it is a user/customer-hostile action. My experience has been that data is a crucial element to understand the health and state of the business with both breadth and depth at a given point in time and identify trends. However, it's mostly used by folks in management as a crutch to try to de-risk decision making, or worse as a political tool to give a faux support to a decision that's already been made but not yet publicized. Decisions carry inherent risk, including the decision to do nothing, you cannot eliminate this, it's one of the components of decision trade-offs. This sort of broken use of data by management is supported by "Data Scientists" that see the field as a cash-cow they can milk while they work on pie-in-the-sky ML strategies which are often unnecessary, even when they actually work. Done correctly a strong data culture in a company can increase decision velocity, empower engineers, and reduce overhead on management to understand the business. Done improperly, data culture in a business can easily destroy decision velocity, empower dysfunctional politics, and increase engineering overhead to understand systems. Getting it right is the main test for businesses in the new era.
  • lamename 6y ago
    This sounds reasonable. The trick is to identify companies/cultures of each type at interview time. But is that even possible?
  • 6y ago
• hehehaha 6y ago
  I want to live in a world where data scientists making nearly $500K can understand and correctly implement simple concepts such as fixed effects. Is that asking for too much?
  • beckingz 6y ago
    Sure, but what about at $80k? None of this discussion is helped by the fact that companies want to get into data science without actually having much data strategy.
• thecolorgreen 6y ago
  I'm a SWE and data engineering actually sounds super interesting to me. Unfortunately, my day-to-day doesn't provide opportunities to work with the massive amounts of data we generate. I've looked into learning this stuff online but courses like DataCamp seem too basic (I have experience with Python and data cleaning in a research setting along with some academic ML experience) or downright a bit scammy. Many of the articles I read online about this also frame data engineering as a way to transition into the tech industry, which isn't my blocker. Does anyone have advice to help me transition away from pure software to a job as a data engineer?
  • nojito 6y ago
    We just hired a SWE turned Data Engineer. You don't need to handle massive data to make the transition. I believe all the person did was build a small but robust pipeline that took some API response data, cleaned it, populated some sqlite dbs, replicated it for a small team to use and kept it updated every few days automatically.
    • thecolorgreen 6y ago
      That's good news for me. For a minute, I genuinely thought I'd have to waste time going through a bunch of stuff I already knew to learn a tiny bit and earn a certificate to prove my skills.
• ska 6y ago
  In many cases, what is needed isn't even more data engineers; it's data janitors.
• jesseryoung 6y ago
  4 years ago I moved from a role where I primarily wrote C# as an architect on a web application, to an architect helping to build a data warehouse. The contrast in tooling, discipline and information available to build anything in the data world is so stark it had me questioning my career decisions. Sure, you can read Kimball and Inmon and I'm sure there are a handful of others out there - but there are drastically fewer than what you can find in the application development space. Things are getting better, Visual ETL tools are falling out of favor to proper coded ETL (spark, dbt, etc) and data teams are starting to see the value of actually engineering a solution instead of just throwing it over the wall to a DBA to deal with. But tooling, and general information on the web is still lacking. Pushing data engineers over "etl developers" or "bi developers" (or "data scientists") will drastically improve any organizations ability to actually deliver real analytics and hopefully an industry wide push will raise all ships.
  • Avalaxy 6y ago
    Why do you think that coded ETL is winning over the click-and-drag variant? I'd say the latter makes things a lot easier no?
    • beckingz 6y ago
      Advances in ETL. Spark and DBT are large improvements over pre 2010 ETL tools. Give it a few years and we'll see really good GUIs for Spark/DBT.
    • jesseryoung 6y ago
      My bias has always been against click-and-drag programming, and I believe it mostly comes from my application developer background as the sentiment towards visual style application development tools is (almost) unanimously negative. Coming over to the data world, I noticed the same type of problems click-and-drag app development had appearing in tools like IBM's DataStage and Informatica's Powercenter. There's only so much you can do by dragging and dropping items on a screen, eventually you need to take their respective escape hatches and do some programming - and when you do it's almost never ideal. I've also yet to see a visual coding tool produce readable concise diffs in any source control provider. Most of these tools also require some sort of centralized server infrastructure and a thick client making it so much more challenging to bootstrap new ETL developers. I do hear others in the data world who have migrated to Spark or DBT share the same sentiments - but that could just be confirmation bias.
• program-iscuous 6y ago
  That's interesting post, however there's large bias in my opinion in how the analysis is done. You have a stratified sample of companies in their early stages, I think it's quite normal for most companies in their early stages to prioritize data engineers rather than data scientists. Data scientist comes after the data engineer, and if you have a data scientist and not a data engineer then probably the data scientist does both jobs. On the other hand, data engineer is not dependent on a data scientist. To conclude, I think that indeed there are more data engineer positions because there are too many "data scientists", however the true difference is not as large as in your analysis.
• mongol 6y ago
  Why is data scientist a profession in IT but not for example computer scientist? Many IT professionals studied computer science but they don't call themselves scientists in their line of work.
  • eeZah7Ux 6y ago
    Don't give developer ideas. People with a javascript bootcamp and 2 years experience are already called "senior engineer".
    • collyw 6y ago
      We have a PhD computer scientist working as as software engineer. Reinvents the wheel constantly. Over engineers everything. Clever algorithms but poor abstractions. Needs to have the "best" (from a CS perspective) solution to everything rather than the most maintainable or pragmatic solution. He is smart, but doesn't write code that is easy for the next developer to pick up.
  • zzbzq 6y ago
    Basically this manager guy had like a dozen job titles under him, such as Business Analyst, Data Analyst, ML Engineer, and sundry more. Then his HR team came to him and told him to just boil it down to 1. So he made up this title "Data Scientist" because it sounded badass, and then everyone from both the business analytics side and the software engineering side decided this sounds awesome and they want it to be the next step of their career. Then the federal government came up with this title something like "Secretary of Data Scientist" and the guy who made up the term worked at that.
  • globular-toast 6y ago
    "Computer science" is a well-known misnomer as it is not about computers nor is it science. It's generally reserved for the academic study of computing. Data science is much closer to science.
• spacemanmatt 6y ago
  Data engineer, here. Or at least, that has been my title a couple of times. Some data is inherently trash but a huge part of the data quality problem is sources who are allowed to produce trash that everyone else has to clean up, when it would be way more efficient for them to quit producing trash. Not to pick on any one institution, but SOAP seems to be a read flag that the service will also deliver some screwy data.
  • jaegerpicker 6y ago
    Every time I see SOAP in a ticket/task I die a little inside. Without fail every SOAP related project I've dealt with in the last 10 years has been a shit show. I've actually and unfortunately gotten pretty good at dealing with it but I so hate it.
• afryer 6y ago
  I recommend the book: Agile Data Science by Russell Jurney[1]. The tech stack is circa 2017, but the chapters on the Agile Data Science Process and Teams are timeless. He clearly articulates team roles: from Biz Devs, marketers, PMs, UX designers, UI designers, Web Developers, API Engineers, Data scientists, Applied researchers, Platform/Data engineers, QA engineers, DevOps Engineers. Then he talks about different ways to increase agility by combining these roles into generalists empowered to iteratively explore the "pyramid of data value" until the right product-market fit is found. Building Data-science Intensive Web Applications is inherently waterfall, not agile, and I find this book to be a fascinating reference. [1] https://www.oreilly.com/library/view/agile-data-science/9781491960103/ https://www.oreilly.com/library/view/agile-data-science/9781...
• shrumm 6y ago
  this post is so timely. How do you guys handle hiring? I’m looking to hire more data engineers (Singapore only for now), especially senior ones and it’s not easy. My gut is to just find good engineers who have good first principles and let them learn on the job. Feedback/experiences welcome!
  • yaster 6y ago
    Curious to hear this from the other side. I'm an SE manager interested in pivoting to Data Engineering. I've built modest pipelines in R / Postgres for an operations analyst job I did before I became a programmer, but I'm not experienced with most of the technologies I see listed on DE roles (e.g. Airflow) and my statistics etc. are rusty.
    • shrumm 6y ago
      I see these experiences as nice-to haves. It just means you'll be able to hit the ground faster. Personally, I'd value someone who shows they can learn quickly. That said, it's trivial these days to spin up a few VM's, install Airflow and try some practice scenarios just to get your hands dirty.
• sbpayne 6y ago
  +1. I generally say that data scientists bring an amazing skillet to the table, but companies can only leverage 10% of it.
• Ansil849 6y ago
  Genuine question: why is there so much pure teeming hatred for data scientists in this comment thread? Almost every comment comes off as full of snark and vitriol against data scientists.
  • astrophysician 6y ago
    I'm guessing just venting personal frustrations due to their own experiences, plus maybe poor hiring and guidance of data scientists in their own teams? I have definitely seen DS people in my experience that fit some of the descriptions here, but I think it's a mistake to trivialize the DS position itself. A good DS is a valuable asset, but depending on your company/data, maybe not worth the cost. Plus there is no "single" DS candidate or role, a lot of these roles (data engineer, DS, analyst, swe) blend together at times, and it's about finding the right balance of skills. Sometimes I think a company (not having the DS experience themselves) mistakenly over-hire DS roles in today's hype of "AI" when their data is mostly run-of-the-mill and only requires simple linear models that can be architected an understood by a stats/math-savvy engineer. Even then, a good DS is still useful (even linear models can be complex: e.g. what priors do you want to use? Do you want a multi-task solution? etc.), but maybe not worth the cost.
  • bartleby_ 6y ago
    Probably just people blowing off steam and the target de jour are data scientists. If people had to sit down with their company's data scientists to air their grievances face to face I doubt they would be so condescending.
  • danbrooks 6y ago
    I would guess that it's a reaction to job title hype. There's a huge variety in DS responsibility and background between companies.
  • andrewnc 6y ago
    I came to this thread interested in the discussion, but I feel now like the homer simpson meme retreating into the hedge. Maybe I'll come back in a few hours, but for now I'll stay away.
  • 6y ago
• giantg2 6y ago
  I don't find it too surprising. I think more people would want to work with the conceptual work in the data science space, like models. The ETL stuff seems extremely boring and it seems it pays less too.
• analog31 6y ago
  Disclosure: I'm a just plain scientist, without the data. ;-) Since "data science" appeared on my radar (in other words, on HN), I've noticed that we fling the "scientist" and "engineer" terms around without asking whether the practitioners have science or engineering backgrounds, or something else such statistics, math, programming, etc. It strikes me that "do we need scientists or engineers" is not unique to data science/engineering. I think we need both, but of course it's an open question as to how many of each are needed. Also, both "scientist" and "engineer" are loosely defined in practice, with some overlap. Painting with a broad brush, a scientist wants to learn how things work. An engineer wants to make things that work. If you're in the business of making things that work in order to sell them, then you need lots of engineers, but maybe a few scientists. It's about 10:1 at my workplace. Overlap? Of course. Engineers use scientific knowledge and methodology. Scientists have to make things work in order to make experiments and theoretical computations work. Also, scientists often show up in emerging fields before they become recognizable engineering disciplines, for many reasons: 1) We need the newest stuff, right away. 2) We are opportunists by nature and necessity. For instance many of the older "software engineers" at my workplace have science degrees, but the younger ones all have computer science degrees. The oldest people I know who did programming, digital logic, and embedded systems, did not have degrees in those areas. The youngest ones all do. At my college, the physics professors had personal computers, and were ripping them apart, while the CS professors used the mainframe. That's part of why I chose to major in physics, though I was interested in programming. Another reason for overlap is that both "scientist" and "engineer" titles include things that are not strictly within either area. A lot of people with science degrees end up working as technicians, marketeers, salesmen, managers, etc. A lot of people with engineering degrees do little or no quantitative engineering, but work as programmers, designers, marketeers, salesmen, managers, etc. Something to ponder is if there are differences between how scientists and engineers think and approach problems. Naturally there's a lot of folklore about that, probably little hard evidence. That's where I think you should start if you want to know whether to hire scientists, engineers, or both.
• deleted 6y ago
  [deleted]
• Dumblydorr 6y ago
  I'm a scientist first, data person second. I'll use the label data scientist if that's what the employer wants. I'll use data analyst, scientist, researcher, or any other term, so long as they pay me. What matters is: are we contributing to the betterment of society with data-driven decision making?
  • Dumblydorr 6y ago
    Another aspect: so many PhDs and postdocs in the sciences do similar work but get paid 1/3 or less of the wages. We shouldn't be surprised they snap up industry jobs in data science, it's way more remuneration than academics, with less stress IMO
• civilized 6y ago
  Just my personal experience, but the people at my company titled "Data Engineers" basically can only be trusted to (very slowly) move data around, while the people titled "Data Scientists" have to do all the cleaning work to make the data suitable for analysis and modeling, in addition to doing that analysis and modeling. Is the point here that data scientists are doing too much of the work that should be handled by data engineers? If so I agree, but there are some org barriers. For one thing, our data engineers are not accountable to any particular project. All we can do is send them tickets to move data around, and it already takes them weeks to move a table from one database to another. I can't imagine the headaches if we gave them anything nontrivial to do.
  • Avalaxy 6y ago
    That sounds... very weird. Something is for sure wrong in your org. Anyway, the point, I think, is that data engineers and data scientists should be integrated in one team and not operate in different silos.
• itsoktocry 6y ago
  My first job as a "Data Scientist" (it wasn't called that, but the work was the same) was for a small gaming shop, around 2011. It involved applying econometric analysis and doing simple statistical testing on the player data sets. I realized quickly that knowing how to do statistical testing was only a very small portion of what it took to create value in such a role. At the time, I didn't even know (but learned) SQL. Everything I wanted to do involved teaming up with a developer, which wasn't efficient in a small operation. So I learned to program. I continue to enjoy skilling-up, most recently learning cloud-tech to enable me to deploy data tools I develop. The most valuable people in the data chain will be those that can take idea to near-production. Running ML libraries over clean datasets is overrated. The fact is, 80% of the value of "Data Science" comes from KPIs and basic stuff.
  • hardtke 6y ago
    > The most valuable people in the data chain will be those that can take idea to near-production. Having hired many data scientists/ML engineers over the years, people that build robust automated intelligence directly into products are extremely rare. I've estimated a maximum of 10K people in the entire world. They also command the highest salaries, not coincidentally. Very few people have both the statistics and engineering backgrounds as well as temperament to be successful, particularly when the problem requires new data sources or new types of models. There are some real simple practical hurdles such as the need to implement robust tracking that allows data snapshotting at the time when a decision needs to be made without affecting product performance, as well as figuring out how to gather data on users/situations that are actually important for moving the needle (first time users, casual users, etc.) There is also a mismatch between the best frameworks for prototyping/research and implementation which (at least at the companies I've worked for) can be summarized as "Java is good for application development, not ML. Python is good for ML, not application development."
• dqpb 6y ago
  workera.ai has a very interesting approach to measuring, categorizing, and visualizing where you are on the skill spectrum and what employers are looking for on that spectrum.
• lordnacho 6y ago
  My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a way that's detrimental to the rest of the team. You get people who just don't appreciate that there's other work to do than finding models, and that models depend on that other work to function. You also get a huge prestige gap, because inevitably management will think that there's a magician and a blacksmith. One guy needs to be paid a lot, and the other guy needs to be paid enough. These effects feed each other. Magician will say "where's my data" and expect blacksmith to make it, promptly. He won't do it himself, because spending time on mundane stuff makes the magic disappear. And not doing it yourself, or taking the time to understand it, will eventually lead to problems with the magic.
  • chadash 6y ago
    I worked in investment banking (as an analyst, not an engineer), so very different part of finance, but this was my take as well. Companies might love to talk about how important engineers are, but at the end of the day, if you can't directly link someone to revenue, they get viewed as a cost center and take on second tier status in the organization. Then the same companies complain that they can't find enough (or retain) engineering talent. Not many places get the balance right. Silicon valley treats engineers well because for the most part, the value they bring is more obvious (and also, they don't threaten the existing hierarchy in the company). Curious to hear if anyone has had the opposite experience.
    • isolli 6y ago
      Yes, quite a few developers left our investment bank and went to work for our suppliers (of trading software), stating they'd rather work somewhere where they're seen as value creators rather than a cost center.
    • AtomicOrbital 6y ago
      I worked for 15 years as a software engineer at Morgan Stanley where they valued the process of taking a 3 martini lunch idea into a production platform so value of engineers was recognized and rewarded as such ... its somewhat easier to whip up a new financial wrinkle its a whole other level of magic to design and implement that idea when it takes 60 software developers 3 years to get that idea to market before the rest of the street ... of course the IT department was/is the largest budgeted portion at the entire bank and for a good reason
• asimjalis 6y ago
  Who gets paid more? Maybe the question to ask is why market forces are not naturally producing more of what is needed?
• hankchinaski 6y ago
  the article is long overdue. so many times i have seen data scientists glue together non production ready code and libraries expecting someone else to finish the job and put the project to production. in various places i worker at they have soon realised that we mostly and exclusively needed data engineers - glorified DevOps people that make production ready data pipelines - more than we needed data scientists to do “modelling”
• brd 6y ago
  As an industry we're letting history repeat itself and making all the same mistakes. There are different kinds of developers. At it's most base form, you have systems focused developers and algorithmic focused developers. Sure there is a grey area but I think those two buckets are pretty defensible. In the data science world you have an exact parallel. Those who build the systems and those who optimize the thing the system supports. In the ML world you have another parallel. Those who build the systems and those who optimize and pioneer the model architectures and parameters. We never reached consensus on the titles for different kinds of developer/programmer/computer scientists. And we're failing now to reach consensus on sane titles for ML and DS.
• tfehring 6y ago
  One annoying thing about being a generalist is that domain experts in any given area that you need familiarity with can't help but complain about how little you know about that domain, ignoring the fact that your job requires equally deep knowledge of several other domains simultaneously. In the case of data scientists, I think the business folks that want them to understand the business domain better generally have the strongest argument, followed by the statisticians - good data scientists need to personally understand both of those things well, while the engineering and ops stuff that data scientists are also expected to do is easier to compartmentalize on other teams. So I agree that we should have more data engineers, but apparently for the opposite reason as most people in this thread.
• nlbrown 6y ago
  What are the typical entry level Data Science/Engineering positions like? Are they available to fresh college or bootcamp graduates?
  • marcinzm 6y ago
    From everything I'm hearing Data Science is right now flooded with entry level applicants to the point where they're applying for even pure Data Analyst jobs. Data Engineering a lot less so.
• deleted 6y ago
  [deleted]
• 6d6b73 6y ago
  I would go even farther. We don't need data, we need knowledge.
• eVoLInTHRo 6y ago
  Agree with the general idea. Data engineers are like the essential workers of the data world, people who today may not receive the appropriate level of appreciation that they deserve. I think the glut of data scientist occurs because we clump so many different skills and disciplines under the single term "data scientist". Data scientists today come from so many different backgrounds that the definition means something different to everyone. Because of this, the surface area of possible skills that could be expected of a data scientist is vast, to the point where it's pretty unlikely to be sufficiently competent in all of them, let alone a majority. I'd like to go back to a world where we had a little more specificity about what kind of data scientist you are (e.g. I had no problem with terms like statistician and data miner), which could help ground expectations that others have of us, and it'd also help clearly define the scope of various career paths for the next generation. Sadly the individual who coined the term shows no contrition for the degree of confusion that the rest of us have been left to deal with: https://observer.com/2019/11/data-scientist-inventor-dj-patil-interview-linkedin-job-market-trend/ https://observer.com/2019/11/data-scientist-inventor-dj-pati...
• noahmoss 6y ago
  What does CV stand for in this context?
  • macksd 6y ago
    Computer vision (i.e. deep learning on images / video)
    • noahmoss 6y ago
      Thanks
• gigatexal 6y ago
  Why then are a data Scientists paid a ton more? Seems they are in more demand.
• a_zaydak 6y ago
  My view is from a small startup with little to no room for single purpose employees. When I first started hiring and working with data scientist my view was this: If you can only manipulate data and run it through pipelines to generate models then you can't do enough to be highly valuable. You either need to have a strong enough background in CS to build the pipelines / tools or a strong enough mathematics background to be able to propose cutting edge new ideas. From my experience it is hard to find someone who has one of these skill just from a University "data science" program. At a small company (at least ones that I have worked with) being only proficient in R and basic Python isn't enough. That being said, I have met and handful of Data Scientist who were very smart and self motivated enough to pick up on the lacking skills when given the chance. My question to HN is this; are there rolls at these larger companies for a Data Scientist who who primarily just crunches data in R and Python without the ability to actually build the pipelines / tools or conduct research?
  • proverbialbunny 6y ago
    I would be cautious about that. I've worked in the startup space for over 10 years now as a data scientist, often the first one hired on, working on the pipes. From my experience, there are two types of data scientists who work who do infrastructure work: 1) Those who do not make the best data scientist because their skill set is too far in engineering land, leaving them weak where it counts. If the startup is relying on the data scientist to be profitable, I'd be cautious with these types. or 2) Someone who is senior, beyond senior really, who has worked both jobs, and doesn't mind doing both jobs. This unicorn is so rare it is mythical. The joke when the terminology was created is they're so rare no one has ever seen one, hence unicorn. Me, I can not do the work I need to do if I'm on call. That is where I draw the line. That means hiring someone to monitor the infrastructure. Furthermore, I'm an okay architect, but you really do want to hire a specialist if you can help it for that. Do I help them with the infrastructure? Absolutely, but they're on call if a server is on fire. They have the admin login credentials, not me. I get wearing multiple hats, but keep in mind to be a data scientist you're already wearing multiple hats. Being a data scientist is like double majoring and getting a phd. At what point are they stretched too thin? The consensus in the industry is they're already stretched too thin and should be broken up into different specialized roles. >My question to HN is this; are there rolls at these larger companies for a Data Scientist who who primarily just crunches data in R and Python without the ability to actually build the pipelines / tools or conduct research? That is the standard role, even at startups. However, the industry consensus these days is data scientists should have more responsibility when it comes to deploying models than previous standards.[1] So data scientists are being pushed in a more engineering direction, not with hosting sql servers and infrastructure, but with working with engineers to make sure the models are monitored properly. This change comes from model deployment being further automated as time goes on, making it easier for the data scientist to have more responsibility during this stage. [1] source: https://www.dominodatalab.com/static/gfx/uploads/domino-managing-ds.pdf https://www.dominodatalab.com/static/gfx/uploads/domino-mana... page 9. Suboptimal organization and incentive structures.
• prionassembly 6y ago
  I'm late to the comment party, but: this is classic "commoditize your complement". This guy would have you believe that Pytorch has Solved the entire, vast field of data analysis as inherited from Newton, de Moivre, Laplace, Bayes, Fisher, Neyman, Pearson, Wald, Savage, Jaynes, Breiman, Pearl. This is a lot like saying that photography has Solved art, and now we need people who can climb ladders and glue the posters on them big billboards. It would be delusional if it didn't have a self-interested angle. What, we with math degrees are fully confident that the plumbing problem is easier to commoditize than the problem of making sense of data.
  • nelsondev 6y ago
    It’s about diminishing returns. E.g. getting a model from 0% accuracy to 70% accuracy might be a couple Pytorch library calls that any dummy who watched Andrew Ng’s course can do. But getting that same model from 70% to 75% accuracy might be deeply mathematical and require the latest and greatest mathmeticians, statisticians. But in this hypothetical example, an engineer who stands up the 70% model and keeps it running 99.9% of the time, with high uptime; is more valuable to the bottom line of the business than the 75% accuracy model hacked together with scripts with 50% uptime.
• jh88 6y ago
  Just like with data science, don't we expect data engineering to get to a point where they codify best practices into a tool, so 1x data engineers can be closer to be as effective as 10x data engineers? The amount of data engineering work won't change, but the number of people needed to do it will reduce, reducing the open headcounts.
• monksy 6y ago
  This has been a bit of an annoying thing for me for quite a while. There is a huge difference between a data engineer and a data scientist. A data scientist is not and should not be a data engineer. These are 2 different specializations that work together. A data engineer is more of what we use to refer to as people who wrote utility processes to process data or do system optimizations. At some point the industry decided to do away with a lot of the things we used to do (desktop apps, distirbuted systems, etc) and moved to REST services only. Then people realized oh wait.. we can't process data on a rest/web app. In typical inexperienced fashion, people tried to cram in there, but it doesn't work. (See Javascript neural networks) What is data engineering? It's all about moving data around efficiently and processing it in a way to is per formant and reactive. A lot of people tie hadoop/spark to being a data engineer. That's a terrible way of going about it. More of the modern approaches to this is using streaming platforms and reacting to events. (Sadly a lot of the ML stuff is tied to either python/tensorflow or spark) At times data engineering is pushed towards data maintenance and pushing all of the data in a bucket. This isn't a very valuable use of effort.. but people want to be buzzword compliant. Note: There are use cases for hadoop and spark.. but those rarer now. (They've better for very large datasets and merging for data that you have a much longer timeframe for the answer).
• tpoacher 6y ago
  We don't need [old but still relatively new definition, whose meaning still isn't fully agreed on or established]! We need [brand new definition of the same, which most people are even more confused what it means and how it's different from the old]!
• shoshin8 6y ago
  I think just replace this with. Need Engineers
• aaroncrayford 6y ago
  What scientist doesn't use data? Such terrible naming.
• LawrenceHecht 6y ago
  Building or running data infrastructure is an important part of 55% of 372 data engineers’ jobs, according to the “2020 Kaggle Machine Learning & Data Science Survey.” ...Check out three charts I created showing differences between data scientists, data engineers and machine learning engineers: https://thenewstack.io/software-engineers-use-spreadsheets-data-engineers-use-the-cloud/ https://thenewstack.io/software-engineers-use-spreadsheets-d...
• donkeyd 6y ago
  This is interesting. I was the technical founder for a data startup that used NLP, elastic and some other stuff for analysis. It's still active, growing and approaching profitability, is used by fortune 500 companies and has had some media attention. However, I've never been approached for a related role and have never been invited after applying for similar roles. Maybe my resume is bad, maybe my experience doesn't really fit anywhere, but I thought it was an interesting observation in light of this article.
• patothon 6y ago
  why not both?
• b0rsuk 6y ago
  Oh, but maybe we don't need either? It seems that data science is used primarily for advertising. Local internet communities are dead. Message boards are dying. Everything is either a reddit / discord / Steam forums / Boardgamegeek. In 2021, GOG forums pass for a small forum. Only the biggest can float in the ocean of spam.
  • adeelk93 6y ago
    Advertising? I would say the biggest sectors for data science are healthcare and finance.
• yters 6y ago
  While we like to pooh pooh at the theoreticians, there are some remarkable results proven just through thought experiments and math, which are only confirmed and used many, many years later. People would not even think to look into such things if theoreticians did not come up with the original abstract proof. For example, there are more combinations of characters for text messages of this length than there are particles in the entire history of all possible multiverses. Clearly I need to use something beside blind trial and error to write a halfway coherent HN comment. Instead I must rely on a little mental theorizing to even just rant on the internet.
• stakkur 6y ago
  We don't need data engineers, we need great tools and storytellers.
• peppery 6y ago
  "When art critics get together they talk about Form and Structure and Meaning. When artists get together they talk about where you can buy cheap turpentine." — Picasso. ^Culture is good at romanticizing the "dreamer" as divorced from (& higher than) the "doer"/implementer. Picasso might protest. This post is helpful in inviting us examine this instinct/tradition. But that good contrariness doesn't excuse us from being more thoughtful about the chain of deduction underlying the titular claim. First, many big ideas—Maxwell's equations of electromagnetism; that software should be Free as in both Beer and Software; that unruly Democracy could possibly be sometimes a (messily, weirdly) good way of organizing people—might naturally require more doers/thinkers to implement than just the mind (few minds?) who happened to crystallize it. Non-unitary stoichiometries for progress are the rule, not the exception. Nudging a culture is already a many-body problem, & because details matter, and details scale exponentially with levels of abstraction, a project's success can improve with the number of engineering minds adding leverage to advance it. If we take the above earnestly—that making ideas useful usually requires more people than who happened to express an idea—then noticing that more job postings exist for implementory/engineering roles than for "science" roles actually says nearly nothing about whether we as a technological culture are out of balance with science vs engineering, up to how poorly we know about the typical ratio of implementers versus dramers. It could be that there are plenty of good "scientific" ideas in circulation; maybe what separates us from progress is earnest implementation, reflected by empirical over-demand for engineers (as this post seems to mainly argue). The aggressive scaling laws for improving AI (along existing paradigms but broader compute) are tempting support for this conclusion. But personally, I think it comes down to your position on this underlying question. Do you believe that fundamentally _better_ data paradigms--eg those that actually compute differently & more (super)humanly--will come from ideas already articulated in the conceptual universe? Or do you think that the key to smarter data science, if it exists, has yet to be invented and may little resemble the ideas dominantly in circulation? If the latter, then we may most desperately need data _scientists_, in addition to engineers! In the sense that society would totally benefit from generating 1 new idea from “science,” even if 1000+ had been funded but not panned out. This is true on a global, pro-social sense, but also likely on an individual basis: if you are a thinker than surely partly what matters is what role could maximize your ∂impact/∂effort, and probabilistically, science of data is at least competitive with engineering if you think the future will look different than the present and needs to be invented.
• stdbrouw 6y ago
  The unspoken assumption in this article (and many of the replies in this thread) is that all statistical or machine learning problems are online problems, where data must be ingested, processed and analyzed continuously and indefinitely. For something like a SaaS app that's a no brainer because you have all of that data anyway, but the overwhelming amount of business analysis / optimization / modeling / prediction still revolves around one-off analyses and specific experiments with specific goals, for which all of this data engineering is overkill. I've worked in teams that didn't get anything done because the data was a mess or there wasn't any, but I've also had the experience of spending so much time on the engineering side of things that we stopped asking what kind of things would actually be useful to know or learn or test. Computer scientists and engineers in general often have this attitude of "okay, let's take things one step at a time, first we focus on the data and get that 100% right" with a good amount of smugness about these data scientists and their messy, unorganized code, but consider that maybe they know something you don't.
• svordsmith918 6y ago
  What about empowering the magician to handle some of the blacksmith's work? We need to start glorifying the more mundane tasks (data quality management, fixing broken ETLs, aggregating data)and tying them to the bottomline. This comes up a lot at fintech companies, as lordnacho mentions: data science makes the data look pretty for your CEO, but the data wouldn't even be there if it wasn't for the data engineer. We need self-service data platforms that hold magicians accountable. I'm seeing this more and more, but there's still a lot of work that needs to be done in this arena: https://towardsdatascience.com/how-to-build-your-data-platform-like-a-product-6677e8abe318?source=friends_link&sk=848d794b11e4d9f33abe4bf7a2a7cd30 https://towardsdatascience.com/how-to-build-your-data-platfo...
• amai 6y ago
  Isn't the new buzzword "machine learning engineer"? https://www.datarevenue.com/de-blog/machine-learning-engineer-vs-data-scientist https://www.datarevenue.com/de-blog/machine-learning-enginee...