12 ms·

Everyone wants to do the model work, not the data work [pdf]

358 points by Anon84 6y ago

ramraj07 6y ago
That’s a long paper to write given it’s all based on a survey of 50 people. Nevertheless the point is true, and importantly it’s probably caused by “AI engineers” who are actually pretty mediocre and were just attracted to this job for the wrong incentives. If interviews for DS and AI focus on recruiting people for not just theoretical middling knowledge but also the drive to solve “the real problem” that will let them tackle the project in a truly fundamental way, the majority of issues pointed in this paper will probably go away.
- wrnr 6y ago
  I wanted to point out the same thing, I'm counting six authors on this piece of science complaining there aren't any doers around. At this point we have reached peak papers, there is more truth to be found in a HN thread and thats saying a lot.
- primitivesuave 6y ago
  Agree, it lost some credibility for me as well. I immediately search for methodology before reading anything nowadays. > We presented a qualitative study of data practices and challenges among 53 AI practitioners in India, East and West African countries, and the US, working on cutting-edge, high-stakes... I'd rather read a well-written opinion article and hear more about what those 53 people have to say.
  - rusty-rust 6y ago
    Are you saying that; quantities studies > qualitative studies? Or are you saying that their qualitative methodology is lacking? Is so, what would be a better way of approaching a topic like this?
- bildung 6y ago
  This is a qualitative, not quantitative study. The goal of qualitative research (in cases like this) is, essentially, finding categories, not the relative distribution of the population in those categories. I've seen well-executed qualitative studies with less than 10 participants, 53 is actually on the higher end for this approach. You usually try to select for a pretty diverse cohort (which the authors did in this case), not a representative one. A diverse cohort means you'll probably have all categories that matter. A representative cohort means you'll have a distribution over these categories that is pretty similar to the real world. The result here is essentially displayed in Figure 1. One could now follow up with a quantitative design to e.g. discern which particular step is the most common obstacle.
avs733 6y ago
What doesn’t help is that these techniques are increasingly taught in undergrad programs and short term data science experiences. More content, more tools, no time for understanding of the full scope of analysis. Statistical education has called for the use of real data for a long time and it is still woefully absent in undergrad statistics and probability courses. Students are given clean/cleanish data so they can be evaluated on the correct application of tools, not on a full stack process of data sense making. When I took over our stats course I took out the coding (instructors were arguing over R vs Matlab) and put in excel. Excel opens their eyes because it can’t make math invisible like R can...they can’t obfuscate their mistakes in abstractions. It’s hard because doing that is at the edge developmentally of young adults...but it’s like giving them a Ferrari as a present for their drivers licenses when we teach this stuff. If experts can’t do it right, think about the implications of teaching them to 19 year olds.
- iujjkfjdkkdkf 6y ago
  But will taking away abstractions address what the paper is talking about? If we can abstract away the models and training, I see an opportunity to focus more of the dataset. If you look at something like Pytorch Lightning (and I think fast.ai but am not familiar) you can deploy very powerful, and proven, models without having to get into the details. What might still be missing are more tools to work with the dataset - easily varying size, composition, using active learning, etc. But it's all supported by more abstraction. (I'm a huge believer in the importance of understanding things from first principles, so I agree with making sure students develop their skills that way. But when it comes to practical work with datasets, abstraction is very important)
  - avs733 6y ago
    At a professional level not directly, but I believe (opinion statement) starting with the abstractions and the abstractions being the focus of all the coursework you typically take contributes to the problem over the long haul.
- selimthegrim 6y ago
  I'm a firm believer Blockpad (https://blockpad.net https://blockpad.net) can be a powerful Excel-like tool in these environments, especially with engineering data.
deleted 6y ago
[deleted]
yunesj 6y ago
> Under-valuing of data work is common to all of AI development. I don’t think that’s true in bioinformatics. For example, people highly respect the laboratory work done to collect data on protein function.
- haram_masala 6y ago
  Exactly, in fact it’s the opposite of anything. I’ve seen plenty of life sciences companies where the bioinformatics team was added on long after founding the company, almost as an afterthought.
ssivark 6y ago
Probably stems from the foundational misnomer of “artificial intelligence” (which glamorizes the algorithm) compared with, say “data-driven algorithms” which emphasizes both aspects suitably. AI might have been a passable term in the era of rule-based expert systems, but both “AI” and “ML” are particularly misleading terms (when applied to modern data-heavy methods), leading to consequences like that pointed out in the article. Further, glossing over the importance of data to emphasize the algorithms is not accidental, given the subject’s roots in CS, and the rhetoric used to promote the field. The sub fields where it plays out differently are those which have a healthy respect for the prices of data gathering.
- visarga 6y ago
  I've also heard of "matrix networks" as an alternative to "deep learning". Immediately becomes more clear and less hyped.
AndrewKemendo 6y ago
Yes, this is extremely common and extremely frustrating. Companies, governments, individuals come to me all the time asking how to "Do AI" without knowing anything about their product, data provenance, testing plans, state observability etc... Sure, you can do a toy problem with some interesting vision or NLP framework and some off the shelf data that isn't ready for production or anything serious. Who cares? That doesn't do anything novel, it doesn't actually change your product function. Chomsky said the most important thing to do as a young person is to follow the phrase "Know Thyself." People who want to do AI need to "Know Thy-data and product architecture" first. I did a talk last year on exactly this: https://www.youtube.com/watch?v=jK-SBu1iiHo https://www.youtube.com/watch?v=jK-SBu1iiHo
- lumost 6y ago
  My 2 cents is that ML oriented job titles are too fine grained with everyone in theory serving a mythic modeler who just needs to be fead data. Everyone finds the 1% of the time they spend building models the rewarding part - however many of the job titles like Data engineer, or ML engineer do not include modeling in the job description. There are companies which differentiate scientists into multiple categoies based on whether they do ML, can code, or just statistics. Now the problem is that any of these folks can get into trouble for doing the others job, and the pay scales are often radically different. Relegating core responsibilities such as data cleaning/collection to support roles. I'd bet that flattening the title structure in these groups would yield better results, and allow team members to more seamlessly move across Backend, Data, ML, and science oriented tasks.
  - ethbr0 6y ago
    I think it's a problem of visibility up the management ladder. (A) If management understands ML enough to launch a feasible project, (B) if management understands that models are a component of ML, (C) if management understands models are created from data, (D) if management understands that production models must continually be fed data. Typically, understanding seems to stop somewhere between A & B.
  - lostdog 6y ago
deleted 6y ago
[deleted]
sanxiyn 6y ago
Real story. I worked on nudity detection for mobile realtime video chat app with on-device deep learning model. (It was successful.) Should be easy: label some data, start from a good network pretrained on ImageNet (we used SqueezeNet), fine tune. The problem was that ImageNet is photos, not video frames, and distribution of video frames from mobile video is somewhat different. Incredibly large proportion of frames are either blank (camera is blocked) or ceiling (camera is misdirected). We ended up adding lots of data for blank and ceiling explicitly labeled as blank and ceiling. It became an excellent ceiling detector, detecting ceilings it never have seen before. Is there a mobile video frames dataset for labeling ceilings? Why not? I can't believe I am the only one who experienced this. Why is not a ceiling dataset worthy of, say, CVPR? It will improve mobile video on-device deep learning more than most of CVPR papers. This is a serious problem. Edit: I understand why niche datasets remain in industry and not academia. Public datasets are better if they are of general interest. But ceiling dataset is of general interest to anyone who wants to process video frames originating from blocked or misdirected camera aka smartphones, and it's hard to imagine topics of more general interest once you have obvious things like face.
- ethbr0 6y ago
  Because fewer PhD are minted by "Improved on SotA ceiling detection rates by 50%."
  - sanxiyn 6y ago
    Maybe not PhD, but as a research topic, I think doing this well is an interesting task. I learned that ceilings are not featureless. (I know, it's a kind of ridiculous to say I "learned" this, since I could have learned this from, oh well, looking at my ceiling just now, but it really was a surprise when I encountered this in machine learning data context.) Ceilings often have lightings! Lightings (either on and off) vary between geography and culture! (They do! Extreme variations!) Lightings, if on, will cause flares! We just sampled from actual distribution (privacy policy included we can monitor connection for anti-abuse and we can use data for research and development of anti-abuse), but dataset collected with care will be valuable. I am sure actually doing this will reveal other considerations, and even just with points I mentioned, it should be good enough for a paper, if not for PhD.
totorovirus 6y ago
Is is because data work doesn't need a degree and the degree holders don't want to risk their career for doing something that is off topic to their degree?
- js8 6y ago
  It's the other way around. Data work does require understanding of the domain, and degree holders don't want to risk their career for doing something that is off topic to their degree, that is, learning the domain. I am just pointing out the your statement has an other side, too. It's kinda like with programming languages, people think that particular programming language, rather than domain details, are the transferable skill for SW developer. In data science, it's the statistical modelling theory that is considered to be the transferable skill, rather than the business domain. But you always need both, and there is no particular reason why would, in hiring, one prefer knowledge of the tooling to knowledge of the domain. It just happens to be, as a kind of spontaneous symmetry breaking.
  - vinay_ys 6y ago
    For a strong engineer today, the most valuable/transferable skills are ability to learn new things (in both tech and domain) and effective problem-solving (in different settings). Such an engineer can learn the most important aspects of statistical modeling needed for their problem at hand. The kind of researchers who are averse – "to getting their hands dirty" – to do data tasks will have a hard time staying relevant in any production setting. Unless the company has dedicated research department which is a million miles away from product teams, they are going to have a tough time being impactful.
dmh2000 6y ago
isn't it typical for data engineers to be paid less (way less?) than the modelers?
- andyxor 6y ago
  Check levels.fyi, for pure SWE/Data/Backend vs. ML, the former often pays more.
angarg12 6y ago
I work in a ML Engineering team, and it has almost become a meme for me to demystify ML to prospect candidates. First of all I have to convey the idea that we are just engineers, so the vast majority of our job is "boring" engineering stuff. Write, maintain and upgrade code, fix bugs, scale our systems... other than the domain, you wont see much difference. What people usually see as the "cool" part of the job is often done by scientists. There is an interface between teams where we get the result of science and need to get it "production ready", but that might be around 10% of the job.
heisenbit 6y ago
Appreciation and renumeration as a rule tends to be higher the closer one is to the final customer.
mustafa_pasi 6y ago
What is modelling in this case? Choosing a sequence of layers and fine tuning a bunch of hyper parameters? Isn't that also kinda boring?
- beforeolives 6y ago
  So true. Deep learning is kind of lazy from the point of view of the model builder. Pretty much any other technique - ML models that require feature engineering, graphical models, differential equations, optimization problems, even running simple statistical tests - each of those is more rewarding to work with than DL. The more interesting aspects of deep learning are often around engineering the whole pipeline and doing it efficiently at large scale.
Traster 6y ago
I'm actually going to take the counter-side of this argument. It's actually very important for people to do the model work first. Yes, you need some amount of decent data, but the first step of a project should always be building a prototype to answer the question "Can we fundamentally do this". It's perfectly reasonable in that case to do a small amount of work picking and cleaning up some data and then really focusing on building the model. Once you've got your model then you can answer the best case scenario of whether the problem is tractable. Once you've got the model then you need to work on operationalise, and that's where work on your data pipeline becomes important. It's always going to be the case though that building your initial prototype is more fun/interesting and probably 1000x easier, than getting your prototype into production. And it's important to remember that operationalizing your model is probably going to involve working on your model, retraining it and refining it to work with your new production environment. The authors have tried to dress their waterfall up as "cascades" but the truth is that they've defined the development process as a waterfall and are trying to fix that rather than adopting modern development practices.
- kfk 6y ago
  I agree we need to start from an mvp and so with a data model, but does this model really need to be AI or ML? My counter point to your counter point is yes, we need a model, but probably the initial one can be done with simple BI/SQL type aggregation methods. Most of AI/ML I see is optimizing an already existing model, not create something completely new.
  - disgruntledphd2 6y ago
    I'd suggest using a lasso, tbh. It's pretty fast, relatively performant and one can interpret the coefficients. That being said, I'd do your simple SQL aggregations first, to ensure that one can actually learn something (if we have 0.0001% positives, a simple approach probably isn't going to work well).
  - Ntrails 6y ago
    A few years back the team that do "Business analysis" came to our team asking for help setting up some machine learning. What for, we asked. Well, we'd like to predict hardware expenditures over the next n years. Ok. Well that sure seems like a thing the business might like an estimate of! Sure. Next question: Have you run a basic regression over the data you've got, maybe a simple model would be close enough to throw some error bars up and get going. Of course not.
johanneskanybal 6y ago
Which is why being in the business of doing the data work is the supreme choice /data engineer consultant
tinco 6y ago
So, I don't have time to read the entire paper, but I thought it would be interesting to share my experience from the other side. I lead the engineering for a company that does the "data work", we gather terabytes of data, mostly photography and lidar, every day. So far it's been a crappy industry to be in, it seems the only way to make money is to provide the entire value chain, and this is the business model we're pivoting to. Just a bunch of photography and lidar scans isn't worth much to anyone, and it's very costly to establish a quality data stream. We had enormous capex on sensor gear, training personnel, running the operations. All this to save a couple bucks on sending an inspector physically to the site. If you do the entire value chain, you gather the data, you process it into information, and then you turn that information into actionable reports. So that means that besides running our operations, we're also doing the modeling work, and we're providing the industry specific expertise for the actual inspection. So our customers pay for knowing what is broken, where it's broken, and what the impact of that defect is. In the end, I believe we'll be very successful, but it will be because our investors were willing to take us all the way, where many other companies have simply given up on the concept.
- ghgr 6y ago
  That's an insightful experience, thank you for sharing. To give a counter-counter point, in my experience working with LIDAR (terrestrial in my case, not airborne) there's a wealth of tools that claim to process large amounts of unstructured pointcloud data and make all kinds of fancy visualizations and 3d models (Pix4D comes to my mind). The "catch" was often to find high quality data to feed these tools (other than the sample data provided by them). I'm convinced that having high-quality proprietary data is the moat the investors are looking for, not the algorithms. After all, universities are cranking improved algorithms all the time for free.
  - tinco 6y ago
    Oh for sure, I suppose that's also why the investors are still interested in us. But high-quality proprietary data is not what the customers are interested in. They just want to see a report that says how much m2 of brickwork needs to be repointed in the next 5 years. And they don't care if it's a machine learning model that deduced that information from high quality drone imagery, or if it was a dude on a ladder. We're also working with terrestrial lidar btw. When we started out we thought we were going to do everything using photogrammetry, but right now we're deriving most of our geometry from the lidar.
elnygren 6y ago
I suppose skillful AI/ML people like to think that great AI/ML is all about their skills and the models they produce. Someone else might think that the true value is in the data, while AI/ML skills can be bought for 120$ per hour :) Perhaps one day we'll have tools that will figure out the right AI model for you given data, but we won't have tools that can as easily come up with the data.
bonoboTP 6y ago
Well, why do they want to? Is it their fault? People follow incentives. Infrastructure work is low status, janitor-adjacent or like the guy who fixes broken toilets. Academic, abstract, scientific, work is less immediate, more rhetoric-focused, more open ended, similar to lawyers and politicians and is about selling ideas. Shoveling data is like shoveling dirt, while academic abstract modeling is like an architect drawing sketches of a palace. Like it or not, infrastructure work is thankless. Complaining is useless, either accept it or switch jobs.
- giantg2 6y ago
  Not to mention it pays far less. I've interviewed for data jockey positions on AI/ML teams that were paying about $85k. The people building the models were making double or triple that.
  - inadequatespace 6y ago
    This is the real issue. It's considered less-skilled and therefore by supply/ demand more can do it and it pays less. I think this is probably only because it's harder to assess skill in it, though. It's easy to test for knowing the vagaries of the ADAM solver or statistics or whatever. What is the equivalent for being a good data cleaner, though?
- hntrader 6y ago
  This is some of the reason, but I've noticed the same behavior in data scientists who are "vertically integrated" and who are only measured on the quality of the final model. In this case, their incentives are aligned with good data work, but they still suck at it and neglect it. I believe there's a few psychological reasons: - People aren't trained to do it in college, so it's undervalued. - It requires domain knowledge, which is hard. - The work is quite gruelling and annoying. It's very detail oriented. - You don't feel like you're making progress while you're doing data work. Modelling work feels closer to the final output. - Self-delusion during modelling is easy since you can overfit on the holdout data which gives you those dopamine hits. - Modelling satisfies curiosity. During data work, the data scientist can't wait to hurry up to the modelling stage and see what the results look like. - Many people are just bad with data work, they don't have the attention to detail required and produce data with glaring errors.
dalbasal 6y ago
I barely skimmed this so criticism is unfair, but... it could be a criticism of many "working papers." There's a type of euphemization that's been irking me in recent years: The vague everyone. No one ever talks about X. "Data largely determines performance, fairness, robustness, safety, and scalability of AI systems [44, 81]. Paradoxically, for AI researchers and developers, data is often the least incentivized aspect, viewed as ‘operational’ relative to the lionized work of building novel models and algorithms" Do they mean that projects (at google? in society?) under resource "data work?" Do people get paid less. Do they mean incentives in academia? Do they mean that companies specialising in these areas are less successful? That universities have priorities wrong? Just for the meander... at the top level, the economy does seem to be valuing data work highly. Adwords & Amazon marketplace antitrust cases suggest that "data work" is at the core of these mega businesses. TSLA's current valuation is partially based on tesla's exclusive dataset. In at least some very substantial examples, data work is very highly valued while models are seen as a temporary advantage... IE it looks like the market (some industrie) expects models/algorithms to be commoditized while the datasets are expected to be valuable intellectual property. The authors do really have some insights here. A lot of "AI project" post mortems do find "data work" was underestimated. I guess that means it's underappreciated by definition. But... I think it would have gone someplace more concrete if they had started someplace more concrete.
- junippor 6y ago
  > at the top level, the economy does seem to be valuing data work highly. Adwords & Amazon marketplace antitrust cases suggest that "data work" is at the core of these mega businesses What is valuable about adwords isn't the "data work", right? It's their pricing power in the advertising space...
Dumblydorr 6y ago
I cleaned data and made the models for MD and PhD researchers. Their work lies more in the thought sphere, they analyze all the work I do, then mold and shape it week to week until it's publishable. They obviously make the basic ideas behind what I should pull and what kind of models to use, and get the grants and funding for my salary so, they are pivotal even if they're not doing the hands on data work.
isolli 6y ago
It reminds me of this [0]: "Before March 2020, the country had no shortage of pandemic-preparation plans. Many stressed the importance of data-driven decision making. Yet these plans largely assumed that detailed and reliable data would simply . . . exist. They were less concerned with how those data would actually be made." [0] https://statmodeling.stat.columbia.edu/2021/03/21/whassup-with-the-haphazard-coronavirus-statistics/ https://statmodeling.stat.columbia.edu/2021/03/21/whassup-wi...
mlthoughts2018 6y ago
Companies don’t pay financial rewards for doing the data plumbing work, yet it’s often much harder, prone to unusual failure domains mixed with high scaling, and carries more stringent on-call and incident triage responsibilities. Given that it’s (a) more difficult, (b) more business critical, (c) more stressful and requiring an incident alerting on-call rotation, then “data work” should be much better paid and offer job security and career growth. Yet no company I know of pays expert modelers & researchers less than expert data platform engineers. So either the companies know something you don’t (e.g. that data platform work is more commodity and easier to replace than rarer modeling talent) or there’s a free lunch you can get by exploiting the arbitrage opportunity to pay data platform experts more and consume correspondingly higher business value that other orgs are missing out on by putting modeler / researcher higher on the status hierarchy than data platform engineer. My perspective after many years of experience managing machine learning teams (both platform/infra and research/modeling) is that data platforming is just a worse job. It’s unpleasant and stressful and business stakeholders who are removed from backend engineering complexity and just want the report or just want the model couldn’t care less about organizational structures and workflows that support healthier lives for intermediate data platform teams. Because of this, the pay and bonuses for data platform roles should be much higher, but politically speaking it’s impossible to advocate for that, so it becomes a turnover mill where everyone burns out to keep the existing shitty system running, with comparatively low pay and low autonomy, and so nobody ends up wanting to join that team or do that work.
yachtman 6y ago
It is similar to an article I read a while back saying we need "data engineers" not "data scientists". I think its generally true. Why my company, while it can do data science consulting, is choosing to focus on data engineering consulting first.
- finnthehuman 6y ago
  I've always been annoyed by the title "data scientist" because I don't consider their methods worthy of the name science. It seems like the winds are turning and more people are questioning data "science." But talking about changing the branding to deliver the product that was advertised all along feels like putting the cart before the horse. Sure, maybe it will work. Or maybe in a few years we'll be talking about how we need more "data sleuths" to replace all those haphazard "data engineers" that just didn't have a job title that emphasized solving the problems rigorously enough.
beyondthecut 6y ago
Someone needs to do the work of making that article readable on mobile...
vizually 6y ago
If a start-up stage company is just getting started on their AI/ML journey has budget for 3 FTE people. The company already has traditional ETL/BI expertise and a "DWH". Who would you hire (data scientist, ML engineer, Dat Engineer) and how would you allocate the division of responsibilities?

358 points by Anon84 6y ago

That’s a long paper to write given it’s all based on a survey of 50 people. Nevertheless the point is true, and importantly it’s probably caused by “AI engineers” who are actually pretty mediocre and were just attracted to this job for the wrong incentives. If interviews for DS and AI focus on recruiting people for not just theoretical middling knowledge but also the drive to solve “the real problem” that will let them tackle the project in a truly fundamental way, the majority of issues pointed in this paper will probably go away.

wrnr 6y ago
I wanted to point out the same thing, I'm counting six authors on this piece of science complaining there aren't any doers around. At this point we have reached peak papers, there is more truth to be found in a HN thread and thats saying a lot.
primitivesuave 6y ago
Agree, it lost some credibility for me as well. I immediately search for methodology before reading anything nowadays. > We presented a qualitative study of data practices and challenges among 53 AI practitioners in India, East and West African countries, and the US, working on cutting-edge, high-stakes... I'd rather read a well-written opinion article and hear more about what those 53 people have to say.
- rusty-rust 6y ago
  Are you saying that; quantities studies > qualitative studies? Or are you saying that their qualitative methodology is lacking? Is so, what would be a better way of approaching a topic like this?
bildung 6y ago
This is a qualitative, not quantitative study. The goal of qualitative research (in cases like this) is, essentially, finding categories, not the relative distribution of the population in those categories. I've seen well-executed qualitative studies with less than 10 participants, 53 is actually on the higher end for this approach. You usually try to select for a pretty diverse cohort (which the authors did in this case), not a representative one. A diverse cohort means you'll probably have all categories that matter. A representative cohort means you'll have a distribution over these categories that is pretty similar to the real world. The result here is essentially displayed in Figure 1. One could now follow up with a quantitative design to e.g. discern which particular step is the most common obstacle.

avs733 6y ago

What doesn’t help is that these techniques are increasingly taught in undergrad programs and short term data science experiences. More content, more tools, no time for understanding of the full scope of analysis. Statistical education has called for the use of real data for a long time and it is still woefully absent in undergrad statistics and probability courses. Students are given clean/cleanish data so they can be evaluated on the correct application of tools, not on a full stack process of data sense making. When I took over our stats course I took out the coding (instructors were arguing over R vs Matlab) and put in excel. Excel opens their eyes because it can’t make math invisible like R can...they can’t obfuscate their mistakes in abstractions. It’s hard because doing that is at the edge developmentally of young adults...but it’s like giving them a Ferrari as a present for their drivers licenses when we teach this stuff. If experts can’t do it right, think about the implications of teaching them to 19 year olds.

iujjkfjdkkdkf 6y ago
But will taking away abstractions address what the paper is talking about? If we can abstract away the models and training, I see an opportunity to focus more of the dataset. If you look at something like Pytorch Lightning (and I think fast.ai but am not familiar) you can deploy very powerful, and proven, models without having to get into the details. What might still be missing are more tools to work with the dataset - easily varying size, composition, using active learning, etc. But it's all supported by more abstraction. (I'm a huge believer in the importance of understanding things from first principles, so I agree with making sure students develop their skills that way. But when it comes to practical work with datasets, abstraction is very important)
- avs733 6y ago
  At a professional level not directly, but I believe (opinion statement) starting with the abstractions and the abstractions being the focus of all the coursework you typically take contributes to the problem over the long haul.
selimthegrim 6y ago
I'm a firm believer Blockpad (https://blockpad.net https://blockpad.net) can be a powerful Excel-like tool in these environments, especially with engineering data.

deleted 6y ago

[deleted]

yunesj 6y ago

> Under-valuing of data work is common to all of AI development. I don’t think that’s true in bioinformatics. For example, people highly respect the laboratory work done to collect data on protein function.

haram_masala 6y ago
Exactly, in fact it’s the opposite of anything. I’ve seen plenty of life sciences companies where the bioinformatics team was added on long after founding the company, almost as an afterthought.

ssivark 6y ago

Probably stems from the foundational misnomer of “artificial intelligence” (which glamorizes the algorithm) compared with, say “data-driven algorithms” which emphasizes both aspects suitably. AI might have been a passable term in the era of rule-based expert systems, but both “AI” and “ML” are particularly misleading terms (when applied to modern data-heavy methods), leading to consequences like that pointed out in the article. Further, glossing over the importance of data to emphasize the algorithms is not accidental, given the subject’s roots in CS, and the rhetoric used to promote the field. The sub fields where it plays out differently are those which have a healthy respect for the prices of data gathering.

visarga 6y ago
I've also heard of "matrix networks" as an alternative to "deep learning". Immediately becomes more clear and less hyped.

AndrewKemendo 6y ago

Yes, this is extremely common and extremely frustrating. Companies, governments, individuals come to me all the time asking how to "Do AI" without knowing anything about their product, data provenance, testing plans, state observability etc... Sure, you can do a toy problem with some interesting vision or NLP framework and some off the shelf data that isn't ready for production or anything serious. Who cares? That doesn't do anything novel, it doesn't actually change your product function. Chomsky said the most important thing to do as a young person is to follow the phrase "Know Thyself." People who want to do AI need to "Know Thy-data and product architecture" first. I did a talk last year on exactly this: https://www.youtube.com/watch?v=jK-SBu1iiHo https://www.youtube.com/watch?v=jK-SBu1iiHo

lumost 6y ago
My 2 cents is that ML oriented job titles are too fine grained with everyone in theory serving a mythic modeler who just needs to be fead data. Everyone finds the 1% of the time they spend building models the rewarding part - however many of the job titles like Data engineer, or ML engineer do not include modeling in the job description. There are companies which differentiate scientists into multiple categoies based on whether they do ML, can code, or just statistics. Now the problem is that any of these folks can get into trouble for doing the others job, and the pay scales are often radically different. Relegating core responsibilities such as data cleaning/collection to support roles. I'd bet that flattening the title structure in these groups would yield better results, and allow team members to more seamlessly move across Backend, Data, ML, and science oriented tasks.
- ethbr0 6y ago
  I think it's a problem of visibility up the management ladder. (A) If management understands ML enough to launch a feasible project, (B) if management understands that models are a component of ML, (C) if management understands models are created from data, (D) if management understands that production models must continually be fed data. Typically, understanding seems to stop somewhere between A & B.
- lostdog 6y ago

deleted 6y ago

[deleted]

sanxiyn 6y ago

Real story. I worked on nudity detection for mobile realtime video chat app with on-device deep learning model. (It was successful.) Should be easy: label some data, start from a good network pretrained on ImageNet (we used SqueezeNet), fine tune. The problem was that ImageNet is photos, not video frames, and distribution of video frames from mobile video is somewhat different. Incredibly large proportion of frames are either blank (camera is blocked) or ceiling (camera is misdirected). We ended up adding lots of data for blank and ceiling explicitly labeled as blank and ceiling. It became an excellent ceiling detector, detecting ceilings it never have seen before. Is there a mobile video frames dataset for labeling ceilings? Why not? I can't believe I am the only one who experienced this. Why is not a ceiling dataset worthy of, say, CVPR? It will improve mobile video on-device deep learning more than most of CVPR papers. This is a serious problem. Edit: I understand why niche datasets remain in industry and not academia. Public datasets are better if they are of general interest. But ceiling dataset is of general interest to anyone who wants to process video frames originating from blocked or misdirected camera aka smartphones, and it's hard to imagine topics of more general interest once you have obvious things like face.

ethbr0 6y ago
Because fewer PhD are minted by "Improved on SotA ceiling detection rates by 50%."
- sanxiyn 6y ago
  Maybe not PhD, but as a research topic, I think doing this well is an interesting task. I learned that ceilings are not featureless. (I know, it's a kind of ridiculous to say I "learned" this, since I could have learned this from, oh well, looking at my ceiling just now, but it really was a surprise when I encountered this in machine learning data context.) Ceilings often have lightings! Lightings (either on and off) vary between geography and culture! (They do! Extreme variations!) Lightings, if on, will cause flares! We just sampled from actual distribution (privacy policy included we can monitor connection for anti-abuse and we can use data for research and development of anti-abuse), but dataset collected with care will be valuable. I am sure actually doing this will reveal other considerations, and even just with points I mentioned, it should be good enough for a paper, if not for PhD.

totorovirus 6y ago

Is is because data work doesn't need a degree and the degree holders don't want to risk their career for doing something that is off topic to their degree?

js8 6y ago
It's the other way around. Data work does require understanding of the domain, and degree holders don't want to risk their career for doing something that is off topic to their degree, that is, learning the domain. I am just pointing out the your statement has an other side, too. It's kinda like with programming languages, people think that particular programming language, rather than domain details, are the transferable skill for SW developer. In data science, it's the statistical modelling theory that is considered to be the transferable skill, rather than the business domain. But you always need both, and there is no particular reason why would, in hiring, one prefer knowledge of the tooling to knowledge of the domain. It just happens to be, as a kind of spontaneous symmetry breaking.
- vinay_ys 6y ago
  For a strong engineer today, the most valuable/transferable skills are ability to learn new things (in both tech and domain) and effective problem-solving (in different settings). Such an engineer can learn the most important aspects of statistical modeling needed for their problem at hand. The kind of researchers who are averse – "to getting their hands dirty" – to do data tasks will have a hard time staying relevant in any production setting. Unless the company has dedicated research department which is a million miles away from product teams, they are going to have a tough time being impactful.

dmh2000 6y ago

isn't it typical for data engineers to be paid less (way less?) than the modelers?

andyxor 6y ago
Check levels.fyi, for pure SWE/Data/Backend vs. ML, the former often pays more.

angarg12 6y ago

I work in a ML Engineering team, and it has almost become a meme for me to demystify ML to prospect candidates. First of all I have to convey the idea that we are just engineers, so the vast majority of our job is "boring" engineering stuff. Write, maintain and upgrade code, fix bugs, scale our systems... other than the domain, you wont see much difference. What people usually see as the "cool" part of the job is often done by scientists. There is an interface between teams where we get the result of science and need to get it "production ready", but that might be around 10% of the job.

heisenbit 6y ago

Appreciation and renumeration as a rule tends to be higher the closer one is to the final customer.

mustafa_pasi 6y ago

What is modelling in this case? Choosing a sequence of layers and fine tuning a bunch of hyper parameters? Isn't that also kinda boring?

beforeolives 6y ago
So true. Deep learning is kind of lazy from the point of view of the model builder. Pretty much any other technique - ML models that require feature engineering, graphical models, differential equations, optimization problems, even running simple statistical tests - each of those is more rewarding to work with than DL. The more interesting aspects of deep learning are often around engineering the whole pipeline and doing it efficiently at large scale.

Traster 6y ago

I'm actually going to take the counter-side of this argument. It's actually very important for people to do the model work first. Yes, you need some amount of decent data, but the first step of a project should always be building a prototype to answer the question "Can we fundamentally do this". It's perfectly reasonable in that case to do a small amount of work picking and cleaning up some data and then really focusing on building the model. Once you've got your model then you can answer the best case scenario of whether the problem is tractable. Once you've got the model then you need to work on operationalise, and that's where work on your data pipeline becomes important. It's always going to be the case though that building your initial prototype is more fun/interesting and probably 1000x easier, than getting your prototype into production. And it's important to remember that operationalizing your model is probably going to involve working on your model, retraining it and refining it to work with your new production environment. The authors have tried to dress their waterfall up as "cascades" but the truth is that they've defined the development process as a waterfall and are trying to fix that rather than adopting modern development practices.

kfk 6y ago
I agree we need to start from an mvp and so with a data model, but does this model really need to be AI or ML? My counter point to your counter point is yes, we need a model, but probably the initial one can be done with simple BI/SQL type aggregation methods. Most of AI/ML I see is optimizing an already existing model, not create something completely new.
- disgruntledphd2 6y ago
  I'd suggest using a lasso, tbh. It's pretty fast, relatively performant and one can interpret the coefficients. That being said, I'd do your simple SQL aggregations first, to ensure that one can actually learn something (if we have 0.0001% positives, a simple approach probably isn't going to work well).
- Ntrails 6y ago
  A few years back the team that do "Business analysis" came to our team asking for help setting up some machine learning. What for, we asked. Well, we'd like to predict hardware expenditures over the next n years. Ok. Well that sure seems like a thing the business might like an estimate of! Sure. Next question: Have you run a basic regression over the data you've got, maybe a simple model would be close enough to throw some error bars up and get going. Of course not.

johanneskanybal 6y ago

Which is why being in the business of doing the data work is the supreme choice /data engineer consultant

tinco 6y ago

So, I don't have time to read the entire paper, but I thought it would be interesting to share my experience from the other side. I lead the engineering for a company that does the "data work", we gather terabytes of data, mostly photography and lidar, every day. So far it's been a crappy industry to be in, it seems the only way to make money is to provide the entire value chain, and this is the business model we're pivoting to. Just a bunch of photography and lidar scans isn't worth much to anyone, and it's very costly to establish a quality data stream. We had enormous capex on sensor gear, training personnel, running the operations. All this to save a couple bucks on sending an inspector physically to the site. If you do the entire value chain, you gather the data, you process it into information, and then you turn that information into actionable reports. So that means that besides running our operations, we're also doing the modeling work, and we're providing the industry specific expertise for the actual inspection. So our customers pay for knowing what is broken, where it's broken, and what the impact of that defect is. In the end, I believe we'll be very successful, but it will be because our investors were willing to take us all the way, where many other companies have simply given up on the concept.

ghgr 6y ago
That's an insightful experience, thank you for sharing. To give a counter-counter point, in my experience working with LIDAR (terrestrial in my case, not airborne) there's a wealth of tools that claim to process large amounts of unstructured pointcloud data and make all kinds of fancy visualizations and 3d models (Pix4D comes to my mind). The "catch" was often to find high quality data to feed these tools (other than the sample data provided by them). I'm convinced that having high-quality proprietary data is the moat the investors are looking for, not the algorithms. After all, universities are cranking improved algorithms all the time for free.
- tinco 6y ago
  Oh for sure, I suppose that's also why the investors are still interested in us. But high-quality proprietary data is not what the customers are interested in. They just want to see a report that says how much m2 of brickwork needs to be repointed in the next 5 years. And they don't care if it's a machine learning model that deduced that information from high quality drone imagery, or if it was a dude on a ladder. We're also working with terrestrial lidar btw. When we started out we thought we were going to do everything using photogrammetry, but right now we're deriving most of our geometry from the lidar.

elnygren 6y ago

I suppose skillful AI/ML people like to think that great AI/ML is all about their skills and the models they produce. Someone else might think that the true value is in the data, while AI/ML skills can be bought for 120$ per hour :) Perhaps one day we'll have tools that will figure out the right AI model for you given data, but we won't have tools that can as easily come up with the data.

bonoboTP 6y ago

Well, why do they want to? Is it their fault? People follow incentives. Infrastructure work is low status, janitor-adjacent or like the guy who fixes broken toilets. Academic, abstract, scientific, work is less immediate, more rhetoric-focused, more open ended, similar to lawyers and politicians and is about selling ideas. Shoveling data is like shoveling dirt, while academic abstract modeling is like an architect drawing sketches of a palace. Like it or not, infrastructure work is thankless. Complaining is useless, either accept it or switch jobs.

giantg2 6y ago
Not to mention it pays far less. I've interviewed for data jockey positions on AI/ML teams that were paying about $85k. The people building the models were making double or triple that.
- inadequatespace 6y ago
  This is the real issue. It's considered less-skilled and therefore by supply/ demand more can do it and it pays less. I think this is probably only because it's harder to assess skill in it, though. It's easy to test for knowing the vagaries of the ADAM solver or statistics or whatever. What is the equivalent for being a good data cleaner, though?
hntrader 6y ago
This is some of the reason, but I've noticed the same behavior in data scientists who are "vertically integrated" and who are only measured on the quality of the final model. In this case, their incentives are aligned with good data work, but they still suck at it and neglect it. I believe there's a few psychological reasons: - People aren't trained to do it in college, so it's undervalued. - It requires domain knowledge, which is hard. - The work is quite gruelling and annoying. It's very detail oriented. - You don't feel like you're making progress while you're doing data work. Modelling work feels closer to the final output. - Self-delusion during modelling is easy since you can overfit on the holdout data which gives you those dopamine hits. - Modelling satisfies curiosity. During data work, the data scientist can't wait to hurry up to the modelling stage and see what the results look like. - Many people are just bad with data work, they don't have the attention to detail required and produce data with glaring errors.

dalbasal 6y ago

I barely skimmed this so criticism is unfair, but... it could be a criticism of many "working papers." There's a type of euphemization that's been irking me in recent years: The vague everyone. No one ever talks about X. "Data largely determines performance, fairness, robustness, safety, and scalability of AI systems [44, 81]. Paradoxically, for AI researchers and developers, data is often the least incentivized aspect, viewed as ‘operational’ relative to the lionized work of building novel models and algorithms" Do they mean that projects (at google? in society?) under resource "data work?" Do people get paid less. Do they mean incentives in academia? Do they mean that companies specialising in these areas are less successful? That universities have priorities wrong? Just for the meander... at the top level, the economy does seem to be valuing data work highly. Adwords & Amazon marketplace antitrust cases suggest that "data work" is at the core of these mega businesses. TSLA's current valuation is partially based on tesla's exclusive dataset. In at least some very substantial examples, data work is very highly valued while models are seen as a temporary advantage... IE it looks like the market (some industrie) expects models/algorithms to be commoditized while the datasets are expected to be valuable intellectual property. The authors do really have some insights here. A lot of "AI project" post mortems do find "data work" was underestimated. I guess that means it's underappreciated by definition. But... I think it would have gone someplace more concrete if they had started someplace more concrete.

junippor 6y ago
> at the top level, the economy does seem to be valuing data work highly. Adwords & Amazon marketplace antitrust cases suggest that "data work" is at the core of these mega businesses What is valuable about adwords isn't the "data work", right? It's their pricing power in the advertising space...

Dumblydorr 6y ago

I cleaned data and made the models for MD and PhD researchers. Their work lies more in the thought sphere, they analyze all the work I do, then mold and shape it week to week until it's publishable. They obviously make the basic ideas behind what I should pull and what kind of models to use, and get the grants and funding for my salary so, they are pivotal even if they're not doing the hands on data work.

isolli 6y ago

It reminds me of this [0]: "Before March 2020, the country had no shortage of pandemic-preparation plans. Many stressed the importance of data-driven decision making. Yet these plans largely assumed that detailed and reliable data would simply . . . exist. They were less concerned with how those data would actually be made." [0] https://statmodeling.stat.columbia.edu/2021/03/21/whassup-with-the-haphazard-coronavirus-statistics/ https://statmodeling.stat.columbia.edu/2021/03/21/whassup-wi...

mlthoughts2018 6y ago

Companies don’t pay financial rewards for doing the data plumbing work, yet it’s often much harder, prone to unusual failure domains mixed with high scaling, and carries more stringent on-call and incident triage responsibilities. Given that it’s (a) more difficult, (b) more business critical, (c) more stressful and requiring an incident alerting on-call rotation, then “data work” should be much better paid and offer job security and career growth. Yet no company I know of pays expert modelers & researchers less than expert data platform engineers. So either the companies know something you don’t (e.g. that data platform work is more commodity and easier to replace than rarer modeling talent) or there’s a free lunch you can get by exploiting the arbitrage opportunity to pay data platform experts more and consume correspondingly higher business value that other orgs are missing out on by putting modeler / researcher higher on the status hierarchy than data platform engineer. My perspective after many years of experience managing machine learning teams (both platform/infra and research/modeling) is that data platforming is just a worse job. It’s unpleasant and stressful and business stakeholders who are removed from backend engineering complexity and just want the report or just want the model couldn’t care less about organizational structures and workflows that support healthier lives for intermediate data platform teams. Because of this, the pay and bonuses for data platform roles should be much higher, but politically speaking it’s impossible to advocate for that, so it becomes a turnover mill where everyone burns out to keep the existing shitty system running, with comparatively low pay and low autonomy, and so nobody ends up wanting to join that team or do that work.

yachtman 6y ago

It is similar to an article I read a while back saying we need "data engineers" not "data scientists". I think its generally true. Why my company, while it can do data science consulting, is choosing to focus on data engineering consulting first.

finnthehuman 6y ago
I've always been annoyed by the title "data scientist" because I don't consider their methods worthy of the name science. It seems like the winds are turning and more people are questioning data "science." But talking about changing the branding to deliver the product that was advertised all along feels like putting the cart before the horse. Sure, maybe it will work. Or maybe in a few years we'll be talking about how we need more "data sleuths" to replace all those haphazard "data engineers" that just didn't have a job title that emphasized solving the problems rigorously enough.

beyondthecut 6y ago

Someone needs to do the work of making that article readable on mobile...

vizually 6y ago

If a start-up stage company is just getting started on their AI/ML journey has budget for 3 FTE people. The company already has traditional ETL/BI expertise and a "DWH". Who would you hire (data scientist, ML engineer, Dat Engineer) and how would you allocate the division of responsibilities?