14 ms·

Ask HN: Machine learning engineers, what do you do at work?

• tambourineman88 2y ago
  The opposite of what you’d think when studying machine learning… 95% of the job is data cleaning, joining datasets together and feature engineering. 5% is fitting and testing models.
  • toephu2 2y ago
    Sounds like a Data Scientist job?
    • moandcompany 2y ago
      This is a large problem in industry: defining away some of the most important parts of a job or role as (should be) someone else's. There is a lot of toil and unnecessary toil in the whole data field, but if you define away all of the "yucky" parts, you might find that all of those "someone elses" will end up eating your lunch.
      • hiatus 2y ago
        > There is a lot of toil and unnecessary toil in the whole data field, but if you define away all of the "yucky" parts, you might find that all of those "someone elses" will end up eating your lunch. See: the use of "devops" to encapsulate "everything besides feature development"
      • tedivm 2y ago
        It's not about "yucky" so much as specialization and only having a limited time in life to learn everything. Should your reseacher have to manage nvidia drivers and infiniband networking? Should your operations engineer need to understand the math behind transformers? Does your researcher really gain any value from understanding the intricacies of docker layer caching? I've seen what it looks like when a company hires mostly researchers and ignores other expertise, versus what happens when a company hires diverse talent sets to build a cross domain team. The second option works way better.
        • AndrewKemendo 2y ago
          My answer is yes to both of those If other peoples work is reliant on yours then you should know how their part of the system transforms your inputs Similarly you should fully understand how all the inputs to your part of the system are generated No matter your coupling pattern, if you have more than 1 person product, knowing at least one level above and below your stack is a baseline expectation This is true with personnel leadership too, I should be able to troubleshoot one level above and below me to some level of capacity.
          • otteromkram 2y ago
            The parent comment had three examples...
            • mrbombastic 2y ago
              2/3 is close enough in ML world
        • moandcompany 2y ago
          > I've seen what it looks like when a company hires mostly researchers and ignores other expertise, versus what happens when a company hires diverse talent sets to build a cross domain team. The second option works way better. I've seen these too, and you aren't wrong. Division into specializations can work "way better" (i.e. the overall potential is higher), but in practice the differentiating factors that matter will come down to organizational and ultimately human-factors. The anecdotal cases I draw my observations from organizations operating at the scale of 1-10 people, as well as 1,000s working in this field. > Should your reseacher have to manage nvidia drivers and infiniband networking? Should your operations engineer need to understand the math behind transformers? Does your researcher really gain any value from understanding the intricacies of docker layer caching? To realize the higher potential mentioned above, what they need to be doing is appreciating the value of what those things are and those who do those things beyond: these are the people that do the things I don't want to do or don't want to understand. That appreciation usually comes from having done and understanding that work. When specializations are used, they tend to also manifest into organizational structures and dynamics which are ultimately comprised of humans. Conway's Law is worth mentioning here because the interfaces between these specializations become the bottleneck of your system in realizing that "higher potential." As another commenter mentions, the effectiveness of these interfaces, corresponding bottlenecking effects, and ultimately the entire people-driven system is very much driven by how the parties on each side understand each other's work/methods/priorities/needs/constraints/etc, and having an appreciation for how they affect (i.e. complement) each other and the larger system.
    • auntienomen 2y ago
      A good DS can double as an MLE.
      • disgruntledphd2 2y ago
        And sometimes, a good MLE can double as a DS. Personally I think we calcified the roles around data a little too soon but that's probably because there was such demand and the space is wide.
    • RSZC 2y ago
      Used to do this job once upon a time - can't overstate the importance of just being knee-deep in the data all day long. If you outsource that to somebody else, you'll miss out on all the pattern-matching eureka moments, and will never know the answers to questions you never think to ask.
    • huygens6363 2y ago
      “Scientist”? Is this like Software Engineer?
      • staunton 2y ago
        I guess it means "someone who has or is about to have a PhD".
    • maxlamb 2y ago
      Sounds like a data engineer job to me
    • jamil7 2y ago
      My partner is a data engineer, from what I’ve gathered the departments are often very small or one person so the roles end up blending together a lot.
  • dblohm7 2y ago
    As somebody whose machine learning expertise consists of the first cohort of Andrew Ng's MOOC back in 2011, I'm not too surprised. One of the big takeaways I took from that experience was the importance of getting the features right.
    • geoduck14 2y ago
      >was the importance of getting the features right. Yeah, but also knowing which features to get right. Right?
    • Animats 2y ago
      I remember that class. Someone from Blackrock taught it at Hacker Dojo. The good old days of support vector machines and Matlab.
    • ismailmaj 2y ago
      This was very important with classical machine learning, now with deep learning, feature engineering became useless as the model can learn the relevant features by itself. However, having a quality and diverse dataset is more important now than ever.
      • Salgat 2y ago
        That depends on the type of data, and regardless, your goal is to minimizing the input data since it has a direct impact on performance overhead and duration of inference.
      • beckhamc 2y ago
        no we just replaced feature engineering with architectural engineering
  • AndrewKemendo 2y ago
    As it was in the beginning and now and ever shall be amen At the staff/principal level it’s all about maintaining “data impedance” between the product features that rely on inference models and the data capture This is to ensure that as the product or features change it doesn’t break the instrumentation and data granularity that feed your data stores and training corpus For RL problems however it’s about making sure you have the right variables captured for state and action space tuple and then finding how to adjust the interfaces or environment models for reward feedback
  • llama_person 2y ago
    Same here, it's tons of work to collect, clean, validate data, followed by a tiny fun portion where you train models, then you do the whole loop over again.
    • gopher_space 2y ago
      > it's tons of work to collect, clean, validate data That's my fun part. The discovery process is a joy especially if it means ingesting a whole new domain and meeting people.
  • whiplash451 2y ago
    In a sense, the data _is_ the model (inductive bias) so splitting « data work » and « model work » like you do is arbitrary.
• hirako2000 2y ago
  The amount of response may be self explaining. Not my main work, but spending a lot of time gluing things together. Tweaking existing open source. Figuring out how to optimize resources, retraining models on different data sets. Trying to run poorly put together python code. Adding missing requirements files. Cleaning up data. Wondering what could in fact really be useful to solve with ML that hasn't been done years ago already. Browsing the prices of the newest GPUs and calculating whether that would be worth it to get one rather than renting overpriced hours off hosting providers. Reading papers until my head hurt, that is just 1 by 1, by the time I finish the abstract and glanced over a few diagrams in the middle.
  • ZenMikey 2y ago
    Where do you locate/how do you select papers?
    • hirako2000 2y ago
      arXiv.org searches Dedicated groups on social. e.g: https://www.linkedin.com/newsletters/top-ml-papers-of-the-week-7020865424875474944 https://www.linkedin.com/newsletters/top-ml-papers-of-the-we...
• davedx 2y ago
  pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Macbook Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python pip install … oh time to go home!
  • makapuf 2y ago
    Maybe pip should not work by default (but python -m venv then pip install should)
    • avmich 2y ago
      Legends say there were times when you'd have a program.c file and just run cc program.c, and then could just execute the compiled result. Funny that programmer's job is highly automatable, yet we invent ourselves tons of intermediate layers which we absolutely have to deal with manually.
      • EnergyAmy 2y ago
        And then you'd have to deal with wrong glibc versions or mysterious segfaults or undefined behavior or the the code assuming the wrong arch or ...
        • KeplerBoy 2y ago
          python solves none of those issues. It just adds a myriad of ways those problems can get to you. All of a sudden you have people with C problems, who have no idea they're even using compiled dependencies.
          • EnergyAmy 2y ago
            In theory you're right, CPython is written in C and it could segfault or display undefined behavior. In practice, you're quite wrong. It's not really much of a counterargument to say that Python is good enough that you don't have to care what's under the hood, except when it breaks because C sucks so badly.
• deleted 2y ago
  [deleted]
• itake 2y ago
  not sure if this counts as ML engineering, but I support all the infra around the ML models: caching, scaling, queues, decision trees, rules engines, etc.
  • selimthegrim 2y ago
    What do you do with decision trees specifically?
  • barrenko 2y ago
    MLOps, sure.
• angarg12 2y ago
  My job title is ML Engineer, but my day to day job is almost pure software engineering. I build the systems to support ML systems in production. As others have mentioned, this includes mostly data transformation, model training, and model serving. Our job is also to support scientists to do their job, either by building tools or modifying existing systems. However, looking outside, I think my company is an outlier. It seems in the industry the expectations for a ML Engineer are more aligned to what a data/applied scientist does (e.g. building and testing models). That introduces a lot of ambiguity into the expectations for each role in each company.
  • hnthrowaway0328 2y ago
    That's really the kind of job I'd love. Whatever the data is, I don't care. I make sure that the users get the correct data quickly.
  • tedivm 2y ago
    In my experience your company is doing it right, and doing it the way that other successful companies do. I gave a talk at the Open Source Summit on MLOps in April, and one of the big points I try to drive home is that it's 80% software development and 20% ML. https://www.youtube.com/watch?v=pyJhQJgO8So https://www.youtube.com/watch?v=pyJhQJgO8So
  • exegete 2y ago
    My company is largely the same. I’m an MLE and partner with data scientists. I don’t train or validate the models. I productionize and instrument the feature engineering pipelines and model deployments. More data engineering and MLOps than anything. I’m in a highly regulated industry so the data scientists have many compliance tasks related to the models and we engineers have our own compliance tasks related to the deployments. I was an MLE at another company in the very same industry before and did everything in the model lifecycle and it was just too much.
• trybackprop 2y ago
  In a given week, I usually do the following: * 15% of my time in technical discussion meetings or 1:1's. Usually discussing ideas around a model, planning, or ML product support * 40% ML development. In the early phase of the project, I'm understanding product requirements. I discuss an ML model or algorithm that might be helpful to achieve product/business goals with my team. Then I gather existing datasets from analysts and data scientists. I use those datasets to create a pipeline that results in a training and validation dataset. While I wait for the train/validation datasets to populate (could take several days or up to two weeks), I'm concurrently working on another project that's earlier or further along in its development. I'm also working on the new model (written in PyTorch), testing it out with small amounts of data to gauge its offline performance, to assess whether or not it does what I expect it to do. I sanity check it by running some manual tests using the model to populate product information. This part is more art than science because without a large scale experiment, I can only really go by the gut feel of myself and my teammates. Once the train/valid datasets have been populated, I train a model on large amounts of data, check the offline results, and tune the model or change the architecture if something doesn't look right. After offline results look decent or good, I then deploy the model to production for an experiment. Concurrently, I may be making changes to the product/infra code to prepare for the test of the new model I've built. I run the experiment and ramp up traffic slowly, and once it's at 1-5% allocation, I let it run for weeks or a month. Meanwhile, I'm observing the results and have put in alerts to monitor all relevant pipelines to ensure that the model is being trained appropriately so that my experiment results aren't altered by unexpected infra/bug/product factors that should be within my control. If the results look as expected and match my initial hypothesis, I then discuss with my team whether or not we should roll it out and if so, we launch! (Note: model development includes feature authoring, dataset preparation, analysis, creating the ML model itself, implementing product/infra code changes) * 20% maintenance – Just because I'm developing new models doesn't mean I'm ignoring existing ones. I'm checking in on those daily to make sure they haven't degraded and resulted in unexpected performance in any way. I'm also fixing pipelines and making them more efficient. * 15% research papers and skills – With the world of AI/ML moving so fast, I'm continually reading new research papers and testing out new technologies at home to keep up to date. It's fun for me so I don't mind it. I don't view it as a chore to keep me up-to-date. * 10% internal research – I use this time to learn more about other products within the team or the company to see how my team can help or what technology/techniques we can borrow from them. I also use this time to write down the insights I've gained as I look back on my past 6 months/1 year of work.
• deleted 2y ago
  [deleted]
• frankPoole 2y ago
  Pretty much the same as the others, building tool, data cleaning, etc. But something I don't see mentioned: experiment design/ data collection protocols
• tenache 2y ago
  Although I studied machine learning and was originally hired for that role, the company pivoted and is now working with LLMs, so I spend most of my day working on figuring out how different LLMs work, what parameters work best for them, how to do RAG, how to integrate them with other bots.
  • KeplerBoy 2y ago
    Would you not consider LLMs as a part of machine learning?
    • Cyclone_ 2y ago
      I'd say deep learning is a subset of machine learning, and LLMs are a subset of deep learning.
    • chudi 2y ago
      Probably it's because we are not training them anymore and just using with prompts. Seems like more of a swe regular type of job
      • aulin 2y ago
        except regular swe is way more fun than writing prompts
    • layer8 2y ago
      They are the result of machine learning.
    • uoaei 2y ago
      There is a vanishingly small percentage of people actually working on the design and training of LLMs vs all those who call themselves "AI engineers" who are just hitting APIs.
• mardifoufs 2y ago
  I work on optimizing our inference code, "productizing" our trained models and currently I'm working on local training and inference since I work in an industry where cloud services just aren't very commonly used yet. It's super interesting too since it's not LLMs, meaning that there aren't as many pre made tools and we have to make tons of stuff by ourselves. That means touching anything from assessing data quality (again, the local part is the challenge) to using CUDA directly as we already have signal processing libs that are built around it and that we can leverage. Sometimes it also involves building internal tooling for our team (we are a mixed team of researchers/MLEs), to visualize the data and the inferences as again, it's a pretty niche sector and that means having to build that ourselves. That allowed me to have a lot of impact in my org as we basically have complete freedom w.r.t tooling and internal software design, and one of the tools that I built basically on a whim is now on the way to be shipped in our main products too.
• burnedout_dc4e3 2y ago
  I've been doing machine learning since the mid 2000s. About half of my time is spent keeping data pipelines running to get data into shape for training and using in models. The other half is spent doing tech support for the bunch of recently hired "AI scientists" who can barely code, and who spend their days copy/pasting stuff into various chatbot services. Stuff like telling them how to install python packages and use git. They have no plan for how their work is going to fit into any sort of project we're doing, but assert that transformer models will solve all our data handling problems. I'm considering quitting with nothing new lined up until this hype cycle blows over.
  • naveen99 2y ago
    You’re living the dream. Why quit ?
    • bowsamic 2y ago
      Is that really your idea of a dream?
      • naveen99 2y ago
        My dreams are usually more disturbing, or fun… But yes. My work is kind of similar… I do some data curation / coding, and help 2 engineers who report to me. I enjoy it.
    • burnedout_dc4e3 2y ago
      I like to feel useful, and like I'm actually contributing to things. I probably didn't express it well in my first post, but the attitude is very much that my current role is obsolete and a relic that's just sticking around until the AI can do everything. It means I'm marginalized in terms of planning. The company has long term goals that involve making good use of data. Right now, the plan is that "AI" will get us there, with no plan B is it doesn't work. When it inevitably fails to live up to the hype, we're going to have a bunch of clobbered together systems that are expensive to run, rather than something that we can keep iterating on. It means I'm marginalized in terms of getting resources for projects. There's a lot of good my team could be doing if we had the extra budget for more engineers and computing. Instead that budget is being sent off to AI services, and expensive engineer time is being spent on tech support for people that slapped "LLM" all over their resume.
• giantg2 2y ago
  I've interviewed for a few of the ML positions and turned them down because they were just data jockey positions.
• redwood 2y ago
  Do people feel like they are more or less in demand with the hyper around genai?
  • uoaei 2y ago
    Demand is higher for flashy things that look good on directors' desks, definitely. But there's less attention on less flashy applications of machine learning, unless your superiors are so clueless that they think what you're doing is GenAI. Which sometimes the systems/models being trained are legitimately generative, but in the more technical, traditional sense.
  • trybackprop 2y ago
    [dead]
• redwood 2y ago
  What are the tools people up to use? Feature platforms like Tecton on the list?
• schmookeeg 2y ago
  Getting my models dunked on by people who can't open MS Outlook more than 3 tries out of 5, however, have a remarkable depth and insight into their chosen domain of expertise. It's rather humbling. Collaborating with nontechnical people is oddly my favorite part of doing MLE work right now. It wasn't the case when I did basic web/db stuff. They see me as a magician. I see them as voodoo priests and priestesses. When we get something trained up and forecasting that we both like, it's super fulfilling. I think for both sides. Most of my modeling is healthcare related. I tease insights out of a monstrous data lake of claims, Rx, doctor notes, vital signs, diagnostic imagery, etc. What is also monstrous is how accessible this information is. HIPAA my left foot. Since you seemed to be asking about the temporal realities, it's about 3 hours of meetings a week, probably another 3 doing task grooming/preparatory stuff, fixing some ETL problem, or doing a one-off query for the business, the rest is swimming around in the data trying to find a slight edge to forecast something that surprised us for a $million or two using our historical snapshots. It's like playing wheres waldo with math. And the waldo scene ends up being about 50TB or so in size. :D
  • soared 2y ago
    3 hours of meetings a week, that’s incredible. Sounds like your employer understands and values your time!
    • visarga 2y ago
      13 meetings/week, at least one full day of work wasted for me
    • schmookeeg 2y ago
      They really do. This has been my longest tenure at any position by far and Engineer QoL is a massive part of it. Our CTO came up through the DBA/Data/Engineering Management ranks and the empathy is solidly there. As we grow, I'm ever watchful for our metamorphosis into a big-dumb-company, but no symptoms yet. :)
  • saulrh 2y ago
    > HIPAA my left foot. That was my experience as well - training documentation for fresh college grads (i.e. me) directed new engineers to just... send SQL queries to production to learn. There was a process for gaining permissions, there were audit logs, but the only sign-off you needed was your manager, permission lasted 12 months, and the managers just rubber-stamped everyone. That was ten years ago. Every time I think about it I find myself hoping that things have gotten better and knowing they haven't.
• primaprashant 2y ago
  Been working as an MLE for the last 5 years and as another comment said most of the work is close to SWE. Depending on the stage of the project I'm working on, day-to-day work varies but it's along the lines of one of these: - Collaboration with stakeholders & TPMs and analyzing data to develop hypotheses to solve business problems with high priority - Framing business problems as ML problems and creating suitable metrics for ML models and business problems - Building PoCs and prototypes to validate the technical feasibility of the new features and ideas - Creating design docs for architecture and technical decisions - Collaborating with the platform teams to set up and maintain the data pipelines based on the needs of new and exiting ML projects - Building, deploying, and maintaining ML microservices for inference - Writing design docs for running A/B tests and performing post-test analyses - Setting up pipelines for retraining of ML models
• deleted 2y ago
  [deleted]
• singularity2001 2y ago
  Teaching others python.
  • reeboo 2y ago
    Underrated comment. At my place of work, I find this to be a huge part of the MLE job. Everyone knows R but none of the cloud tools have great R support.
• exe34 2y ago
  90% of the time it's figuring out what data to feed into neural networks, 2% of the time figure out stuff about neural networks and the other 8% of the time figure out why on earth the recall rate is 100%.
• jackspawn 2y ago
  50%+ of my time is spent on backend engineering because the ML is used inside a bigger API. I take responsibility for the end to end experience of said API, so I will do whatever gives the best value per time spent. This often has nothing to do with the ML models.
• Xenoamorphous 2y ago
  I’m a regular software dev but I’ve had to do ML stuff by necessity. I wonder how “real” ML people deal with the stochastic/gradient results and people’s expectations. If I do ordinary software work the thing either works or it doesn’t, and if it doesn’t I can explain why and hopefully fix it. Now with ML I get asked “why did this text classifier not classify this text correctly?” and all I can say is “it was 0.004 points away to meet the threshold”, and “it didn’t meet it because of the particular choice of words or even their order” which seems to leave everyone dissatisfied.
  • xtagon 2y ago
    Not all ML is built on neural nets. Genetic programming and symbolic regression is fun because the resulting model is just code, and software devs know how to read code.
    • nchfgsj1 2y ago
      Genetic programming however isn’t machine learning, but instead it’s an AI algorithm. An extremely interesting one as well! It was fun to have my eyes opened after being taught genetic algorithms, to then be brought into genetic programming
    • aiforecastthway 2y ago
      Symbolic regression has the same failure mode; the reasons why the model failed can be explained in a more digestible way, but the actual truth of what happened is fundamentally similar -- some coefficient was off by some amount and/or some monomial beat out another in some optimization process. At least with symbolic regression you can treat the model as an analyzable entity from first principles theories. But that's not really particularly relevant to most failure modes in practice, which usually boil down to either missing some qualitative change such as a bifurcation or else just parameters being off by a bit. Or a little bit of A and a little bit of B.
  • hkt 2y ago
    This seems to be the absolute worst of all worlds: the burden of software engineering with the tools of an English Language undergrad.
• npalli 2y ago
  Clean data and try to get people to understand why they need to have clean data.
• rurban 2y ago
  Highly paid cleaning lady. With dirty data you get no proper results. BTW: perl is much better than python on this. Highly paid motherboard troubleshooter, because those all those H100's really get hot, even with watercooling, and we have no dedicated HW guy. Fighting misbehaving third-party deps, as everyone else.
  • shoggouth 2y ago
    Could you talk more about “BTW: perl is much better than python on this.”?
    • eb0la 2y ago
      I haven't touched Perl in more than 20 years... ... but I (routinely) miss something like: $variable = something() if sanity_check() And do_something() unless $dont_do_that
      • jononor 2y ago
        There exists a ternary if statement? foo = something() if sanity_check else None Can replace None with foo (or any other expression), if desired.
      • reportgunner 2y ago
        Both work in python: variable = something() if sanity_check() else None do_something() if not dont_do_that else None
      • c6401 2y ago
        Additionally to the provided answers: if sanity_check(): variable = something() variable = sanity_check() and something (but the standard python two line way is more readable unless it's a throwaway single use code, then whatever)
• rldjbpin 2y ago
  junior level role, but currently it is a mix of working like a proxy product owner and half software engg. the users are researchers and have deep technical knowledge of their use case. it is still a challenge to map their needs into design decisions of what they want in the end. thanks to open-source efforts, the model creation is rather straightforward. but everything around making that happen and shaping it like a tool is a ride. especially love the ever-changing technical stack of "AI" services by major cloud providers rn. it makes mlops nothing more than a demo imho.
  • reportgunner 2y ago
    Junior level role product owner ?
    • rldjbpin 2y ago
      no junior ml engineer in a small team
• lemursage 2y ago
  In larger companies, and, specifically, bigger projects, systems tend to have multiple ML components, and those are usually a mix of large NN models and more classical (ML) algorithms, so you end up tweaking multiple parts at once. In my case optimising for such systems is ~90% of the work. For instance, can I make the model lighter or go faster and keep the performance? Or, can I make it go faster? Loss change, pruning, quantisation, dataset optimisation etc. -- most of the time I'm testing out those options & tweaking parameters. There is of course the deployment part, but this one is usually a quickie if your team has specific processes/pipelines for this. There's a checklist of what you must do while deploying, along with cost targets. In my case, there are established processes and designated teams for cleaning & collecting data, but you still do a part of it yourself to provide guidelines. So, even though data is always a perpetual problem, I can shed off most of that boring stuff. Ah, and of course you're not a real engineer if you don't spend at least 1-2% of your time explaining to other people (surprisingly often to a technical staff, but not ML-oriented) why doing X is a really bad idea. Or, just explaining how ML systems work with ill-fitted metaphors.
• ivylee 2y ago
  The same job title could have very different responsibilities. I have been more of a "full-stack" MLE, I work from research and prototyping all the way to production software engineering and ML infra/dev-ops. I have also been mentoring MLEs and ML tech leads/managers - schedule a free 30-min intro call at: https://cal.com/studioxolo/intro https://cal.com/studioxolo/intro.
• usgroup 2y ago
  We live in a time when there are many more alleged machine learning roles than real ones. I'd argue that if you are not spending >50% of your time in model development and research then it is not a machine learning role. I'd also say that nothing necessitates the vast majority of an ML role being about data cleaning, etc. I'd suggest that indicates that the role is de facto not a machine learning role, although it may say so on paper.
• rapjr9 2y ago
  I worked as an engineer on an academic project that used sound to detect chewing. I acted as a domain expert to help the grad student doing the project understand how to select an appropriate microphone to couple to skin, how to interface that microphone to an ADC, offered insights into useful features to explore based on human biology, found a sound processing library that gave us 100's of new features for very little work, helped design a portable version of the electronics, did purchasing, helped troubleshoot during the times we were getting poor results, monitored human safety so we weren't connecting student volunteers to earth ground or AC power, helped write up human subjects committee review application forms, tracked down sources of electrical noise in the system, helped analyze results, reviewed 3D models for a headset to hold the microphone and electronics, wrote some signal processing/feature extracting code, selected an appropriate SBC system, explored confounding factors that were different between lab tests and field tests, and more. The grad student organized tests to record chewing sounds and ground truth video (a cheap camera in the bill of a baseball hat), arranged for a Mechanical Turk-like service to mark up the sound data based on the ground truth using a video analysis app I found, tried different pre-written ML algorithms to figure out which worked the best across different types of food, and analyzed which features contributed most to accuracy so we could eliminate most of them and get the thing to run on a wearable system with reasonable battery life. Then we collaborated with some medical doctors (our target customer for the device, it was not intended for consumers) to run tests on kids in their eating labs and wrote papers about it. The project ended when the grad student graduated. I worked on other ML projects as well. A system that analyzed the syslogs of dozens of computers to look for anomolies. I wrote the SQL (256 fields in the query! Most complex SQL I've ever written) that prefiltered the log data to present it to the ML algorithm. And built a server that sniffed encrypted log data we broadcast on the local network in order to gather the data in one place continuously. Another system used heart rate variability to infer stress. I helped design a smartwatch and implemented the drivers that took in HRV data from a Bluetooth chest strap. We tested the system on ourselves. None of our ML projects involved writing new ML algorithms, we just used already implemented ones off the shelf. The main work was getting the data, cleaning the data, and fine tuning or implementing new feature extractors. The CS people weren't familiar with the biological aspects (digging into Grey's Anatomy), sensors, wireless, or electronics, so I handled a lot of that. I could have done all the work, it's not hard to run ML algorithms (we often had to run them on computing clusters to get results in a reasonable amount of time, which we automated) or figure out which features are important, but then the students wouldn't have graduated :-) Getting labeled data for supervised learning is the most time consuming part. If you're doing unsupervised learning then you're at the mercy of what is in the data and you hope what you want is in there with the amount of detail you need. Interfacing with domain experts is important. Depending on the project a wide variety of different skills can be required, many not related to coding at all. You may not be responsible for them all, but it will help if you at least understand them.