11 ms·
The Machine Learning Software Engineering Interview
- brendanw 7y agoIt would be great if companies would allow you to bypass the code challenge if you grant them read git access to a relevant project that you have ownership of.
- codingslave 7y agoProblem is its easy to cheat about whats yours
- shantly 7y agoNot much harder than with a take-home challenge. Either’d be tough to do convincingly.
- jorblumesea 7y agoAs if regurgitating someone else's leetcode answers is any better? Or googling or stackoverflowing answers? Or a take home challenge? Even at the ML level, engineering is more about applying known good techniques and less about new innovative ideas.
- codingslave 7y agoCompanies think regurgitating leetcode is better... the number of problems that can be asked is quite large. Memorizing that search space in itself takes tenacity and a good memory...of which are signs of a good employee (regardless of their current coding skill set)
- davnicwil 7y agoI think the trick here is have people talk through what it does, how and why (as in what are the tradeoffs, what other approaches could have been taken, etc). You can't fake this. Or to put it better, even if the code isn't yours and you can do this well, it doesn't even matter that the code isn't yours as in doing this you by definition have the skills and knowledge to reimplement it anyway.
- pmikesell 7y agoYou can absolutely fake this. There’s a huge difference between coming up with and implementing something vs understanding it enough to be convincing. Have you actually tested this out?
- davnicwil 7y agoYes I've done this both as an interviewer and interviewee, it's a pretty valuable exercise I've found. Not a silver bullet, not the only thing you can do, not appropriate for every situation, but generally good. Regarding your other point I agree with you, but if the main point is to evaluate understanding then the difference doesn't really matter. If you're testing for ability to invent concepts etc, then this might be more important to you.
- dcolkitt 7y agoOkay, but at this point it sounds pretty much the same as a traditional interview where the candidate discusses their prior experience.
- dahart 7y agoThis is a common sentiment among applicants, but after many years of hiring people, just FYI, looking at people’s github projects is one of the lowest quality signals I get from an applicant. It takes me forever to wade through someone’s code to figure out what they did or how good they are, and when there are multiple people on the project, it’s not easy to see who’s driving or what the cooperation/conflict situation is by looking at the git history. It can also backfire in a variety of ways you might not want and not know about, so keep this in mind. Your personal project code sometimes has lots of decisions in it that don’t look particularly good to a very experienced engineer. I can’t even count how many people have sent me github links for code they’re proud of that when I study it makes me second-guess the candidate a little bit. As an interviewer, I honestly prefer to hear the candidate talk through their code so I can find out they’re learning and hopeful rather than read code that says more about today’s limitations than tomorrow’s potential.
- sh87 7y agoI had to read through your comment twice to get it. My experience leads me to agree with your idea of letting people talk through their own code. That category of conversations has usually been very rich and telling, much better than one over imaginary binary tree rotation algorithms or a "design a url shortener"-ish question. My favorite part is where a skillful candidate can lead the interviewer to interesting and real problems and solutions, demonstrate ability and both get to enjoy the conversation.
- EpicEng 7y agoOn a whim I responded to a recruiter last week about a job. The first step was an online coding challenge. The company wanted a "Java developer", and I told the recruiter I have never used Java regularly, but given my experience she said it wasn't a problem. I figured the code test would be some simple filter stuff. Instead, it was five algorithm quizzes... all in Java. I had an hour to finish it and, after spending the first fifteen minutes searching for Java standard library stuff just so I could write the code I wanted, I decided this was a waste of my time and closed the tab. This was all before I could even speak with someone who works at the company to find out if I cared to move forward. I worked as a hiring manager for a couple of years and have had input into the hiring process for more than a decade. I've never found it difficult to screen someone out who had no place applying to begin with by taking a fifteen minute call. This at least gives them a chance to ask some questions and they don't feel like you're yanking them around.
- ptd 7y agoEnjoyed the post but I found this sentence interesting. >As a candidate, it’s easy to get a foot in the door and be evaluated by an interviewer. Do people really feel that way about a ML SWE postion at Lyft? I would be interested to see what percent of applications they receive make it to the door.
- dahart 7y agoThe article does hint at the percent: “YMMV: healthy numbers for an interview loop for the phone screen to onsite, to offer, and to offer acceptance are, for example, 50%, 25%, and 70%.”
- thundergolfer 7y ago> In the context of the modeling onsite, we ask open-ended problems with sufficient business and problem context such that the candidate can clearly identify an ML-based approach to solve it. I'm disappointed the author wasn't more specific about where the line is drawn between "ML SWE" and "Research Scientist"/"Data Scientist" when it comes to the core ML competencies like model selection, evaluation, and design. Having worked in teams with Data Scientists and "ML Engineers" it's been murky whether me and others in my team that were not the former were Data Engineers, Backend Software Engineers, "ML SWE", or "Software Engineer - Machine Learning". This area of software is rapidly developing and there's no semblance of the bright line that tends to exist between Backend Engineer and Frontend Engineer for ML-involved engineers. I'm personally interested in moving into the "Software Engineer - ML" space (or is it "ML SWE"?) and thus I have to find out what the right balance between Software Eng skills and researcher skills is. I think I have a decent sense of real-world ML basics but would quickly flounder when pressed for details on mathematical technicalities of different modelling approaches. Lyft's job listings have these items for "Software Engineer - Machine Learning": > 5+ years (or Ph.D. with 2+ years) of industry or research experience developing ML models This really seems like the purview of a research scientist or a data scientist, if I understand the meaning of "developing" correctly. > Proven ability to quickly and effectively turn research ML papers into working code This seems fair enough, but my experience has been that the data scientists are doing this mostly, while the "Software Engineer - ML" people are preparing data pipelines or batch training systems. > Deep knowledge of ML libraries like scikit-learn, Tensorflow, PyTorch, Keras, MXNet, etc I know it's a job ad and thus it is a 'wish list', but this seems unreasonable. Guess I'll just keeping studying 'all the things'. Brb in 5 years.
- mlthoughts2018 7y agoI think you underestimate the level of software architecture and engineering skill that people with formal training in graduate level statistics bring to these jobs. I manage a team of machine learning engineers in a mid-size ecommerce company and I can tell you that the same person who is optimizing Dockerfiles for better layer reuse & figuring out how our CI pipeline will safely get secrets needed to retrieve model files for integration tests is the same person researching new variations of triplet loss in a paper from arxiv they present in our team’s journal club & developing Bayesian hierarchical regression models and explaining why partial pooling using an industry-specialized prior distribution actually produces coefficients in the model fit that have a meaningful improvement over OLS for some business outcome. The same people that are refactoring a collection of unit tests to be parallelizable and save us 45 seconds on every test run are also running huge hyperparameter tuning experiments to get a learning rate scheduler that allows us to reduce training time for an in-house deep GRU neural network from 24 hours to 15 hours, and they are defining infrastructure as code tooling to even create the very GPU environments where this training is taking place. The skill set really is extremely different from general backend engineer with a proclivity for hacking around ML models and also is really different from “research developer” who rarely deals with end to end systems or necessary concerns of production or quality code factoring, and also is super different from data science which is effectively just ad hoc business analytics but with more impressive pedigree on the resume. I’d define a machine learning engineer as taking someone with several years of graduate experience in statistics / machine learning inclusive of formal probability theory, analysis, topology, Bayesian stats (or work + research experience that is equivalent, though it’s super rare for this to be able to replace formal graduate math training), and then adding senior level skill set in high-performance computing, generalist backend engineering, system architecture, and full management of the lifecycle of complex production systems. The only piece of common modern engineering that I would say ML engineers typically don’t have a senior-ish level of command over is frontend development (though some do out of just hobbyist interest).
- yanilkr 7y agoThis blog post was so painful for me to read. This is a symptom of "bullshit" going on around in big tech companies. "bullshit" here is an economic term defined in the book "bullshit jobs". https://www.amazon.com/Bullshit-Jobs-Theory-David-Graeber/dp/150114331X https://www.amazon.com/Bullshit-Jobs-Theory-David-Graeber/dp... Reading through the post, I was noticing So much corporate Jargon which really does not mean anything important. Dehumanizing language when describing people interviewing and being interviewed and its process. Too much obfuscation of ideas that can be very simply explained. glorification of simpler problems into heroic challenges. Delusions of Grandeur. Today's such jobs are tomorrows layoffs. I think I will stop here. I have crossed my negativity threshold for the day.
- deleted 7y ago[deleted]
- sidlls 7y ago"Obfuscation" and "delusions of grandeur" are practically synonyms for ML and Data "Science" in this industry. I've been around for a while and I've never quite seen something as over-hyped and hyper-glamorized as these two specializations.
- danielmg 7y agoDoes "blockchain" get an honourable mention?
- sidlls 7y agoDefinitely.
- codesushi42 7y agoReally? Were you around during the dotcom era? Although I'm not old enough, I've heard that OR in the 80s was the same crap.
- mr_toad 7y ago
- sportanova 7y ago> Most companies are open about the expectations for the role being interviewed for, the interview process, and preparation tips When did this happen? I haven't interviewed for a few years, but nothing was open or clear then
- shantly 7y ago“Most” is certainly wrong. I’d allow “some”.
- FpUser 7y agoSummary: we do not really want you to work for us, we are too busy trying to understand WTF do we really want. Amount of BS the article is staggering IMO
- thundergolfer 7y agoI thought it was because I was reading it right as I woke up, but I've re-read it now and it's almost dizzying how it's cloaking the details but written as if it's open and clear.
- FpUser 7y agoWhen I was trying to decipher it I thought it was written by somebody in a state of delirium .
- aloknnikhil 7y agoThis just reeks of mostly inexperience with a touch of arrogance. > We looked at other companies’ equivalent roles but they weren’t exactly contextualized to Lyft’s business setting. Ok, so what does Lyft need then? > What are Lyft’s challenges (and can a specific role help)? > What should the role be with respect to the organization’s goals? > What are the desired skills, knowledge, and talents given the expectations for the role? Umm. Isn't that what every company hiring looks to address? > What’s left is to define the necessary ingredients for what a successful hire looks like vis-a-vis (3) in Lyft’s context: > > Skills acquired through practice, > Knowledge learned through study and personal experience, and > Talents that make each candidate unique. Ok, I'll stop reading now since "Lyft's context" is no different from any other company let alone one looking to hire a Machine Learning Engineer/Scientist.
- starpilot 7y agoI think this airy, self-aggrandizing post full of BS will be very offputting to data scientists and MLE's. Very low information density here.
- PeCaN 7y agothat sounds perfect for data scientists and machine learning 'engineers'
- applecrazy 7y agoI'm confused at the animosity towards ML SWEs and data scientists within this thread. Yes, there's a lot of hype/BS surrounding the field, but there's also some very highly technical people making very novel discoveries. My question is: why the hate? Isn't a machine learning engineer just as valid as any other engineer?
- starpilot 7y agoGood source of data for training a de-bullshitter. Input: A year and a half ago when we began scouting for this type of machine learning-savvy engineer —something we now call the machine learning Software Engineer (ML SWE) — it wasn’t something we knew much about. We looked at other companies’ equivalent roles but they weren’t exactly contextualized to Lyft’s business setting. This need motivated an entirely new role that we set up and started hiring for. Target: We invented a position called a machine learning software engineer. Input: First, candidates on the ML SWE loop go through Lyft’s hiring review. The review is a regularly scheduled session for a committee to study candidates with an unbiased perspective and decide whether to hire them. Working alongside the review committee is a separate panel of interviewers that provides technical feedback. This feedback is designed to help the committee decide if there’s a fit and, if so, the candidate’s technical level. At first glance, this review process may seem cumbersome. Examining the checks and balances more carefully, however, we notice that they are intentionally introduced to put friction on the hiring process. Having a consistent review committee unifies standards and eliminates bias. Target: A committee and a group of interviewers evaluate a candidate for fit and technical level. Input: Despite the what-ifs, being transparent about how we design interviews can improve our interviews. Call it enlightened self-interest: candidates invest time to talk to us and we mutually benefit from learning if there is a good fit. Even if there isn’t an immediate fit, positive experiences build brand and improves candidate sourcing. Maybe the candidate can reapply when the timing is better. Practically, hiring an engineer easily costs tens of thousands of dollars. By showing how we iterate on our interviews, we reveal what we truly care about and how we try to probe at them, hopefully adding to the virtuous cycle for the hiring pipeline. Target: We don't respect the time of our readers and are hopelessly unenlightened on this fact.
- fishingisfun 7y agowhat an absolute mess. this is the private corps equivalence inefficiency of the same madness i see with union jobs in the public sector.
- the_cat_kittles 7y agoim all for the absolute beating this article is taking. what confuses me if why soooo many equally bad if not worse articles make it through here without a scratch
- syspec 7y agoWow this post is really up it’s own butt, talk about self important
- the_enigma 7y agoWow that was a waste of time reading it.
- high_derivative 7y agoTo save some people the read, it's the same leetcode bs performance as anywhere else.