12 ms·
The cold start problem: how to build your machine learning portfolio
- ziont 8y agoSeems to me more of a case where companies no longer train grads, and instead tell them to work on a proof of concept on solving a hard problem without paying for it, a sad state of affairs.
- InterestBazinga 8y agoPlaying Devil's advocate: The other side is they hire a false positive, and suffer through the different sad state of affairs?
- ziont 8y agoSkunkworks: omg this kid has replicated our Auto-GAS system using only computer vision Lockheed Martin CEO: So what the fuck is this youtube video? https://www.youtube.com/watch?v=WkZGL7RQBVw https://www.youtube.com/watch?v=WkZGL7RQBVw
- tensor 8y agoThe reality is that there are enough people with experience in machine learning that companies don't need to train people.
- soraki_soladead 8y agoThis is not remotely true. It's true that there are lots of data scientists who can string together sklearn or R models. However, very few people know how these models work, when to apply them _and why_ or what concerns need to be addressed on data, metrics and deployment. Even fewer know how to improve these models once they catch all of the low-hanging fruit.
- simonw 8y agoI disagree. I think it is absolutely immoral (and dumb from an IP risk point of view) for companies to try to get interviewees to solve real problems for free. This is different though: this is about potential candidates finding creative ways to demonstrate their skills. The companies didn't even ask people to do this: these people chose to take on projects that would demonstrate their skills in a new space in which which they did no yet have commercial experience. I think that's commendable and a very smart strategy.
- munk-a 8y agoThis isn't different though. This article was written as "How to get into ML" and is basically selling the idea that getting into ML requires (or is aided by) doing some independent open source work to bulk up a profile. A very competent developer who is interested in doing some ML work may have just recently read this article and gotten to work on an ML project because he feels that not doing so will hurt his chances of getting a job. As a note, I have no idea what the right solution to this problem is, it is good to confirm candidate's knowledge (I prefer creative ways similar to the parent) and part of that knowledge certainly can include open source work, but going down this road too much leads to people feeling obligated to make open source work to put on their resume and even to people faking open source work to try and land a job.
- jimbokun 8y ago"but going down this road too much leads to people feeling obligated to make open source work to put on their resume" ...which makes the world a better place. Why do you object to people writing open source code?
- autokad 8y agoi agree, and I recommend doing it for those wishing to get into the field. i believe one of the things that helped me get where i am now is my performance in kaggle competitions. i do have one concern if it ever became an industry prereq, it becomes a filter for those who have more free time to work on side projects.
- minimaxir 8y agoThe current problem with things like data science thought pieces and MOOCs is that they don't prepare for the realities of machine learning/data science in the real world. (longer blog post on the subject: https://minimaxir.com/2018/10/data-science-protips/ https://minimaxir.com/2018/10/data-science-protips/) Doing a unique project is much better for learning cost/benefits of implementing AI/ML, although this post may be overoptimistic on how that can lead to a job offer.
- technobabble 8y agoThanks for posting the article. While I've read up on ML, I don't work in the industry. I'm curious to see what other people think of the article. Can other people weigh in?
- solveit 8y agoI'm a junior SWE at a mid-sized AI startup. The article is pretty much spot on, but I'm not sure that the romantic view of data science the article argues against is actually that prevalent. It seems like you would need a very specific level of knowledge to romanticize data science in that way. Most people know too little (So you're basically trying to build skynet?), and most of the rest are either in the industry or know someone that is, and so have a more realistic view. I don't know, I did maths in undergrad, maybe some of the more clueless CS majors thought this way.
- achompas 8y agoI agree with my sibling comment and would add: Max is correct to point out the irony in his anti-thoughtpiece thoughpiece as he falls into the same trap of vagueness as those other articles. Specifically, he rails against general “black box” approaches to modeling, then takes a general “black box” approach to the work of operationalizing a model (much harder than building the prototype to begin with!). The discussion of “pulling data” does not match the practical reality, since pulling via BI tools is not scalable and rarely automatable. SQL may cover this insofar as you dump data from SQL to...what, though? A Python session on your laptop? Automating this process allows a data scientist to scale their impact. For more specificity on engineering practices required for data science, I recommend Robert Chang’s series of posts: https://link.medium.com/CG7c7mQdyS https://link.medium.com/CG7c7mQdyS For details on how a data scientist can impact an organization, I recommend this from the FirstMark blog by Jeremy Stanley and Daniel Tunkelang: https://firstround.com/review/doing-data-science-right-your-most-common-questions-answered/ https://firstround.com/review/doing-data-science-right-your-...
- smrtinsert 8y agoI had a hunch. I'm sitting here perusing theory after theory for years now. I need to just sit down with some data and a python ide. It's more fun that way anyway. This doesn't mean I think I'll get or even want a job, but I'll get a better sense of what the work is and how to best use it.
- aabhay 8y agoThe title of this article is misleading. I thought it was a discussion of the "Cold Start Problem" (https://en.wikipedia.org/wiki/Cold_start_(computing) https://en.wikipedia.org/wiki/Cold_start_(computing)), which is a common technical challenge. Instead, it is a recollection of two stories for how to get a job in ML.
- nostromo 8y agoIt's a playful title -- using a term from the field in a new way.
- dotancohen 8y agoIt is using a term from the field in an incorrect way.
- gdy 8y agoYou've just failed the Turing test :)
- cwilkes 8y agoHuh that’s an interesting way of thinking about the Turing Test. State something that is categorically wrong according to a dictionary lookup (“cold start problem” means X in machine learning and ML only) and see if the respondent can be maliable enough to use that in a different context. Course that’s the definition of a Turing test but in a way I hadn’t thought about.
- gdy 8y agoWell, you know, this common technical challenge got its name from another problem.
- danielmorozoff 8y agoOmg yes ! I was confused having read the first paragraph and did not understand why it was titled this way.
- garysieling 8y agoThis is also a great way to become a better engineer - a lot of jobs are improving existing systems, so you get a different category of lessons building something from scratch to completion. Doing a write-up at the end is also teaches you to communicate better.
- nightski 8y agoI learned two things from this - 1. Apparently modeling is a solved problem. No need for any knowledge of math/stats folks, just use the latest python packages. 2. Data collection is more important than actually knowing how a model works and even more importantly doesn't work. What happens when he is hired and can't put together a reliable model? I think his approach is solid (using projects to learn and being ambitious), but it feels like trying to run before you can walk.
- DrNuke 8y agoI am not afraid to outsource domain-agnostic projects to smart junior practitioners, though. Your conclusion is a cultural barrier where metrics is the real king. Domain expertise is absolutely needed in a few specific cases only, the rest being gatekeeping.
- nightski 8y agoI was maybe a bit too snarky, apologize if so. But the crux of it is that his model never achieved > 50% accuracy. It was never necessary because he already got the job. But the thing is, that is the hard part. Closing those gaps in not just accuracy, but also generalization is the data science portion of the task (and requires much more knowledge than what is demonstrated by this blog post - although they could of left out a lot of detail). They make it seem like if he just had a little bit more time this would of been straightforward. But I am not sure about that. I am all for giving junior practitioners a chance. But this is like hiring an english major for aerospace engineering because they built a model airplane in my opinion. But maybe I vastly underestimate the amount of extremely low hanging fruit out there for ML projects.
- DrNuke 8y agoYou’re welcome and not snarky at all. I am not sure you work with commercially-driven ML projects? It really depends on the metrics only. I can’t trust a ML black box for cancer screening (both false negatives and false positives have a big cost) or complex industrial failures (stopping a plant starts a lot of expensive compensating manouvres) but nobody sane in his mind has a real problem with 88, 90 or 92% accuracy for online retail recommenders (all the field is a kind of magic beyond a certain baseline and money never earned is not money lost from a bookkeeping perspective) or language translation (which is still human-proofed wherever it gets legal value of any kind). Hope I made my point clearer. Cheers.
- Jerry2 8y ago>But it was similar enough that they quickly asked Ron to make his repo private. How is this infringement? Why did he oblige? Unless he got paid by them, he should not have obliged with their request.
- max76 8y agoRon wanted a job with the organization that asked him to make his repo private. Cooperating is the best strategy given the goal. Ron can always make his repo public later.
- jldugger 8y ago> Alex planned to improve his accuracy, of course, but he was hired before he got the chance. Reading these posts I get the impression people prefer portfolio projects over studying the mathematical fundamentals; "dazzle hiring managers with these three easy tricks!" Seems like a great strategy to getting hired, but one wonders how long people using these lifehacks persist in the role.
- joshwcomeau 8y ago(not an ML person, take with grain of salt) I think someone without experience training a model to high-accuracy might struggle at a startup where they're the first hire in the department, but at a larger company, presumably there are senior folks to help newcomers along. I think having motivation is a much better signal, and you need to be motivated to develop a side-project of significant magnitude.
- llampx 8y ago> Reading these posts I get the impression people prefer portfolio projects over studying the mathematical fundamentals It didn't appear as though these two didn't train their mathematical fundamentals though. I believe the gist here is to find an interesting project tailored to the field you want to work in, and make it visual for the non-technical interviewers. Once you do get their attention, you would naturally need to pass a real DS interview, which would test your mathematical fundamentals.
- drawkbox 8y agoCompanies want people that can come in and hit the ground running and are self-starters. It is always better to go into the interview, meeting, pitch with projects that are shipped or working to demonstrate. For instance, if you want to get into the game industry, build games. Same with any field, when you create/build you can rise above education, experience and more competitive metrics. Created/functional projects, and especially shipped products, are the best qualifications.
- technobabble 8y agoIs this true outside of the SV bubble?
- solveit 8y agoCertainly it's true everywhere. It's also not necessary in most places (including many places in SV). But you still want to convince the company that you can learn quickly and well enough to be worth it. If you don't have any of the traditional ways of signalling that (fancy degree, reference from somebody already in the game, etc), then this is probably the best way to go.
- drawkbox 8y agoIn addition to development, any creative field is this way including movies, writing, gaming, design etc. It is also true of vocational skills and probably even legal/accounting/executive and other corporate fields.
- runeb 8y agoI agree with you, but would argue that this doesn't rise above experience, this shows experience. The value of having shipped something, maybe even supported it and maintained it is invaluable experience as a software developer, and not something taught at school, at least not my university.
- drawkbox 8y ago> doesn't rise above experience, this shows experience. Experience is highly valuable but concrete examples of that which are done individually or completely self propelled, that is the way to be. Creating/shipping/projects to show does demonstrate experience and skills. There can be people that have experience that can't ship, but if you created/projects/shipped and have less experience it truly demonstrates that you do have experience enough to be worthwhile and improve the place you are going to. I'd argue even when I have had to interview or been on the hiring side, showing projects beats experience and education, it can also make interviews focused on your projects and what you bring rather than just whiteboard and talking about what you have done. For instance when I went into gaming I had multiple games made for every place I was aiming for, i.e. a racing game at a racing game company, promotional games at advertising/promotional jobs, and networking/multiplayer shooters/online for more hardcore gaming on console and mobile. Ultimately projects are the only way to get clients as a freelancer/small business pitching to clients, so that also works when going for full time employment and jobs.
- hnuser355 8y agoI don’t know why everyone cares about machine learning so much. Frankly there seem to be so many prerequisites to knowing what goes on, when I imagine a software developer could just get better at java and then learn about parallel code or something to get lots of money for less effort. I’m not in software so I don’t know if any of that is actually true
- t0astbread 8y agoThere are lots of ways to make money easier and faster. I guess this is for people who are genuinely interested in the subject.
- visarga 8y agoWell, because ML is fun.
- timdellinger 8y agoThe number of prospective employers that will actually pull up your github / portfolio project website / etc (let alone read it) is not especially different from zero. It's important to have stories to tell during an interview, but getting the interview is the hard part.
- scrollaway 8y agoGetting an interview when you have a consequent Github portfolio is very easy. Prospective employers not only do actually pull up your github profile, many of them will find you because of it. As an aside, writing this makes me realize how successful Github has been at implementing their motto of "social coding" when none of us had any idea wtf that could actually mean or look like.
- timdellinger 8y agoI'd be interested in the numbers on this. To get a rough number, I typed "data scientist" "San Francisco, CA" into indeed.com and it's reporting 2,832 job listings. So of those 2,832 openings that companies are trying to fill, how many employers are out actively recruiting candidates based on their github profiles? It seems to me that Github / social coding is an interesting mix of networking (the social kind) and coding (which demonstrates abilities), but like most networking, it's only one of many ways into the door, and it's unclear how many of the 2,832 positions will be filled via that particular flavor of networking. My guess is that it's the minority.
- jonahx 8y agoThis is false at the company I work for now (an large enterprise) and was false when I was hiring for my startup.
- timdellinger 8y agoI'd be curious to hear your experiences hiring - I've been on the hiring side in the chemical industry, but not in data science. Let's say that the funnel of applicants for one position starts with 250 resumes. How do you proceed? Among the applicants with github profiles, are all of the github profiles looked at?
- munchbunny 8y agoThere's another important trend in these two examples that generalizes well beyond just machine learning. If you show a willingness to scrape resources together to work on difficult problems, that's scrappier than the majority of your competition for the job. Startups love that. Of course actual technical skill matters, but in my experience, the willingness to reach for aspirational goals is rarer than baseline competence, so I'll happily interview someone who shows that specific trait.
- sheeshkebab 8y agoSide projects directly related to the area someone is interested in getting a job in are very useful, or those that use trendy technologies. However if it’s in unrelated technology (or no longer trendy), then I’d argue they should be removed from the profile altogether. These might raise unnecessary questions and trigger biases.
- ScoutOrgo 8y agoI had a similar experience with a personal project. I scrapped NFL player/game stats dating back past 2000 and spent a bunch of time on the data, feature engineering, and modeling. Although my original goal was to beat vegas models (got to even in some metrics, but not enough to cover the juice), it ended up being the best line on my resume. Every interviewer during my last job search asked about it, and it was very easy to talk about. I did make money in the end however, but only because I was buying bitcoins to gamble with in the 2017 season.
- ptd 8y agoSerendipity? Or just good planning.
- pragmacoders 8y agoI agree with the general point of this article. Showing that you have the skill to accomplish a job effectively will get you a job most anywhere. There's a major issue when this is applied to a certain class of problems in ML and Data Science that people tend to ignore. If you could get a job as a civil or mechanical engineer (building bridges or whatnot) by showing that you built a small bridge in your backyard... We'd have some unstable bridges. If hospitals just let residents run the hospital... We'd have a lot of mistreated illnesses. If you could show a realty company that you can build a recommendation engine and they hire you to build their advertisement algorithm... Suddenly you're breaking housing discrimination laws. I am all for folks being able to get jobs from their cool projects. But we need ethical standards and educational standards before folks are given large, real-world problems to work with. We need to take a page out of engineering and medical playbooks and build official education or apprenticeship requirements. We need to have licenses that can be revoked if someone fails to follow ethical or quality standards. So - love creative people getting jobs. Now give them a high-quality education program along with those jobs.
- berti 8y agoA good start would be only actual professional engineers calling themselves engineers.
- klipt 8y agoYou'd need a professional accreditation body for software engineers then.
- umichguy 8y agoWasn't this tried/mooted a few years ago but got nowhere in the end? But you also have to understand that for many/most of these types of professional bodies in other fields, you need to have a basic BS type degree to be accredited and this is where I think it fell down flat as quite a few SWEs (compared to other fields) were self-taught, dropped out etc. and didn't necessarily have a college degree. Maybe they can have a different set of criteria for SW field.
- nharada 8y agoOne tricky thing I've noticed when trying to hire ML people is that it's very difficult to tell when someone is okay at ML and when someone is great at ML from their past projects. Because machine learning is just statistics, it is often resilient to errors in design and implementation. You can mess things up and still get reasonable (but suboptimal) results. This means that someone can easily claim "this problem is difficult to learn for machines" when they fail or claim "we got X% accuracy look how great we are!" when they do okay. But a really good engineer or scientist would have succeeded in the same task, or have gotten X+10% accuracy with the right models, data, or engineering.
- lstmemery 8y agoOn my resume, I compare my results to either the previously implemented model or the state-of-the-art for that specific problem. If you bring someone in for an interview, ask them what the previous best was.