24 ms·
Business questions engineers should ask when interviewing at ML/AI companies
- inetsee 9y agoSome other questions I would want the answers to: 1) Is a Non-Compete Agreement required and what are the terms (time and geographic coverage; compensation during the non-compete time frame)? 2) Is the company going to assert ownership over IP created before employment began? 3) Is the company going to assert ownership over IP created that is unrelated to current responsibilities (or IP in a area unrelated to the company's current business), especially IP created on the employee's own time, not using any company resources?
- vadimberman 9y agoIn California, the non-compete is not worth the paper it's written on: http://prodigylaw.com/CA_Death_of_Non-Compete.html http://prodigylaw.com/CA_Death_of_Non-Compete.html and the state with the strictest non-compete enforcement, Massachusetts, seems to be close to the long-anticipated reform: https://www.faircompetitionlaw.com/2017/10/10/massachusetts-noncompete-reform-what-you-need-to-know-by-october-31/ https://www.faircompetitionlaw.com/2017/10/10/massachusetts-... and https://www.faircompetitionlaw.com/2017/10/31/taking-aim-at-the-hearing-on-massachusetts-trade-secrets-and-noncompete-bills/ https://www.faircompetitionlaw.com/2017/10/31/taking-aim-at-.... With the reformed bill in place, the non-compete clause in Massachusetts will also be a hollow threat unless multiple (unrealistic) conditions are met.
- ryandrake 9y agoThese are great questions to ask any company, not just AI/ML companies. Also, if you're planning to do it: 4) What is the companies policy on employees moonlighting / doing side contract work for other companies not related to it's business? Inquire about this before you get your offer. Ask me how I know... EDIT: Another one, which is kind of risky but I personally think is important and would ask any small company: 5) How transparent/open are you about the company's financial health? Are the books open or semi-open? Keeping in mind that some companies will not share this information and may even find the question offensive. One time I asked the owner/founder this, and he would not share--I found out later through a side channel that he was very offended that I asked for this information (but hired me anyway because he needed someone like me).
- abraae 9y agoRed flag if asking this question courteously causes offence.
- lkrubner 9y ago"Why does anyone need this? Like all advice, this sounds deceptively simple. But make sure you get a very compelling answer here." Right now I'm at the Web Summit, in Lisbon. I saw a few ML/AI startups here, but I was surprised there weren't a lot more. There was some imbalance. There were many startups trying to create new online social networks for niche groups -- business professionals, women, entrepreneurs, sales professionals, etc. I do think there is some innovation that can happen in social networks, but generally I regard it as an over-crowded space. For the most part, the benefits offered by the Internet seemed to have been mostly absorbed by the 1994-2008 cycle, and what's left is fairly minor compared to what happened previously. One of the few areas where I still see the possibility of significant traction (the creation of large companies) is with information gathering and a combination of ML and NLP. We've already seen a wave of startups which were no more than pretty GUIs over existing technology. At the risk of being unfair, BigML.com is just a nice interface over existing ML tools, and API.ai is just a nice interface over some NLP tools. AmenityAnalytics goes a bit further combining their web scraping and NLP scripts with a nice interface that customers can use to filter the incoming data. But there is still a wide space for companies to go much further in this field. All the same, I agree this is a good question: "Why does anyone need this?" You should really ask it whenever you are joining any startup.
- PeachPlum 9y ago"It's like Facebook, but for $group" Groan
- minimaxir 9y agoShouldn't the interviewing engineer know the answers to these business questions beforehand during their research of the company? (particularly if the company is upfront about their ML/AI-usage)
- danicgross 9y agoGood question! Many pre-launch startups lack an online presence, so it’d be hard to tease out answers before talking to someone.
- justonepost 9y agoUsually you're only going to get good answers to these questions after they've made you an offer.
- dagw 9y agoIn my experience the answers I find on the company home page and the answer I get when asking the question in an interview only rarely sync up perfectly and the difference can be very illuminating.
- vadimberman 9y agoThese questions should be asked about every startup, not only ML/AI.
- Eridrus 9y ago> Google got great because of PageRank, but it stayed great due to network effects. I think this overstates network effects and under-appreciates branding and simply reinvesting significant capital into continued R&D.
- danicgross 9y agoAgreed. There are many flavors of moats. I originally wrote something more complex but it quickly turned into an amateur HBS study about Google. I hope the point is interpreted more metaphorically.
- Eridrus 9y agoI've heard the line that Google succeeds because of network effects so many times on HN that I feel like that's the only business concept commenters know about.
- deepnotderp 9y agoI would suggest that in many cases the "data network effect" advantage of ML startups is overappreciated relative to the branding advantage they have. For example, DDG and others can, for the most part, match Google in quality of search, but almost no one will use them because of entrenched branding, this is especially true IMO in smaller ML vertical industries ideal for startups. e.g., if there is only room for one ML company that analyzes farm pipe leaks through camera, then branding and market dominance helps considerably there. I may be wrong, I'm (mostly) a DL hardware guy, this is just my two cents :)
- cycrutchfield 9y ago>For example, DDG and others can, for the most part, match Google in quality of search Even for the long tail? I'm pretty skeptical of this.
- mcny 9y agoGoogle will I'm sure vehemently deny that it is unbeatable but I think with 60k+ searches every second, Google will keep getting better. I think Google is unbeatable in search as we know it.
- murukesh_s 9y agoShould the engineers ask the same question when a high profile VC, like YC have already invested on the company? The questions look more like what a VC should (have) asked before funding. Many ML/AI companies especially if funded and therefore highly visible may get acquired as the market is very hot. Isn't that good for the engineers?
- pjmorris 9y ago> Many ML/AI companies especially if funded and therefore highly visible may get acquired as the market is very hot. Isn't that good for the engineers? Only if the potential acquirers are happy with the answers to the same questions. As a percentage of net worth, individual engineers have a lot more at stake than VC's or potential acquirers.
- murukesh_s 9y agoMost of the big companies(apple/google/microsoft/fb/amazon) at current stage are acquiring for talent/IP, not necessarily market fit.. they do have enough money to do the R&D without worrying about finding a market fit soon
- OliverJones 9y agoIf you're a friend of the VC and he suggests you work for a portfolio company, then maybe you can rely on the VC's judgment without asking questions. General it's always legitimate to ask these questions. Why? Because they cut straight to how the company will be valued, by a customer or by a potential acquirer.
- taneq 9y agoIf equity is part of your compensation, you're an investor and should be doing the appropriate due diligence.
- bluetwo 9y agoOr, you can't figure out how AI is helping solve the problem they claim to solve, and you realize they are just throwing every buzzword out there in hopes you'll be impressed. https://www.youtube.com/watch?v=kzT3yfe2o-I https://www.youtube.com/watch?v=kzT3yfe2o-I
- justonepost 9y agoNot sure why ML/AI is in the title. Seems applicable to any startup.
- logicallee 9y agoit's 2017. half of YC startups have AI in the pitch.
- OscarTheGrinch 9y agoAI is the new UI...
- agibsonccc 9y agoThe gotcha here is that most desirable AI startups SV wants to fund now (understandably) follow a pattern more closely described here: http://www.bradfordcross.com/blog/2017/6/13/vertical-ai-startups-solving-industry-specific-problems-by-combining-ai-and-subject-matter-expertise http://www.bradfordcross.com/blog/2017/6/13/vertical-ai-star... Generally, their advantage is in the way they use data to solve a vertical specific problem. They more "bolt on" AI. AI isn't really the focus of most of these companies. This is just what investors want to hear about. The article here isn't calling them out specifically, but AI infra startups are what are being discussed here as the "hammer looking for a nail" startup. YC funded a few of them including Skymind (my company) and deepgram (mainly audio). There are a lot more of these startups such as vicarious that mainly publish papers but raise tons of funding hoping for a deepmind style exit (they didn't really have a business model). With that context out of the way, I'll also maybe just state some learnings from the other side of the table. Skymind has typically made money trying to reduce dependencies on cloud providers by decoupling the AI infra from google cloud and co from the cloud provider itself. A horizontal play like this is by its nature very hard. We maintain the whole stack including our own framework. We also started in 2014 and have a decent foothold in enterprise. I wouldn't recommend trying to do this in 2017. That being said, one other semi similar horizontal AI startup that is being started are the chip companies. Those are significantly harder to run than even what we're doing. The most common "failed" type of startup that I think of when we think "horizontal" plays are Machine Learning as a service, which is hugely a loss leader for the various cloud providers (some companies are building on top of these providers though). This is where you see Metamind, Alchemy API, Scaled Inference (founded in 2015), even Nervana before they sold were trying to a "nervana asic cloud" among others. Maybe a lot of investors from SV view this as a waste of time dissecting, but I would personally love to see a bit more content on acknowledging some of these trends in the market. The allure of "AI as a service" and the horizontal dev tools infra play is that you can try to build the next AWS similar to what the container and database companies are trying to do. Execution is definitely key for this to work though. Research also can't be the primary focus. I won't comment on what will or won't work there
- okabat 9y agoThe fastest way to make progress on these business questions is often to build hacky MVPs that look like they're doing something smart, but behind the scenes are powered by humans or dead-simple algorithms, and get them in front of customers ASAP. I recently joined a seed-stage startup solving a business problem via audio analysis in the manner I described above. I'm not spending much time doing ML yet, but I'm banking on my belief that we're solving a valuable problem (customers want to buy our hacky MVP) and that ML can and will be needed to scale our solution. By deeply understanding the customer as a first step, I think the ML systems we build will be business critical and enduring. Time will tell
- cirgue 9y ago> dead-simple algorithms The most successful models I've ever built have been logistic regression models. If you can rephrase your problem in a way that's amenable to run-of-the-mill statistical techniques, you can frequently achieve much better results than you can with 'deep learning'.
- madcaptenor 9y agoIf you're going to do logistic regression, at least call it a single-layer neural network with a sigmoid activation function.
- zebraflask 9y agoI'm totally "borrowing" that for later use.
- inputcoffee 9y agoSo true. I have had to do this. I had a pretty good regression model but it was not taken seriously. So I wrote it using "a neural network in TensorFlow" and the next thing you know the whole company is asking me how it works and what it does.
- cirgue 9y agoThis feels so dirty, but these kinds of tricks work. Your stakeholders get to participate in the titillating fiction that they are on the bleeding edge of technology, and you get to deploy a scalable, explainable, and (hopefully) high-performing solution. That's 95% of a win.
- thisisit 9y agoQuestion 6. How often do people talk about ARPUs with potential hires? I am sure people will tell you that there are 150 million potential users but not their revenue models. It is easy to under shoot or overshoot ARPU.
- deleted 9y ago[deleted]
- ordinaryradical 9y agoThis is really solid advice when looking at it backwards too--if you, as a founder, cannot produce reasonable and compelling answers to these questions, you don't know what you're doing, where you're going, or how to actually get there. Great read.
- OliverJones 9y agoThis advice applies to all kinds of fledgling businesses, not just ML/AI.
- alexasmyths 9y agoYes, quite good actually. Unfortunately, sometimes the hype is so strong, the cart comes before the horse and gets acquired for quite a lot - which is why these companies exist.
- siliconc0w 9y agoThe best use of these questions is to get an idea whether your equity might be worth anything. Ask how many customers they have, how fast they're growing, how long does it take to onboard customers, etc. What do they see as an exit (is it a 10M, 100M, 500M a unicorn?). When do they see it happening? If, let's say, you have 0.01% of a Series A that exits at a billion in five years. You'd expect to be diluted by half and there is around a 20% chance of exit that high so roughly ((0.01%billion)/2).20 /5 or about 2k a year in 'expected value'(or why, numerically at least, it's likely not worth it to take a paycut to work at a startup unless you're seeing substantial equity).
- dfee 9y agoPlease let me know where I can find today’s Series A ready companies that have a 20% chance of being worth $1b in 5 years time. That figure probably oughta be at least one order of magnitude less.
- rrdharan 9y agoAgree with the other commenter that this still seems overly optimistic, but also I think as stated your math is off and it’d be 20k/year?
- notyourday 9y agoSo. Much. B.S. There's only one question they need to ask: "How much are you going to pay me?" Edit: The reason anyone would wax poetically about everything else is because he or she realizes that they are not being paid enough. It is like day trading - the moment it goes against a bad trader he or she calls it "investment" Edit: Thanks for the down votes! It makes me feel warm and fuzzy inside to know that it is still possible to swindle people!
- wpietri 9y agoNah. This is a good list of questions. Partly because startups will include equity, and you need to know these things (and more) to make a guess at valuing the equity. And partly because a company without strong answers is likely to be out of business soon; changing jobs unexpectedly can be expensive, especially if you get stiffed on wages. But mostly because it really sucks to spend a couple of years building stuff that never gets used. There are a few people who seem happy to get paid even if nothing is achieved. But most engineers I know do the work because the like building things that get used, things that are valuable.
- 0xbear 9y agoUnless you are employee #1-5, the correct approach to valuing equity is to not consider it to be worth anything. For employees 1-5, divide the numbers the CEO gives you by at least 10 to account for risk, and then demand a competitive base pay.
- wpietri 9y agoYes and no. I agree should expect zero to come out of it, as it's a pretty low probability you'll see anything. But you still have to do the work of valuing it. First off, the people you're negotiating with see it as valuable, so it's a very bad negotiating move to say, "Keep your dumb equity and just give me cash." You'd be slightly better off just pointing to their family photo and saying, "Wow, your kids are ugly." Second, going through the work of valuing equity is necessary to see how the investors (that is, they people actually paying your salary) look at the company. Third, on the off chance you are successful, you're going to really wish you took the equity part of the negotiation seriously.
- udba 9y agoTypical of SV to focus solely on the business viability of a firm to the detriment of everything else. I’d like to work somewhere knowing that I have a clean conscience. Why not add: Where do you get your data from? If your data is sourced from users of your product, do you tell them what you’re collecting?
- jjeaff 9y agoThey essentially all say "tell" their users where they are getting the data. It's in the privacy policy. But people don't read it.
- DeRock 9y agoExactly. Informing the user does nothing except maybe clear your conscience. Expecting the user to extrapolate to how you will use that data, what will be inferred from it, what risks come with centralizing and storing it, all of that is not the users job. It’s the job of the collector to be as conservative as possible. Any company that treats users privacy as a resource to be extracted is not somewhere I want to work.
- tzs 9y agoHuh? There are dozens, if not hundreds, of things one should consider when contemplating taking a job. That an article focuses on things on one particular category (business viability) does not imply that the author thinks that things from other categories (such as ethical considerations) are less important.
- troels 9y agoThere many other factors to optimize for, like the people you’ll work with, the technologies you’ll work on, commute, etc.
- Hyperbolic 9y agoAfter working for a few companies who (tried to) make ML products, I think one of the most important questions to ask is "who owns the data you are building models on?". It is way harder for a company to build good models off data they don't have complete access to and full knowledge about. The worst (and unfortunately common) scenario for companies trying to do AI is that data scientists don't have full access to all priors necessary to build good models, and the owners of the data don't really know much about it either. Spells death of company
- alvis 9y agoWell said. It is a good tester of knowing how clear the people understand the problem themselves. Many people (particularly non-science background ones, no offence) don't know which kind of data they should access to build a product and, even worse, whether they have access to the data. Not knowing the answer of this question = they do not know they are travelling on Titanic at best, if not a boat full of holes.
- deleted 9y ago[deleted]
- anoother 9y agoAnd here I was thinking these would be questions about the moral / ethical / social implications of the ways in which many companies use these technologies..
- justtopthings 9y agoclick here to more about it https://www.youtube.com/watch?v=RiVkFDSPqfs&t=337s https://www.youtube.com/watch?v=RiVkFDSPqfs&t=337s
- justtopthings 9y agoclick here for more https://www.youtube.com/watch?v=RiVkFDSPqfs&t=337s https://www.youtube.com/watch?v=RiVkFDSPqfs&t=337s
- known 9y agoIs it different from https://en.wikipedia.org/wiki/Wisdom_of_the_crowd https://en.wikipedia.org/wiki/Wisdom_of_the_crowd
- raverbashing 9y agoGood questions, and they apply to many more companies than just ML/AI
- _Codemonkeyism 9y agoIf startup: When do you need the next financing round?
- danieldk 9y agoI am in academia, but some of my students have returned somewhat disgruntled from their internships in companies claiming to do machine learning, while in practice their systems were primarily rule-based and the work consisted of writing rules [1]. I now recommend students to find out whether companies are just name dropping ML or doing serious ML work. Typically, the best way is to ask former employees/interns. But I question directed at this also would not hurt. [1] There is nothing wrong with that, but it sets the wrong expectations.
- nl 9y agoI work in industry (sort of) and this is very true. One of the reasons for this is that the people designing the systems don't know how to do anything else. There's such a huge, huge gap between using ML to do the same thing everyone else is doing (say... image classification, or maybe regression on volume vs sales...) and solving unique business problems using ML.
- erikb 9y agoAnybody recognizes how similar these questions are to what an investor would ask? This is what I'm always telling developers. You are an investor. You invest your life, health, best hours of 5+ days a week, ambition. Your investment is not as important to the company as a few million bucks would be, but to you it should be more valuable than a few million bucks. It's all you have. And you want to give it to someone in exchange for money, you better make sure that someone is worth your investment. So read what investors are asking a prospect company, and ask the same questions. Only accept if you would give them a million bucks.
- kemiller2002 9y agoRandy Pausch said it best (paraphrase), "You can always make more money, you can never get more time." It's the most valuable thing you have, and one way or another you are investing in a place where you work.
- delibes 9y agoThanks for reminding me. The Last Lecture is good. Since it's about 9 years ago, some younger HN readers may have missed it: https://www.youtube.com/watch?v=ji5_MqicxSo https://www.youtube.com/watch?v=ji5_MqicxSo Cancer sucks.
- kthejoker2 9y agoLet's give props to Seneca: For we are mistaken when we look forward to death; the major portion of death has already passed. Whatever years be behind us are in death’s hands. https://www.lettersfromastoic.net/letter-1/ https://www.lettersfromastoic.net/letter-1/
- shanwang 9y agoSo, if I'm interviewing with a startup in AI, and asked all these questions, how many good answers I should expect? Is it ok for a startup to have a defensible business to solve a problem 10x better, knows how to make money in a big market, but have no experience in marketing and haven't talked to many potentially users? As a wanna be startup founder, I found my ideas have bad answers to at least 2-3 of these questions.
- purerandomness 9y ago> haven't talked to many potentially users? Any bad answer to these questions is a showstopper, yes. It's an indicator that this company is definitely going to fail. In your example, not having talked to many customers is the classical Juicero approach: Build a solution, then search for a problem that this solution is solving. Not talking to customers means not understanding if there is demand for something, and if there are customers who would like to pay for a solution. (Problems exist in markets where target groups are not used or not willing to pay for digital solutions) If you're starting out, I highly recommend reading the blog of Amy Hoy and Alex Hillmann (https://stackingthebricks.com/ https://stackingthebricks.com/) - they defend the idea to first go on a "sales safari" where you simply obvserve your target group, and then find a solution for their pains. As a founder, you want to continuously talk to potenttial customers about your idea or your prototype: make them use it and perform thinking-aloud tests.
- shanwang 9y agoGood points. But how many customers you talk to is enough? If I'm building a product for enterprise, is talking to one keen potential customer enough? For a first time founder, I can't imagine someone without much connections can get the chance to talk to multiple potential enterprise customers. What about a product targeting the mass market, if I'm building a mobile app, is talking to a dozen friends enough? or should I pay for a market research which can reach hundreds of strangers. Do you talk to customers with prototypes of the features or do you just have a basic skeleton and say 'what if I have this and that'? I actually subscribed to Amy Hoy's mailing list but haven't checked it out for a while...
- YeGoblynQueenne 9y ago>> 2. How was this problem being solved before the AI came around? Was the pre-AI “manual” solution good enough? Common answer: “we’re replacing humans.” That isn’t enough. Often having a human is desirable (bedside manner, dexterity, perfection a requirement). Often a human is affordable due to margin structure. You’re looking to get a sense that the product provided is something that was never possible before, 10X better, or just-as-good but 10X cheaper. Not 20% cheaper. 10X. That "10X" sounds like a bit of a heuristic, but is it a good one? Surely, what you're looking for is some assurance that the rewards from the use of machine learning justify its cost and that the profit the company can make out of it is higher than the profit they would be (or were) making without it? It doesn't even matter even the profit is really much higher. In business, as in war (and, er, board games) any edge you can get is enough to push ahead. If a company can make even 1% more of what it did before, thanks to some new technology or whatever other trick, then that trick is worth pursuing in earnest.
- hamilyon2 9y agoAi experts and practitioners are among smartest people you ever meet. Opportunity cost is huge
- sharemywin 9y agoI was going to say something about that my self. I'm thinking it's more like this TED talk about something being remarkable: https://www.ted.com/talks/seth_godin_on_sliced_bread https://www.ted.com/talks/seth_godin_on_sliced_bread
- bald 9y ago> "Reverse engineering from the technology to the market almost never works." Completely disagree. If the technology is creating some value for someone, and if you have only one smart business dude that knows how to approach the right people and validate markets, reverse engineering from technology to market _does_ work. Source: Did that myself.
- tboyd47 9y agoYou can't really ask the hard questions during the interview phase. Before you get a job offer, they have all the leverage, and people don't like being asked hard questions when they're the ones interviewing. Best thing to do is try to get a contact inside the company, then when you get the offer, call back and start asking those questions. Even better still - don't agree to an interview unless you know in advance that the job will move your career in a good direction, then don't even bother asking questions. It's not like they have to tell you the truth anyway.
- jerf 9y ago"Before you get a job offer, they have all the leverage" Only if you are in a situation where you need this job, right now. Try to avoid being in that situation. It isn't always possible, no, but there are reasonably steps you can take to try, such as prioritizing saving up a cushion over creature comforts and applying for multiple jobs. And if you are good at a hot tech, which ML is right now, you probably do have some leverage just from rarity. Yes, the perfectly free market of labor may be a theoretical construct, but don't give in to the despair that you have no control over your employment and that they have all the power. It'll become a self-fulfilling prophecy.
- tboyd47 9y agoTrue; I mean to say until you have a job offer, you have no idea whether or not they are willing to negotiate anything with you. It's easy to fool yourself based on how friendly your interviewers are. So many times I've come out of the interview feeling like I got the job and then been told that they're going in a different direction.
- monktastic1 9y agoDuring the "do you have any questions for me?" phase (which every good interviewer should make time for), these kinds of questions will make you stand out (in a good way) from the rest of the crowd. Just don't twist the thumbscrews, y'know?
- Brushfire 9y agoWhile I appreciate the nature of these questions, many of them exclude network-driven consumer businesses. You know, those like Tinder, Facebook, Twitter, Medium, Yelp, FourSquare... If you're spending time with a consumer company like that, I'd focus less on LTV and more on organic growth rate.
- aecs99 9y agoDid any of you succeed in asking such questions to companies (ML/AL or not) and ended up receiving reasonable responses to your questions? My experiences have never been great when asking such loaded questions. To be clear, I interviewed extensively at small startups and I'm only taking those interviews into consideration for this comment. The HR and technical interviewers dedicate a large amount of time for questioning you, and reserve the last few minutes to answer your questions. I lost count of how many times the interviewers would just blabber something meaningless in a rush, rather than answer a questions with patience and honesty. Of course, I made my decisions on whether or not to join the team based on such experiences. However, I cannot think of any remarkable instances where the interviewers answered such questions without getting impatient. Any chats with directors/PMs/C-level executives after receiving an offer were also not very informative. I walked out several times going in and coming back from such chats with no questions properly answered because: (a) they are still figuring out, or (b) they cannot discuss certain details because you haven't said yes to them.
- lmeyerov 9y agoI always answered this for Graphistry candidates. We take a vertical data product mindset to AI -- so just one solid piece of solving the human-in-the-loop investigation problem for security and fraud teams trying to help their responders get a handle on incidents/APIs/logs -- so perhaps we're less concerned about image. For startups with real tech, competition shouldn't be the thing that matters, but execution. For us, a great interview is one where the candidate teaches us and thereby demonstrates the potential to be a force multiplier for the team. Also, someone asking these kinds of questions suggests we may be able to put them in front of a customer!
- agjacobson 9y agoThis is silly. You’re the one being evaluated here. Can you code, or are you just a “sophisticated” smartass? The trick is to find out all these answers WITHOUT asking the interviewer.