7 ms·

Data Science Challenges at Instacart

93 points by apoorvamehta 11y ago

jeremystan 11y ago
Behind the scenes, Instacart is a revolutionary new e-commerce marketplace, an incredibly sophisticated last-mile logistics engine, and a dynamic source of work for thousands of personal shoppers. Each of these aspects of Instacart could be a whole company elsewhere, and data science plays a key role in our success in each endeavor. In this article, I (our VP Data Science) highlight some challenges the data science team is tackling at Instacart ranging from logistics to personalization. I also go into detail on how we have organized data science to have maximal impact and what we look for when recruiting data scientists.
- jsprogrammer 11y ago
  >“We will take full ownership of our projects. We take pride in our work and relentlessly execute to get things completely finished.” Does this mean that as an employee, or ex-employee, I can take my owned projects with me and use them for my own purposes?
  - jeremystan 11y ago
    We've worked hard to open source projects whenever we think they'll be useful broadly: https://www.instacart.com/opensource https://www.instacart.com/opensource. There is definitely more of this we can (and I hope will) do in the future.
    - sandGorgon 11y ago
      How do you guys run R in production? Just getting started with R based datascience and it has been a struggle to figure out how to build a production data science stack. Do you snapshot the computed models as RData and stream them to s3, etc
      - disgruntledphd2 11y ago
        If I were running R in production, then I'd probably fit models on some kind of batch process and then serve up the predictions/output from a DB or something. In general, R is not well-suited for DB-backed websites in real-time, but you can certainly use the outputs in production. You can do it, but I'm not sure it's worth the effort. You could probably provide a predict() interface in real-time if it was reasonably quick.
        sandGorgon 11y ago
        So I have seen a couple of large data science driven startups (like consumer finance) to throw R on 128gb machines and call it a day. That's reasonably going to be my plan except that I can't make it work very well. I really wish pandas had a "save workspace" feature - R does that very well. No point in saving to dB if you're going to need the data set in memory anyway.... Or use Hadoop.
      - sgt101 11y ago
        We run udfs in Hive to invoke R models, which is fine for compiling dashboards and reports but I wouldn't run it for something that needed instant responses.
      - jeremystan 11y ago
        We use R in production in two ways: 1. For batch processes that run daily, hourly or minutely, where the models are rebuilt on every run, and outputs (often predictions) are written to a database 2. For computation of coefficients in large sparse regularized models, where the coefficients are written to a database and scoring is done in another language in real-time For situations where we want real-time predictions, recommendations or optimizations, we tend to setup Python services instead. For batch processes, you can definitely store models in S3 to re-use them, and I've done that at other companies. But in general I've found it better to rebuild models frequently and cache them for short periods of time only if they are cost-prohibitive to rebuild.
        sandGorgon 11y ago
    - cballard 11y ago
      Why don't you pay your workers? https://news.ycombinator.com/item?id=11121092 https://news.ycombinator.com/item?id=11121092
- x0x0 11y ago
  fyi I interviewed for this team and 2 notes: 1 - I've been doing ad optimization / user classification / propensity scoring / product recommendations, warned them that I took a couple stochastic processes classes but haven't used them for a decade, and that I was entirely unsuited for OR type problems. They said that was ok and they where hiring for things I was suited for. Great. My in-person interview was primarily an OR problem best solved with stochastic processes. 2 - they were very responsive at first but after the interview, went radio silent for a week. After promising a response in a day. This was particularly annoying since I told them I had a written offer that I was pushing off for them. My guess is they were waiting to see if another candidate would accept. Which is fine, but the recruiter should have been honest with me. They ignored me for 5 business days after the interview -- 4 after their promised response -- before finally telling me no thanks. I'm not grumpy about being told no -- that's definitely happened before -- but their crappy behavior. Fortunately I'd already accepted the other offer after reading between the lines, but still, the experience left me grumpy. I debated posting this for a while, but bluntly, I kind of felt like they wasted my time and was really not happy their internal recruiter blew me off after repeated promises otherwise. I'm sure they'll be along to say your experience will be different (and it may well be!) but here's a data point for your consideration. I'm just sharing my experience.
  - randycupertino 11y ago
    > they were very responsive at first but after the interview, went radio silent for a week. After promising a response in a day. This was particularly annoying since I told them I had a written offer that I was pushing off for them. ugh, I hate that. I was jerked around by an a-list firm like that this fall and it was totally frustrating, especially because I turned down another offer while I was waiting to hear back. Really screwed me over and left me very bitter/annoyed. There's no respect and everyone's looking out for #1.
  - jeremystan 11y ago
    We work hard to get back to candidates quickly - in some cases the same day as they interview, but at least the day after if not. We know the market is very competitive and want candidates to make the best decisions they can - so it's in our best interest to act quickly! We are always working to improve how we screen, interview and respond to candidates in hiring, so will take this feedback to heart. Thank you for providing it. Regarding the focus on OR, that was definitely the case for our first few years, and while it's still important, we have definitely expanded our focus beyond it.
randycupertino 11y ago
My friend works at Instacart, she says it totally sucks. Mainly because there's no guaranteed hourly pay and what she gets paid depends entirely on what the customers decide to give out as a tip. So for example sometimes she spends an hour grocery shopping for someone, then 30 minutes to drive to their house, and then the person can just arbitrarily decide to tip her $10. So she makes $10 for 90 minutes of work? That's below minimum wage. She says MOST people will tip $20 or $25 however not all. So all it takes is one cheapskate to not understand how long it takes for you to go and pick out and deliver all their groceries and you get totally screwed. Apparently when she works in SF proper is the only place with a guaranteed wage, all surrounding areas you are at the mercy of what people decide to tip. Sounds outrageous, all it takes is one cheap idiot to completely ruin your shift and make it not worth it to work there.
- iamse7en 11y ago
  Then she should quit. There are certainly other people out there willing to do this type of simple work for a low wage. Instacart is basically passing it to the consumers so that the economics of the business work.
  - randycupertino 11y ago
    Yeah, I'm pretty sure she is going to, I know she and the other people that do it hate it. Shouldn't they be guaranteed minimum wage? I mean, if they don't make minimum wage they can sue, or report them to the Department of Labor, right?
    - dlgeek 11y ago
      Not if they're classified as contractors. (Whether the contractor classification is reasonable is a whole other ball of wax)
- dang 11y ago
  Your comment implies that Instacart pays their employees $0. That can't be true.
hathym 11y ago
we are hiring would have been enough.
minimaxir 11y ago
I'm all for data science, but I think this article romanticizes the field a bit too much (the random pictures of employees being thoughtful do not help). Listing the problems Instacart has which can be solved through applied statistics is one thing. The other, more important thing, is how exactly data science works to solve problems, with technical detail, as opposed to data science being some mystical unexplained power. (Especially since the intended audience for this post is data scientists Instacart wants to hire and presumably are already knowledgable in data science)
- jeremystan 11y ago
  I find that great data scientists care a tremendous amount about the problems they tackle and the impact they have, and less so the specific methods used. But we will definitely write more in the future on details. Here I wanted to share the range of problems, how we organize (not commonly discussed) and what we look for in candidates. That can be useful to other startups looking to build data science teams.
JasonCEC 11y ago
On this note: My team and I at Analytical Flavor Systems[1] wrote a blog post on how we go about hiring data science interns[2]. It's heavier on the technical details, and suffers from less... romanticism.... [1] www.gastrograph.com [2] https://gastrograph.com/blogs/gastronexus/interviewing-data-science-interns.html https://gastrograph.com/blogs/gastronexus/interviewing-data-...
- minimaxir 11y ago
  There is definitely more technical detail and statistical process in that post, although I strongly question the use of a hiring test that intricate and time consuming for an internship.
  - JasonCEC 11y ago
    It's a 3 month paid internship, and 99% of the students have been from Princeton. Did you see the work we linked to? That's intern work here - we treat our interns as full members of the team, and they've delivered.
    - asdfologist 11y ago
      So you had 100 interns and 99 of them were from Princeton??
      - JasonCEC 11y ago
        That's now how percentages work....
        grayclhn 11y ago
        I'm really curious what numerator and denominator you have in mind, then... are you somehow using fractional internship units? :) (Ordinarily this would be off topic, but....)
        asdfologist 11y ago
SixSigma 11y ago
People like shopping
grayclhn 11y ago
> Many of our best data science ideas have come from Instacart employees in the field – working directly with our shoppers in our stores, or interacting directly with our customers. I have no idea what this could mean. Either you're getting algorithm suggestions from your shoppers and customers, or (more likely) "data science" means "user interface."
- jeremystan 11y ago
  Agreed, that wasn't worded well - i'll try to explain. By 'employees in the field' we mean people who work in operations and management roles in the cities we operate in. They work with shoppers in stores and respond to shopper and customer feedback. They have ideas about how to improve our logistics and our apps - and while many ideas will be about user interfaces, many others will relate to how the algorithms operate behind the UIs.
CPLX 11y ago
So we no longer use the word scientist to describe people who do science? What a shame, I think science is really neat and scientists deserve unique respect. As far as I can tell what these people do every day is called "business" or maybe "logistics"
- yummyfajitas 11y ago
  What do you believe distinguishes "science" from "data science"? I.e., why can't "logistics" be a subset of "science"?
  - CPLX 11y ago
    Because science is an academic pursuit designed to create and test generalizable hypotheses and add to our collective knowledge, while the people in the article are trying to figure out how to optimize the act of underpaying someone to go grab some cans off a supermarket shelf and bring them to me. They're not scientists, they're engineers perhaps, or business analysts.
    - yummyfajitas 11y ago
      So a person doing basic biology research for Monsanto isn't a scientist? And whether or not Kantorovich qualifies as a scientist depends on whether he was working for the military or a university at the time he came up with linear programming? That's an interesting definition.
      - CPLX 11y ago
        No, basic biology research is science of course. It doesn't depend on where the person works (though sure that's a relevant sign) it matters what they're doing. Analyzing business related data and optimizing KPI's isn't science. At best it's applied science, which we have names for, such as engineering or statistics or financial analysis.
  - bmm6o 11y ago
    There's the old quip that if your discipline has "science" in the name, it probably isn't really a science. More seriously, scientists follow the scientific method, formulating hypotheses and designing experiments to test them. And that's a broad enough definition that it includes A/B testing, so it must apply to some of what they do. But science typically goes another step, generalizing observations and hypotheses into theories; I would be surprised if there was a lot of that going on at Instacart. Which isn't a slight against them or the field in the least, it's just a debate about definitions.

93 points by apoorvamehta 11y ago

Behind the scenes, Instacart is a revolutionary new e-commerce marketplace, an incredibly sophisticated last-mile logistics engine, and a dynamic source of work for thousands of personal shoppers. Each of these aspects of Instacart could be a whole company elsewhere, and data science plays a key role in our success in each endeavor. In this article, I (our VP Data Science) highlight some challenges the data science team is tackling at Instacart ranging from logistics to personalization. I also go into detail on how we have organized data science to have maximal impact and what we look for when recruiting data scientists.

jsprogrammer 11y ago
>“We will take full ownership of our projects. We take pride in our work and relentlessly execute to get things completely finished.” Does this mean that as an employee, or ex-employee, I can take my owned projects with me and use them for my own purposes?
- jeremystan 11y ago
  We've worked hard to open source projects whenever we think they'll be useful broadly: https://www.instacart.com/opensource https://www.instacart.com/opensource. There is definitely more of this we can (and I hope will) do in the future.
  - sandGorgon 11y ago
    How do you guys run R in production? Just getting started with R based datascience and it has been a struggle to figure out how to build a production data science stack. Do you snapshot the computed models as RData and stream them to s3, etc
    - disgruntledphd2 11y ago
      If I were running R in production, then I'd probably fit models on some kind of batch process and then serve up the predictions/output from a DB or something. In general, R is not well-suited for DB-backed websites in real-time, but you can certainly use the outputs in production. You can do it, but I'm not sure it's worth the effort. You could probably provide a predict() interface in real-time if it was reasonably quick.
      - sandGorgon 11y ago
        So I have seen a couple of large data science driven startups (like consumer finance) to throw R on 128gb machines and call it a day. That's reasonably going to be my plan except that I can't make it work very well. I really wish pandas had a "save workspace" feature - R does that very well. No point in saving to dB if you're going to need the data set in memory anyway.... Or use Hadoop.
    - sgt101 11y ago
      We run udfs in Hive to invoke R models, which is fine for compiling dashboards and reports but I wouldn't run it for something that needed instant responses.
    - jeremystan 11y ago
      We use R in production in two ways: 1. For batch processes that run daily, hourly or minutely, where the models are rebuilt on every run, and outputs (often predictions) are written to a database 2. For computation of coefficients in large sparse regularized models, where the coefficients are written to a database and scoring is done in another language in real-time For situations where we want real-time predictions, recommendations or optimizations, we tend to setup Python services instead. For batch processes, you can definitely store models in S3 to re-use them, and I've done that at other companies. But in general I've found it better to rebuild models frequently and cache them for short periods of time only if they are cost-prohibitive to rebuild.
      - sandGorgon 11y ago
  - cballard 11y ago
    Why don't you pay your workers? https://news.ycombinator.com/item?id=11121092 https://news.ycombinator.com/item?id=11121092
x0x0 11y ago
fyi I interviewed for this team and 2 notes: 1 - I've been doing ad optimization / user classification / propensity scoring / product recommendations, warned them that I took a couple stochastic processes classes but haven't used them for a decade, and that I was entirely unsuited for OR type problems. They said that was ok and they where hiring for things I was suited for. Great. My in-person interview was primarily an OR problem best solved with stochastic processes. 2 - they were very responsive at first but after the interview, went radio silent for a week. After promising a response in a day. This was particularly annoying since I told them I had a written offer that I was pushing off for them. My guess is they were waiting to see if another candidate would accept. Which is fine, but the recruiter should have been honest with me. They ignored me for 5 business days after the interview -- 4 after their promised response -- before finally telling me no thanks. I'm not grumpy about being told no -- that's definitely happened before -- but their crappy behavior. Fortunately I'd already accepted the other offer after reading between the lines, but still, the experience left me grumpy. I debated posting this for a while, but bluntly, I kind of felt like they wasted my time and was really not happy their internal recruiter blew me off after repeated promises otherwise. I'm sure they'll be along to say your experience will be different (and it may well be!) but here's a data point for your consideration. I'm just sharing my experience.
- randycupertino 11y ago
  > they were very responsive at first but after the interview, went radio silent for a week. After promising a response in a day. This was particularly annoying since I told them I had a written offer that I was pushing off for them. ugh, I hate that. I was jerked around by an a-list firm like that this fall and it was totally frustrating, especially because I turned down another offer while I was waiting to hear back. Really screwed me over and left me very bitter/annoyed. There's no respect and everyone's looking out for #1.
- jeremystan 11y ago
  We work hard to get back to candidates quickly - in some cases the same day as they interview, but at least the day after if not. We know the market is very competitive and want candidates to make the best decisions they can - so it's in our best interest to act quickly! We are always working to improve how we screen, interview and respond to candidates in hiring, so will take this feedback to heart. Thank you for providing it. Regarding the focus on OR, that was definitely the case for our first few years, and while it's still important, we have definitely expanded our focus beyond it.

randycupertino 11y ago

My friend works at Instacart, she says it totally sucks. Mainly because there's no guaranteed hourly pay and what she gets paid depends entirely on what the customers decide to give out as a tip. So for example sometimes she spends an hour grocery shopping for someone, then 30 minutes to drive to their house, and then the person can just arbitrarily decide to tip her $10. So she makes $10 for 90 minutes of work? That's below minimum wage. She says MOST people will tip $20 or $25 however not all. So all it takes is one cheapskate to not understand how long it takes for you to go and pick out and deliver all their groceries and you get totally screwed. Apparently when she works in SF proper is the only place with a guaranteed wage, all surrounding areas you are at the mercy of what people decide to tip. Sounds outrageous, all it takes is one cheap idiot to completely ruin your shift and make it not worth it to work there.

iamse7en 11y ago
Then she should quit. There are certainly other people out there willing to do this type of simple work for a low wage. Instacart is basically passing it to the consumers so that the economics of the business work.
- randycupertino 11y ago
  Yeah, I'm pretty sure she is going to, I know she and the other people that do it hate it. Shouldn't they be guaranteed minimum wage? I mean, if they don't make minimum wage they can sue, or report them to the Department of Labor, right?
  - dlgeek 11y ago
    Not if they're classified as contractors. (Whether the contractor classification is reasonable is a whole other ball of wax)
dang 11y ago
Your comment implies that Instacart pays their employees $0. That can't be true.

hathym 11y ago

we are hiring would have been enough.

minimaxir 11y ago

I'm all for data science, but I think this article romanticizes the field a bit too much (the random pictures of employees being thoughtful do not help). Listing the problems Instacart has which can be solved through applied statistics is one thing. The other, more important thing, is how exactly data science works to solve problems, with technical detail, as opposed to data science being some mystical unexplained power. (Especially since the intended audience for this post is data scientists Instacart wants to hire and presumably are already knowledgable in data science)

jeremystan 11y ago
I find that great data scientists care a tremendous amount about the problems they tackle and the impact they have, and less so the specific methods used. But we will definitely write more in the future on details. Here I wanted to share the range of problems, how we organize (not commonly discussed) and what we look for in candidates. That can be useful to other startups looking to build data science teams.

JasonCEC 11y ago

On this note: My team and I at Analytical Flavor Systems[1] wrote a blog post on how we go about hiring data science interns[2]. It's heavier on the technical details, and suffers from less... romanticism.... [1] www.gastrograph.com [2] https://gastrograph.com/blogs/gastronexus/interviewing-data-science-interns.html https://gastrograph.com/blogs/gastronexus/interviewing-data-...

minimaxir 11y ago
There is definitely more technical detail and statistical process in that post, although I strongly question the use of a hiring test that intricate and time consuming for an internship.
- JasonCEC 11y ago
  It's a 3 month paid internship, and 99% of the students have been from Princeton. Did you see the work we linked to? That's intern work here - we treat our interns as full members of the team, and they've delivered.
  - asdfologist 11y ago
    So you had 100 interns and 99 of them were from Princeton??
    - JasonCEC 11y ago
      That's now how percentages work....
      - grayclhn 11y ago
        I'm really curious what numerator and denominator you have in mind, then... are you somehow using fractional internship units? :) (Ordinarily this would be off topic, but....)
      - asdfologist 11y ago

SixSigma 11y ago

People like shopping

grayclhn 11y ago

> Many of our best data science ideas have come from Instacart employees in the field – working directly with our shoppers in our stores, or interacting directly with our customers. I have no idea what this could mean. Either you're getting algorithm suggestions from your shoppers and customers, or (more likely) "data science" means "user interface."

jeremystan 11y ago
Agreed, that wasn't worded well - i'll try to explain. By 'employees in the field' we mean people who work in operations and management roles in the cities we operate in. They work with shoppers in stores and respond to shopper and customer feedback. They have ideas about how to improve our logistics and our apps - and while many ideas will be about user interfaces, many others will relate to how the algorithms operate behind the UIs.

CPLX 11y ago

So we no longer use the word scientist to describe people who do science? What a shame, I think science is really neat and scientists deserve unique respect. As far as I can tell what these people do every day is called "business" or maybe "logistics"

yummyfajitas 11y ago
What do you believe distinguishes "science" from "data science"? I.e., why can't "logistics" be a subset of "science"?
- CPLX 11y ago
  Because science is an academic pursuit designed to create and test generalizable hypotheses and add to our collective knowledge, while the people in the article are trying to figure out how to optimize the act of underpaying someone to go grab some cans off a supermarket shelf and bring them to me. They're not scientists, they're engineers perhaps, or business analysts.
  - yummyfajitas 11y ago
    So a person doing basic biology research for Monsanto isn't a scientist? And whether or not Kantorovich qualifies as a scientist depends on whether he was working for the military or a university at the time he came up with linear programming? That's an interesting definition.
    - CPLX 11y ago
      No, basic biology research is science of course. It doesn't depend on where the person works (though sure that's a relevant sign) it matters what they're doing. Analyzing business related data and optimizing KPI's isn't science. At best it's applied science, which we have names for, such as engineering or statistics or financial analysis.
- bmm6o 11y ago
  There's the old quip that if your discipline has "science" in the name, it probably isn't really a science. More seriously, scientists follow the scientific method, formulating hypotheses and designing experiments to test them. And that's a broad enough definition that it includes A/B testing, so it must apply to some of what they do. But science typically goes another step, generalizing observations and hypotheses into theories; I would be surprised if there was a lot of that going on at Instacart. Which isn't a slight against them or the field in the least, it's just a debate about definitions.