7 ms·

> "How would you pull in data from an arbitrarily long -- maybe never-ending -- source of data in Python?" The answer to this question is to use a generator. I

> "How would you pull in data from an arbitrarily long -- maybe never-ending -- source of data in Python?" The answer to this question is to use a generator. If you don't know this, then both people are going to wish that they had that half hour of their life back. Is this a good question? If you want somebody with a lot of Python expertise, then knowing about generators is a good thing. If I was interviewing a candidate, and they didn't know this, then I would just find a pleasant way to move on after a minute or two. If this is the only interviewing question, then that makes me happy, because this company has a flawed process, and is leaving some great talent for me to hire. It's not a big deal if you didn't know this at the time. You might not have known what a generator was, and the person interviewing probably wasn't very good at interviewing people.

jph 5y ago
How about asyncio streams.py?
- mateo411 5y ago
  That seems like a good answer too. I think that's only applicable if you are reading from a socket. Although I don't know too much about asyncio streams.py, so I'm not going to get a job at this company.
timr 5y ago
> > "How would you pull in data from an arbitrarily long -- maybe never-ending - source of data in Python?" > The answer to this question is to use a generator. If you don't know this, then both people are going to wish that they had that half hour of their life back. That's funny...I've implemented more than my share of such things in a long career, and my first intuition for an answer to the question, as phrased, was to implement stream sampling. It never even occurred to me that the answer might be a syntactical feature of the language (and yes, I know what generators are). According to you, I'd be so wrong that I'm a waste of time. Do you see the problem yet? It isn't theoretical. I've hit this kind of situation with so many interviewers that I'm actually more nervous when I'm sitting across from someone who is clearly on their first or second job out of school. They're asking (unclear) questions in the "python syntax" category, and I'm thinking about the engineering consequences of consuming an endless stream of data...
- mateo411 5y ago
  > According to you, I'd be so wrong that I'm a waste of time. I don't think so, because you actually know about generators. If you were hiring for people who had a lot of Python experience, but didn't know about generators, that would be a bad sign. You would probably want to ask a some other questions, so you would have a more complete picture of what the candidate knows. My main point is that the company's interview process is flawed.
  - timr 5y ago
    > I don't think so, because you actually know about generators. Sure, but will the OP give me a chance to demonstrate, or simply flip the idiot bit as soon as I don't give the expected response? In the context of a phone interview, I probably wouldn't get anywhere near discussion of python generators in an answer to this question. I'd probably say something like: "For an infinite stream, my main concern is system resources, which are never infinite...I'd want to use something like stream sampling to calculate the desired metrics, but it's hard to know the right solution without more information about the problem." Far too many interviewers hear that as weasel-wording, and simply disengage. It's even worse when the interviewer thinks they're asking a clear question about python syntax. It takes a high level of self-awareness to know that you might be the problem when you're the interviewer. An experienced interviewer might engage with that, but many folks don't have the patience to listen to an unexpected response, nor the respect for the candidate to give them the benefit of the doubt. As I said, this pattern isn't theoretical. I've been on the interviewer end of this kind of pattern many times before. > My main point is that the company's interview process is flawed. Agreed on this.
- cratermoon 5y ago
  This is a sign that the interview questions aren't there to actually assess technical ability, but are geared towards excluding anyone who hasn't travelled in the right circles. The employer isn't looking for someone skilled, they are looking for someone who is pre-filtered to think a certain way and know certain things.
  - 5y ago
jodrellblank 5y ago
How does "use a generator" answer this question? A generator can yield an endless amount of data, but if they ask "our customers buy things 24/7 on our website, how would you pull the endless orders into Python?" and you said "a generator", I'd think you were trying to guess the teacher's password ( https://www.lesswrong.com/posts/NMoLJuDJEms7Ku9XS/guessing-the-teacher-s-password https://www.lesswrong.com/posts/NMoLJuDJEms7Ku9XS/guessing-t... ) It surely matters if the data source is push or poll or something else, matters if it's a lot of data or a trickle, matters if you need all of it or if you can miss some (and how much), if the source buffers it or doesn't. I'd expect discussions about message busses, queueing, buffering (maybe ring buffers), data stores (maybe SQL or No-SQL), as well as concurrency or parallelism expecting the question could move to "and what if there was 1000x more data?". "Process stock market data" needs more than "read a temperature every minute". Or at least for a candidate to ask about some of them and be told "this isn't so complicated", not "it's up to you". If it's only asking whether you can loop over a generator instead of trying to read all the data up front, the question could surely be more focused?
- Banana699 5y ago
  The question is garbage as phrased to be sure, but I think the auxiliary context matters a lot in this type of questions. Mentioning "python", when databases are language agnostic, heavily implies the answer is a language feature. Furthermore the way the question is asked (reading from a script) also implies there is a specific non-open-ended answer. I still answered the question in my head on a whim, half-expecting it to be false, it's a really vague formulation. Sounds like the interviewer was randomly browsing language features and cutting and pasting random descriptions from whatever tutorial. Or maybe the company was using python generators in a really specific "data pulling" niche that the answer seemed extremely obvious to the technical team writing the questions and they forgot that outsiders are not wired to think of the same patterns.
kamaal 5y ago
>>The answer to this question is to use a generator. Wrong answer. The heap data structure would be a right way to go about representing such data. >>If you don't know this, then both people are going to wish that they had that half hour of their life back. Same in your case now.

jph 5y ago

How about asyncio streams.py?

mateo411 5y ago
That seems like a good answer too. I think that's only applicable if you are reading from a socket. Although I don't know too much about asyncio streams.py, so I'm not going to get a job at this company.

timr 5y ago

> > "How would you pull in data from an arbitrarily long -- maybe never-ending - source of data in Python?" > The answer to this question is to use a generator. If you don't know this, then both people are going to wish that they had that half hour of their life back. That's funny...I've implemented more than my share of such things in a long career, and my first intuition for an answer to the question, as phrased, was to implement stream sampling. It never even occurred to me that the answer might be a syntactical feature of the language (and yes, I know what generators are). According to you, I'd be so wrong that I'm a waste of time. Do you see the problem yet? It isn't theoretical. I've hit this kind of situation with so many interviewers that I'm actually more nervous when I'm sitting across from someone who is clearly on their first or second job out of school. They're asking (unclear) questions in the "python syntax" category, and I'm thinking about the engineering consequences of consuming an endless stream of data...

mateo411 5y ago
> According to you, I'd be so wrong that I'm a waste of time. I don't think so, because you actually know about generators. If you were hiring for people who had a lot of Python experience, but didn't know about generators, that would be a bad sign. You would probably want to ask a some other questions, so you would have a more complete picture of what the candidate knows. My main point is that the company's interview process is flawed.
- timr 5y ago
  > I don't think so, because you actually know about generators. Sure, but will the OP give me a chance to demonstrate, or simply flip the idiot bit as soon as I don't give the expected response? In the context of a phone interview, I probably wouldn't get anywhere near discussion of python generators in an answer to this question. I'd probably say something like: "For an infinite stream, my main concern is system resources, which are never infinite...I'd want to use something like stream sampling to calculate the desired metrics, but it's hard to know the right solution without more information about the problem." Far too many interviewers hear that as weasel-wording, and simply disengage. It's even worse when the interviewer thinks they're asking a clear question about python syntax. It takes a high level of self-awareness to know that you might be the problem when you're the interviewer. An experienced interviewer might engage with that, but many folks don't have the patience to listen to an unexpected response, nor the respect for the candidate to give them the benefit of the doubt. As I said, this pattern isn't theoretical. I've been on the interviewer end of this kind of pattern many times before. > My main point is that the company's interview process is flawed. Agreed on this.
cratermoon 5y ago
This is a sign that the interview questions aren't there to actually assess technical ability, but are geared towards excluding anyone who hasn't travelled in the right circles. The employer isn't looking for someone skilled, they are looking for someone who is pre-filtered to think a certain way and know certain things.
- 5y ago

jodrellblank 5y ago

How does "use a generator" answer this question? A generator can yield an endless amount of data, but if they ask "our customers buy things 24/7 on our website, how would you pull the endless orders into Python?" and you said "a generator", I'd think you were trying to guess the teacher's password ( https://www.lesswrong.com/posts/NMoLJuDJEms7Ku9XS/guessing-the-teacher-s-password https://www.lesswrong.com/posts/NMoLJuDJEms7Ku9XS/guessing-t... ) It surely matters if the data source is push or poll or something else, matters if it's a lot of data or a trickle, matters if you need all of it or if you can miss some (and how much), if the source buffers it or doesn't. I'd expect discussions about message busses, queueing, buffering (maybe ring buffers), data stores (maybe SQL or No-SQL), as well as concurrency or parallelism expecting the question could move to "and what if there was 1000x more data?". "Process stock market data" needs more than "read a temperature every minute". Or at least for a candidate to ask about some of them and be told "this isn't so complicated", not "it's up to you". If it's only asking whether you can loop over a generator instead of trying to read all the data up front, the question could surely be more focused?

Banana699 5y ago
The question is garbage as phrased to be sure, but I think the auxiliary context matters a lot in this type of questions. Mentioning "python", when databases are language agnostic, heavily implies the answer is a language feature. Furthermore the way the question is asked (reading from a script) also implies there is a specific non-open-ended answer. I still answered the question in my head on a whim, half-expecting it to be false, it's a really vague formulation. Sounds like the interviewer was randomly browsing language features and cutting and pasting random descriptions from whatever tutorial. Or maybe the company was using python generators in a really specific "data pulling" niche that the answer seemed extremely obvious to the technical team writing the questions and they forgot that outsiders are not wired to think of the same patterns.

kamaal 5y ago

>>The answer to this question is to use a generator. Wrong answer. The heap data structure would be a right way to go about representing such data. >>If you don't know this, then both people are going to wish that they had that half hour of their life back. Same in your case now.