3 ms·

Well, two things there - First, the case you describe is one that's covered by using Spark for batch processing. Parent was criticizing using it and other tool

Well, two things there - First, the case you describe is one that's covered by using Spark for batch processing. Parent was criticizing using it and other tools for stream processing. The two are very different use cases. Second, just gotta call out that 2nd paragraph. A qualified data scientist should have a solid training in statistics. And someone who has a solid training in statistics should rarely if ever make the assumption that one day is representative of the whole week in the first place, regardless of whether they subsequently check that assumption. They would probably start with the presumption that one day is not representative of the whole week, because that is self-evidently going to be near-universally the case for any data that measures something about human behavior.

deepsun 8y ago
Right, for batch models and training evaluation. Re. assumptions -- it all depends on the data. If your data, say, collection of bird songs on Galapagos, then assuming there's correlation between day-of-week is counter productive. If a data scientist says me that they need to spend a day or two checking that assumption is correct, then I would look for a more productive scientist. Time to market is critical. And there are many of such assumptions in every project, just try to do something, anything with data science checking every assumption you make. Basically as I said above, you won't get far. By the way, majority of statistics theorems start with "take N independently distributed variables". In real world nothing is independent, but we still make useful predictions.
- lenticular 8y ago
  Birdsong actually isn't a good example, because birds sing less the more noise there is. If there is more activity during certain days of the week, it will be biased. You really can't predict these kind of correlations ahead of time, which is why it is always critical that your data is not collected in a biased way. The data scientists should have randomly subsampled from each day of the week. This is extremely common practice, and there's nothing wrong with it if the subsampled data gives acceptably credible results. I've never heard of people only using certain days of the week. As a practicing and productive data scientist, that sounds completely insane to me. I would never hire a data scientist who does not understand why such practices are bad. Data scientist is just a buzzy name for statistician. Data scientists who do not understand basic good statistical techniques are not competent.
- bunderbunder 8y ago
  Sorry, wasn't clear enough on the "human behavior" bit. Where data scientists are dealing with business problems, you can pretty much always assume that the data describes human behavior at least in part. Doubly so if you're talking about data that's big enough to not be processable on a single computer, and also needs to be handled in short order, since that implies we're in Big Data (emphasis on the capital B and D) territory. The counter-hypothetical you're giving just isn't plausible for the situation at hand. Analyzing bird songs on the Galapagos is scientist work, not data scientist work. Analyzing bird songs in an urban area might more plausibly be data scientist work, in that it gets you back toward business applications, but also gets us back into a spot where there's no way you could just assume there's no day-of-week effect. I'm gonna stand firm on that one. That sort of thing is worrisome - I don't care if it's someone who holds a PhD or someone who just took a MOOC or two, data scientists have a professional responsibility to be way less sloppy than that.

deepsun 8y ago

Right, for batch models and training evaluation. Re. assumptions -- it all depends on the data. If your data, say, collection of bird songs on Galapagos, then assuming there's correlation between day-of-week is counter productive. If a data scientist says me that they need to spend a day or two checking that assumption is correct, then I would look for a more productive scientist. Time to market is critical. And there are many of such assumptions in every project, just try to do something, anything with data science checking every assumption you make. Basically as I said above, you won't get far. By the way, majority of statistics theorems start with "take N independently distributed variables". In real world nothing is independent, but we still make useful predictions.

lenticular 8y ago
Birdsong actually isn't a good example, because birds sing less the more noise there is. If there is more activity during certain days of the week, it will be biased. You really can't predict these kind of correlations ahead of time, which is why it is always critical that your data is not collected in a biased way. The data scientists should have randomly subsampled from each day of the week. This is extremely common practice, and there's nothing wrong with it if the subsampled data gives acceptably credible results. I've never heard of people only using certain days of the week. As a practicing and productive data scientist, that sounds completely insane to me. I would never hire a data scientist who does not understand why such practices are bad. Data scientist is just a buzzy name for statistician. Data scientists who do not understand basic good statistical techniques are not competent.
bunderbunder 8y ago
Sorry, wasn't clear enough on the "human behavior" bit. Where data scientists are dealing with business problems, you can pretty much always assume that the data describes human behavior at least in part. Doubly so if you're talking about data that's big enough to not be processable on a single computer, and also needs to be handled in short order, since that implies we're in Big Data (emphasis on the capital B and D) territory. The counter-hypothetical you're giving just isn't plausible for the situation at hand. Analyzing bird songs on the Galapagos is scientist work, not data scientist work. Analyzing bird songs in an urban area might more plausibly be data scientist work, in that it gets you back toward business applications, but also gets us back into a spot where there's no way you could just assume there's no day-of-week effect. I'm gonna stand firm on that one. That sort of thing is worrisome - I don't care if it's someone who holds a PhD or someone who just took a MOOC or two, data scientists have a professional responsibility to be way less sloppy than that.