7 ms·
No. Data scientists exist at the mercy of programmers: without the tooling and the pipes, data science would not be a going concern.
by sigfubar 8y ago
No. Data scientists exist at the mercy of programmers: without the tooling and the pipes, data science would not be a going concern.
- closeparen 8y agoHah! At my company a decent proportion of engineers spend their lives scrambling to productionalize and operate the Lovecraftian concoctions of R and Python that our data scientists cook up on their laptops.
- stochastic_monk 8y agoEven the Stanford NLP Java implementations don’t always match the code they’re meant to be a translation of. I think R is a worse offender than Python.
- darkhorn 8y agoIn Python 0.1 + 0.2 is not equal to 0.3 because the result is 0.30000000000000004. In R 0.1 + 0.2 is equal to 0.3.
- detaro 8y agoR: > 0.1+0.2 == 0.3 [1] FALSE ?
- darkhorn 8y agoWhy do you compare point to point, it is meaningless? The result for 0.1 + 0.2 shows 0.3. It doesn't show you wrong result. If you print it prints 0.3, nothing else. Why do you compare point to point? I have never once in my Statistics education compared point to point. You always need to see the probability of the result if it is within 2 points. But if you want to compare use all.equal(0.1 + 0.2, 0.3)
- detaro 8y agoI thought that's what you mean by "equal". It didn't appear to me that you were merely talking about the default formatting of numbers printed on the REPL, which is utterly inconsequential to developing and porting applications.
- darkhorn 8y agoR is not used for developing applications. https://www.r-project.org/about.html https://www.r-project.org/about.html
- detaro 8y agoThis entire subthread is under a comment talking about "productionalizing" code developed in R. If you want to make an argument that R should not be used to make products, but only as an interactive notebook and to make nice plots, maybe make that instead of just mentioning small UI details.
- sampo 8y agoNot on my R: > (0.1 + 0.2) == 0.3 [1] FALSE
- darkhorn 8y agoThe result for 0.1 + 0.2 is 0.3. https://imgur.com/xWpx1Cg https://imgur.com/xWpx1Cg Why do you compare point to point? I have never once in my Statistics education compared point to point. You always need to see the probability of the result if it is within 2 points. But if you want to compare use all.equal(0.1 + 0.2, 0.3)
- sampo 8y ago> all.equal(0.1 + 0.2 + 0.000000003, 0.3) [1] TRUE
- j7ake 8y agoTry this: > print(.1+.2, digits=18) [1] 0.300000000000000044
- darkhorn 8y agoIn R 0.1 + 0.2 returns 0.3. In Python 0.1 + 0.2 returns 0.30000000000000004 Why do you compare point to point? I have never once in my Statistics education compared point to point. You always need to see the probability of the result if it is within 2 points. But if you want to compare use all.equal(0.1 + 0.2, 0.3)
- dekhn 8y agoNot only is this technically wrong, but Python has a decimal module for performing these kinds of calculations. It uses floats/doubles for the native float type which produces exactly the type of results you see above.
- darkhorn 8y agoWhat is the result of 3/2? In R it is 0.5. In Python it is 1.
- mlthoughts2018 8y agoWhere I work, the data scientists are more educated and experienced on containerization, CI tooling, unit testing, profiling tools, web service prototyping including API validation tools, caching layers, queues, GPU systems programming, etc. etc. We are constantly thwarted by infrastructure teams that use superficial policy basically so they can whine and complain that they don’t want to have to provide support for the extremely heavily researched and tested implementation we create. They don’t care that different technologies, database systems, whatever are chosen to solve customer use cases and that growing our business means supporting “Lovecraftian concoctions of R and Python” — they just don’t want to do their jobs (which indeed requires providing infrastructure support for crazy screwball data services that repeatedly break all the assumptions)...
- dasil003 8y agoIf the data scientists know more about CI, unit testing, profiling, and caching than the engineers then they are better engineers and I'd wonder a bit about their math/stats chops and whether their role was just re-branded "data scientist" to keep up with trends.
- mlthoughts2018 8y agoIt’s quite common to start out with a PhD / masters in math / stats, with deep specialization in fields like NLP, computer vision, MCMC sampling, and then to become an experienced expert in GPU computing, containerization, web service layers, etc., while working on implementations of ML models. This was true for me anyway. The main thing I do is deep learning for computer vision and image search, but I think it’s fair to say I have significant experience with Docker, GPU architectures, various CI tooling, linux system programming, deep internals of CPython, internals of MySQL and Postgres, lots of frustrating performance tradeoffs with py4j in the pyspark world, as well as all the usual crap with pandas, sklearn, data visualization tools, and a lot more. I’d say almost all data scientists I’ve worked with are just like me, just with maybe different specialization areas, except possibly for very young data scientists right out of undergrad.
- 8y ago
- hsndmoose 8y agoIt's scary how much this comment applies to my current job. Literally just spent today discussing with entire engineering org how to steer away from this behavior.
- itronitron 8y agowhy?
- natalyarostova 8y agoAs a data scientists I view other data scientists who need engineers to productionize their code with contempt. Perhaps (probably?) their data pipelines and systems are sufficiently more complicated than mine, but I'd feel embarrassed if I couldn't write production Python code
- blt 8y agoThey are different skills. Not saying that it's hard to learn both, but there are standardized career paths that will lead you to be good at the modeling / techniques side of "data science" without learning much about software engineering. For example, studying math in undergrad. And there are certainly lots of people capable of productionizing messy R scripts without fully understanding the statistical ideas behind them. So I think, as a team leader, you are restricting yourself to some degree if you only hire people who can do both.
- closeparen 8y agoWe don’t allow (new) Python code in production, which has something to do with it. Feature engineering is also a very different game with streaming systems and online datastores than with CSVs.
- thousandautumns 8y agoJust what the data science needed more of: gatekeeping. Data science is an extremely broad, vague buzzword encompassing a variety of jobs and skillsets, most of which have existed for decades under different names. You do work that involves putting models into production in Python, congrats. The insistence that all data scientists must also do so is silly, especially considering that there are surely many skills used by many data scientists that you are incapable of.
- acchow 8y agohedge fund?
- nschampions2004 8y agoAs a data analyst who is pushing towards data science, do you have any resources or advice on how to steer away from these "Lovecraftian concoctions" you're speaking of?
- contem 8y agohttps://ai.google/research/pubs/pub43146 https://ai.google/research/pubs/pub43146 http://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf http://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf https://en.wikipedia.org/wiki/Functional_programming https://en.wikipedia.org/wiki/Functional_programming
- engi_nerd 8y agoAt my company, I don't have a staff of engineers to productionalize things so I have to do it myself.
- shoo 8y agoIa! Ia! Prototype into production! Research Quality code! RPy2 Fhtagn!
- dschuetz 8y agoWell, say that to an actual computer scientist. They will argue without data science/informatics there wouldn't be any computers in the first place. Steven Levy's Hackers is a showcase that it's true. Coders/Programmers/Hackers emerged from that open computer science lab culture. And so on.
- cle 8y agoThese kinds of "arguments" are so tiring. Who cares who's better than who? Focus on solving problems, not stroking egos.
- smus 8y agoAnd programmers exist at the mercy of farmers. What's your point?