38 ms·

Ask HN: As a data scientist, what should be in my toolkit in 2018?

341 points by mxgr 9y ago

trevz 9y ago
A couple of thoughts, off the top of my head: Programming languages: - python (for general purpose programming) - R (for statistics) - bash (for cleaning up files) - SQL (for querying databases) Tools: - Pandas (for Python) - RStudio (for R) - Postgres (for SQL) - Excel (the format your customers will want ;-) ) Libraries: - SciPy (ecosystem for scientific computing) - NLTK (for natural language) - D3.js (for rendering results online)
- jxub 9y ago
  Maybe SpaCy for NLP. Way more intuitive and fast too. Good list.
- ktpsns 9y ago
  I make the claim that you can go very far in the SciPy ecosystem without ever touching R. It is worth understanding the concepts of numpy and pandas. Furthermore, try out IPython/Jupyter, especially for rapid publishing (people run their blogs on jupyter notebooks). I think certain libraries depend very much on where you focus. Machine learning? Native language processing? Visualization? Something in economics? Fundamental sciences? For instance, I never need NLTK in theoretical astrophysics ;-) Instead, I need powerful GPU based visualization, which is however very old school with VTK and Visit/Amira/Paraview (also very much pythonic).
  - marmaduke 9y ago
    I second that R is nice to have, but not needed. I’ve been doing science in Python for a decade without ever needing R. For powerful GPU viz, have you considered vispy? Four authors of four independent Python science visualization libs got together to build it.
  - albertgoeswoof 9y ago
    Agree, I would drop R, Python has you mostly covered now. Julia is also worth learning.
    - threeseed 9y ago
      I wouldn't be recommending to drop R at all. Very few enterprise data science teams are 100% Python (in fact none I've heard of). R is still very heavily used (and in fact all data science teams I've worked in it has been the dominant technology). There is a reason Microsoft purchased Revolution.
    - kirillseva 9y ago
      R, python and Julia are all Turing-complete languages, so of course you can drop any two and get by with just the third. The real selection happens when you consider what's available in opensource world. What code you don't have to write? What high-quality libraries are available vs which ones you will have to write yourself? On this topic, R has vast advantage over python in some domains, such as bioinformatics for example, while python definitely shines when it comes to deep learning (and using for loops). You can't just claim that one shouldn't look at R because you personally know one language better the other, quite likely because in your domain it's not being used as much. I do prefer the deep learnin, NLP and production serving story in python, but you will have to pry dplyr+ggplot from my cold dead hands for quick analysis and charting. Not to mention that pandas's API is a clusterfuck compared to R's native data frames.
  - closed 9y ago
    I disagree, even though python is the language I do most of my development in. But it probably depends on the problems we're thinking of a data scientist solving. If you're doing a lot of work with matrices, model fitting in production, then python seems fine. However, a lot of data scientists I see are more like scrappy data analysis / visualization types, who are churning out small dashboards. In that case R's tidy verse and shiny are just incredibly fast to develop with.
- Ultimatt 9y ago
  I'd gently suggest basic CLI Perl over BASH for cleaning up files, as it combines grep/sed/awk in a language thats more generally useful.
  - jonathankoren 9y ago
    Agreed. Perl was designed for text munging, and is superior to pretty much everything for this task. WRT bash, where to begin? In the past 40 years, there’s pretty much a better tool for everything someone tries to do with bash. It lives on pretty much through inertia and pride.
    - gkya 9y ago
      FreeBSD sh(1) (not bash(1)) man page. That's just how I understood how to shell. Nowadays I'm running Debian and my $SHELL is /bin/bash, but when I was on FreeBSD I really learnt tools like make(1), sh(1); the man pages were pieces of art. Having read sh(1), I do have a nice grasp of how shell works in general, to which knowledge I can add anytime the higher-level goodies bash has to offer (though I generally prefer keeping it POSIX, and using an actual programming language when it doesn't cut it).
- cmdlinetips 9y ago
  good list. I would add tidyverse in R ecosystem to it
  - p33p 9y ago
    I would go as far as saying the tidyverse is an essential piece of working with R. Base R sans tidyverse is not a pleasant experience.
    - disgruntledphd2 9y ago
      It's not that bad. It's inconsistent and clunky, but all of the tools are there (and tend to be faster than the tidyverse versions). Don't get me wrong, I love the tidyverse but R is very, very usable without it.
- pytonslange 9y ago
  Most of these are conveniently packaged in: $ docker run -it --rm -p 8888:8888 jupyter/datascience-notebook
eggie5 9y ago
a lot of people using spark?
- sandGorgon 9y ago
  same question that i have. Anyone using pyspark in production ? Would you use pyspark mllib in a webservice instead of scikit ?
  - wenc 9y ago
    1) Yes, PySpark is great if you're mostly just doing dataframe manipulation in Spark, using built-in functions. PySpark actually has similar performance to Scala Spark for dataframes. (We've moved away from RDDs) However, if you use a lot of UDFs where Spark has to serialize your Python functions, you might consider rewriting those UDFs in a JVM language. Serialization overhead is still fairly substantial. Arrow is trying to address this by implementing a common in-memory format, but it's still early days. I would still recommend PySpark to most people. It's more than good/fast enough for most data munging tasks. Scala does buy you two things: type safety and low serialization overhead (i.e. significant!), which can be critical in some situations, but not all. Also, the Python way has always been to prototype fast, profile, and rewrite bottlenecks in a faster language, and PySpark conforms to that pattern. 2) Spark MLLib is still fairly rudimentary in its coverage of major ML algorithms, and Spark's linear algebra support, while serviceable, is currently not very sophisticated. There are a few functions that are useful in the data prep stage (encoding, tokenizers, etc.) but overall, we don't really use MLlib very much. Companies that have simple needs (e.g. a simple recommender) and that don't have a lot of in-house expertise, might use MLlib though -- I believe someone from a startup said that they did at a recent meetup. Most of us need better algorithmic coverage and Scikit's coverage is currently much better, plus it is more mature. We also have Numpy at our disposal, which lets us do matrix-vector manipulation easily. There is some serialization cost, but we can usually just throw cloud computational power at it. Also note that for most workloads, the majority of the cost is incurred in training. For models in production, one is typically processing a much smaller amount of data using a trained model, so less horsepower is required.
greyman 9y ago
If you will work in some bigger company doing data analytics, you can also come across Tableau instead of Excel. Apart from SQL, if there is more data, you might want to use Bigquery or something similar.
latenightcoding 9y ago
If you use Python: scikit-learn, Pandas, NumPy, Tensorflow or PyTorch Language agnostic: XGBoost, LibLinear, Apache Arrow, MXNet
ever1 9y ago
Python: Jupyter, pandas, numpy, scipy, scikit-learn Numba for custom algorithms. Dataiku (amazing tool for preprocessing and complex flows) Amazon RDS (postgress), but thinking about redshift. Spark Tableau or plotly/seaborn
eps 9y ago
You probably mean "data analyst". "Data scientist" title would apply only if you are applying scientific method to discover new fact about natural world exclusively through data analysis (as opposed to observation and experiments).
- Twisell 9y ago
  Define natural world... And gather a consensus around your definition... Or maybe whole humanities should be considered as « not science ». Beside a data analyst that don’t use scientific method is just a bad analyst. Some media outlet showcase blatantly lying charts made by people that understand the technicals but get everything wrong about the concepts. So this is my advice, focus on understanding the concepts before the tooling. That is what will really make your value.
- sgt101 9y ago
  Designing experiments is a key part of Data Science work. Another key part is determining where & how revealing observations can be made. The analysis part is usually quite simple, often if it gets really complex then that's a sign that the data is being tortured. Sometimes the marginal gains that complex methods create (vs simple but good approaches) are not worthwhile even if they are valid - simply in terms of time spent and difficulty in communications.
kome 9y ago
Excel, VBA, SPSS ;)
- babayega2 9y ago
  OpenRefine has helped me a lot in data cleaning tasks.
sureaboutthis 9y ago
As a data scientist, you should already know that answer and be telling us.
piqufoh 9y ago
> what tools should be in my arsenal A sound understanding of mathematics, in particular statistics. It's amazing how many people will talk endlessly about the latest python/R packages (with interactive charting!!!) who can't explain the student's t-test.
anc84 9y ago
Any programming language that you are proficient in. A solid understanding how a computer works. Solid basis of statistics. Anything else is just sprinkles, trends and field-specific.
- greyman 9y ago
  Hard to say... I was more proficient in PHP than python, but when doing AI, we use python anyway, since in PHP some necessary libraries just aren't there...
- wenc 9y ago
  > Any programming language that you are proficient in. Oh I don't know about that. Programming languages are force multipliers, and each language has a different force coefficients for different problem domains. They are not all equivalent. They have their different points of leverage, and simply being good in one does not mean you can solve problems in any domain with ease. In fact the wrong programming language can often be harmful if it's ill-suited to the problem at hand, and especially if it contorts your mental model of what you can do with the data. One example I encounter a lot in industry is Excel VBA. I'm fairly good at VBA and have seen very sophisticated code in VBA. I've also seen many basic operations implemented badly in VBA that should not have been written in VBA at all. By solving the problem in VBA, the solution is often "hemmed in" by the constraints of VBA. For instance, unpivoting data is often done badly in VBA (with for-loops), but is trivial to do well in dplyr or pandas. So I would say one has to choose one's programming language somewhat carefully. Not any language will do.
SFjulie1 9y ago
Methodology, Signal processing, probability and statistic. Any monkey can make nice plots with R or python. Using a tool will not make you efficient. What matters is the brain of the one using the tools. In fact for the same reasons astronauts are scientific trained to pilot and not pilots trained in science, data scientists/analyst are scientist trained to use tools and thus the science part is the only one that matters. Major problem I have encountered in IT is that monkey hate science, they just go on internet, see a blog post and reproduce nice biased manipulation (the more obscure, the deeper it will be perceived). It is basically cargo cult. Mimicking science instead of doing science. Science is boring and is all about being a data janitor, and cleaning data so much, it sparkles with simplicity. How to clean data is the most vital part of data "science". IT data are often plagued with a lot of errors typo, poor encoding translations, heterogeneous sources which quality and relevance varies a lot due to IRL bias. Having a background in being a DBA, a coal faced coder helps assessing fast the quality of data and transforming them. Also, IT is plagued with floats. Where science matters, the uncertainty inherent with float representation can introduce artefacts that are a pain in the a.. when it comes to phenomenon that are sensitive to initial conditions or when measure instruments are put at the edge of their resolutions. So at my opinion, the "measure theory" (Lebegue) is a must. It helps you understand the confidence you can have from your data, the margin error you have, and assess if data are faked. In IT a lot of cheating is involved, so being able to detect fraud is required, for this you have to know the shape of "randomness" (Poisson, Student...) and know your estimators. Also, signal processing basics are important for seasonal/recurrent activities (trend analysis). However, some phenomeon are non linear, hence using fourier transform, moving average sometimes are the wrong (overused) tool, thus a little knowledge of complex systems (stochastic) is a plus. Then after a LOT of work (the last 5%) you may have to decide if you face a correlation, a causation, or just noise. For this there are 2 schools : bayesians and frequentists. I am a frequentist because the tools are far more robusts, easy to understand and to use. The market want bayesian because it is so obscure you can easily make your data say whatever you want. My intuition is bayesian are at the core of the problem of reproducibility in science. And last but not least, your work is useless if you cannot communicate. There are standards : removing all useless elements from plots, quoting your sources the right way... For the tools ? It depends. They are domain specific. Some use matlab, mathematica, excel, R, python, proprietary software. The scientific background is so harder to find than a programming background no serious companies will turn you down for not knowing their tools. But clearly having a factual knowledge of accessing, transforming, cleaning data is a must and this can a combination of languages and tools (C, python, Perl, bash, sed, awk, grep, C++, pandoc, imagemagick) since every languages have their strengths and weaknesses and you don't know where you data will come from (CSV, DBs (SQL or not), web pages, scans, pictures, instruments, simulations, text files, structured documents, syscalls....). Thus, being a data analyst is first of all knowing the intellectual and computer tools to assess, gather, transform, clean the datas and methodic about their source and how you did it so that someone else can try to reproduce your claims.
fredley 9y ago
IPython/Jupyter, Pandas/Numpy and Python will get you everywhere you need to go. Currently, until maybe Go gets decent DataFrame support, in terms of the total time to get to your solution, I'd be amazed if any other setup got you there quicker.
- threeseed 9y ago
  > get you everywhere you need to go No it won't. That combination can't handle large datasets that are typical for most data science teams i.e. maybe include PySpark. And then it's very limited so far as ML/DL technologies.
  - fredley 9y ago
    > i.e. maybe include PySpark Pandas and Spark are both DataFrame libraries, and seem to offer very similar functionality to me. Why do you prefer Spark over Pandas? > very limited so far as ML/DL technologies I mean, getting Tensorflow up and running with GPU support isn't trivial, but it's not exactly hard, and Keras[1] provides excellent support for a wide variety of other backends. What, in your experience, is less limited? [1]: https://keras.io/ https://keras.io/
    - jwilbs 9y ago
      Spark sits on top of YARN/Mesos, and is used for data processing scalability that pandas can't handle. Personally, I think two areas often lacking are software development skills and general statistics knowledge. The former is necessary for writing production-quality code, assisting with an sort of data engineering pipeline, writing reliable, reusable code, and creating custom solutions. Unfortunately, the latter is often skimped on (if not skipped entirely) in favor of more 'hot' fields like ml/dl, with the result being a fuzzy understanding across the board. (You'd be amazed at the quantity of candidates lacking fundamental knowledge about glm's, basic nonparametric stats, popular distributions, etc).
  - ppod 9y ago
    >typical for most data science teams I would bet that the mean size of dataset people are dealing with is a lot bigger than the median size.
ms013 9y ago
Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you can go surprisingly far with few and relatively simple ones. Knowing how, when, and where to apply them is the hard part: and that often boils down to understanding the mathematics and domain you are working in. And don’t over use viz. Pictures do effectively communicate, but often people visualize without understanding. The result is pretty pictures that eventually people realize communicate little effective domain insight. You’d be surprised that sometimes simple and ugly pictures communicate more insight than beautiful ones do. My arsenal of tools: python, scipy/matplotlib, Mathematica, Matlab, various specialized solvers (eg, CPLEX, Z3). Mathematical arsenal: stats, probability, calculus, Fourier analysis, graph theory, PDEs, combinatorics. (Context: Been doing data work for decades, before it got its recent “data science” name.)
- gaius 9y ago
  Indeed. Too many people when asked about their skills or experience just rattle off a list of tools or libraries. Usually the same ones as everyone else!
- anewhnaccount2 9y ago
  Out of interest, can you give an example of a problem you've solved using Z3?
  - ms013 9y ago
    One data problem boiled down to being an instance of the set cover problem (https://en.m.wikipedia.org/wiki/Set_cover_problem https://en.m.wikipedia.org/wiki/Set_cover_problem). Pretty easy to pose as an integer constraint problem, and Z3 solved it in about 20 minutes for me.
- edem 9y ago
  I really like to get a degree in Mathematics but I simply don't have the time to throw at it (work, children, etc). What do you suggest I should do to have something on my resume? MOOC maybe?
trollied 9y ago
What does "Data Scientist" actually mean these days? Does it mean "Write 10 lines of Python or R, and not fully understand what it actually does"? Or something else? I just see the term flinged around so much recently, and applied to so many different roles, it has all become a tad blurred. Maybe we need a Data Scientist to work out what a Data Scientist is?
- threeseed 9y ago
  I hire data scientists so can tell you. It means someone who can work with business stakeholders to break down a problem e.g. "we don't know why customers are churning", produce a machine learning model or some adhoc analysis (usually the former) and either communicate the results back or assist in deploying the model into production. Typically there will be data engineers who will be doing acquisition and cleaning and so the data scientists are all about (a) understanding the data and (b) liaising with stakeholders. As for technologies it is typically R/Python with Spark/H20 on top of a data lake i.e. HDFS, S3. Every now and again on top of an SQL store e.g. EDW, Presto or a Feature store e.g. Cassandra.
- jxub 9y ago
  That's a good meta reflection. Let's make an Y Combinator of Data Scientist a and Data Scientist b (recursive data scientist) to prove they can support recursion if Data Scientists a and b are first class functions, just because we can: const Y = a => (b => b(b))(b => a(x => b(b)(x)));
amelius 9y ago
Any book recommendations?
severo 9y ago
I'd say: 1. You need research skills that will allow you to ask the right questions, define the problem and put it in a mathematical framework. 2. Familiarity with math (which? depends on what you are doing) to the point where you can read articles that may have a solution to your problem and the ability to propose changes, creating proprietary algorithms. 3. Some scripting language (Python, R, w/e) 4. (optional) Software Engineering skills. Can you put your model into production? Will your algorithm scale? Etc.
closed 9y ago
I would think about which of these you see yourself doing more.. * statistical methods (more math) * big, in-production model fitting (more python) * quick, scrappy data analyses for internal use (more R) For example, I would feel weird writing a robust web server in R, but it's straightforward in python. On the other hand R's shiny lets you put up quick, interactive web dashboards (that I wouldn't trust in exposing to users).
justusw 9y ago
Dealing with large data processing problems my main tools are as follows: Libs: - Dask for distributed processing - matplotlib/seaborn for graphing - IPython/Jupyter for creating shareable data analyses Environment: - S3 for data warehousing, I mainly use parquet files with pyarrow/fastparquet - EC2 for Dask clustering - Ansible for EC2 setup My problems usually can be solved by 2 memory-heavy EC2 instances. This setup works really well for me. Reading and writing intermediate results to S3 is blazing fast, especially when partitioning data by days if you work with time series. Lots of difficult problems require custom mapping functions. I usually use them together with dask.dataframe.map_partitions, which is still extremely fast. The most time-consuming activity is usually nunique/unique counting across large time series. For this, Dask offers hyperloglog based approximations. To sum it up, Dask alone makes all the difference for me!
drej 9y ago
grep, cut, cat, tee, awk, sed, head, tail, g(un)zip, sort, uniq, split; curl; jq, python3
- proc0 9y ago
  So unix? lol
  - deleted 9y ago
    [deleted]
chewxy 9y ago
My toolkit hasn't changed since 2016: - Jupyter + Pandas for exploratory work, quickly define a model - Go (Gonum/Gorgonia) for production quality work. (here's a cheatsheet: https://www.cheatography.com/chewxy/cheat-sheets/data-science-in-go-a/ https://www.cheatography.com/chewxy/cheat-sheets/data-scienc... . Additional write-up on why Go: https://blog.chewxy.com/2017/11/02/go-for-data-science/ https://blog.chewxy.com/2017/11/02/go-for-data-science/) I echo ms013's comment very much. Everything is just tools. More important to understand the math and domain
- ZeroCool2u 9y ago
  I'm a big Go fan, but this is the first time I've seen someone recommend Go for data science. After looking at this cheat sheet you've got me convinced though. Would you mind pointing me to any other less cheat sheet style and more in depth examples that you particularly like?
  - chewxy 9y ago
    Working on it. Part of my goal for 2018 is to write a lot more soft documentation - tutorials etc. Go is quite straightforwards though - WYSIWYG for the most parts, hence you probably won't find a lot of sexy tutorials. Almost everything is just a loop away, and in the next version of Gorgonia, even more native looping capability is coming
    - ZeroCool2u 9y ago
      Awesome, thank you!
  - samuell 9y ago
    You might also want to have a look at: - http://gopherdata.io http://gopherdata.io ... and in particular the resources lists at - https://github.com/gopherdata/resources https://github.com/gopherdata/resources Also, Dan's GopherCon talk on Go for data science is a great way to get yourself convinced enough to try it out: - https://www.youtube.com/watch?v=D5tDubyXLrQ https://www.youtube.com/watch?v=D5tDubyXLrQ
pentium10 9y ago
As 1TB is free for processing every month, using SQL 2011 standard + combined with Javascript UDFs, the winner solution is Google BigQuery for us, combined with Dataprep
sdfjkl 9y ago
numpy, Jupyter (formerly IPython Notebook) and probably Mathematica anyways.
ChrisRackauckas 9y ago
A good understanding of calculus (probability), linear algebra, and your dataset/domain. Anything else can be picked up as you need it. Oh, and test-driven development in some programming language, otherwise you can't develop code you know is correct.
schaunwheeler 9y ago
A lot of people in this thread are focusing on technical tools, which is normal for a discussion of this type, but I think that focus is misplaced. Most technical tools are easily learnable and are not the limiting factor is creating good data science products. https://towardsdatascience.com/data-is-a-stakeholder-31bfdb650af0 https://towardsdatascience.com/data-is-a-stakeholder-31bfdb6... (Disclaimer: I wrote the post at the above link). If you have a sound design you can still create a huge amount of value even with a very simple technical toolset. By the same token, you can have the biggest, baddest toolset in the world and still end up with a failed implementation if you have bad design. There are resources out there for learning good design. This is a great introduction and points to many other good materials: https://www.amazon.com/Design-Essays-Computer-Scientist/dp/0201362988 https://www.amazon.com/Design-Essays-Computer-Scientist/dp/0...
Xcelerate 9y ago
As a data scientist who has been using the language for 5 years now, Julia is by far the best programming language for analyzing and processing data. That said, it’s common to find many Julia packages that are only half-maintained and don’t really work anymore. (I still don’t know how to connect to Postgres in a bug-free way using Julia.) And you’d be hard pressed to find teams of data scientists that use Julia. So in that sense, Python has much more mature and stable libraries, and it’s used everywhere. (But I really hope Julia overtakes it in the next couple of years because it’s such a well-designed language.) Aside from programming languages, Jupyter notebooks and interactive workflows are invaluable, along with maintaining reproducible coding environments using Docker. I think memorizing basic stats knowledge is not as useful as understanding deeper concepts like information theory, because most statistical tests can easily be performed nowadays using a library call. No one asks people to program in assembler to prove they can program anymore, so why would you memorize 30 different frequentist statistical tests and all of the assumptions that go along with each? Concepts like algorithmic complexity, minimum description length, and model selection are much more valuable.
- chubot 9y ago
  Has Julia converged on a solution for data frames? I watched some JuliaCon videos and got the impression that they hadn't. There seem to be a lot of different overlapping efforts.
  - 0kto 9y ago
    Well, only the DataFrames.jl package comes to my mind. However, there exist a few packages that extend this package (DataFramesMeta.jl or Query.jl; these overlap to some extend, but the newer Query package seems to go beyond DataFrames and offers some piping capabilities to interface with plotting packages). In general: During the three years of my PhD some language / package upgrades broke some of my scripts (during 0.4 -> 0.5 and -> 0.6), but the language (and its extensive documentation, online and from the source code of the packages) is very pleasant to use - the deprecation warnings usually help you to adjust your code in time. I have been relaying heavily on said DataFrame package, and am quite happy - the community is usually responsive and helpful in case of problems or questions.
bitL 9y ago
Spark + MLlib, Python + Pandas + NumPy + Keras + TensorFlow + PyTorch, R, SQL, top placement in some Kaggle competitions. This would get you long way.
- geebee 9y ago
  Good tool set recommendations (+1 for mentioning SQL, immensely helpful), and I enjoy Kaggle. Not sure how critical top placement is, though. It seems like getting into the upper echelons of Kaggle is a matter of refining your model, and I do wonder how much value these refinements offer over a more basic and general approach in a real world scenario. To be clear, when I say I wonder, I'm not saying I'm rejecting the value, I really do mean it, I'm uncertain about the value. I think it's probably very scenario specific. Think of it this way - a predictive value of 90% vs 95% could be the difference between placing in the top 10% and the bottom third. Now, 5% isn't nothing, it could be very valuable. It really depends. But Kaggle is an environment where the question is already posed, the data has been collected, the test and train sets are already split apart for you, and winning model is the one that scores best on a hidden test set by a predefined goodness of fit score. In a real world scenario, suppose someone does a great job figuring out the question to ask, gathering the data, and determining the most effective way to act on the results, but uses a fairly basic, unrefined model. Someone else does a middling job on those things, but builds a very accurate model as measured by the data that has been collected. I'd say the first scenario is likely to be more valuable, but again, it depends of course. A couple other things, since I am a fan of Kaggle and do highly recommend it. First, these things aren't necessarily exclusive - you can have a particularly well conceived and refined model as well as a thorough and excellent businesss and data collection process (though you may have to decide where to put your time and resources). Also, refining a model with Kaggle can be an exceptional training opportunity to really understand what drives these things. So go for it! (I also find these things kinda fun).
  - bitL 9y ago
    Top placement in Kaggle attracts recruiters for higher positions; i.e. I observed a top 10 person getting a job of Head/VP of analytics in a large European company even if let's say formal education wasn't top 100. I agree real-world it is often useless, but people are drawn to proven winners.
nrjames 9y ago
There are two "poles" in data science: math/modeling and backend/data-wrangling. Most of the time, the backend/data-wrangling piece is a prerequisite to the math/modeling. The vast majority of small and medium sized companies have not set up the systems they would need to support a data scientist who knows only math/modeling. Depending on the domain, it's not uncommon to find that a small/medium company outsourced analytics to Firebase, Flurry, etc... That's fine, but when it comes time to create some customer segmentation models (or whatever) the data scientist they hire is going to need to know how to get the raw data. Questions become: how do I write code to talk to this API? How do I download 6 months of data, normalize it (if needed) and store it in a database? Those questions flow over into: how do I set up a hosted database with a cloud provider? What happens if I can't use the COPY command to load in huge CSV files? How do I tee up 5 TB of data so that I can extract from it what I need to do the modeling? Then you start looking at BigQuery or Hadoop or Kafka or NiFi or Flink and you drown for a while in the Apache ecosystem. If you take a job at a place that has those needs, be prepared to spend months or even up to a year to set up processes that allow you to access the data you need for modeling without going through a painful 75 step process each time. Case in point: I recently worked on a project where the raw data came to me in 1500 different Excel workbooks, each of which had 2-7 worksheets. All of the data was in 25-30 different schemas, in Arabic, and the Arabic was encoded with different codepages, depending on whether it came from Jordan, Lebanon, Turkey, or Syria. My engagement was to do modeling with the data and, as is par for the course, it was an expectation that I would get the data organized. Well - to be more straightforward, the team with the data did not even know that the source format would present a problem. There were ~7500 worksheets, all riddled with spelling errors and the type of things that happen when humans interact with Excel: added/deleted columns, blank rows with ID numbers, comments, different date formats, PII scattered everywhere, etc. A data scientist's toolkit needs to be flexible. If you have in mind that you want to do financial modeling with an airline or a bank, then you probably can focus on the mathematics and forget the data wrangling. If you want the flexibility to move around, you're going to have to learn both. The only way to really learn data wrangling is through experience, though, since almost every project is fundamentally different. From that perspective, having a rock solid understanding of some key backend technologies is important. You'll need to know Postgres (or some SQL database) up and down; how to install, configure, deploy, secure, access, query, tweak, delete, etc. You really need to know a very flexible programming language that comes with a lot of libraries for working with data of all formats. My choice there was Python. Not only do you need to know the language well, you need to know the common libraries you can use for wrangling data quickly and then also for modeling. IMO, job descriptions for "Data Scientist" positions cover too broad of a range, often because the people hiring have just heard that they need to hire one. Think about where you want to work and/or the type of business. Is it established? New? Do they have a history of modeling? Are you their first "Data Scientist?" All of these questions will help you determine where to focus first with your skill development.
Jeff_Brown 9y ago
Static typing lets you catch errors before running the code. Pattern matching helps you write code faster (that is, spending less human time). Algebraic data types, particularly sum types, let you represent complicated kinds of data concisely. Coconut is an extension of Python that offers all of those. Test driven development also helps you write more correct code.
ak_yo 9y ago
Experimental design and observational causal inference would be excellent skills to have. Especially if you’re working with people who are asking you “why” questions, ML is helpful but isn’t going to cut it alone.
elsherbini 9y ago
I'm a scientist (PhD student in microbiolgy) that works with lots of data. My data is on the order of hundreds of gigabytes (genome collections and other sequencing data) or megabytes (flat files). I use the `tidyverse` from R[0] for everything people use `pandas` for. I think the syntax is soooo much more pleasant to use. It's declarative and because of pipes and "quosures" is highly readable. Combined with the power of `broom`,fitting simple models to the data and working with the results is really nice. Add to that that `ggplot` (+ any sane styling defaults like `cowplot`) is the fastest way to iterate on data visualizations that I've ever found. "R for Data Science" [1] is great free resource for getting started. Snakemake [2] is a pipeline tool that submits steps of the pipeline to a cluster and handles waiting for steps to finish before submitting dependent steps. As a result, my pipelines have very little boilerplate, they are self documented, and the cluster is abstracted away so the same pipeline can work on a cluster or a laptop. [0] https://www.tidyverse.org/ https://www.tidyverse.org/ [1] http://r4ds.had.co.nz/ http://r4ds.had.co.nz/ [2] http://snakemake.readthedocs.io/en/stable/ http://snakemake.readthedocs.io/en/stable/
- nonbel 9y ago
  Sometimes I think I'm the only one who isn't really a fan of the tidyverse. I've found it slower, more prone to dependency issues, more prone to silent errors, and less well documented than most R packages (ie most of what you find on CRAN).
  - swarchal 9y ago
    You're not the only one. Though I've found there seems to be a bit of a cult surrounding the tidyverse, a mere hint of criticism usually results in outrage and attacking other tools/packages (by users, not the authors).
    - anarchimedes 9y ago
      I like the Tidyverse. My only complaint is that it presents another headache when moving from one language (R) to another (usually Python/SQL). Using the base R functions while integrating loops and functions lessens the fatigue of moving between R and Python.
cwyers 9y ago
You can get a lot of mileage out of just using R, dplyr, ggplot2 and lm/glm. OLS still performs well in a lot of problem spaces. Understanding your data is the key there, and a lot of exploratory visualization there will help a lot.
ellisv 9y ago
Counting and dividing.
xitrium 9y ago
If you care about quantifying uncertainty, knowing about Bayesian methods is a good idea I don't see represented here yet. I care so much about uncertainty quantification and propagation that I work on the Stan project[0] which has an extremely complete manual (600+ pages) and many case studies illustrating different problems. Full Bayesian inference such as that provided by Stan's Hamiltonian Monte Carlo inference algorithm is fairly computationally expensive so if you have more data than fits into RAM on a large server, you might be better served by some approximate methods (but note the required assumptions) like INLA[1]. [0] http://mc-stan.org/ http://mc-stan.org/ [1] http://www.r-inla.org/ http://www.r-inla.org/
- peatmoss 9y ago
  I think this is an important point. Having worked in / proximate to public policy kinds of problems, Bayesian methods have some really great properties: 1. easier interpretation of results than frequentist methods for lay people (business strata, elected officials, or other decision makers) 2. Uncertainty can be quantified and visualized reasonably well, which helps decision makers not think of stats as a magic box that produces a single answer. 3. Sensitivity analysis can be placed right up front: selection of priors representative of the beliefs of differing opinions / ideologies can inform decision makers of when they should consider changing their minds, and when they might still hold out. Downsides of Bayesian methods: 1) Conceptually more involved than typical maximum likelihood estimation methods 2) Computationally expensive 3) Methods might not be as well known to a nominally stats-savvy audience.
- hobolord 9y ago
  do you have a recommended guide/textbook on learning stan? I've recently started doing more bayesian analysis, mainly bayesian estimation supercedes the t-test.
  - wishart_washy 9y ago
    As someone who uses Stan - I would recommend reading the Stan reference documentation, it's essentially a textbook. Also, get used to reading the Stan forums on Discourse. Happy Stanning
kmax12 9y ago
One crucial skill you will need is feature engineering. Formal methods for it aren’t typically in data science classes. Still, it’s worth understanding in order to build ML applications. Unfortunately, there aren't many available tools today, but I expect that to change this year. Deep learning addresses it to some extent, but isn’t always the best choice if you don’t have image / text data (eg tabular datasets from databases, log files) or a lot of training examples. I’m the developer of a library called Featuretools (https://github.com/Featuretools/featuretools https://github.com/Featuretools/featuretools) which is a good tool to know for automated feature engineering. Our demos are also a useful resource to learn using some interesting datasets and problems: https://www.featuretools.com/demos https://www.featuretools.com/demos
in9 9y ago
I saw a simple tool somewhere a while ago (maybe a month or so ago) of a simple cli for data inspection in the terminal. It seemed very useful for inspecting data ssh'ed into a machine. However, I can't seem to recall the name. Has any one seen what I'm talking about?
larrykwg 9y ago
Nobody mentioned this yet: ETE: http://etetoolkit.org/docs/latest/tutorial/tutorial_trees.html http://etetoolkit.org/docs/latest/tutorial/tutorial_trees.ht... a fantastic tree visualization framework, its intended for phylogenetic analysis but can really be used for any type of tree/hierarchical structure
spdustin 9y ago
OpenRefine (openrefine.org) is definitely a handy (and automate-able) part of my data-cleansing workflow.
dxbydt 9y ago
> What’s the fizzbuzz test for data scientists anyway? Here's 3 questions I was recently asked on a bunch of DS interviews in the Valley. 1. Probability of seeing a whale in the first hour is 80%. What's the probability you'll see one by the next hour ? Next two hours ? 2. In closely contested election with 2 parties, what's the chance only one person will swing the vote, if there are n=5 voters ? n = 10 ? n = 100 ? 3. Difference between Adam and SGD.
innovather 9y ago
Hey everyone, I'm not a data scientist or a developer but I work with a lot of them. My company, Introspective Systems, recently released xGraph, an executable graph framework for intelligent and collaborative edge computing that solves big problems: those that have massive decision spaces, tons of data, are highly distributed, dynamically reconfigure, and need instantaneous decision making. It's great for the modeling work that data scientists do. Comment if you want more info.
vaibkv 9y ago
You need to buckle up on Mathematics. Do not look at fancy stuff like PyTorch, Keras, Tensorflow, etc. Libraries and languages are a means to an end. Good, intuitive understanding of mathematical concepts behind a certain ML algorithm will go a long way. As mentioned in one of the comments, statistics shows up everywhere.
topologie 9y ago
Random Matrix Theory.

341 points by mxgr 9y ago

A couple of thoughts, off the top of my head: Programming languages: - python (for general purpose programming) - R (for statistics) - bash (for cleaning up files) - SQL (for querying databases) Tools: - Pandas (for Python) - RStudio (for R) - Postgres (for SQL) - Excel (the format your customers will want ;-) ) Libraries: - SciPy (ecosystem for scientific computing) - NLTK (for natural language) - D3.js (for rendering results online)

jxub 9y ago
Maybe SpaCy for NLP. Way more intuitive and fast too. Good list.
ktpsns 9y ago
I make the claim that you can go very far in the SciPy ecosystem without ever touching R. It is worth understanding the concepts of numpy and pandas. Furthermore, try out IPython/Jupyter, especially for rapid publishing (people run their blogs on jupyter notebooks). I think certain libraries depend very much on where you focus. Machine learning? Native language processing? Visualization? Something in economics? Fundamental sciences? For instance, I never need NLTK in theoretical astrophysics ;-) Instead, I need powerful GPU based visualization, which is however very old school with VTK and Visit/Amira/Paraview (also very much pythonic).
- marmaduke 9y ago
  I second that R is nice to have, but not needed. I’ve been doing science in Python for a decade without ever needing R. For powerful GPU viz, have you considered vispy? Four authors of four independent Python science visualization libs got together to build it.
- albertgoeswoof 9y ago
  Agree, I would drop R, Python has you mostly covered now. Julia is also worth learning.
  - threeseed 9y ago
    I wouldn't be recommending to drop R at all. Very few enterprise data science teams are 100% Python (in fact none I've heard of). R is still very heavily used (and in fact all data science teams I've worked in it has been the dominant technology). There is a reason Microsoft purchased Revolution.
  - kirillseva 9y ago
    R, python and Julia are all Turing-complete languages, so of course you can drop any two and get by with just the third. The real selection happens when you consider what's available in opensource world. What code you don't have to write? What high-quality libraries are available vs which ones you will have to write yourself? On this topic, R has vast advantage over python in some domains, such as bioinformatics for example, while python definitely shines when it comes to deep learning (and using for loops). You can't just claim that one shouldn't look at R because you personally know one language better the other, quite likely because in your domain it's not being used as much. I do prefer the deep learnin, NLP and production serving story in python, but you will have to pry dplyr+ggplot from my cold dead hands for quick analysis and charting. Not to mention that pandas's API is a clusterfuck compared to R's native data frames.
- closed 9y ago
  I disagree, even though python is the language I do most of my development in. But it probably depends on the problems we're thinking of a data scientist solving. If you're doing a lot of work with matrices, model fitting in production, then python seems fine. However, a lot of data scientists I see are more like scrappy data analysis / visualization types, who are churning out small dashboards. In that case R's tidy verse and shiny are just incredibly fast to develop with.
Ultimatt 9y ago
I'd gently suggest basic CLI Perl over BASH for cleaning up files, as it combines grep/sed/awk in a language thats more generally useful.
- jonathankoren 9y ago
  Agreed. Perl was designed for text munging, and is superior to pretty much everything for this task. WRT bash, where to begin? In the past 40 years, there’s pretty much a better tool for everything someone tries to do with bash. It lives on pretty much through inertia and pride.
  - gkya 9y ago
    FreeBSD sh(1) (not bash(1)) man page. That's just how I understood how to shell. Nowadays I'm running Debian and my $SHELL is /bin/bash, but when I was on FreeBSD I really learnt tools like make(1), sh(1); the man pages were pieces of art. Having read sh(1), I do have a nice grasp of how shell works in general, to which knowledge I can add anytime the higher-level goodies bash has to offer (though I generally prefer keeping it POSIX, and using an actual programming language when it doesn't cut it).
cmdlinetips 9y ago
good list. I would add tidyverse in R ecosystem to it
- p33p 9y ago
  I would go as far as saying the tidyverse is an essential piece of working with R. Base R sans tidyverse is not a pleasant experience.
  - disgruntledphd2 9y ago
    It's not that bad. It's inconsistent and clunky, but all of the tools are there (and tend to be faster than the tidyverse versions). Don't get me wrong, I love the tidyverse but R is very, very usable without it.
pytonslange 9y ago
Most of these are conveniently packaged in: $ docker run -it --rm -p 8888:8888 jupyter/datascience-notebook

eggie5 9y ago

a lot of people using spark?

sandGorgon 9y ago
same question that i have. Anyone using pyspark in production ? Would you use pyspark mllib in a webservice instead of scikit ?
- wenc 9y ago
  1) Yes, PySpark is great if you're mostly just doing dataframe manipulation in Spark, using built-in functions. PySpark actually has similar performance to Scala Spark for dataframes. (We've moved away from RDDs) However, if you use a lot of UDFs where Spark has to serialize your Python functions, you might consider rewriting those UDFs in a JVM language. Serialization overhead is still fairly substantial. Arrow is trying to address this by implementing a common in-memory format, but it's still early days. I would still recommend PySpark to most people. It's more than good/fast enough for most data munging tasks. Scala does buy you two things: type safety and low serialization overhead (i.e. significant!), which can be critical in some situations, but not all. Also, the Python way has always been to prototype fast, profile, and rewrite bottlenecks in a faster language, and PySpark conforms to that pattern. 2) Spark MLLib is still fairly rudimentary in its coverage of major ML algorithms, and Spark's linear algebra support, while serviceable, is currently not very sophisticated. There are a few functions that are useful in the data prep stage (encoding, tokenizers, etc.) but overall, we don't really use MLlib very much. Companies that have simple needs (e.g. a simple recommender) and that don't have a lot of in-house expertise, might use MLlib though -- I believe someone from a startup said that they did at a recent meetup. Most of us need better algorithmic coverage and Scikit's coverage is currently much better, plus it is more mature. We also have Numpy at our disposal, which lets us do matrix-vector manipulation easily. There is some serialization cost, but we can usually just throw cloud computational power at it. Also note that for most workloads, the majority of the cost is incurred in training. For models in production, one is typically processing a much smaller amount of data using a trained model, so less horsepower is required.

greyman 9y ago

If you will work in some bigger company doing data analytics, you can also come across Tableau instead of Excel. Apart from SQL, if there is more data, you might want to use Bigquery or something similar.

latenightcoding 9y ago

If you use Python: scikit-learn, Pandas, NumPy, Tensorflow or PyTorch Language agnostic: XGBoost, LibLinear, Apache Arrow, MXNet

ever1 9y ago

Python: Jupyter, pandas, numpy, scipy, scikit-learn Numba for custom algorithms. Dataiku (amazing tool for preprocessing and complex flows) Amazon RDS (postgress), but thinking about redshift. Spark Tableau or plotly/seaborn

eps 9y ago

You probably mean "data analyst". "Data scientist" title would apply only if you are applying scientific method to discover new fact about natural world exclusively through data analysis (as opposed to observation and experiments).

Twisell 9y ago
Define natural world... And gather a consensus around your definition... Or maybe whole humanities should be considered as « not science ». Beside a data analyst that don’t use scientific method is just a bad analyst. Some media outlet showcase blatantly lying charts made by people that understand the technicals but get everything wrong about the concepts. So this is my advice, focus on understanding the concepts before the tooling. That is what will really make your value.
sgt101 9y ago
Designing experiments is a key part of Data Science work. Another key part is determining where & how revealing observations can be made. The analysis part is usually quite simple, often if it gets really complex then that's a sign that the data is being tortured. Sometimes the marginal gains that complex methods create (vs simple but good approaches) are not worthwhile even if they are valid - simply in terms of time spent and difficulty in communications.

kome 9y ago

Excel, VBA, SPSS ;)

babayega2 9y ago
OpenRefine has helped me a lot in data cleaning tasks.

sureaboutthis 9y ago

As a data scientist, you should already know that answer and be telling us.

piqufoh 9y ago

> what tools should be in my arsenal A sound understanding of mathematics, in particular statistics. It's amazing how many people will talk endlessly about the latest python/R packages (with interactive charting!!!) who can't explain the student's t-test.

anc84 9y ago

Any programming language that you are proficient in. A solid understanding how a computer works. Solid basis of statistics. Anything else is just sprinkles, trends and field-specific.

greyman 9y ago
Hard to say... I was more proficient in PHP than python, but when doing AI, we use python anyway, since in PHP some necessary libraries just aren't there...
wenc 9y ago
> Any programming language that you are proficient in. Oh I don't know about that. Programming languages are force multipliers, and each language has a different force coefficients for different problem domains. They are not all equivalent. They have their different points of leverage, and simply being good in one does not mean you can solve problems in any domain with ease. In fact the wrong programming language can often be harmful if it's ill-suited to the problem at hand, and especially if it contorts your mental model of what you can do with the data. One example I encounter a lot in industry is Excel VBA. I'm fairly good at VBA and have seen very sophisticated code in VBA. I've also seen many basic operations implemented badly in VBA that should not have been written in VBA at all. By solving the problem in VBA, the solution is often "hemmed in" by the constraints of VBA. For instance, unpivoting data is often done badly in VBA (with for-loops), but is trivial to do well in dplyr or pandas. So I would say one has to choose one's programming language somewhat carefully. Not any language will do.

SFjulie1 9y ago

Methodology, Signal processing, probability and statistic. Any monkey can make nice plots with R or python. Using a tool will not make you efficient. What matters is the brain of the one using the tools. In fact for the same reasons astronauts are scientific trained to pilot and not pilots trained in science, data scientists/analyst are scientist trained to use tools and thus the science part is the only one that matters. Major problem I have encountered in IT is that monkey hate science, they just go on internet, see a blog post and reproduce nice biased manipulation (the more obscure, the deeper it will be perceived). It is basically cargo cult. Mimicking science instead of doing science. Science is boring and is all about being a data janitor, and cleaning data so much, it sparkles with simplicity. How to clean data is the most vital part of data "science". IT data are often plagued with a lot of errors typo, poor encoding translations, heterogeneous sources which quality and relevance varies a lot due to IRL bias. Having a background in being a DBA, a coal faced coder helps assessing fast the quality of data and transforming them. Also, IT is plagued with floats. Where science matters, the uncertainty inherent with float representation can introduce artefacts that are a pain in the a.. when it comes to phenomenon that are sensitive to initial conditions or when measure instruments are put at the edge of their resolutions. So at my opinion, the "measure theory" (Lebegue) is a must. It helps you understand the confidence you can have from your data, the margin error you have, and assess if data are faked. In IT a lot of cheating is involved, so being able to detect fraud is required, for this you have to know the shape of "randomness" (Poisson, Student...) and know your estimators. Also, signal processing basics are important for seasonal/recurrent activities (trend analysis). However, some phenomeon are non linear, hence using fourier transform, moving average sometimes are the wrong (overused) tool, thus a little knowledge of complex systems (stochastic) is a plus. Then after a LOT of work (the last 5%) you may have to decide if you face a correlation, a causation, or just noise. For this there are 2 schools : bayesians and frequentists. I am a frequentist because the tools are far more robusts, easy to understand and to use. The market want bayesian because it is so obscure you can easily make your data say whatever you want. My intuition is bayesian are at the core of the problem of reproducibility in science. And last but not least, your work is useless if you cannot communicate. There are standards : removing all useless elements from plots, quoting your sources the right way... For the tools ? It depends. They are domain specific. Some use matlab, mathematica, excel, R, python, proprietary software. The scientific background is so harder to find than a programming background no serious companies will turn you down for not knowing their tools. But clearly having a factual knowledge of accessing, transforming, cleaning data is a must and this can a combination of languages and tools (C, python, Perl, bash, sed, awk, grep, C++, pandoc, imagemagick) since every languages have their strengths and weaknesses and you don't know where you data will come from (CSV, DBs (SQL or not), web pages, scans, pictures, instruments, simulations, text files, structured documents, syscalls....). Thus, being a data analyst is first of all knowing the intellectual and computer tools to assess, gather, transform, clean the datas and methodic about their source and how you did it so that someone else can try to reproduce your claims.

fredley 9y ago

IPython/Jupyter, Pandas/Numpy and Python will get you everywhere you need to go. Currently, until maybe Go gets decent DataFrame support, in terms of the total time to get to your solution, I'd be amazed if any other setup got you there quicker.

threeseed 9y ago
> get you everywhere you need to go No it won't. That combination can't handle large datasets that are typical for most data science teams i.e. maybe include PySpark. And then it's very limited so far as ML/DL technologies.
- fredley 9y ago
  > i.e. maybe include PySpark Pandas and Spark are both DataFrame libraries, and seem to offer very similar functionality to me. Why do you prefer Spark over Pandas? > very limited so far as ML/DL technologies I mean, getting Tensorflow up and running with GPU support isn't trivial, but it's not exactly hard, and Keras[1] provides excellent support for a wide variety of other backends. What, in your experience, is less limited? [1]: https://keras.io/ https://keras.io/
  - jwilbs 9y ago
    Spark sits on top of YARN/Mesos, and is used for data processing scalability that pandas can't handle. Personally, I think two areas often lacking are software development skills and general statistics knowledge. The former is necessary for writing production-quality code, assisting with an sort of data engineering pipeline, writing reliable, reusable code, and creating custom solutions. Unfortunately, the latter is often skimped on (if not skipped entirely) in favor of more 'hot' fields like ml/dl, with the result being a fuzzy understanding across the board. (You'd be amazed at the quantity of candidates lacking fundamental knowledge about glm's, basic nonparametric stats, popular distributions, etc).
- ppod 9y ago
  >typical for most data science teams I would bet that the mean size of dataset people are dealing with is a lot bigger than the median size.

ms013 9y ago

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you can go surprisingly far with few and relatively simple ones. Knowing how, when, and where to apply them is the hard part: and that often boils down to understanding the mathematics and domain you are working in. And don’t over use viz. Pictures do effectively communicate, but often people visualize without understanding. The result is pretty pictures that eventually people realize communicate little effective domain insight. You’d be surprised that sometimes simple and ugly pictures communicate more insight than beautiful ones do. My arsenal of tools: python, scipy/matplotlib, Mathematica, Matlab, various specialized solvers (eg, CPLEX, Z3). Mathematical arsenal: stats, probability, calculus, Fourier analysis, graph theory, PDEs, combinatorics. (Context: Been doing data work for decades, before it got its recent “data science” name.)

gaius 9y ago
Indeed. Too many people when asked about their skills or experience just rattle off a list of tools or libraries. Usually the same ones as everyone else!
anewhnaccount2 9y ago
Out of interest, can you give an example of a problem you've solved using Z3?
- ms013 9y ago
  One data problem boiled down to being an instance of the set cover problem (https://en.m.wikipedia.org/wiki/Set_cover_problem https://en.m.wikipedia.org/wiki/Set_cover_problem). Pretty easy to pose as an integer constraint problem, and Z3 solved it in about 20 minutes for me.
edem 9y ago
I really like to get a degree in Mathematics but I simply don't have the time to throw at it (work, children, etc). What do you suggest I should do to have something on my resume? MOOC maybe?

trollied 9y ago

What does "Data Scientist" actually mean these days? Does it mean "Write 10 lines of Python or R, and not fully understand what it actually does"? Or something else? I just see the term flinged around so much recently, and applied to so many different roles, it has all become a tad blurred. Maybe we need a Data Scientist to work out what a Data Scientist is?

threeseed 9y ago
I hire data scientists so can tell you. It means someone who can work with business stakeholders to break down a problem e.g. "we don't know why customers are churning", produce a machine learning model or some adhoc analysis (usually the former) and either communicate the results back or assist in deploying the model into production. Typically there will be data engineers who will be doing acquisition and cleaning and so the data scientists are all about (a) understanding the data and (b) liaising with stakeholders. As for technologies it is typically R/Python with Spark/H20 on top of a data lake i.e. HDFS, S3. Every now and again on top of an SQL store e.g. EDW, Presto or a Feature store e.g. Cassandra.
jxub 9y ago
That's a good meta reflection. Let's make an Y Combinator of Data Scientist a and Data Scientist b (recursive data scientist) to prove they can support recursion if Data Scientists a and b are first class functions, just because we can: const Y = a => (b => b(b))(b => a(x => b(b)(x)));

amelius 9y ago

Any book recommendations?

severo 9y ago

I'd say: 1. You need research skills that will allow you to ask the right questions, define the problem and put it in a mathematical framework. 2. Familiarity with math (which? depends on what you are doing) to the point where you can read articles that may have a solution to your problem and the ability to propose changes, creating proprietary algorithms. 3. Some scripting language (Python, R, w/e) 4. (optional) Software Engineering skills. Can you put your model into production? Will your algorithm scale? Etc.

closed 9y ago

I would think about which of these you see yourself doing more.. * statistical methods (more math) * big, in-production model fitting (more python) * quick, scrappy data analyses for internal use (more R) For example, I would feel weird writing a robust web server in R, but it's straightforward in python. On the other hand R's shiny lets you put up quick, interactive web dashboards (that I wouldn't trust in exposing to users).

justusw 9y ago

Dealing with large data processing problems my main tools are as follows: Libs: - Dask for distributed processing - matplotlib/seaborn for graphing - IPython/Jupyter for creating shareable data analyses Environment: - S3 for data warehousing, I mainly use parquet files with pyarrow/fastparquet - EC2 for Dask clustering - Ansible for EC2 setup My problems usually can be solved by 2 memory-heavy EC2 instances. This setup works really well for me. Reading and writing intermediate results to S3 is blazing fast, especially when partitioning data by days if you work with time series. Lots of difficult problems require custom mapping functions. I usually use them together with dask.dataframe.map_partitions, which is still extremely fast. The most time-consuming activity is usually nunique/unique counting across large time series. For this, Dask offers hyperloglog based approximations. To sum it up, Dask alone makes all the difference for me!

drej 9y ago

grep, cut, cat, tee, awk, sed, head, tail, g(un)zip, sort, uniq, split; curl; jq, python3

proc0 9y ago
So unix? lol
- deleted 9y ago
  [deleted]

chewxy 9y ago

My toolkit hasn't changed since 2016: - Jupyter + Pandas for exploratory work, quickly define a model - Go (Gonum/Gorgonia) for production quality work. (here's a cheatsheet: https://www.cheatography.com/chewxy/cheat-sheets/data-science-in-go-a/ https://www.cheatography.com/chewxy/cheat-sheets/data-scienc... . Additional write-up on why Go: https://blog.chewxy.com/2017/11/02/go-for-data-science/ https://blog.chewxy.com/2017/11/02/go-for-data-science/) I echo ms013's comment very much. Everything is just tools. More important to understand the math and domain

ZeroCool2u 9y ago
I'm a big Go fan, but this is the first time I've seen someone recommend Go for data science. After looking at this cheat sheet you've got me convinced though. Would you mind pointing me to any other less cheat sheet style and more in depth examples that you particularly like?
- chewxy 9y ago
  Working on it. Part of my goal for 2018 is to write a lot more soft documentation - tutorials etc. Go is quite straightforwards though - WYSIWYG for the most parts, hence you probably won't find a lot of sexy tutorials. Almost everything is just a loop away, and in the next version of Gorgonia, even more native looping capability is coming
  - ZeroCool2u 9y ago
    Awesome, thank you!
- samuell 9y ago
  You might also want to have a look at: - http://gopherdata.io http://gopherdata.io ... and in particular the resources lists at - https://github.com/gopherdata/resources https://github.com/gopherdata/resources Also, Dan's GopherCon talk on Go for data science is a great way to get yourself convinced enough to try it out: - https://www.youtube.com/watch?v=D5tDubyXLrQ https://www.youtube.com/watch?v=D5tDubyXLrQ

pentium10 9y ago

As 1TB is free for processing every month, using SQL 2011 standard + combined with Javascript UDFs, the winner solution is Google BigQuery for us, combined with Dataprep

sdfjkl 9y ago

numpy, Jupyter (formerly IPython Notebook) and probably Mathematica anyways.

ChrisRackauckas 9y ago

A good understanding of calculus (probability), linear algebra, and your dataset/domain. Anything else can be picked up as you need it. Oh, and test-driven development in some programming language, otherwise you can't develop code you know is correct.

schaunwheeler 9y ago

A lot of people in this thread are focusing on technical tools, which is normal for a discussion of this type, but I think that focus is misplaced. Most technical tools are easily learnable and are not the limiting factor is creating good data science products. https://towardsdatascience.com/data-is-a-stakeholder-31bfdb650af0 https://towardsdatascience.com/data-is-a-stakeholder-31bfdb6... (Disclaimer: I wrote the post at the above link). If you have a sound design you can still create a huge amount of value even with a very simple technical toolset. By the same token, you can have the biggest, baddest toolset in the world and still end up with a failed implementation if you have bad design. There are resources out there for learning good design. This is a great introduction and points to many other good materials: https://www.amazon.com/Design-Essays-Computer-Scientist/dp/0201362988 https://www.amazon.com/Design-Essays-Computer-Scientist/dp/0...

Xcelerate 9y ago

As a data scientist who has been using the language for 5 years now, Julia is by far the best programming language for analyzing and processing data. That said, it’s common to find many Julia packages that are only half-maintained and don’t really work anymore. (I still don’t know how to connect to Postgres in a bug-free way using Julia.) And you’d be hard pressed to find teams of data scientists that use Julia. So in that sense, Python has much more mature and stable libraries, and it’s used everywhere. (But I really hope Julia overtakes it in the next couple of years because it’s such a well-designed language.) Aside from programming languages, Jupyter notebooks and interactive workflows are invaluable, along with maintaining reproducible coding environments using Docker. I think memorizing basic stats knowledge is not as useful as understanding deeper concepts like information theory, because most statistical tests can easily be performed nowadays using a library call. No one asks people to program in assembler to prove they can program anymore, so why would you memorize 30 different frequentist statistical tests and all of the assumptions that go along with each? Concepts like algorithmic complexity, minimum description length, and model selection are much more valuable.

chubot 9y ago
Has Julia converged on a solution for data frames? I watched some JuliaCon videos and got the impression that they hadn't. There seem to be a lot of different overlapping efforts.
- 0kto 9y ago
  Well, only the DataFrames.jl package comes to my mind. However, there exist a few packages that extend this package (DataFramesMeta.jl or Query.jl; these overlap to some extend, but the newer Query package seems to go beyond DataFrames and offers some piping capabilities to interface with plotting packages). In general: During the three years of my PhD some language / package upgrades broke some of my scripts (during 0.4 -> 0.5 and -> 0.6), but the language (and its extensive documentation, online and from the source code of the packages) is very pleasant to use - the deprecation warnings usually help you to adjust your code in time. I have been relaying heavily on said DataFrame package, and am quite happy - the community is usually responsive and helpful in case of problems or questions.

bitL 9y ago

Spark + MLlib, Python + Pandas + NumPy + Keras + TensorFlow + PyTorch, R, SQL, top placement in some Kaggle competitions. This would get you long way.

geebee 9y ago
Good tool set recommendations (+1 for mentioning SQL, immensely helpful), and I enjoy Kaggle. Not sure how critical top placement is, though. It seems like getting into the upper echelons of Kaggle is a matter of refining your model, and I do wonder how much value these refinements offer over a more basic and general approach in a real world scenario. To be clear, when I say I wonder, I'm not saying I'm rejecting the value, I really do mean it, I'm uncertain about the value. I think it's probably very scenario specific. Think of it this way - a predictive value of 90% vs 95% could be the difference between placing in the top 10% and the bottom third. Now, 5% isn't nothing, it could be very valuable. It really depends. But Kaggle is an environment where the question is already posed, the data has been collected, the test and train sets are already split apart for you, and winning model is the one that scores best on a hidden test set by a predefined goodness of fit score. In a real world scenario, suppose someone does a great job figuring out the question to ask, gathering the data, and determining the most effective way to act on the results, but uses a fairly basic, unrefined model. Someone else does a middling job on those things, but builds a very accurate model as measured by the data that has been collected. I'd say the first scenario is likely to be more valuable, but again, it depends of course. A couple other things, since I am a fan of Kaggle and do highly recommend it. First, these things aren't necessarily exclusive - you can have a particularly well conceived and refined model as well as a thorough and excellent businesss and data collection process (though you may have to decide where to put your time and resources). Also, refining a model with Kaggle can be an exceptional training opportunity to really understand what drives these things. So go for it! (I also find these things kinda fun).
- bitL 9y ago
  Top placement in Kaggle attracts recruiters for higher positions; i.e. I observed a top 10 person getting a job of Head/VP of analytics in a large European company even if let's say formal education wasn't top 100. I agree real-world it is often useless, but people are drawn to proven winners.

nrjames 9y ago

There are two "poles" in data science: math/modeling and backend/data-wrangling. Most of the time, the backend/data-wrangling piece is a prerequisite to the math/modeling. The vast majority of small and medium sized companies have not set up the systems they would need to support a data scientist who knows only math/modeling. Depending on the domain, it's not uncommon to find that a small/medium company outsourced analytics to Firebase, Flurry, etc... That's fine, but when it comes time to create some customer segmentation models (or whatever) the data scientist they hire is going to need to know how to get the raw data. Questions become: how do I write code to talk to this API? How do I download 6 months of data, normalize it (if needed) and store it in a database? Those questions flow over into: how do I set up a hosted database with a cloud provider? What happens if I can't use the COPY command to load in huge CSV files? How do I tee up 5 TB of data so that I can extract from it what I need to do the modeling? Then you start looking at BigQuery or Hadoop or Kafka or NiFi or Flink and you drown for a while in the Apache ecosystem. If you take a job at a place that has those needs, be prepared to spend months or even up to a year to set up processes that allow you to access the data you need for modeling without going through a painful 75 step process each time. Case in point: I recently worked on a project where the raw data came to me in 1500 different Excel workbooks, each of which had 2-7 worksheets. All of the data was in 25-30 different schemas, in Arabic, and the Arabic was encoded with different codepages, depending on whether it came from Jordan, Lebanon, Turkey, or Syria. My engagement was to do modeling with the data and, as is par for the course, it was an expectation that I would get the data organized. Well - to be more straightforward, the team with the data did not even know that the source format would present a problem. There were ~7500 worksheets, all riddled with spelling errors and the type of things that happen when humans interact with Excel: added/deleted columns, blank rows with ID numbers, comments, different date formats, PII scattered everywhere, etc. A data scientist's toolkit needs to be flexible. If you have in mind that you want to do financial modeling with an airline or a bank, then you probably can focus on the mathematics and forget the data wrangling. If you want the flexibility to move around, you're going to have to learn both. The only way to really learn data wrangling is through experience, though, since almost every project is fundamentally different. From that perspective, having a rock solid understanding of some key backend technologies is important. You'll need to know Postgres (or some SQL database) up and down; how to install, configure, deploy, secure, access, query, tweak, delete, etc. You really need to know a very flexible programming language that comes with a lot of libraries for working with data of all formats. My choice there was Python. Not only do you need to know the language well, you need to know the common libraries you can use for wrangling data quickly and then also for modeling. IMO, job descriptions for "Data Scientist" positions cover too broad of a range, often because the people hiring have just heard that they need to hire one. Think about where you want to work and/or the type of business. Is it established? New? Do they have a history of modeling? Are you their first "Data Scientist?" All of these questions will help you determine where to focus first with your skill development.

Jeff_Brown 9y ago

Static typing lets you catch errors before running the code. Pattern matching helps you write code faster (that is, spending less human time). Algebraic data types, particularly sum types, let you represent complicated kinds of data concisely. Coconut is an extension of Python that offers all of those. Test driven development also helps you write more correct code.

ak_yo 9y ago

Experimental design and observational causal inference would be excellent skills to have. Especially if you’re working with people who are asking you “why” questions, ML is helpful but isn’t going to cut it alone.

elsherbini 9y ago

I'm a scientist (PhD student in microbiolgy) that works with lots of data. My data is on the order of hundreds of gigabytes (genome collections and other sequencing data) or megabytes (flat files). I use the `tidyverse` from R[0] for everything people use `pandas` for. I think the syntax is soooo much more pleasant to use. It's declarative and because of pipes and "quosures" is highly readable. Combined with the power of `broom`,fitting simple models to the data and working with the results is really nice. Add to that that `ggplot` (+ any sane styling defaults like `cowplot`) is the fastest way to iterate on data visualizations that I've ever found. "R for Data Science" [1] is great free resource for getting started. Snakemake [2] is a pipeline tool that submits steps of the pipeline to a cluster and handles waiting for steps to finish before submitting dependent steps. As a result, my pipelines have very little boilerplate, they are self documented, and the cluster is abstracted away so the same pipeline can work on a cluster or a laptop. [0] https://www.tidyverse.org/ https://www.tidyverse.org/ [1] http://r4ds.had.co.nz/ http://r4ds.had.co.nz/ [2] http://snakemake.readthedocs.io/en/stable/ http://snakemake.readthedocs.io/en/stable/

nonbel 9y ago
Sometimes I think I'm the only one who isn't really a fan of the tidyverse. I've found it slower, more prone to dependency issues, more prone to silent errors, and less well documented than most R packages (ie most of what you find on CRAN).
- swarchal 9y ago
  You're not the only one. Though I've found there seems to be a bit of a cult surrounding the tidyverse, a mere hint of criticism usually results in outrage and attacking other tools/packages (by users, not the authors).
  - anarchimedes 9y ago
    I like the Tidyverse. My only complaint is that it presents another headache when moving from one language (R) to another (usually Python/SQL). Using the base R functions while integrating loops and functions lessens the fatigue of moving between R and Python.

cwyers 9y ago

You can get a lot of mileage out of just using R, dplyr, ggplot2 and lm/glm. OLS still performs well in a lot of problem spaces. Understanding your data is the key there, and a lot of exploratory visualization there will help a lot.

ellisv 9y ago

Counting and dividing.

xitrium 9y ago

If you care about quantifying uncertainty, knowing about Bayesian methods is a good idea I don't see represented here yet. I care so much about uncertainty quantification and propagation that I work on the Stan project[0] which has an extremely complete manual (600+ pages) and many case studies illustrating different problems. Full Bayesian inference such as that provided by Stan's Hamiltonian Monte Carlo inference algorithm is fairly computationally expensive so if you have more data than fits into RAM on a large server, you might be better served by some approximate methods (but note the required assumptions) like INLA[1]. [0] http://mc-stan.org/ http://mc-stan.org/ [1] http://www.r-inla.org/ http://www.r-inla.org/

peatmoss 9y ago
I think this is an important point. Having worked in / proximate to public policy kinds of problems, Bayesian methods have some really great properties: 1. easier interpretation of results than frequentist methods for lay people (business strata, elected officials, or other decision makers) 2. Uncertainty can be quantified and visualized reasonably well, which helps decision makers not think of stats as a magic box that produces a single answer. 3. Sensitivity analysis can be placed right up front: selection of priors representative of the beliefs of differing opinions / ideologies can inform decision makers of when they should consider changing their minds, and when they might still hold out. Downsides of Bayesian methods: 1) Conceptually more involved than typical maximum likelihood estimation methods 2) Computationally expensive 3) Methods might not be as well known to a nominally stats-savvy audience.
hobolord 9y ago
do you have a recommended guide/textbook on learning stan? I've recently started doing more bayesian analysis, mainly bayesian estimation supercedes the t-test.
- wishart_washy 9y ago
  As someone who uses Stan - I would recommend reading the Stan reference documentation, it's essentially a textbook. Also, get used to reading the Stan forums on Discourse. Happy Stanning

kmax12 9y ago

One crucial skill you will need is feature engineering. Formal methods for it aren’t typically in data science classes. Still, it’s worth understanding in order to build ML applications. Unfortunately, there aren't many available tools today, but I expect that to change this year. Deep learning addresses it to some extent, but isn’t always the best choice if you don’t have image / text data (eg tabular datasets from databases, log files) or a lot of training examples. I’m the developer of a library called Featuretools (https://github.com/Featuretools/featuretools https://github.com/Featuretools/featuretools) which is a good tool to know for automated feature engineering. Our demos are also a useful resource to learn using some interesting datasets and problems: https://www.featuretools.com/demos https://www.featuretools.com/demos

in9 9y ago

I saw a simple tool somewhere a while ago (maybe a month or so ago) of a simple cli for data inspection in the terminal. It seemed very useful for inspecting data ssh'ed into a machine. However, I can't seem to recall the name. Has any one seen what I'm talking about?

larrykwg 9y ago

Nobody mentioned this yet: ETE: http://etetoolkit.org/docs/latest/tutorial/tutorial_trees.html http://etetoolkit.org/docs/latest/tutorial/tutorial_trees.ht... a fantastic tree visualization framework, its intended for phylogenetic analysis but can really be used for any type of tree/hierarchical structure

spdustin 9y ago

OpenRefine (openrefine.org) is definitely a handy (and automate-able) part of my data-cleansing workflow.

dxbydt 9y ago

> What’s the fizzbuzz test for data scientists anyway? Here's 3 questions I was recently asked on a bunch of DS interviews in the Valley. 1. Probability of seeing a whale in the first hour is 80%. What's the probability you'll see one by the next hour ? Next two hours ? 2. In closely contested election with 2 parties, what's the chance only one person will swing the vote, if there are n=5 voters ? n = 10 ? n = 100 ? 3. Difference between Adam and SGD.

innovather 9y ago

Hey everyone, I'm not a data scientist or a developer but I work with a lot of them. My company, Introspective Systems, recently released xGraph, an executable graph framework for intelligent and collaborative edge computing that solves big problems: those that have massive decision spaces, tons of data, are highly distributed, dynamically reconfigure, and need instantaneous decision making. It's great for the modeling work that data scientists do. Comment if you want more info.

vaibkv 9y ago

You need to buckle up on Mathematics. Do not look at fancy stuff like PyTorch, Keras, Tensorflow, etc. Libraries and languages are a means to an end. Good, intuitive understanding of mathematical concepts behind a certain ML algorithm will go a long way. As mentioned in one of the comments, statistics shows up everywhere.

topologie 9y ago

Random Matrix Theory.