10 ms·
Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand t
by ms013 9y ago
Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis.
Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you can go surprisingly far with few and relatively simple ones. Knowing how, when, and where to apply them is the hard part: and that often boils down to understanding the mathematics and domain you are working in.
And don’t over use viz. Pictures do effectively communicate, but often people visualize without understanding. The result is pretty pictures that eventually people realize communicate little effective domain insight. You’d be surprised that sometimes simple and ugly pictures communicate more insight than beautiful ones do.
My arsenal of tools: python, scipy/matplotlib, Mathematica, Matlab, various specialized solvers (eg, CPLEX, Z3). Mathematical arsenal: stats, probability, calculus, Fourier analysis, graph theory, PDEs, combinatorics.
(Context: Been doing data work for decades, before it got its recent “data science” name.)
- gaius 9y agoIndeed. Too many people when asked about their skills or experience just rattle off a list of tools or libraries. Usually the same ones as everyone else!
- anewhnaccount2 9y agoOut of interest, can you give an example of a problem you've solved using Z3?
- ms013 9y agoOne data problem boiled down to being an instance of the set cover problem (https://en.m.wikipedia.org/wiki/Set_cover_problem https://en.m.wikipedia.org/wiki/Set_cover_problem). Pretty easy to pose as an integer constraint problem, and Z3 solved it in about 20 minutes for me.
- edem 9y agoI really like to get a degree in Mathematics but I simply don't have the time to throw at it (work, children, etc). What do you suggest I should do to have something on my resume? MOOC maybe?
- deleted 9y ago[deleted]
- deleted 9y ago[deleted]
- CoVar 9y agoUsually MOOC for resume don't help as everyone does them. The advice that I found useful for resume building is working on projects that you can catalog in a portfolio. With regards to gaining math skills, this upcoming MOOC from Microsoft on EdX looks promising[1]. [1] https://www.edx.org/course/essential-mathematics-for-artificial-intelligence https://www.edx.org/course/essential-mathematics-for-artific...
- edem 9y agoSo you suggest that I should learn from MOOC then go on and work on some projects so that I can prove I really know it.
- CoVar 9y agoExactly. And to take it one step further, choose one industry you are interested in. That way you will gain invaluable domain experience as you add relevant portfolio projects. If you don't have an industry in mind, you can use a site like glassdoor.com and search for data scientist positions by city and industry to get a feel for demand.
- edem 9y agoFull disclosure: I'm in the industry for 10+ years as a programmer. I just realized that if I want to move in the AI direction I'll need some math education. I don't want to become a data scientist.
- p33p 9y agoAnd don’t over use viz. Pictures do effectively communicate, but often people visualize without understanding. The result is pretty pictures that eventually people realize communicate little effective domain insight. You’d be surprised that sometimes simple and ugly pictures communicate more insight than beautiful ones do. I don't necessarily agree with this. Yes, a sound understanding of the domain and knowledge of the mathematics and statistics are vital to gaining insights. But. I would make a very clear distinction between exploratory data viz and explanatory data viz. Data visualization when presenting those insights is an important part of driving decision making.
- barskern 9y ago> And don’t over use viz. [...] I don't fully agree with this neither. Especially for mathematical concepts, visualization can give insight into how theorems are constructed and combined. This can prove to be vital when applying concepts and theorems to new problems. I would especially like to bring forth 3blue1brown[1]. He is a creator of videos which beautifully visualizes and explains complex mathematical problems. His efforts has given me an insight into math which theorems explained in text and variables could never do. However I do see your point that visualizations without understanding can be misleading. Hence the pure, written math is important to read and reason about, but I do believe that some concepts need to be visualized to be fully understood. [1]: http://www.3blue1brown.com http://www.3blue1brown.com
- biswaroop 9y agoI think what he/she means is poorly designed visualizations. Just because a plot is grayscale and not interactive doesn't mean it's worse than a cluttered poorly-designed super interactive web widget. It's a poor choice of wording, but I think by "overuse" they might mean "unclear but eye-catching". Besides "overuse" is literally the quantity that is excessive.
- gesman 9y agoGood viz is what connects non-ML/AI users to the "magical" results of ML/AI
- fsloth 9y agoDo people get careers as 'data scientists' without masters degrees? I'm heavily biases towards experts... but still would call it fair in the general case to call anyone doing data science without at least a masters degree or the equivalent mental toolkit more like a 'data quack'.
- booleandilemma 9y agoI think data scientist, much like software engineer, is something you can call yourself without having any credentials whatsoever. It’s why technical interviews can be so brutal, unfortunately. There are a lot of frauds out there. Money attracts frauds. What’s the fizzbuzz test for data scientists anyway?
- red_hare 9y agoI'm a data engineer for a startup that's trying to hire its first data scientist. The range of candidates that apply with this title is massive. Defining our expectations has been challenging. My phone screen "fizzbuzz" is having them calculate a standard deviation from an array of data w/out with only basic operators (no numpy.std). Then explain why they choose population/sample and explain the difference. I studied math in undergrad so one of my requirements is "knows more math than me".
- arca_vorago 9y agoI have to admit this scares me just a little bit. I'm a senior sysadmin who is trying to lateral transition into data science, but I'm no math whiz, I'm just good at pragmatic use of tech stacks and have a generally analytical mind. If you are a math undergrad how could I ever expect to know more math than you? Of course a standard deviation should be easy, but your comment on math just stuck out to me.
- shepardrtc 9y agoMaybe consider being a data engineer or a systems engineer? There's a pretty big demand for people that can set up, maintain, and assist the data scientists with the more complex tech stacks out there. In a former job as a systems engineer, I set up Hadoop clusters and helped manage data going into and out of it. And if you do decide to continue learning to become a data scientist, you'll already have a solid footing on the tech they actually use.
- Aardwolf 9y agoR is not present in your list, did you ever try it and what's your opinion about it?
- asafira 9y agoI think it is safe to say that they don't think it's too important for them given their main message, but hey, maybe they have an opinion anyway...
- throwaway7645 9y agoIf you have Mathematica, you might not need R as both are like Swiss Army Chainsaws for Data Analytics.
- throwaway7645 9y agoNot sure y I was downvoted here. I've used both products in this problem area. Mathematica is definitely more than a CAS. Both are great in their own ways.
- mojoe 9y agoR is good for one-off analysis problems, but is really bad for large distributed production systems. My team moved a large internal analysis application from R to Python because Python works well for both statisticians and software engineers.
- ms013 9y agoI know R and have used it in the past. I just don’t like the language. I keep RStudio around though because on rare occasions I do look around in it to see if it has something I need. So rarely though that I forgot to list it...
- Balgair 9y agoHonestly, in any STEM major, esp. the physics heavy ones, those maths areas should be well understood. Is any (physics heavy) STEM major also a Data Scientist then too?
- fhk 9y ago@ms013 interested to know how you are using the solvers, are you willing to share any further details?
- ms013 9y agoResponded to someone else earlier about this. Used solvers for problems that end up requiring solutions to problems like minimum set cover or schedule optimization problems. Basically, problems where a naïve or brute force approach will take forever to run and you need to use a real solver to attack it. These usually are data problems that end up looking like what would traditionally be considered under the umbrella of operations research.
- maayank 9y agoI have good background in graph theory (IMHO) but don't know many data science use-cases (I'm amateur at that). Could you point to some good start points?
- ms013 9y agoGraphs show up all over the place. Social media: who is connected, which people interact. Cybersecurity: which computers/programs/users interact with which other computers/programs/users. Retail analytics: which products are bought with which other products; which products are more important in a graph than others. Basically, any problem where you can establish relations between elements can be treated as a graph. I've used graphs for image analysis before too: pixels are vertices, edges represent neighborhood relations - especially useful when you make nonlocal connections (e.g., nonlocal means; graph-cut methods for segmentation; etc...) I've worked with them in three of the above contexts: cybersecurity (my current projects), retail analytics, and image analysis. I've avoided social network stuff - never cared for that area much.
- uptownfunk 9y agoI think visualization can be a helpful tool to understand the data. I have seen some DS's get caught up in visualization for visualization's sake which I think can be wasteful. I definitely think a solid mathematical understanding helps to build quantitative and critical thinking skills which are very key in data science.