6 ms·
A Gentle Visual Intro to Data Analysis in Python Using Pandas
- jalammar 8y agoHello HN, author here. If you've ever wanted to get into data analysis, this is my best attempt at getting you past that first hump. A lot of these concepts are easier than you might think.
- bussiere 8y agoThanks for the tutorial, it's really what i was looking to give to some friends.
- pyrenan 8y agoThanks for this. This is very useful for a beginner.
- lesss365 8y agoThis is a perfect out-of-tutor-session reference for my novice data analysis pupils. Will be sharing with them later today. Thank you!
- qwerty456127 8y agoThank you very much. I would certainly love to read more and more about Pandas (or anything) written this style and go deeper in the subject. Are you going to write more? Can you (or anybody) recommend where (a book, a YouTube channel, a website or whatever) do I continue from the point where you intro ends? As for now all I use of Pandas is a datetime-indexed array of real numbers + simple vector operations on its columns but I feel like I would like to take a learning/career path to becoming a Pandas expert.
- jalammar 8y agoA reasonable next step would be pick up a dataset that interests you (in a domain you're comfortable with) and explore it with pandas. Kaggle has a bunch of data sets (https://www.kaggle.com/datasets https://www.kaggle.com/datasets) in various domains. You can look at the "Kernels" where other users often use pandas to uncover insights and show you their process. Thanks for the kind words!
- happy-go-lucky 8y agoYou may want to check this out: > Short hands-on challenges to perfect your data manipulation skills https://www.kaggle.com/learn/pandas https://www.kaggle.com/learn/pandas Also this: > Things in Pandas I Wish I'd Known Earlier http://nbviewer.jupyter.org/github/rasbt/python_reference/blob/master/tutorials/things_in_pandas.ipynb http://nbviewer.jupyter.org/github/rasbt/python_reference/bl...
- skadamat 8y agoHey there, I'm involved with Dataquest and we have a Pandas and NumPy fundamentals course where we dive into more intermediate concepts like vectorization, key data structures, and the key functions. https://www.dataquest.io/course/pandas-fundamentals https://www.dataquest.io/course/pandas-fundamentals We use a similar approach to the OP. Lots of diagrams and visual aids and you always work with a real dataset.
- reacharavindh 8y agoVery pleasant to read. I will pass this to my wife who is an accounting professor trying to break ground into using Python/Pandas/Numpy instead of Stata. I really enjoyed your style of writing and use of visual examples. I wish for such an explainer for SQL. If you made that as a book, you can tun with my money.
- Bishonen88 8y agoA very gentle intro, indeed ;) Pandas is such a vast monster, that even after going through the book from the original author of Pandas (https://www.safaribooksonline.com/library/view/python-for-data/9781491957653/ https://www.safaribooksonline.com/library/view/python-for-da...), I was absolutely unprepared for doing real analysis. Whilst I understood the basics, such as data loading, (simple) cleaning, selections, functions, groupby', indexes etc., I spent most of my time on stackoverflow looking for solution to actual problems I was facing. I reckon that many other users have made the same experience - there is lot's of general info out there when it comes to pandas, but every data is different and the devil lies in the details. Long Story Short: learning pandas is all about trial-and-error and will take months (years even), to be efficient in.
- manojlds 8y agoI still see myself going to spark and Spark SQL for some tasks like stratified sampling which I haven't been able to properly do with pandas. Somehow the spark DF API feels more intuitive and I was able to figure out a lot by myself.
- xiao_haozi 8y agoCan you use groupby for your stratified sampling work?
- mharrison 8y agoThat's interesting as the Spark api was inspired by Pandas
- dorfsmay 8y agoIt might take months to be be able to use its full power and be efficient (delivering at a high velocity), but as someone who was proficient in python but had never used pandas it took me just over a week to write a process to clean data and produce graphs (with seaborn) to compare sets with boxplots. This incldes anonymizing the data properly and play with different Turkey's fence values for the graphs to make the most sense. This is after people spent weeks, and failed, to try to get a similar process with excel. It's good to warn people, but let's not scare them.
- Mefis 8y agoHow does Pandas compare to R's Tidyverse? Tidyverse was super easy to pick up, and I can do almost anything I want with. Why would I want to switch to Panda? Has anyone tired the python tydiverse port? How does it compare to the original?
- alexcnwy 8y agoNeural net universe is in Python and you can use Python to build production pipelines.
- jalammar 8y agoI haven't used tidyverse myself, but I know that pandas is heavily influenced and inspired by R. Most analysis tasks are doable in both platforms. If later stages of your pipeline involve deep learning (or machine learning, generally), then it could pay to be in the python universe given the wide adoption of python ML/DL tools. I generally wouldn't advise switching unless you have a certain pain point, though.
- wenc 8y ago> pandas is heavily influenced and inspired by R. Is it? How so?
- thanatropism 8y agoPandas is inspired by R's dataframes, which I'm told are native.
- minimaxir 8y agoNative doesn't necessarily mean it's the best option. (tidyverse/dplyr leverages Rcpp for data transformation, which makes it a lot faster at common ETL tasks)
- peatmoss 8y agoThe Tidyverse is more coherent and is generally bigger than what’s just in Pandas (R’s Tidyverse; I haven’t used the Python port). If you already have a good grasp of Python, sure why not learn Pandas too? In my case, I’m reasonably ambidextrous in Python and R but find myself not reaching for Python unless there are colleague / deployment considerations that remove R as an option. The reason? R’s Tidyverse is pretty awesome, and reflects one of the better parts of the R language, namely the meta programming that is a holdover from Scheme’s influence on R. Now, if you don’t already know Python and don’t have some other reason (such as specific deployment considerations or a team of Python collaborators) to learn? I don’t think so. Python is a fine language, just as R is a fine language. You’re already getting things done in R. If you want a mental challenge, or to get in on the ground floor of something that might be the future, learn Julia, or F#, or (my favorite) Racket. Or heck, learn Spark, or a new modeling method.
- happy-go-lucky 8y agoNote to author: > We can select one or multiple rows using their numbers (inclusive of both bounding row numbers): > df[1:3] That will slice beginning from the row with integer location 1 up to 3, exclusive of the last element. So, just two rows, not three as shown.
- jalammar 8y agoThanks for the heads up! Indeed it's df.loc[1:3] that would return three rows, not straight-up df[1:3] which indeed returns two rows. Edit: Corrected in the post. Thanks again!
- yufeng66 8y agoBut why design it that way? Seems to be a sure way to confuse new user.
- EForEndeavour 8y agoI'd say that's because df[start:stop] mimics python's builtin list slicing, which includes start but excludes end, so this dictates the behaviour of indexing without .loc. By contrast, df.loc[start:stop] is a label-based indexer, and labels can be anything (integers, strings, datetimes, categories, etc.), so it doesn't always make sense to exclude the right endpoint of the interval.
- happy-go-lucky 8y ago(＾＾)ｂ
- rapfaria 8y agoGenuine question, did you run the original before publishing it on the website?
- skadamat 8y agoLove this intro! All of the popular dataframe oriented tools (tidyverse, pandas, etc) all require familiarity with vectorization and thinking with related mental models. I'm involved with Dataquest Labs and we teach data science interactively in the browser. We're pretty big believers in using diagrams and visual aids to help people learn these concepts as well. We've had a pandas course (https://www.dataquest.io/course/pandas-fundamentals https://www.dataquest.io/course/pandas-fundamentals) for a while and we just launched some R courses that teach a lot of vectorization (https://www.dataquest.io/path/data-analyst-r https://www.dataquest.io/path/data-analyst-r).
- pleasecalllater 8y agoThe last time I wanted to use Pandas, it ate 32GB or RAM and then I just killed it, and made all the analysis in Postgres.
- NightlyDev 8y agoWhat kind of syntax is var['string'] in these examples? Haven't really used python for anything and I'm just wondering, since it looks like an array or map, but clearly seems to have some logic behind it as it seems to reference the specified column at each row. What is this functionality, something that's built in to python or use of some sort of magic functions?
- xapata 8y agoIt's key-lookup syntax, like for a dict (map or hashmap in other languages). A Pandas DataFrame can be thought of as a dict of columns.
- cgriswald 8y agovar['string'] means get the item of object var with key 'string'. Any Python class can define the magic method __getitem__ to define the behavior of/overload the [] operator.
- NightlyDev 8y agoThanks! Didn't know python had magic functions so I was really confused.
- catacombs 8y agoThis is great. I hope the author posts more Pandas visual guides.
- Scarbutt 8y agoCurious, for stuff like this, why not just use sql(sqlite)?
- joelschw 8y agoThere are some things which Pandas is just better at, such as: extracting content via RegEx and pivoting... However, there are also some situations where you should use SQL such as UPSERT or date-range joins.
- BrandoElFollito 8y agoI wanted to try to analyze application logs (usually a timestamp and some text) but all examples in pandas deal with numbers. Is this useful for the analysis of such data (with a machine learning mid term goal (clustering and anomaly detection)?