11 ms·
Pandas vs. Julia – cheat sheet and comparison
- _aaed 3y agoNah, I'll do it with SQL
- rectang 3y agoOne nice thing about solutions like pandas or Julia is that they’re much easier to write tests for or otherwise validate. I can’t tell you how many times I’ve been handed a big ball of SQL which doesn’t behave like its author thinks it does, diverging in subtle or not-so-subtle ways.
- ivirshup 3y agoibis in python is a really nice middle ground. Nice API + in a programming language, but executes on database backends (which could be polars or duckdb on in memory arrow tables).
- akdor1154 3y agoI agree, and interestingly so does the author of the article so it's a bit weird that you're receiving downvotes. Julia's DataFrames library is more consistent than Pandas by a mile, but it's still a bit weird. JuliaDB's IndexedTable and NDTable was a really awesome API design, it's quite a pity that JuliaDB is now unmaintained. :(
- pjmlp 3y agoSame here, I don't get the point of this other that "don't want to learn SQL".
- smabie 3y agoYou're saying all Pandas usage (an incredibly popular library) is because people don't want to use SQL?
- cookieperson 3y agoIt's a broken argument with some truth too it. IE you can run a SQL query put the result in a dataframe to dump it to an interchange format. But in the same breathe... If you learn to use SQL a great deal of workloads often used via dataframes APIs kind of disappear, and in doing so learning a new language to use a new dataframes API isn't really worth it.
- pjmlp 3y agoAs far as I am aware, plenty of use cases can also be done via OLAP.
- laratied 3y ago[dead]
- cookieperson 3y agoA lot of data scientists don't know SQL and don't understand why people use it. That said, there are cases where in memory workloads and certain manipulations are less encumbered by dataframes APIs. But in a lot of cases... They get abused by people who really do need to push themselves a little bit to learn something new.
- cjalmeida 3y agoYou use both. Once your data fits comfortably in memory it's naive to try to build histograms, pivots and charts using pure SQL.
- cookieperson 3y agoSQLite is often much faster then dataframes jl and pandas.
- ayhanfuat 3y agoI highly doubt that SQLite is faster than pandas, let alone dataframes.jl, for analytical workloads.
- cookieperson 3y agoMight surprise you how many people are using pandas or dataframes for OLTP on a daily basis because they don't know better.
- cjalmeida 3y agoSQLite is good for a bunch of stuff, but it's terrible for analytic workloads. Not even in the same ballpark for pandas, let alone Julia
- agacera 3y agoSame here. But have you tried duckdb? You can do sql in the pandas dfs and it is fast af. https://duckdb.org/2021/05/14/sql-on-pandas.html https://duckdb.org/2021/05/14/sql-on-pandas.html
- cookieperson 3y agoDuckdb is sick. You can also do queries on parquet, etc.
- avnigo 3y agomydf = pd.DataFrame({'a' : [1, 2, 3]}) print(duckdb.query("SELECT SUM(a) FROM mydf").to_df()) I can see the appeal, but if you're working in Python, something doesn't sit right with me when having to write out variable names as strings. E.g., if I want to refactor the code, my LSP or parser won't pick up those references. > The SQL table name mydf is interpreted as the local Python variable mydf [...] Not only is this process painless, it is highly efficient. It might be painless and convenient at first, but I feel like this could get you in trouble down the line. Is there a way to avoid this?
- freilanzer 3y agohttps://duckdb.org/docs/guides/python/ibis.html https://duckdb.org/docs/guides/python/ibis.html
- retrochameleon 3y agoUnreadable on mobile
- marginalia_nu 3y agoYou do a lot of software development on mobile?
- Alifatisk 3y agoHow’s that related to the article not being mobile friendly?
- marginalia_nu 3y agoIt's a reference sheet for a programming language. Like it's clearly designed to be read as you're writing code. I don't understand in which circumstance you'd optimize such a document for mobile use.
- arijun 3y agoYou have never read documentation on something you’re not currently writing code with _right_ _now_? Or looked at documentation for something you worked on in the day as you’re on your way home from work? I opened it on mobile because I was interested in seeing how they differ, even though I’m not using either right now.
- marginalia_nu 3y agoWell, I typically don't read a lot on mobile. Screen is too small and you can't block ads, which makes the screen even smaller.
- ivvve 3y agoBet your code always works on your machine too ;)
- jstx1 3y agoThis seems very poor - the comparison is between pandas and DataFrames.jl, not Julia; syntax comparison is very surface-level; cheatsheats are low resolution; the learning curve section says nothing about the learning curve; and the conclusion is "do whatever you like".
- xgdgsc 3y agoBut there's CSV.jl so I didn't change title. I don't see any low resolution, maybe a font choice issue? I'd say the conclusion is the right thing to say for such a short comparison.
- misja111 3y agoWell Pandas is a framework, not a language, so it only makes sense to compare it to DataFrames.jl and not to Julia as a lanugage. But I agree this should have been reflected in the title of the article.
- ChrisRackauckas 3y agoI both agree and disagree. It does look weird as library vs language. However at the same time, in Pandas everything tends to be in the Pandas library, whereas when using DataFrames.jl you tend to mix it with a lot of features that are external. Most of the calls just use overloaded functions from Julia's Base library (mean, first, last, findall). The Pandas model is to look at the Pandas docs and find the Pandas dataframe function that does your job. The DataFrames.jl model is to do whatever you would have done normally in Julia, like use the sort function, but now just use it on a DataFrame. The idea of DataFrames.jl is that you know the language and so it extends/adds as few functions as possible (joins, groupby, split-apply-combine I think are it?). This plus many other calls use functions from the more general Julia data science ecosystem (CSV.jl, JSON.jl, ...). So the title ends up being a bit apples and oranges, but the usage is also quite apples and oranges and the cheat sheet does accurately reflect that.
- SilverBirch 3y agoYeah this is basically why I keep trying and bouncing off Julia. I understand the real performance reasons why you'd choose to use Julia but the syntax is the perfect distance from python to make it extremely difficult to me. It's just close enough to get constantly confused. So if I really wanted to do much work in it I'd have swear off python - and I can't do that because for trivial stuff python is more convenient.
- tombert 3y agoI have not done anything even remotely significant in Julia, but the little I played with didn't seem to indicate to me that it would be bad for trivial stuff...what trivial stuff is hard in Julia but easy in Python?
- deleted 3y ago[deleted]
- Nanana909 3y agoI responded in detail him about, but honestly I can't think of many that answer your qiestion. It's amost always the opposite. A I like to present this as stereotypical example of the types of differences you find the two, IMO. Generating a 3x3 matrix of random booleans. >Julia A = rand(Bool,3,3) Python: No standard support for Matrices. I could really do it a disservsice and comapre the "core language", but that''s obivously stupid so we'll bring in some external libraries to make it easier. Of many ways to skin the cat here's one.. >Python import numpy.numpy as np gen = np.random.default.default_rng() B = gen.choice([True,False],(3,3)) I find myself just taking `rand(["Msg1",.....,"MsnN"])` to get a random string often. And small things like this are why you see people Julia is so nice to write and become so defendant of it.
- sundarurfriend 3y agoI think they just mean that with trivial projects, it's not worth trying them in a new language since the performance benefits are probably going to be minimal, and wouldn't really show off Julia's strengths.
- seaparter 3y agoApples vs Oranges
- Quindecillion 3y agoWhich are both fruit!
- adammarples 3y agoIt's strange, I would think that Julia's multiple dispatch would make something more like this desirable ``` df = DataFrame(CSV(File("name.csv"))) data = JSON(File("name.json")) ``` instead of the usual hodge podge of methods ``` df = CSV.read("file.csv", DataFrame) data = JSON.parsefile("file.json") ```
- affinepplan 3y agoActually, that works too :) I think `DataFrame(CSV.File("name.csv"))` is what you're looking for
- rgavuliak 3y agoThe cheatsheet goes wrong already for the first example of declaring a df: - you could do a range in python (range(11, 14)) - columns are called col_1 & col_2 vs a & b (both sets are horrible names) - pandas defines index of 0, 1, 3, while Julia would most likely have 0, 1, 2?
- jstx1 3y agoAlso df.loc[1:3, :] doesn't get the first N rows. First because of 0-indexing, second because when your index isn't ordered integers, you'll get completely unexpected results with .loc.
- joeman1000 3y agoWhat do you mean? Julia is 1-indexed.
- rgavuliak 3y agoOk, in that case Python would be 0, 1, 3 and Julia 1, 2, 3. My point is that the example explicitly skips an index in the definition of a data frame for Python, but it doesn't for Julia.
- markkitti 3y agoSurely Python would not skip the 2.
- rgavuliak 3y agoThe code snippet explicitly defines the index to be 0, 1, 3. It may just be a typo, but if even simple examples are written sloppily, how can I trust the sheet: df = pd.DataFrame( {'col_1': [11, 12, 13], 'col_2': [21, 22, 23]}, index=[0, 1, 3])
- henlab 3y agoDataFrames.jl was for me on of the primary reasons to use Julia. I personally the syntax way easier than pandas, especially for more complex operations. I find this cheat sheet doesn't do in justice.
- lukego 3y agoTip: If you want these capabilities and your favorite language doesn't have them natively then you can consider embedding DuckDB for something comparable.
- joeman1000 3y agoThe thing that keeps me coming back to Julia is the ability to pipe (or whatever you want to call it). It makes DataFrame operations a lot cleaner since I don't need to modify in place or create new DFs at intermediate steps in a process. Here's a video showing this sort of workflow in R: https://youtu.be/W3e8qMBypSE https://youtu.be/W3e8qMBypSE
- stoniejohnson 3y agoIs that not also available in Pandas? https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.pipe.html https://pandas.pydata.org/docs/reference/api/pandas.DataFram...
- joeman1000 3y agoYes, but try using this and then try Julia's way. I tried this pandas implementation once and never touched it again.
- ChrisRackauckas 3y agoJulia has a pipe syntax (|>). But I think the bigger part here is more generally APIs built around it, which people are doing some things to port tidy syntax (https://github.com/TidierOrg/Tidier.jl https://github.com/TidierOrg/Tidier.jl).
- joeman1000 3y agoThis is very similar to DataFramesMeta: https://github.com/JuliaData/DataFramesMeta.jl https://github.com/JuliaData/DataFramesMeta.jl
- chaxor 3y agoIn pandas you can chain commands by wrapping the whole command in (). Personally IMO looks far 'cleaner' than all of the ugly %>% everywhere.
- civilized 3y ago
- xigoi 3y agoThe site's cookie banner doesn't give an option to reject tracking cookies. Isn't that illegal?
- nologic01 3y agoAnxiously waiting for pandas to get its mojo. If there is any Python library that needs it, this is it.
- make3 3y agoI wish Python catches up to Julia in performance. No sense rewriting a trillion lines of code for what is a really pleasant syntax & ecosystem already. But this is a language flamewar thing, probably not a constructive comment, sorry.
- wdroz 3y agoYou can just use Polars[0] instead of Pandas and easily beat both Pandas and DataFrames.jl Pure Julia is faster than pure python, but there are non-pure python tools available in the python ecosystem for a ton of things. [0] -- https://www.pola.rs/ https://www.pola.rs/
- affinepplan 3y agoPolars definitely doesn't "easily" beat DF.jl on all tasks. Yes, I agree, on average polars is a bit faster for many of the simple workflows, but I certainly don't think that's unconditionally true. It's especially less true when you might want to do something out of the ordinary with your series --- in Julia it's trivial to just extract that as a vector and loop over it (fast!). In polars, one would have to make sure their function can be appropriately vectorized.
- clircle 3y agoThanks, but I'm sticking with R.
- kloch 3y agoI've just started getting into Julia for one of it's best use cases: It's super easy to do arbitrary precision math. But you have to be very careful when using string literals with BigInt or BigFloat: julia> setprecision(1024) julia> a=BigFloat(1.0E-300) 1.000000000000000025059091835208759685696146807703705249925342319900466043184051484676302812181950100894962306270278254148910311464998804130812246091606190182719426627934584275510414782787015070222639260603793613924359775094030143866141479125513590882591017341692222921220404918621822029155619541859418525883262e-300 Notice without quotes on the literal you only get ~15 decimal digits of precision because the parser treats the literal as a double and then passes that to the BigFloat variable. julia> a=BigFloat("1.0E-300") 9.999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999988e-301 With quotes we get the full ~308 decimal digits of precision for the configured 1024-bit binary precision. Now we can add it to 1.0 to validate the precision of a calculation and use the @printf macro for C-style formatting to round the output to 308 decimal digits: julia> b=BigFloat("1.0") julia> using Printf julia> @printf("%.308f\n", (a+b)) 1.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000 I'm not sure why this is the default behavior, it seems like a really easy way for people to screw up their calculations, especially scientists that don't do a lot of programming.
- DNF2 3y agoI actually find the string macro syntax even more convenient: `big"1e-300"`.
- wnoise 3y ago> I'm not sure why this is the default behavior, Um. You said the answer earlier: > because the parser treats the literal as a double BigFloats aren't built in to the syntax of the language (and probably should't be, so you need to escape the parser somehow -- either pass a string to the constructor, or use the @big_str macro to get a non-standard string parsed into a BigFloat.