8 ms·
FireDucks: Pandas but Faster
- viraptor 2y ago> 100% compatibility with existing Pandas code: check. Is it actually? Do people see that level of compatibility in practice?
- jeroenhd 2y agoI don't think it's exactly 100%: https://fireducks-dev.github.io/docs/user-guide/04-compatibility/ https://fireducks-dev.github.io/docs/user-guide/04-compatibi... It should be pretty close, though.
- thecleaner 2y agoSure but single node performance. This makes it not very useful IMO since quite a few data science folks work with Hadoop clusters or Snowflake clusters or DataBricks where data is distributed and querying is handled by Spark executors.
- chaxor 2y agoThe comparison is to pandas, so single node performance is understood in the scope. This is for people running small tasks that may only take a couple days on a single node with a 32 core CPU or something, not tasks that take 3 months using thousands of cores. My understanding for the latter is that pyspark is a decent option, while ballista is the better option for which to look forward. Perhaps using bastion-rs as a backend can be useful for an upcoming system as well. Databricks et al are cloud trash IMO, as is anything that isn't meant to be run on a local single node system and a local HPC cluster with zero code change and a single line of config change. While for most of my jobs I ended up being able to evade the use of HPC by simply being smarter and discovering better algorithms to process information, I recall like pyspark decently, but preferring the simplicity of ballista over pyspark due to the simpler installation of Rust over managing Java and JVM junk. The constant problems caused by anything using JVM backend and the environment config with it was terrible to add to a new system every time I ran a new program. In this regard, ballista is a enormous improvement. Anything that is a one-line install via pip on any new system, runs local-first without any cloud or telemetry, and requires no change in code to run on a laptop vs HPC is the only option worth even beginning to look into and use.
- Kalanos 2y agoHadoop hasn't been relevant for a long time, which is telling. Unless I had thousands of files to work with, I would be loathe to use cluster computing. There's so much overhead, cost, waiting for nodes to spin up, and cloud architecture nonsense. My "single node" computer is a refurbished tower server with 256GB RAM and 50 threads. Most of these distributed computing solutions arose before data processing tools started taking multi-threading seriously.
- markhahn 2y agounderstood: big facilities get shared; sharing requires arbitration and queueing. an interesting angle on 50 threads and 256G: your data is probably pretty cool (cache-friendly). if your threads are merely HT, that's only 25 real cores, and might be only a single socket. implying probably <100 GB/s memory bandwidth. so a best-case touch-all-memory operation would take several seconds. for non-sequential patterns, effective rates would be much lower, and keep cores even less busy. so cache-friendliness is really the determining feature in this context. I wonder how much these packages are oriented towards cache tuning. it affects basic strategy, such as how filtering is implemented in an expression graph...
- E_Bfx 2y agoVery impressive, the Python ecosystem is slowly getting very good.
- BiteCode_dev 2y agoSpent the last 20 years hearing that. At some point I think it's more honest to say "the python ecosystem keeps getting more awesome".
- Kalanos 2y agoContinues to be the best by far
- i_love_limes 2y agoI have never heard of FireDucks! I'm curious if anyone else here has used it. Polars is nice, but it's not totally compatible. It would be interesting how much faster it is for more complex calculations
- bratao 2y agoUnfortunately it is not Opensource yet - https://github.com/fireducks-dev/fireducks/issues/22 https://github.com/fireducks-dev/fireducks/issues/22
- Y_Y 2y agoWouldn't it be nice if GitHub was just for source code and you couldn't just slap up a README that's an add for some proprietary shitware with a vague promise of source some day in the glorious future?
- rad_gruchalski 2y agoYou'd slap that in a comment then?
- thecopy 2y ago>proprietary shitware Is this shitware? It seems to be very high quality code
- yupyupyups 2y agoI think the anger comes from the fact that we expect Github repositories to host the actual source code and not be a dead-end with a single README.md file.
- ori_b 2y agoHow can you tell?
- pplonski86 2y agoHow does it compare to Polars? EDIT: I've found some benchmarks https://fireducks-dev.github.io/docs/benchmarks/ https://fireducks-dev.github.io/docs/benchmarks/ Would be nice to know what are internals of FireDucks
- rich_sasha 2y agoIt's a bit sad for me. I find the biggest issue for me with pandas is the API, not the speed. So many foot guns, poorly thought through functions, 10s of keyword arguments instead of good abstractions, 1d and 2d structures being totally different objects (and no higher-order structures). I'd take 50% of the speed for a better API. I looked at Polars, which looks neat, but seems made for a different purpose (data pipelines rather than building models semi-interactively). To be clear, this library might be great, it's just a shame for me that there seems no effort to make a Pandas-like thing with better API. Maybe time to roll up my sleeves...
- martinsmit 2y agoCheck out redframes[1] which provides a dplyr-like syntax and is fully interoperable with pandas. [1]: https://github.com/maxhumber/redframes https://github.com/maxhumber/redframes
- otsaloma 2y agoBuilding on top of Pandas feels like you're only escaping part of the problems. In addition to the API, the datatypes in Pandas are a mess, with multiple confusing (and none of them good) options for e.g. dates/datetimes. Does redframes do anything there?
- ljosifov 2y ago+1 Seconding this. My limited experience with pandas had a non-trivial number of moments "?? Is it really like this? Nah - I'm mistaken for sure, this can not be, no one would do something insane like that". And yet and yet... Fwiw since I've found that numpy is a must (ofc), but pandas is mostly optional. So I stick to numpy for my writing, and keep pandas read only. (just execute someone else's)
- TeenGirlza17 2y ago[flagged]
- omnicognate 2y ago
- omnicognate 2y ago> Then came along Polars (written in Rust, btw!) which shook the ground of Python ecosystem due to its speed and efficiency Polars rocked my world by having a sane API, not by being fast. I can see the value in this approach if, like the author, you have a large amount of pandas code you don't want to rewrite, but personally I'm extremely glad to be leaving the pandas API behind.
- ralegh 2y agoI personally found the polars API much clunkier, especially for rapid prototyping. I use it only for cemented processes where I could do with speed up/memory reduction. Is there anything specific you prefer moving from the pandas API to polars?
- benrutter 2y agoNot OP but the ability to natively implement complex groupby logic is a huge plus for me at least. Say you want to take an aggergation like "the mean of all values over the 75th percentile" algonside a few other aggregations. In pandas, this means you're gonna be in for a bunch of hoops and messing around with stuff because you can't express it via the api. Polars' api lets you express this directly without having to implement any kind of workaround. Nice article on it here: https://labs.quansight.org/blog/dataframe-group-by https://labs.quansight.org/blog/dataframe-group-by
- adrian17 2y agoAny explanation what makes it faster than pandas and polars would be nice (at least something more concrete than "leverage the C engine"). My easy guess is that compared to pandas, it's multi-threaded by default, which makes for an easy perf win. But even then, 130-200x feels extreme for a simple sum/mean benchmark. I see they are also doing lazy evaluation and some MLIR/LLVM based JIT work, which is probably enough to get an edge over polars; though its wins over DuckDB _and_ Clickhouse are also surprising out of nowhere. Also, I thought one of the reasons for Polars's API was that Pandas API is way harder to retrofit lazy evaluation to, so I'm curious how they did that.
- ayhanfuat 2y agoIn its essence it is a commercial product which has a free trial. > Future Plans By providing the beta version of FireDucks free of charge and enabling data scientists to actually use it, NEC will work to improve its functionality while verifying its effectiveness, with the aim of commercializing it within FY2024.
- graemep 2y agoIts BSD licensed. They do not way what the plans are but most likely a proprietary version with added support or features.
- ayhanfuat 2y agoThey say the source code for the part “where the magic happens” is not available so I am not sure what BSD implies there.
- HelloNurse 2y agoIt serves as a ninja's smoke bomb until the "BSD" binary blob is suddenly obsoleted by a proprietary binary blob.
- ori_b 2y agoIt's a BSD licensed binary blob. There's no code provided.
- deleted 2y ago[deleted]
- graemep 2y agoWow! That is so weird. Its freeware under an open source license. Really misleading. It looks like something you should stay away from unless you need it REALLY badly. Its a proprietary product with unknown pricing and no indication of what their plans are. Does the fact that the binary is BSD licensed allow reverse-engineering?
- imranq 2y agoThis presentation does a good job distilling why FireDucks is so fast: https://fireducks-dev.github.io/files/20241003_PyConZA.pdf https://fireducks-dev.github.io/files/20241003_PyConZA.pdf The main reasons are * multithreading * rewriting base pandas functions like dropna in c++ * in-built compiler to remove unused code Pretty impressive especially given you import fireducks.pandas as pd instead of import pandas as pd, and you are good to go However I think if you are using a pandas function that wasn't rewritten, you might not see the speedups
- faizshah 2y agoIt’s not clear to me why this would be faster than polars, duckdb, vaex or clickhouse. They seem to be taking the same approach of multithreading, optimizing the plan, using arrow, optimizing the core functions like group by.
- mettamage 2y agoMaybe it isn’t? Maybe they just want a fast pandas api?
- geysersam 2y agoAccording to their benchmarks they are faster. Not by a lot, but still significantly.
- maleldil 2y agoNone of those drop-in replacements for Pandas. The main draw is "faster without changing your code".
- faizshah 2y agoI’m asking more about what techniques did they use to get the performance improvements in the slides. They are showing a 20-30% improvement over Polars, Clickhouse and Duckdb. But those 3 tools are SOTA in this area and generally rank near eachother in every benchmark. So 20-30% improvement over that cluster makes me interested to know what techniques they are using to achieve that over their peers.
- Kalanos 2y agoLinux only right now https://github.com/fireducks-dev/fireducks/issues/27 https://github.com/fireducks-dev/fireducks/issues/27
- Kalanos 2y agoRegarding compatibility, fireducks appears to be using the same column dtypes: ``` >>> df['year'].dtype == np.dtype('int32') True ```
- DonHopkins 2y agoFireDucks FAQ: Q: Why do ducks have big flat feet? A: So they can stomp out forest fires. Q: Why do elephants have big flat feet? A: So they can stomp out flaming ducks.
- deleted 2y ago[deleted]
- short_sells_poo 2y agoLooks very cool, BUT: it's closed source? That's an immediate deal breaker for me as a quant. I'm happy to pay for my tools, but not being able to look and modify the source code of a crucial library like this makes it a non-starter.
- KameltoeLLM 2y agoShouldn't that be FirePandas then?
- benrutter 2y agoAnyone here tried using FireDucks? The promise of a 100x speedup with 0 changes to your codebase is pretty huge, but even a few correctness / incompatibility issues would probably make it a no-go for a bunch of potential users.
- safgasCVS 2y agoI’m sad that R’s tidy syntax is not copied more widely in the python world. Dplyr is incredibly intuitive most don’t ever bother reading the instructions you can look at a handful of examples and you’ve got the gist of it. Polars despite its speed is still verbose and inconsistent while pandas is seemingly a collection of random spells.
- dr_kiszonka 2y agoI don't know why, but it seems pipes are hard to do in Python without extra parentheses, etc.
- softwaredoug 2y agoThe biggest advantage of pandas is its extensibility. If you care about that, it’s (relatively) easy to add your own extension array type. I haven’t seen that in other system like Polars, but maybe I’m wrong.
- ssivark 2y agoSetting aside complaints about the Pandas API, it's frustrating that we might see the community of a popular "standard" tool fragment into two or even three ecosystems (for libraries with slightly incompatible APIs) -- seemingly all with the value proposition of "making it faster". Based on the machine learning experience over the last decade, this kind of churn in tooling is somewhat exhausting. I wonder how much of this is fundamental to the common approach of writing libraries in Python with the processing-heavy parts delegated to C/C++ -- that the expressive parts cannot be fast and the fast parts cannot be expressive. Also, whether Rust (for polars, and other newer generation of libraries) changes this tradeoff substantially enough.
- tgtweak 2y agoI think it's a natural path of software life that compatibility often stands in the way of improving the API. This really does seem like a rare thing that everything speeds up without breaking compatability. If you want a fast revised API for your new project (or to rework your existing one) then you have a solution for that with Polars. If you just want your existing code/workloads to work faster, you have a solution for that now. It's OK to have a slow, compatible, static codebase to build things on then optimize as-needed. Trying to "fix" the api would break a ton of existing code, including existing plugins. Orphaning those projects and codebases would be the wrong move, those things take a decade to flesh out. This really doesn't seem like the worst outcome, and doesn't seem to be creating a huge fragmented mess.
- SiempreViernes 2y ago> Based on the machine learning experience over the last decade, this kind of churn in tooling is somewhat exhausting. Don't come to old web-devs with those complains, every single one of them had to write at least one open source javascript library just to create their linkedin account!
- cmcconomy 2y agoEvery time I see a new better pandas, I check to see if it has geopandas compatibility
- PhasmaFelis 2y ago"FireDucks: Pandas but Faster" sounds like it's about something much more interesting than a Python library. I'd like to read that article.
- dkga 2y agoReading all pandas vs polars reminded me of the tidyverse vs data.table discussion some 10 years ago.
- flakiness 2y ago> FireDucks is released on pypi.org under the 3-Clause BSD License (the Modified BSD License). Where can I find the code? I don't see it on GitHub. > contact@fireducks.jp.nec.com So it's from NEC (a major Japanese computer company), presumably a research artifact? > https://fireducks-dev.github.io/docs/about-us/ https://fireducks-dev.github.io/docs/about-us/ Looks like so.
- xbar 2y agoGreat work, but I will hold my adoption until c++ source is available.
- uptownfunk 2y agoIf they could just make a dplyr for py it would be so awesome. But sadly I don’t think the python language semantics will support such a tool. It all comes down to managing the namespace I guess
- gigatexal 2y agoOn average only 1.5x faster than polars. That’s kinda crazy.
- geysersam 2y agoWhy is that crazy? (I think the crazy thing is that they are faster at all. Taking an existing api and making it fast is harder than creating the api from scratch with performance in mind)
- gigatexal 2y ago100x faster than pandas yet 1.5x faster than polars. Polars is stupid fast. It’s got a far better api too — much more ergonomic.
- OutOfHere 2y agoDon't use it: > By providing the beta version of FireDucks free of charge and enabling data scientists to actually use it, NEC will work to improve its functionality while verifying its effectiveness, with the aim of commercializing it within FY2024. In other words, it's free only to trap you.
- ladyanita22 2y agoImportant to upvote this. If there's room for improvement for Polars (which I'm sure there is), go and support the project. But don't fall for a commercial trap when there are competent open source tools available.
- maleldil 2y agoWhile I agree, it's worth noting that this project is a drop-in replacement (they claim that, at least), but Polars has a very different API. I much prefer Polars's API, but it's still a non-trivial cost to switch to it, which is why many people would instead explore Pandas alternatives instead.
- binoct 2y agoNo shade to the juggernaut of the open source software movement and everything it has/will enabled, but why the hate for a project that required people’s time and knowledge to create something useful to a segment of users and then expect to charge for using it in the future? Commercial trap seems to imply this is some sort of evil machination but it seems like they are being quite upfront with that language.
- floatrock 2y agoIt's not hate for the project, it's hate for the deceptive rollout. Basically it's a debate about how many dark patterns can you squeeze next to that "upfront language" before "marketing" slides into "bait-n-switch."
- papichulo2023 2y agoNot sure if evil or not, but it is unprofessional to use a tool that you dont know how much it will cost for your company in the future.
- caycep 2y agoJust because I haven't jumped into the data ecosystem for a while - is Polars basically the same as Pandas but accelerated? Is Wes still involved in either?
- breakds 2y agoI understand `pandas` is widely used in finance and quantitative trading, but it does not seem to be the best fit especially when you want your research code to be quickly ported to production. We found `numpy` and `jax` to be a good trade-off between "too high level to optimize" and "too low level to understand". Therefore in our hedge fund we just build data structures and helper functions on top of them. The downside of the above combination is on sparse data, for which we call wrapped c++/rust code in python.
- liminal 2y agoLots of people have mentioned Polars' sane API as the main reason to favor it, but the other crucial reason for us is that it's based on Apache Arrow. That allows us to use it where it's the best tool and then switch to whatever else we need when it isn't.
- rcarmo 2y agoThe killer app for Polars in my day-to-day work is its direct Parquet export. It's become indispensable for cleaning up stuff that goes into Spark or similar engines.
- hinkley 2y agoTIL that NEC still exists. Now there’s a name I have not heard in a long, long time.
- insane_dreamer 2y agosurprised not to see any mention of numpy (our go-to) here edit: I know pandas uses numpy under the hood, but "raw" numpy is typically faster (and more flexible), so curious as to why it's not mentioned
- __mharrison__ 2y agoMany of the complaints about Pandas here (and around the internet) are about the weird API. However, if you follow a few best practices, you never run into the issue folks are complaining about. I wrote a nice article about chaining for Ponder. (Sadly, it looks like the Snowflake acquisition has removed that. My book, Effective Pandas 2, goes deep into my best practices.)
- otsaloma 2y agoI don't quite agree, but if this was true, what would you tell a junior colleague in a code review? You can't use this function/argument/convention/etc you found in the official API documentation because...I don't like it? I think any team-maintained Pandas codebase will unavoidably drift into the inconsistent and bad. If you're always working alone, then it can of course be a bit better.
- __mharrison__ 2y agoI have strong opinions about Pandas. I've used it since it came out and have coalesced on patterns that make it easy to use. (Disclaimer: I'm a corporate trainer and feed my family teaching folks how to work with their data using Pandas.) When I teach about "readable" code, I caveat that it should be "readable for a specific audience". I hold that if you are a professional, that audience is other professionals. You should write code for professionals and not for newbies. Newbies should be trained up to write professional code. YMMV, but that is my bias based on experience seeing this work at some of the biggest companies in the world.
- __mharrison__ 2y agoLots of Pandas hate in this thread. However, for folks with lots of lines of Pandas in production, Fireducks can be a lifesaver. I've had the chance to play with it on some of my code it queries than ran in 8+ minutes come down to 20 seconds. Re-writing in Polars involves more code changes. However, with Pandas 2.2+ and arrow, you can use .pipe to move data to Polars, run the slow computation there, and then zero copy back to Pandas. Like so... (df # slow part .groupby(...) .agg(...) ) to: def polars_agg(df): return (pl.from_pandas(df) .group_by(...) .agg(...) .to_pandas() ) (df .pipe(polars_agg) )
- dr_kiszonka 2y agoVery effective!
- __mharrison__ 2y agoI'm hoping there's a double meaning in that. ;)
- nooope6 2y agoPretty cool, but where's the source at?
- Gepsens 2y agoIt'll be polars and datafusion for me thanks