10 ms·
Polars Cloud and Distributed Polars now available
- jkkola 1y agoI absolutely love Polars. I work on some unholy dirty data and the ease of use, chaining, speed are a godsend. One dataset that previously took 40 minutes in Pandas now takes two minutes in Polars. Granted, the Pandas query could be optimized, but out of the box, Polars eats pandas when it comes to speed and efficiency. I basically ditched SQL for most of my analytical work because it's way easier to understand for my juniors (we're not technically a tech team) so it's a total win in my eyes.
- willvarfar 1y agoHmm so how does the polars SQLContext stack up against duckdb? And can both cope with a distributed polars? It feels like we are on the path to reinventing BigQuery.
- ritchie46 1y agoHi, I am the original author and CEO of Polars. We are not focused on SQL at this time and provide a DataFrame native API. Polars cloud will for the moment only support our DataFrame API. SQL might come later on the roadmap, but since this market is very saturated, we don't feel there is much need there.
- anonu 1y agoSo competing with SnowFlake?
- rorads 1y agoEDIT: I think the below is correct, but I’ve just seen in the main product landing page that for a certain benchmark it’s an order of magnitude cheaper AND faster than AWS glue, so that’s the target market by the looks of things. —— I don’t think so - probably more in the realms of spark and, based on the roadmap, airflow. For me it would be about doing big data analytics / dashboarding / ML or DS data prep. My understanding is that Snowflake plays a lot in the data warehouse/lakehouse space, so is more central to data ops / cataloguing / SSOT type work. But hey that’s all first impressions from the press release.
- dkdcio 1y agomoreso competing with Coiled Computing (Dask version, very similar, you can run Polars there too). and then Databricks more than Snowflake, but all of these data platforms converge on similar features. also competing with Fivetran eventually after their acquisition yesterday
- cantdutchthis 1y agoBeen a polars fan for a loooong time. Happy to see the team ship their product and I hope it does well!
- weinzierl 1y agoOut of curiosity and because I don't want to create a test account right now: How does billing with "Deploy on AWS" work? Do I need to bring my own AWS account and Polars is payed for the image through AWS or am I billed by Polars and they pass a share to AWS. In other words do I have a contract primarily with AWS or Polars?
- ritchie46 1y agoYour billing partner is AWS. Polars' markup is on your AWS bill.
- gigatexal 1y agoCool. But abstract away the infra knowledge to the actual instance types. Instead I’d expect the polars cloud abstraction to find me the most cost effective (spot instance) that meets my cpu and memory reqs and disk reqs. Why do I have to give it — looking at the example — the AWS instance type?
- ritchie46 1y agoYou don't have to. Passing cpu and memory works as well. pc.ComputeContext{ cpus=4, memory=16 } We are working on a minimal cluster and auto-scaling based on the query.
- gigatexal 1y agoNice! Ritchie, curious you mentioned in other responses that the SQL context stuff is out of scope for now. But I thought the SQL things were basically syntactic sugar to the dataframes in other words they both “compile” down to the same thing. If true then being able to run arbitrary SQL queries should be doable out of the box?
- ritchie46 1y agoNot right now. Our current SQLContext locally inspects schema's to convert the SQL to Polars LazyFrames (DSL). However, this should happend during IR-resolving. E.g. the SQL should translate directly to Polars IR, and not LazyFrames. That way we can inspect/resolve all schema's server-side. It requires a rewrite of our SQL translation in OSS. This should not be too hard, but it is quite some work. Work we eventually get to.
- gigatexal 1y agoThanks for the context.
- cbb330 1y agocan you dive a bit deeper into the comparison with spark rdd
- ritchie46 1y agoI am not an expert on Spark RDDs, but AFAIK they are a more low-level data structure that offer resilience and a lower level map-reduce API. Polars Cloud maps the Polars API/DSL to distributed compute. This is more akin to Spark's high level DataFrame API. With regard to implementation, we create stages that run parts of Polars IR (internal representation) on our OSS streaming engine. Those stages run on 1 or many workers create data that will be shuffled in between stages. The scheduler is responsible for creating the distributed query plan and work distribution.
- ayhanfuat 1y agoCan you tell a little about the status of Iceberg write support? Partitioning, maintenance etc.
- ritchie46 1y agoWe have full iceberg read support. We have done some preliminary work for iceberg write support. I think we will ship that once we have decided which Catalog we will add. The iceberg write API is intertwined with that.
- robertkoss 1y agoOG polars announcement: https://news.ycombinator.com/item?id=23768227 https://news.ycombinator.com/item?id=23768227
- dvko 1y agoNever forget! Crazy to see how far it's come. And how lackluster the initial reception on HN was back then.
- robertkoss 1y agoLove it! Still don't get why one of the biggest player in the space, Databricks is overinvesting in Spark. For startups, Polars or DuckDB are completely sufficient. Other companies like Palantir already support bring your own compute.
- whyever 1y agoThat's a good question! Especially after Frank McSherry's COST paper [1], it's hard to imagine where the sweet spot for Spark is. I guess for Databricks it makes sense to push Spark, since they are the ones who created it. In a way, it's their competitive advantage. [1]: https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos...
- mr_toad 1y agoDatabricks is targeting large enterprises, who have a variety of users. Having both Python and SQL as first class languages is a selling point.
- jpcompartir 1y agoPolars is great, absolute best of luck with the launch
- lvl155 1y agoPolars is certainly better than pandas doing things locally. But that is a low bar. I’ve not had great experience using Polars on large enough datasets. I almost always end up using duckdb. If I am using SQL at the end of the day, why bother starting with Polars? With AI these days, it’s ridiculously fast to put together performant SQLs. Heck you can even make your own grammar and be done with it.
- dkdcio 1y agoyou can use Ibis if you want a dataframe UI on top of DuckDB (or a number of other query engines, including Polars)
- sirfz 1y agoSQL is definitely easier and faster to compose than any dataframe syntax but I think pandas syntax (via slicing API) is faster to type and in most cases more intuitive but I still use polars for all df-related tasks in my workflow since it's more structured and composable (although needs more time to construct but that's a cost I'm willing to take when not simply prototyping). When in an ipython session, sql via duckdb is king. Also: python -m chdb "describe 'file.parquet'" (or any query) is wonderful
- mr_toad 1y ago> SQL is definitely easier and faster to compose Sometimes. But sometimes Python is just much easier. For example transposing rows and columns.
- infecto 1y agoI guess if it’s too large to be performant than SQL can be the way to go. I avoid sql for one off tasks though as I can more easily grok transformations in polars code than sql queries.
- ritzaco 1y agoMaybe just me, but for anyone else who was confused - Polars (Pola.rs) - the DataFrames library that now has a cloud version - Polar (Polar.sh) - Payments and MoR service built on top of Stripe
- forks 1y ago- Polar, the authorization DSL created by Oso It's a common name
- drej 1y agoHaving done a bit of data engineering in my day, I'm growing more and more allergic to the DataFrame API (which I used 24/7 for years). From what I've seen over the past ~10 years, 90+% of use cases would be better served by SQL, both from the development perspective as well as debugging, onboarding, sharing, migrating etc. Give an analyst AWS Athena, DuckDB, Snowflake, whatever, and they won't have to worry about looking up what m6.xlarge is and how it's different from c6g.large.
- robertkoss 1y agoThat is a false dichotomy. You can use SQL tools but still have to choose the instance type. Especially when considering testability and composability, using a DataFrame API inside regular languages like Python is far superior IMO.
- drej 1y agoAs a user? No, I don't have to choose. What I'm saying is that analysts (who this Polars Cloud targets, just like Coiled or Databricks) shouldn't worry about instance types, shuffling performance, join strategies, JVM versions, cross-AZ pricing etc. In most cases, they should just get a connection string and/or a web UI to run their queries, everything abstracted from them. Sure, Python code is more testable and composable (and I do love that). Have I seen _any_ analysts write tests or compose their queries? I'm not saying these people don't exist, but I have yet to bump into any.
- ayhanfuat 1y agoI really doubt that Polars Cloud targets analysts doing ad-hoc analyses. It is much more likely towards people who build data pipelines for downstream tasks (ML etc).
- ritchie46 1y agoWe also target ad-hoc analysis. If your data doesn't fit on your laptop, you can spin up a larger box or a cluster and run interactive queries.
- boomer_joe 1y agoI don't understand. Can I use distributed Polars with my own machines or do I have to buy cloud compute to run distributed queries (I don't want that). If not, is this planned?
- ritchie46 1y agoOn-premises is in the works. We expect this in a couple of months. Currently it is managed on AWS only.
- boomer_joe 1y agoThanks! Will it be paid or open source?
- rubenvanwyk 1y agoPaid
- Arnechos 1y agoHow does it relate to Apache DataFusion/Ballista?
- yencabulator 1y agoIt's an open-core competitor that started more from the "DataFrames for Python" end of the spectrum, where DataFusion went pretty strong into "we can handle SQL" (while still having a dataframes API).
- cmollis 1y agocan i run a distributed computation in pola.rs cloud on my own AWS infra? or do I need to run it on-prem?
- blackhaz 1y agoHow does Polars compare to FireDucks?
- nivekney 1y agohttps://fireducks-dev.github.io/docs/benchmarks/ https://fireducks-dev.github.io/docs/benchmarks/ insert obama awards obama meme
- sureglymop 1y agoI recently had to create a reproducible version of incredibly complicated and messy R concoctions our data scientists came up with. I did it with pandas without much experience with it and a lot of AI help (essentially to fill in the blanks the data scientists had left, because they only had to do the calculation once). I then created a polars version which uses lazyframes. It ended up being about 20x faster than the first version. I did try to do some optimizations by hand to make the execution planner work even better which I believe paid off. If you have to do a large non interactive analytical calculation (i.e. not in a notebook) polars seems to be way ahead imo! I do wish that it was just as easy to use as a rust library though.. the focus however seems to be on being competitive in python land mainly.
- tomtom1337 1y agoOut of curiosity, what makes a rust library easier to use? Could you expand on that?
- ritchie46 1y agoHe means that he wants our Rust library as easy as our Python lib. Which I understand as our focus has been mostly on Python. It is where most of our userbase is and it is very hard for us to have a stable Rust API as we have a lot of internal moving parts which Rust users typically want access to (as they like to be closer to the metal), but has no stability guarantees from us. In python, we are able to abstract and provide a stable API.
- lucasyvas 1y agoI understand the user pool comment but don’t understand why you wouldn’t be able to have a rust layer that’s the same as the Python one API-wise. I say this as a user of neither - just that I don’t see any inherent validity to that statement. If you are saying Rust consumers want something lower level than you’re willing to make stable, just give them a higher level one and tell them to be happy with it because it matches your design philosophy.
- raoulj 1y agoIs there any distributed polars for non Polars Cloud? EDIT: nevermind see same question in this thread. The answer is no!
- curtisszmania 1y ago[dead]
- dbacar 1y agoSnowFlake, Polars, DucksDB, FireBase, FireDuck... I guess the next product will be IceDuck. What is wrong with you DB people :))).
- qrush 1y agoI thought this was about my favorite sparkling water brand at first glance.
- ohnoesjmr 1y agoI guess could be a good contender for replacing spark, however, I suspect the fact spark is free and open source, which forms a community around it, means that dpolars might struggle to gain traction, when it's gated by a credit card.
- yencabulator 1y agoI would expect a better contender for Spark to be something that's actually open source, such as https://github.com/apache/datafusion-ballista https://github.com/apache/datafusion-ballista