7 ms·
Polars Cloud: The Distributed Cloud Architecture to Run Polars Anywhere
- deleted 2y ago[deleted]
- LaurensBER 2y agoThis is very impressive and definitely fills a huge hole in the whole data frame ecosystem. I've been quite impressed with the Polars team and after using Pandas for years, Polars feels like a much needed fresh wind. Very excited to give this a go sometime soon!
- ydjje 2y ago[flagged]
- DrBenCarson 2y agoFalse, very random SO link that has been posted repeatedly
- 0cf8612b2e1e 2y agoI’ll bite- what’s the pitch vs Dask/Spark/Ray/etc? I am admittedly a tough sell when the workstation under my desk has 192GB of RAM.
- benrutter 2y agoDoesn't look like benchmarks are there yet, but knowing polars, I'd guess performance will be front and centre. I think the best selling point speaks to your workstation size- just start with polars vanilla. It'll work great for ages, and if you do need to scale, you can use polars cloud. That solves what I see as one if the big issues with a lot of these types of projects, which is the really poor performance at smaller sizes, meaning practically you end up using completely different frameworks based on size, which is a bif hassle if you want to rewrite in one direction.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- __mharrison__ 2y agoYeah, you can process 99% of tabular workloads with that. I generally advise my clients to work on a single node before attempting to scale out.
- film42 2y agoI think this will be a hit with the big name audit companies. I know some use databricks for pyspark on the M&A side. As deals move forward and they get more data, they have to scale up their instances which isn't cheap. If polars enables serverless compute where you pay by the job, that could be a big win. And sure, databricks has an idle shutdown feature, but suppose it takes ~6 hours to process the deal report, and only the first hour needs the scaled up power to compute one table, and the rest of the jobs only need 1/10th the mem and cores. Polars could save these firms a lot of money.
- serced 2y agoMay I ask what part in M&A needs this much data processing? I am quite familiar with the field but did not yet see such tasks.
- lmeyerov 2y agoI thought databricks has serverless recently already? Or is it by the notebook, while this is by the job?
- Centigonal 2y agoDatabricks supports serverless for both interactive notebooks and jobs.
- tfehring 2y agoThe obvious one is that you can handle bigger workloads than you can fit in RAM on a single machine. The more important but less obvious one is that it right-sizes the resources needed for each workload, so you're not running an 8GB job on an 8TB machine, and your manually-allocated 8GB server doesn't OOM when that job grows to 10GB next year.
- orlp 2y agoDisclaimer: I work for Polars Inc, but my opinions are my own. If you have a very beefy desktop machine and no giant datasets, there isn't a strong reason to use Polars Cloud. Are you a data scientist running a Polars data pipeline against a subsampled dataset in a notebook on your laptop? With just changing a couple lines of code you can run that same pipeline against your full dataset on a beefy cloud machine which is automatically spun up and spun down for you. If you have so much data that one machine doesn't cut it, you can start running distributed. In a nutshell, the pitch is very similar to Dask/Ray/Spark, except that it's Polars. A lot of our users say that they came for the speed but stayed for the API, and with Polars Cloud they can use our API and semantics on the cloud. No need to translate it to Dask/Ray/Spark.
- fastasucan 2y agoI think being able to run the same code locally and on the "cloud" is a great selling point. Developing on Spark feels hillariously ineffective.
- dwagnerkc 2y agothey came for the speed but stayed for the API This is exactly how I would describe my experience. When I talk to others about polars now I usually quickly mention its fast up front, but then mostly talk about the API, its composability, small surface area, etc. are really what make it great to work with. Having these same semantics backed by eager execution, query optimized lazy API, streaming engine, GPU engine, and now distributed auto-magical ephemeral boxes in the sky engine just make it that much better of a tool.
- gardnr 2y agoBeing both eager and lazy does make it sound magical.
- __mharrison__ 2y agoReally excited for the Polars team. I've always been impressed by their work and responsiveness to issues I've filed in the past. The world is lifted when there is good competition like this.
- TheAlchemist 2y agoHaving switched from Pandas to Polars recently, this is quite interesting and I guess performance wise it will be excellent.
- whalesalad 2y agoNever understood these kinds of cloud tools that deal with big data. You are paying enormous ingress/egress fees to do this.
- tfehring 2y agoThat's almost certainly the main reason they're offering this on all 3 major public clouds from day 1.
- tomnipotent 2y ago> You are paying enormous ingress/egress fees to do this. It looks like their offering runs on the same cloud provider as the client, so no bandwidth fees. Right now it looks to be AWS, but mentions Azure/GCP/self-hosted.
- ritchie46 2y agoDisclosure, I wrote this post. The compute-plane (cluster) will run in your own VPC.
- Starlord2048 2y agoI can appreciate the pain points you guys are addressing. The "diagonal scaling" approach seems particularly clever - dynamically choosing between horizontal and vertical scaling based on the query characteristics rather than forcing users into a one-size-fits-all model. Most real-world data workloads have mixed requirements, so this flexibility could be a major advantage. I'm curious how the new streaming engine with out-of-core processing will compare to Dask, which has been in this space for a while but hasn't quite achieved the adoption of pandas/PySpark despite its strengths. The unified API approach also tackles a real issue. The cognitive overhead of switching between pandas for local work and PySpark for distributed work is higher than most people acknowledge. Having a consistent mental model regardless of scale would be a productivity boost. Anyway, I would love to apply for the early access and try it out. I'd be particularly interested in seeing benchmark comparisons against Ray, Dask, and Spark for different workload profiles. Also curious about the pricing model and the cold start problem that plagues many distributed systems.
- scrlk 2y agoIbis also solves this problem by providing a portable dataframe API that works across multiple backends (DuckDB by default): https://ibis-project.org/ https://ibis-project.org/
- codydkdc 2y agowithout locking you into a single cloud vendor ;)
- Starlord2048 2y agowow, ibis supports nearly 20 backends, that's impressive
- ZeroTalent 2y agoI've played around a bit with ibis for some internal analytics stuff, and honestly it's pretty nice to have one unified api for duckdb, postgres, etc. saves you from a ton of headaches switching context between different query languages and syntax quirks. but like you said, performance totally depends on the underlying backend, and sometimes that's a mixed bag—duckdb flies, but certain others can get sluggish with more complex joins and aggregations. polars cloud might have an advantage here since they're optimizing directly around polars' own rust-based engine. i've done a fair bit of work lately using polars locally (huge fan of the lazy api), and if they can translate that speed and ergonomics smoothly into the cloud, it could be a real winner. the downside is obviously potential lock-in, but if it makes my day-to-day data wrangling faster, it might be worth the tradeoff. curious to see benchmarks soon against dask, ray, and spark for some heavy analytics workloads.
- tfehring 2y agoThis is really cool, not sure how I missed it. I assume catalog support will be added fairly quickly. But ironically I think the biggest barrier to adoption will be the lack of an off-ramp to a FOSS solution that companies can self-host. Obviously Polars itself is FOSS, but it understandably seems like there's no way to self-host a backend to point a `pc.ComputeContext` to. That will be an especially tough selling point for companies that are already on Spark. I wonder how much they'll focus on startups vs. trying to get bigger companies to switch, and whether they'll try a Spark compatibility layer like DataFusion (https://github.com/apache/datafusion-comet https://github.com/apache/datafusion-comet).
- orlp 2y agoDisclaimer: I work for Polars Inc, but my opinions are my own. Polars itself is FOSS and will remain FOSS. Self-hosted/on-site Polars Cloud is something we intend on developing as there is quite a bit of demand, but it is unlikely to be FOSS. It most likely will involve licensing of some sort. Ultimately we do have to make money, and we intend on doing that through Polars Cloud, self-hosted or not (as well as other ventures such as offering training, commercial support, etc).
- tfehring 2y agoYep I totally get it and would probably go the same route in Polars' situation. Just sharing how some of the data teams I'm familiar with would likely be thinking about the tradeoffs.
- whyho 2y agoHow does this integrate into existing services like aws glue? I fear that despite polars being good/better it will lack adoption since it cannot easily be integrated.
- th0ma5 2y agoI think this is the main problem with this, like H2O offers Spark integration as well their own clustering solution, but most people with this problem have their own opinionated and bespoke needs.
- melvinroest 2y agoI just got into data analysis recently (former software engineer) and tried out pandas vs polars. I like polars way more because it feels like SQL but then sane, and it's faster. It's clear in what it tries to do. I didn't really have that with pandas.
- epistasis 2y agoI've been doing data analysis for decades, and stayed on R for a long time because Pandas was so bad. People complain about R, but compared to the multitude of import lice and unergonomic APIs in Pandas, R always felt like living in the future. Polars is a much much more sane API, but expressions are very clunky for doing basic computation. Or at least I can't find anything less clunky than pl.col("x") or pl.literal(2) where in R it's just x or 2. Still, I'm using Python a ton more now that polars has enough steam for others to be able to understand the code.
- minimaxir 2y ago> Or at least I can't find anything less clunky than pl.col("x") or pl.literal(2) where in R it's just x or 2. In many cases you can pass a string or numeric literal to a Polars function instead of the pl.col (e.g. select()/group_by()). Overall I agree it's less convenient than in dplyr in the cases where pl.col is required, sure, but not terrible and has the benefit of making the code less ambigious which reduces bugs.
- epistasis 2y agoI think compsci people can appreciate R as a language itself, because it has really beautiful language features. I think programmers hate it, because it's so different and lispy, with features that they can't really appreciate when coming from a C-style OOP mindset.
- theLiminator 2y agoI think if that's too painful, you can introduce a convention of: ``` from polars import col as c, lit as l ``` For anything production though, I just stick to pl.col and pl.lit as it's widely used.
- efxhoy 2y agoLooks great! Can I run it on my own bare metal cluster? Will I need to buy a license?
- marxisttemp 2y agoWhat does this project have to do with Serbia? They’re based in the Netherlands. They must have made a mistake when registering their domain name.
- ritchie46 2y agoNothing. Polars -> pola.rs The Polars name and a hint to the .rs file extension.
- marxisttemp 2y agoI’m aware. I personally wouldn’t want to tie my infrastructure, nor provide funding, to the government of Serbia at this particular juncture in geopolitical time, but hey, you gotta have a cutesy ccTLD hack or you aren’t webscale.
- unit149 2y ago[dead]
- marquisdepolis 2y agoThis is very interesting, clearly there's a major pain point here to be addressed, especially the delta between local pandas work and distributed [pyspark] work! Would love to test this out and do benchmarks against us/ Dask/ Spark/ Ray etc which have been our primary testing ground. Full disclosure, work at Bodo which has similar-ish aspirations (https://github.com/bodo-ai/Bodo https://github.com/bodo-ai/Bodo), but FOSS all the way.
- noworriesnate 2y agoEvery time I build something complex with dataframes in either R or Python (Pandas, I haven't used Polars yet), I end up really wishing I could have statically typed dataframes. I miss the security of knowing that when I change common code, the compiler will catch if I break a totally different part of the dashboard for instance. I'm aware of Pandera[1] which has support for Polars as well but, while nice, it doesn't cause the code to fail to compile, it only fails at runtime. To me this is the achilles heel of analysis in both Python and R. Does anybody have ideas on how this situation could be improved? [1] https://pandera.readthedocs.io/en/stable/ https://pandera.readthedocs.io/en/stable/
- ants_everywhere 2y agoI agree, and I suspect there are large numbers of unknown bugs in a lot of data frame based applications. But to do it right you'd need a pretty good type system because these applications implicitly use a lot of isomorphisms between different mathematical objects. The current solution is just to ignore types and treat everything as a bag of floats with some shape. If you start tracking types you need a way to handle these isomorphisms.
- lamp_book 2y agoScala Spark - a bit absurd if you don't need the parallelism, though. Most of the development can be done simply in quick compilation iterations or copied from the sbt REPL. Python/pandas feels Stone Age in comparison - you absolutely waste a lot of time iterating with run-time testing.
- Centigonal 2y agoWhy scala spark over pyspark?
- smu3l 2y agoScala (and Java) has a typed Dataset api.[0] pyspark only provides the Dataframe API, which is not typed. [0] https://spark.apache.org/docs/latest/sql-programming-guide.html#datasets-and-dataframes https://spark.apache.org/docs/latest/sql-programming-guide.h...
- c7THEC2DDFVV2V 2y agowho covers egress costs?
- ritchie46 2y agoThe cluster runs in your own VPC.
- Larrikin 2y agoAs a hobbyist, I describe polars as pandas if it was planned for humans to use. It's great to use, I just hate running into issues trying to use it. I wish them luck
- appleaday1 2y agoI applied :D just now hehehe
- otteromkram 2y agoHow is this not an advertisement? Does HN tag those or nah?
- robertkoss 2y agoLove it! Competition for Databricks is always appreciated and I think having a competitor that is not running on the JVM is amazing. Working with polars feels always insanely lightweight compared to Spark. If you would provide Workflows / Scheduling out of the box, I would migrate my Spark jobs today :)
- jt_b 2y agoPolars seems cool, but not willing to invest in adoption until Geo support is more mature. I find I'm preferring to run most operations I'd use dataframe libraries for in local SQL via DuckDB anyways.