7 ms·

Having done a bit of data engineering in my day, I'm growing more and more allergic to the DataFrame API (which I used 24/7 for years). From what I've seen over

• robertkoss 1y ago
  That is a false dichotomy. You can use SQL tools but still have to choose the instance type. Especially when considering testability and composability, using a DataFrame API inside regular languages like Python is far superior IMO.
  • drej 1y ago
    As a user? No, I don't have to choose. What I'm saying is that analysts (who this Polars Cloud targets, just like Coiled or Databricks) shouldn't worry about instance types, shuffling performance, join strategies, JVM versions, cross-AZ pricing etc. In most cases, they should just get a connection string and/or a web UI to run their queries, everything abstracted from them. Sure, Python code is more testable and composable (and I do love that). Have I seen _any_ analysts write tests or compose their queries? I'm not saying these people don't exist, but I have yet to bump into any.
    • ayhanfuat 1y ago
      I really doubt that Polars Cloud targets analysts doing ad-hoc analyses. It is much more likely towards people who build data pipelines for downstream tasks (ML etc).
      • ritchie46 1y ago
        We also target ad-hoc analysis. If your data doesn't fit on your laptop, you can spin up a larger box or a cluster and run interactive queries.
    • robertkoss 1y ago
      You were talking about data engineering. If you do not write tests as a data engineer what are you doing then? Just hoping that you don't fuck up editing a 1000 > line SQL script? If you use Athena you still have to worry about shuffling and joining, it is just hidden.. It is Trino / Presto under the hood and if you click explain you can see the execution plan, which is essentially the same as looking into the SparkUI. Who cares about JVM versions nowadays? No one is hosting Spark themselves. Literally every tool now supports DataFrame AND SQL APIs and to me there is no reason to pick up SQL if you are familiar with a little bit of Python
      • drej 1y ago
        I was talking about data engineering, because that was my job and all analysts were downstream of me. And I could see them struggle with handling infrastructure and way too many toggles that our platform provided them (Databricks at the time). Yes, I did write tests and no, I did not write 1000-line SQL (or any SQL for that matter). But I could see analysts struggle and I could see other people in other orgs just firing off simple SQL queries that did the same as non-portable Python mess that we had to keep alive. (Not to mention the far superior performance of database queries.) But I knew how this all came to be - a manager wanted to pad their resume with some big data acronyms and as a result, we spent way too much time and money migrating to an architecture, that made everyone worse off.
      • datadrivenangel 1y ago
        Way too many data engineers are running in clown mode just eyeballing the results of 1000 line SQL scripts.... https://ludic.mataroa.blog/blog/get-me-out-of-data-hell/ https://ludic.mataroa.blog/blog/get-me-out-of-data-hell/
    • ritchie46 1y ago
      With Polars Cloud you don't have to choose those either. You can pick cpu/memory and we will offer autoscaling in a few months. Cluster configuration is optional if you want this control. Anyhow, this doesn't have much to do with the query API, be it SQL or DataFrame.
    • mr_toad 1y ago
      Analysts don’t because it’s not part of the training & culture. If you’re writing tests you’re doing engineering. That said the last Python code I wrote as a data engineer was to run tests on an SQL database, because the equivalent in SQL would have been tens of thousands of lines of wallpaper code.
    • gigatexal 1y ago
      Again the issue you’re having is the skill level of the audience you keep bringing up not the tool.
      • drej 1y ago
        I find it much more beneficial to lower the barrier for entry (oftentimes without any sacrifices) instead of spending time and money on upskilling everyone, just because I like engineering.
        • gigatexal 1y ago
          Right but nobody is saying polars or data frames is to replace SQL or is even for the masses. It’s a tool for skilled folks. I personally think the api makes sense but SQL is easier to pick up. Use whatever tools work best. But coming into such a discussion dunking on a tool cuz it’s not for the masses makes no sense.
          • drej 1y ago
            Read my posts again, I'm not complaining it's not for the masses, I know it isn't. I'm complaining that it's being forced upon people when there are simpler alternatives that help people focus on business problems rather than setting up virtual environments. So I'm very much advocating for people to "[u]se whatever tools work best". (That is - now I'm doing this. In the past I taught a course on pandas data analytics and spoke at a few PyData conferences and meetups, partly about dataframes and how useful they are. So I'm very much guilty of what all of the above.)
            • gigatexal 1y ago
              Who is doing the forcing? I’ve not found a place in my decade as a data engineer that such places forced dataframes on would be and capable SQL analysts.
    • riku_iki 1y ago
      > analysts (who this Polars Cloud targets, just like Coiled or Databricks) shouldn't worry about instance types, shuffling performance, join strategies, I think this part(query optimizations) in general not solved/solvable, and it is sometimes/often(depending on domain) necessary to digg into details to make data transformation working.
  • gigatexal 1y ago
    Yeah it makes no sense. Why is the dataframe approach getting hate when you’re talking about runtime details? That folks understand the almost conversational aspect of SQL vs. that of the dataframe api but the other points make no difference. If you’re a competent dev/data person and are productive with the dataframe then yay. Also setup and creating test data and such it’s all objects and functions after all — if anything it’s better than the horribad experience of ORMs.
• drej 1y ago
  Fun aside - I actually used polars for a bit - first time I tried it, I actually thought it was broken, because it finished processing so quickly I thought it silently exited or something. So I'm definitely a fan, IF you need the DataFrame API. My point was that most people don't need it and it's oftentimes standing in the way. That's all.
  • orochimaaru 1y ago
    Polars is very nice. I’ve used it off and on. The option to write rust udf’s for performance, easy integration of rust with Python with pyo3 will make it a real contender. Yes, I know spark and scala exist. I use it. But the underlying Java engines and the tacky Python gateway impact performance and capacity usage. Having your primary processing engine in the same process compiled natively always helps.
• mrtimo 1y ago
  I agree with this 100%. The creator of duckdb argues that people using pandas are missing out of the 50 years of progress in database research, in the first 5 minutes of his talk here [1]. I've been using Malloy [2], which compiles to SQL (like Typescript compiles to Javascript), so instead of editing a 1000 line SQL script, it's only 18 lines of Malloy. I'd love to see a blog post comparing a pandas approach to cleaning to an SQL/Malloy approach. [1] https://www.youtube.com/watch?v=PFUZlNQIndo https://www.youtube.com/watch?v=PFUZlNQIndo [2] https://www.malloydata.dev/ https://www.malloydata.dev/
  • esafak 1y ago
    Have you used Malloy in a pipeline, e.g., with Airflow? If so, how was the experience?
  • fumeux_fume 1y ago
    In the same talk, Mark acknowledges that "for data science workflows, database systems are frustrating and slow." Granted DuckDB is an attempt to fix that, most data scientists don't get to choose what database the data is stored in.
    • willvarfar 1y ago
      (I use duckdb to query data stored in parquet files)
      • mrtimo 1y ago
        Same. But, I use Malloy which uses duckdb to query data stored in hundreds of parquet files (as if they were one big file).
        • willvarfar 1y ago
          I haven't looked at Mallory, but I do regularly scan lots of parquet files using wildcards etc from duckdb. Its a neat builtin duckdb feature.
  • 1y ago
• RobinL 1y ago
  100% agree. I've also worked as a data engineer and came to the same conclusion. I wrote up a blog which went into a bit more depth on the topic here: https://www.robinlinacre.com/recommend_sql/ https://www.robinlinacre.com/recommend_sql/
• spenczar5 1y ago
  I agree, but there are other possibilities in between those two extremes, like Quivr [1]. Schemas are good, but they can be defined in Python and you get a lot more composability and modularity than you would find in SQL (or pandas, realistically). 1: https://github.com/B612-Asteroid-Institute/quivr https://github.com/B612-Asteroid-Institute/quivr
• fumeux_fume 1y ago
  We all have allergies. I'm allergic to 1000 line SQL queries which include functions that are only usable for a specific flavor and version of SQL.
• ketozhang 1y ago
  I think your argument focuses a lot on the scenario where you already have cleaned data (i.e., data warehouse). I and many other data engineers agree, you're better off with hosting it on SQL RDBMS. However, before that, you need a lot of code to clean the data and raw data does not fit well into a structured RDBMS. Here you choose to either map your raw data into row view or a table view. You're now left with the choice of either inventing your own domain object (row view) or use a dataframe (table view).