wenc

58 ms·

961.
by wenc 4y ago
Point taken but most data wrangling these days — especially at scale — is of the long and thin variety (what is also known as 3rd normal form or tidy format — which actually allows for more flexibility if you think in terms of coordinatized
962.
by wenc 4y ago
It depends on what you do with Pandas. The semantics are different so you’ll have to rewrite stuff — that said, for me it was worth it for the most part because I work with massive columnar data in Parquet. So no, not a drop in replacement.
963.
by wenc 4y ago
Read_csv - yes. I just tried loading a 5MM row csv. Pandas took 5.5 secs. Polars took 0.6 secs. .apply - might be faster in Polars but will not be as fast as using native expressions. That’s because applying a Python function invokes the GI
964.
by wenc 4y ago
Yes that support is still not complete. When you pass a Pandas dataframe into Scikit you are implicitly doing df.values which loses all the dataframe metadata. There is a library called sklearn-pandas which doesn’t seem to be mainstream and
965.
by wenc 4y ago
Although Polars is written in Rust, most people will use Polars from Python since that what most data wranglers use. And Polars’ Python interface is excellent.
966.
by wenc 4y ago
Yup here’s a list of optimizations that Polars does to achieve the speed that it does https://pola-rs.github.io/polars-book/user-guide/#current-st... Polars’ lazy evaluation is a big deal — this lets it do query p
967.
by wenc 4y ago
To be honest I haven’t actually worked with any third party libraries that actually supported Pandas directly — most ML libs like Scikit require Numpy arrays (not Pandas dataframes) as inputs so I’ve always had to cast to Numpy from Pandas
968.
by wenc 4y ago
It works as well as Pandas (realize that scikit actually doesn’t support Pandas — you have to cast a dataframe into a Numpy array first) I generally work with with Polars and DuckDB until the final step, when I cast it into a data structure
969.
by wenc 4y ago
I’m using DuckDB in Jupyter and Python. DuckDB is the SQLite equivalent for complex analytic queries on columnar data (Parquet, CSV etc.)
970.
by wenc 4y ago
Polars is already built on Arrow and the advantages are tremendous.
971.
by wenc 4y ago
I’ve moved entirely to Polars (which is essentially Pandas written in Rust with new design decisions) with DuckDB as my SQL query engine. Since both are backed by Arrow, there is zero copy and performance on large datasets is super fast (du
972.
by wenc 4y ago
I have been inside a McDonald's recently. Prices are up but not outrageous. You can still get a meal for under $10. I don't see why an average person cannot afford this. Here's how much Big Mac costs in every state. https:&#
973.
by wenc 4y ago
I started using Azure in 2018 -- it sucked but I had to use it because my company was entrenched in the MS ecosystem (as are many companies). Over time it sucked less, that when I left in 2021, Azure was in a semi-usable state. AWS had an 8
974.
by wenc 4y ago
It might not have in the past but in recent years especially during 45’s tenure there many pauses, government shutdowns and backlogs that lengthened th duration, not to mention the amount of time it takes for the lawyers to even start the p
975.
by wenc 4y ago
This is the court decision: https://www.bloomberglaw.com/public/desktop/document/SAVEJOB... To folks who aren't familiar with the intricacies of US visas, they are jumping to the conclusion that all spou
976.
by wenc 4y ago
Polars and DuckDB have replaced Pandas for me for the most part.
977.
by wenc 4y ago
The QR scanner should be on the turnstile itself. It’s that little white thing next to the palm reader. It even has an Amazon app icon. https://www-ocregister-com.cdn.ampproject.org/i/s/www.ocregi...
978.
by wenc 4y ago
I can't speak for everyone but I did get obese eating rice. I ate a lot of it. In Asian diets, the one thing people often overindulge in is rice -- not everyone of course, but for those who do overindulge, their preferred carb is rice.
979.
by wenc 4y ago
I see what you mean. I was referring the half the size in terms of population — SK is 50m vs FL with 21m. SK has great transportation connectivity so I’m not sure how being topographically disconnected has an effect in modern times.
980.
by wenc 4y ago
If we take the simplistic assumption that 2.1 children (either through natural birth or immigration) is the replacement rate, we can surmise that for zero growth we would need to hover around that number. 0.78 YoY is negative growth and fas
981.
by wenc 4y ago
South Korea is not half the size of Florida — it’s the other way around. FL is less than half the size of SK. Also it’s not an island nation — it’s on the Korean Peninsula. If you mean it’s an island nation in terms of connectivity then mos
982.
by wenc 4y ago
> bad social problems that further suppress birth rates, which Western countries don't have. Genuinely curious what you think these are? (not looking to argue, just wondering because I’m a closet sociologist)
983.
by wenc 4y ago
The problem is thinking it’s as simple as resources divided by population, so shrinking the denominator means more per capita resources right? But a country’s infrastructure is built for a certain population size assumption, and the numerat
984.
by wenc 4y ago
It still can be. Most of my friends are still in FB (I'm also of a certain generation where none of the people I care about are on IG or TikTok, so those networks mean nothing to me). I ruthlessly unfollow people who post garbage links
985.
by wenc 4y ago
Learning languages is a statistical problem. You have to learn the high frequency words/phrases in context (forget 500 most common vocabulary lists -- those don't work). As a child, you do this naturally. However, there are cert
986.
by wenc 4y ago
To me it does fill a big gap. It’s a local columnar engine that I can use inside a Jupyter notebook. This lowers my cost of iteration tremendously. Yes I can query data from Postgres and munge with Pandas. But what if I need to iterate on a
987.
by wenc 4y ago
Clickhouse local is very fast and good for many purposes but DuckDB is equally fast (or faster in some cases but this is a moving target) and supports a wider range of SQL that people like me often use. DuckDB is great for data scientists a
988.
by wenc 4y ago
The 8am class I had in my sophomore year of college was the only C I had. The professor who taught that class thought I was one of those low performers because I did so badly in his class. Later he became undergrad chair. I needed to retrie
989.
by wenc 4y ago
Try playing brown noise on top of noise canceling. It's more soothing than just pure noise, plus it masks intermittent uncanceled noises.
990.
by wenc 4y ago
At work, the closest thing to approaching a working Cone of Silence ("Get Smart" reference) for me are a pair of AirPods Pro 2. I'm literally in my own world when I'm working in an open-plan office. My level of annoyance

More ›