MrPowers

8 ms·

31.
by MrPowers 3y ago
There are a few features missing from the FOSS Scala/Spark implementation of Delta Lake, but I wouldn't say a lot. The FOSS version supports all the table features in the Delta Lake protocol. The Delta Rust implementation is miss
32.
by MrPowers 3y ago
Yea, there is a Rust implementation of the Delta Lake protocol that lets you do upserts without Spark too. This allows pandas, Polars, DataFusion, and PyArrow users to easily do upserts as well.
33.
Delta Lake vs. Parquet: A Comparison(delta.io)
32 points by MrPowers 3y ago|54 comments
34.
by MrPowers 3y ago
Data Lakes (i.e. Parquet files in storage without a metadata layer) don't support transactions, require expensive file listing operations, and don't support basic DML operations like deleting rows. Delta Lake stores data in Parque
35.
by MrPowers 3y ago
Delta Lake solves a lot of the Parquet limitations mentioned in this post. Disclosure: I work on the Delta Lake project. Parquet files store metadata about row groups in the file footer. Delta Lake adds file-level metadata in the transact
36.
by MrPowers 3y ago
Modern query engines are designed to read data in parallel because it's so much faster. The data could be stored in 50 different one-gig files that were read in parallel.
37.
by MrPowers 3y ago
I think these benchmarks are great, but also quite misleading and should be updated: * the 1 billion row benchmarks are run on a single, uncompressed 50 GB CSV file. 50 GB should be stored in multiple files. * the benchmarks only show the
38.
by MrPowers 3y ago
I think most companies don't understand a large part of the Databricks offering and it should be used by way more organizations. Disclaimer: I was a Databricks user for 6 years and now work at Databricks. Yea, you can create your own
39.
Why it's time to embrace Rust for writing native Python package extensions(datawill.io)
4 points by MrPowers 4y ago|0 comments
40.
by MrPowers 4y ago
I've been having a great time using LinkedIn. I only like and follow data people. My feed is only data and there are a lot of high-quality creators. I unfollow anyone that posts personal content or low-quality data "self-help&q
41.
by MrPowers 4y ago
This is cool, but would like to give some higher level context about querying JSON files. JSON is a row based file format. It doesn't allow query engines to skip rows or skip columns when running queries, so all data needs to get read
42.
by MrPowers 4y ago
Yes, I agree with your assessment that technologies that can deal with larger than memory datasets (e.g. Polars) can be used to filter data so there are less rows for technologies that can only handle datasets a fraction of the data (e.g. p
43.
by MrPowers 4y ago
Dask sidesteps the problem by splitting up the data into smaller pandas DataFrames and processing them individually, rather than all at once. But the fundamental issue of pandas storing strings as objects in DataFrames (which is really ine
44.
by MrPowers 4y ago
The Arrow revolution is particularly important for pandas users. pandas DataFrames are persisted in memory. The rule of thumb was for RAM capacity / dataset size in memory to be 5-10x as of 2017. Let's assume that pandas has mad
45.
by MrPowers 4y ago
PyO3 is being used to expose the Python bindings to the delta-rs project: https://github.com/delta-io/delta-rs It's a great way to expose Python bindings because it "feels" Pythonic. Most users run pip
46.
by MrPowers 4y ago
I've been working in the data space for 8 years and find it incredibly fulfilling. Open source community is great (I've built and contributed to a lot of projects), LinkedIn has tons of great data content, meetups are fun. The wo
47.
by MrPowers 4y ago
Sure, this blog post explains the dependency injection design pattern with Spark: https://mrpowers.medium.com/dependency-injection-with-spark-... You can structure your code to read from paths when run in the production env
48.
by MrPowers 4y ago
Spark makes it easy to wrap SQL in functions that are easy to test. I am the author of the popular Scala Spark (spark-fast-tests) and PySpark (chispa) testing libraries. Some additional tips to speed up Spark tests (can speed up tests bet
49.
by MrPowers 4y ago
You're understandably confused by the industry terminology that's ambiguous and morphing over time. Data lakes are typically CSV/JSON/ORC/Avro/Parquet files stored in a storage system (cloud storage like AWS S3
50.
by MrPowers 4y ago
Some high level context for those less familiar with the Lakehouse storage system space. For various reasons, several companies moved from data warehouses to data lakes starting around 7-10 years ago. Data lakes are better for ML / AI
51.
by MrPowers 4y ago
I think vendor published benchmarks are fine if the dataset is open / accessible, the benchmark code is published, all software versions are disclosed, and the exact hardware is specified. I definitely wouldn't consider an audite
52.
by MrPowers 4y ago
Web Framework versions go EOL and then you're forced to upgrade. So constant maintenance is required with Rails, even if you just want to put the app in "security patch only mode".
53.
Ask HN: What's the best technology for a CRUD website to minimize maintenance
4 points by MrPowers 4y ago|9 comments
54.
by MrPowers 4y ago
> The trend has been to shift as much work possible to the current generation of Data Warehouses, that abstract the programming model that Spark on columnar storage provided with only a SQL interface, reducing the space where you'd
55.
by MrPowers 4y ago
I am working on some blogs / videos that will hopefully help clarify the differences. I'm working on a Delta Lake vs Parquet blog post right now and gave a 5 Reasons Parquet files are better than CSV talk last year: https:/
56.
by MrPowers 4y ago
> A predilection for SQL, yielding "get the answer right once" big-ball-of-SQL solutions which are infeasible to debug or modify without causing regressions. Yea, thankfully some frameworks have Python / Scala APIs that le
57.
by MrPowers 4y ago
Data warehouse: bundles compute & storage and comes at a comparatively high price point. Great option for certain workflows. Not as great for scaling & non-SQL workflows. Data lake: typically refers to Parquet files / CSV fil
58.
by MrPowers 4y ago
Great article. > data engineers have been increasingly adopting software engineering best practices I think the data engineering field is starting to adopt some software engineering best practices, but it's still really early days.
59.
by MrPowers 4y ago
There is an excellent Real Life Lore video on "How Russia is Crippling Germany with Gas" that has quite a different take: https://www.youtube.com/watch?v=Xi74gPrn24Y&ab_channel=RealL... Peter Zeihan also has q
60.
by MrPowers 4y ago
I personally like the Polars docs because they have a detailed user guide that makes it so easy to get started with the project: https://pola-rs.github.io/polars-book/user-guide/

More ›