dancrystalbeach

8 ms·

61.
by dancrystalbeach 1y ago
I was recently working on a PySpark pipeline in which I was using the JDBC option to write about 22 million records from a Spark DataFrame into a Postgres RDS database. Hey, why not use the built in method provided by Spark, how bad could i
62.
by dancrystalbeach 1y ago
I can no longer hold the boiling and frothing mess of righteous anger that starts to rumble up from within me when I hear the words “Medallion Architecture” in the context of Data Modeling, especially when it’s used by some young Engineer w
63.
by dancrystalbeach 1y ago
Just when you thought that Rust-based GOAT data processing tool known as Polars could not get any faster … they go and add GPU to the mix.
64.
by dancrystalbeach 1y ago
I don’t know about you, but I grew up and cut my teeth in what feels like a special and Golden age of software engineering that is now relegated to the history books, a true onetime Renaissance of coding that was beautiful, bright, full of
65.
Databricks Workflows vs. Apache Airflow(dataengineeringcentral.substack.com)
3 points by dancrystalbeach 1y ago|1 comments
66.
by dancrystalbeach 1y ago
I’m such a cheerful guy, I figured I would try to write a positive and inspiring post while comparing two tools. Just kidding. I guess we will see how it goes. Who knows, we'll just see where it leads us, I suppose.
67.
by dancrystalbeach 1y ago
You know, after literally multiple decades in the data space, writing code and SQL, at some point along that arduous journey, one might think this problem would be solved by me, or the tooling … yet alas, not to be.
68.
by dancrystalbeach 1y ago
Wondering what all the AI agent hype is about? Read on.
69.
by dancrystalbeach 1y ago
I’m going to give you the spoiler up front. AI isn’t taking your job anytime soon. If you think otherwise, you’ve been watching too much YouTube and reading the wrong stuff on LinkedIn. Heck, maybe you simply HAVEN’T been using AI enough on
70.
by dancrystalbeach 1y ago
Cloudflare enters the Lake House war with Apache Iceberg
71.
by dancrystalbeach 2y ago
ometimes, I force my old caveman self to do things I do not want. Believe it or not, reviewing the landscape as it relates to data orchestration frameworks is not something on the top of my list. People, including me, have their own pet fav
72.
by dancrystalbeach 2y ago
Bugs are to software what SQL is to r/dataengineering.
73.
by dancrystalbeach 2y ago
When it comes to the AI hype, I pretty much have tried to ignore the constant roll of never ending models and other hoopla that is mostly meaningless. Better to let the dust settle.
74.
by dancrystalbeach 2y ago
Combining the two titans of data.
75.
by dancrystalbeach 2y ago
I simply can’t resist the urge to put my ring in the hat on this one. It's mostly because I see the glut of AI nonsense put out by talking heads who’ve never written a single line of usable code in their lives. But what’s also intrigui
76.
by dancrystalbeach 2y ago
I’ve never been one who’s much for self-help, you know, all those books you walk past in the airport or at Barnes and Noble. “10 Ways to Transform Yourself …” Yeah, right, I usually snort to myself and walk on.
77.
by dancrystalbeach 2y ago
When it comes to building modern Lake House architecture, we often get stuck in the past, doing the same old things time after time. We are human; we are lemmings; it’s just the trap we fall into. Usually, that pit we fall into is called Sp
78.
by dancrystalbeach 2y ago
DuckDB vs Polars for Unity Catalog integration with Delta Lake.
79.
by dancrystalbeach 2y ago
The best Data Quality tool ever created for Pyspark.
80.
by dancrystalbeach 2y ago
You know, for all the hoards of content, books, and videos produced in the “Data Space” over the last few years, famous or others, it seems I find there are volumes of information on the pieces and parts of working in Data. It could be Data
81.
by dancrystalbeach 2y ago
DuckDB to process remote JSON files! Amazing!
82.
by dancrystalbeach 2y ago
An awesome and simple Data Stack using AWS Lambda and DuckDB, along with Delta Lake, to show the Lake House architecture doesn't have to be complex.
83.
by dancrystalbeach 2y ago
S3 Tables are not a Lake House, they are also not commodity file storage on which you can build your Lake House. They are an Amazon table lock-in.
84.
by dancrystalbeach 2y ago
First, anyone who says there is NOT a Lake House battle brewing and bubbling most likely has an agenda tied to one or other of those dueling sides.
85.
by dancrystalbeach 2y ago
AWS recently announced S3 Tables, and the interwebs blew up. But is it just fools gold?
86.
by dancrystalbeach 2y ago
Sometimes I find myself lying in my sunroom, staring out the window in the blue sky above me while the sun plays on the maple tree, empty of most all but a few red leaves … wondering what else I can do to make the already angry readers of m
87.
by dancrystalbeach 2y ago
How many of you little hobbits are thinking more than 5 minutes from now?
88.
by dancrystalbeach 2y ago
Maybe it’s my age, I’m not totally sure, but it’s getting harder and harder for this old guy to keep up with all the new things that come trickling out of the quagmire that is the Data Engineering space these days.
89.
by dancrystalbeach 2y ago
This is an interesting question indeed, is it not? What to use, what to use? Both DuckDB and Polars seem to be flying high at the moment, the new cool kids on the block. Everyone talking about them, and 1% of the people actually using them.
90.
by dancrystalbeach 2y ago
This is an interesting one indeed, it’s one that teases and puzzles the brain to no end. Has the Data Warehouse finally died, has that unruly upstart the Lake House finally taken its place atop the seething mass of data we call home?

More ›