dancrystalbeach

5 ms·

31.
by dancrystalbeach 8mo ago
I can see all you old-school SQL Server buggers just squirming in your seats, hot under the collar. How dare they speak of such sacred things, holy, without reference and a nod to the past? Methinks someone, somewhere, once said that there
32.
by dancrystalbeach 8mo ago
So, you want to join the cool kids on the block and don't know where to start? Do you have stars in your eyes and think adopting Databricks as a Data Platform is the cure-all for the data woes and problems you have today? Maybe. Yes, a
33.
by dancrystalbeach 8mo ago
In this episode, I sit down with industry veteran Robin Moffatt — Sr. Principal Advisor in Streaming Data Technologies (Kafka, etc.) and a longtime voice in the data engineering community, to unpack the journey from old-school data architec
34.
by dancrystalbeach 9mo ago
I rarely get excited about happenings in the data world. When you’ve been staring at CSV files for 20 years, heard the same problems argued over and over again with only a different tool name at the front of the sentence, things can get a l
35.
by dancrystalbeach 9mo ago
Andy’s career didn’t start in software at all. It started with physical circuits, literally wiring systems as an electrician, before moving into programming, databases, and eventually decades of hands-on data engineering work.
36.
by dancrystalbeach 9mo ago
It’s an interesting time to be in software and data; the world of generative AI is changing the landscape beneath our feet. I don’t see this as a bad thing for software folk, but as an opportunity to learn new technologies and BUILD /
37.
by dancrystalbeach 9mo ago
Every day I go out and do the Lord’s work for y’all. On an average evening or weekend, you can find me setting up cloud instances, generating data, deploying code, and running tests. I don’t answer to anyone. I write what I please and what
38.
by dancrystalbeach 9mo ago
Anyone who’s been around for more than a decade or so in the programming, development, and data world might get a slight eye twitch when the word database driver appears. Before the modern times we live in came along, the entire data world
39.
by dancrystalbeach 9mo ago
It seems we have several cadres of people when it comes to “clean code.” I know there is a lot of previous baggage that comes with that nomenclature, good and bad. But, I think we can think about “clean code” from a simplistic point of view
40.
by dancrystalbeach 9mo ago
Well, don’t pretend like y’all didn’t see it coming. If I close my eyes and listen to the winter wind blowing through the maple trees, I can see Josue Bogran dancing in the moonlight with joy. There is truth in what he says, “All roads lead
41.
1TB of Parquet Files. Single Node Benchmark. (DuckDB Style)(dataengineeringcentral.substack.com)
2 points by dancrystalbeach 10mo ago|1 comments
42.
by dancrystalbeach 10mo ago
I give the people what they want; I’m a slave to clicks and the buzz of internet anger and mayhem. Not so long ago, I tried, with limited success, to convince a legion of engineers who were raised on the Snowflake and Databricks teat, that
43.
by dancrystalbeach 10mo ago
I’ve been a Polars bro for most of the last few years. Why? It’s Rust-based, fast, DataFrame-centric, just the way I like it. It also had the excellent feature, right from the start, of Lazy Execution. A few years ago, maybe two, I actually
44.
by dancrystalbeach 10mo ago
I am sort of an addict for a good Lake House; it’s just an incredible architecture that is a joy to build and use daily. No more SQL Server or Oracle, you know what I’m saying if you were born in the right decade.
45.
by dancrystalbeach 10mo ago
It's a fast-paced and ever-changing world we live in; nothing we can do about it. I grew up in the middle of the prairie, when the internet became mainstream, the age of Doom, Myst, MSN Messenger, Yahoo Pool, and that irreplaceable Gol
46.
by dancrystalbeach 10mo ago
Can you, should you, and what will happen, if we use LLMs not just to puke out the next code snippet … but what if we use an LLM mixed with a little Agentic AI, to actually to BE the data pipeline????
47.
by dancrystalbeach 10mo ago
I’m getting to the point where I’m a little burnt out on hearing about AI this and AI that. Work, LinkedIn, YouTube, Substack … it’s a never-ending glut of AI-generated content talking about AI. I get the sense that the closest most develop
48.
by dancrystalbeach 10mo ago
Recently, I had to migrate a few hundred Delta Lake tables that were partitioned over to Liquid Clustering. It seems straightforward on the surface, but everything usually does, until it isn’t. There was no rocket science involved here, but
49.
by dancrystalbeach 10mo ago
Meh. That's how I feel about the languishing and short-lived ideology called “Data Mesh.” The Modern Data Stack has seen its fair share of prophets seeking the loving embrace of the community, a fickle thing, only to fall out of favor
50.
by dancrystalbeach 11mo ago
Oh, that age-old topic of Data Quality, often discussed but never implemented. Poor little blighter. Who knows what it’s like to be the old, beat-up rock that always gets kicked, gets no love or attention, just told about everything you do
51.
by dancrystalbeach 11mo ago
Not only is it not cheap to crunch 650 GB of data on a Spark cluster —piling up DBUs, truth be told — but it’s not complicated either; they’ve made it easy to spend money. Especially when you simply don’t need a cluster anymore for *most da
52.
by dancrystalbeach 11mo ago
I am certainly no master of the minutiae of Databricks compute, able to unwind the wound-up ball of string that leads us to where we are today. Yet DBUs matter in the world of the Lake House. It’s the difference between an angry CTO and a h
53.
by dancrystalbeach 11mo ago
in an effort to bring value and push the boundaries of what is possible, we must at times stray from the safe path. Of course, the minute we stray from the well trodden way, we can expect trouble around every corner.
54.
by dancrystalbeach 11mo ago
We lived in a world where we hear the words ... "Terabyte", or "Iceberg" or "Lake House" and we think Cluster. We ... must ... have ... Cluster ... give ... me ... cluster. DBUs start piling up. We smile, our b
55.
by dancrystalbeach 1y ago
Things have changed a lot in the last year related to LLMs and AI; on the one hand, it seems the AI skeptics for coding are increasingly confined to the corners of the internet. Everyone is dancing around in the middle, not sure of where ev
56.
by dancrystalbeach 1y ago
The real question is, do you think Spark is the bloody British, the AWS EC2 instance is Boston, and DuckDB + Polars are the John Adams and George Washington, bent on freedom or death? You paid attention in history class, didn’t you?
57.
by dancrystalbeach 1y ago
If I had to guess, it was the Illuminati (since they clearly own all the SaaS vendors) that brought us the Semantic Layer, but we will find won’t we?
58.
What Is a Semantic Layer?(dataengineeringcentral.substack.com)
13 points by dancrystalbeach 1y ago|6 comments
59.
by dancrystalbeach 1y ago
Data Modeling is truly dead. I want a resurrection.
60.
The Fastest Way to Insert Data to Postgres(confessionsofadataguy.com)
3 points by dancrystalbeach 1y ago|1 comments

More ›