danthelion

6 ms·

1.
A decade of Gazette: a streaming broker that doesn't store data(estuary.dev)
2 points by danthelion 10d ago|0 comments
2.
FOMO-Driven Development(estuary.dev)
2 points by danthelion 3mo ago|0 comments
3.
Exactly-Once Delivery Is a Spectrum, Not a Checkbox: Part 1(medium.com)
3 points by danthelion 3mo ago|0 comments
4.
Why Is Everyone Buying Change Data Capture?(estuary.dev)
2 points by danthelion 1y ago|0 comments
5.
I Changed My Mind: AI Will Replace Us(medium.com)
14 points by danthelion 1y ago|20 comments
6.
HTAP: Still the Dream, a Decade Later(medium.com)
3 points by danthelion 1y ago|0 comments
7.
A New Reference Architecture for Change Data Capture (CDC)(estuary.dev)
4 points by danthelion 1y ago|0 comments
8.
Show HN: Interactive Change Data Capture (CDC) Playground(change-data-capture.com)
2 points by danthelion 1y ago|0 comments
9.
Fast JSON Processing in Real-Time Systems: Simdjson and Zero-Copy Design(estuary.dev)
22 points by danthelion 2y ago|2 comments
10.
by danthelion 2y ago
At Estuary, we’re creating a real-time data streaming platform that doesn’t rely on Kafka and uses JSON as the primary data format stored in an object storage. Many people are interested in how we achieve millisecond-level latency in our da
11.
Welcome to the Data Engineering Dream Factory(arecadata.com)
3 points by danthelion 2y ago|0 comments
12.
Apache Iceberg: The Hadoop of the Modern Data Stack?(medium.com)
2 points by danthelion 2y ago|0 comments
13.
Data Is Only Useful Outside of the Database(medium.com)
11 points by danthelion 2y ago|1 comments
14.
by danthelion 2y ago
Imagine a system monitoring payment transactions. Each transaction stream (e.g., purchase events) could be joined with customer account data (e.g., past purchasing patterns or blacklist flags). Streaming joins enable flagging potentially fr
15.
by danthelion 2y ago
Materialize is powered by timely dataflow and differential dataflow, Estuary Flow uses a streaming mapreduce architecture: https://estuary.dev/why-mapreduce-is-making-a-comeback/
16.
Streaming joins are hard(estuary.dev)
117 points by danthelion 2y ago|57 comments
17.
by danthelion 2y ago
Gazette is at the core of Estuary Flow ( https://estuary.dev ), a real-time data platform. Unlike Kafka, Gazette’s architecture is simpler to reason about and operate. It plays well with k8s and is backed by S3 (or any object stor
18.
Gazette: Cloud-native millisecond-latency streaming(github.com)
139 points by danthelion 2y ago|52 comments
19.
by danthelion 2y ago
With the 2-pass strategy, we can write arbitrary row group sizes while using a fixed amount of memory, with probably 100-200 MiB of overhead for the parquet file processing, depending on how large the metadata is for the scratch file. witho
20.
Memory Efficient Data Streaming to Parquet Files(estuary.dev)
28 points by danthelion 2y ago|2 comments
21.
Change data capture done correctly(estuary.dev)
2 points by danthelion 2y ago|0 comments
22.
A Mental Framework for Side Projects(arecadata.com)
1 points by danthelion 2y ago|0 comments
23.
Running Doom on Snowflake(arecadata.com)
2 points by danthelion 3y ago|0 comments
24.
SQL for Google Sheets with DuckDB(arecadata.com)
3 points by danthelion 3y ago|1 comments
25.
What I would do differently getting into Data Engineering [in 2024](arecadata.com)
1 points by danthelion 3y ago|0 comments
26.
by danthelion 3y ago
I wrote git-genie to automate commit message writing with GPT & pre-commit hooks, works surprisingly well (most of the time) - https://github.com/danthelion/git-genie
27.
Show HN: git-genie: pre-commit hook that writes commit messages for you(github.com)
4 points by danthelion 3y ago|1 comments
28.
by danthelion 3y ago
Git-genie can now be installed as a pre-commit hook which enables commit message generation automatically based on the staged changes using GPT.
29.
Data Engineering Sucks(arecadata.com)
1 points by danthelion 3y ago|0 comments
30.
Data Debt Is Not Evil: A Pragmatic Perspective(arecadata.com)
1 points by danthelion 3y ago|0 comments

More ›