gunnarmorling

10 ms·

91.
by gunnarmorling 2y ago
Had you read on, you'd have seen that I am discussing this very point: > leader election will only ever be eventually correct... So you’ll always need to be prepared to detect and fence off work done by a previous leader.
92.
Leader Election with S3 Conditional Writes(morling.dev)
122 points by gunnarmorling 2y ago|70 comments
93.
by gunnarmorling 2y ago
Things got finally a bit easier with Postgres 16, which allows you to create replication slots on stand-by servers. It still requires manual intervention to make sure they don't fall too far behind their primary counterpart and to do t
94.
by gunnarmorling 3y ago
This actually doesn't fit the rules. I've designed the challenge so that disk I/O is not part of the measured runtime (initially, by relying on the fact that the first run which would pull the file into the page cache will be
95.
by gunnarmorling 3y ago
Debezium supports all kinds of data formats. For instance, ProtoBuf is often used these days, but it's easily customizable/configurable to have other formats, too. > "At least a few months" Can you back this up? This
96.
by gunnarmorling 3y ago
> you don't have any transactional consistency across those joins Debezium gives you all the information you need for establishing those transactional guarantees (transaction metadata topic), so you can implement a buffering logic f
97.
A Taxonomy of Data Change Events(decodable.co)
2 points by gunnarmorling 3y ago|0 comments
98.
by gunnarmorling 3y ago
One way for addressing this concern is (stateful) stream processing, for instance using Kafka Streams of Apache Flink: using a state store, you ingest the CDC streams for all the tables and incrementally update joins between them whenever t
99.
by gunnarmorling 3y ago
There's the "Show & Tell", where folks have shared implementations in many different languages: https://github.com/gunnarmorling/1brc/discussions/categories... .
100.
by gunnarmorling 3y ago
There is one entry which uses a trie, but it's not at the top, IIRC (which may or may not be related to using a trie, there's many other relevant design decisions).
101.
by gunnarmorling 3y ago
What an excellent post, one of the best on 1BRC I've come across so far. Big shout-out to Marko for participating in the challenge, making a strong push for clarifying corner cases of the rules, and sharing his experiences in this amaz
102.
by gunnarmorling 3y ago
Turns out you don't need GC for processing that 13 GB file, with relatively small heap sizes even: some folks have disabled GC by means of using EpsilonGC (i.e. a no-op collector). That said, the right collector will give you sub-ms pa
103.
by gunnarmorling 3y ago
Results actually have improved quite a bit since I've answered the questions for that interview. Fastest contenders now are below two seconds, and that's running on 8 CPU cores of the evaluation machine. When I ran the entries on
104.
by gunnarmorling 3y ago
You can join against a static (or slowly changing) table in Flink, including AS OF joins [1], i.e. joining the correct version of the records from the static table, as of the stream's event time for instance. You need to keep an eye on
105.
by gunnarmorling 3y ago
It's only silly if you miss the point of the challenge :) Which is to learn something new and have fun along the way.
106.
by gunnarmorling 3y ago
Just clarified this in the README: * Input value ranges are as follows: - Station name: non null UTF-8 string of min length 1 character and max length 100 characters - Temperature value: non null double between -99.9 (inclusive) and 99.9 (i
107.
by gunnarmorling 3y ago
Yes, exactly. Retrieving the source data file from the page cache != storing and reloading precomputed results.
108.
by gunnarmorling 3y ago
Their should be no optimizations which rely on a-priori knowledge of the dataset. I.e. if you calculate a perfect hash function at runtime by inspecting the dataset, that's fine (not sure whether it's beneficial), but hard coding
109.
by gunnarmorling 3y ago
Yeah, and I'd count caching results between runs into the same category, i.e. against the spirit of this challenge. Folks should keep in mind that the goal is to learn something new, rather than "winning" by cheating the rule
110.
by gunnarmorling 3y ago
See the "Show & Tell", where folks are discussing solutions in different languages, including Rust: https://github.com/gunnarmorling/1brc/discussions/categories... .
111.
by gunnarmorling 3y ago
Yes, you are right. This came up yesterday and indeed two solutions were violating the "must work for all station names" rule by relying on specific hash functions optimized for the specific data set, which I unfortunately missed
112.
by gunnarmorling 3y ago
I don't expect min/max to make a substantial difference to the overall runtime of the program. I've added them to the challenge to have a bit of assurance that contenders process the entire data set, rather than doing some ki
113.
by gunnarmorling 3y ago
Debezium (a platform for change data capture) supports CloudEvents as one format [1] for emitting data change events to Kafka, etc. Judging by interactions with users, there seems to be some interest in this, but it's not huge. [1] h
114.
Logical replication from Postgres 16 stand-by servers(decodable.co)
23 points by gunnarmorling 3y ago|0 comments
115.
by gunnarmorling 3y ago
I have never used the Elgato, so I can't speak to its qualities. When I was on the market for a mic though, I've found the Procaster to be on a completely different level than any USB mic I had tried (the Procaster connects via XL
116.
by gunnarmorling 3y ago
Thanks for that post, quite the journey! If you're looking for a new mic, I can recommend the Rode Procaster, it's a a dynamic mic which hardly does pick up any ambient noise whatsoever. I am very happy with it, it should be quite
117.
by gunnarmorling 3y ago
Author of the post here, so nice to see it being discussed here. For folks interested to learn more about JFR, here are a few other post on that topic: * https://www.morling.dev/blog/finding-java-thread-leaks-with-... :
118.
by gunnarmorling 3y ago
Can you expand on what those problems were you had with reconnecting? Would love to better understand this and see whether there's anything which can be improved in Debezium.
119.
Getting Started with PyFlink on Kubernetes(decodable.co)
2 points by gunnarmorling 3y ago|0 comments
120.
by gunnarmorling 3y ago
I've written about two applications of custom JFR events in the past: * Monitoring invocations of a REST API: https://www.morling.dev/blog/rest-api-monitoring-with-custom... * Tracking SQL queries issued by your O

More ›