enether

6 ms·

61.
by enether 11mo ago
The question is the organizational overhead in adopting yet another specialized distributed system, which btw frequently is about scalability at its core. Kafka's original paper emphasizes this ("We introduce Kafka, a distributed
62.
by enether 11mo ago
I’m making the example of a pub sub system. I’m most familiar with Kafka so drew parallels to it. I didn’t actually implement everything Kafka offers - just two simple pub sub like queries.
63.
by enether 11mo ago
It would be nice if a library like pgmq got built. Not sure what the demand for that is, but it feels like there may be a niche
64.
Kafka is Fast – I'll use Postgres(topicpartition.io)
561 points by enether 11mo ago|401 comments
65.
by enether 11mo ago
weird you have to adopt Kafka AND Vector just to batch a bit of writes into Clickhouse...
66.
Confluent Explores Sale(reuters.com)
1 points by enether 1y ago|0 comments
67.
by enether 1y ago
!!! Thanks for calling that out. And sorry for falling prey to the numbers I saw (from the AWS talk) And for the 1/2 vs 1/3rd - I'm just dumb. Thanks again. Super cool paper too
68.
Sonnet 4.5 ranks #25 (below other Claude models) in generating SQL(tinybird.co)
2 points by enether 1y ago|1 comments
69.
by enether 1y ago
> We assessed the ability of popular LLMs to generate accurate and efficient SQL from natural language prompts using a 200 million record dataset from the GitHub Archive. The original link I posted ended with `#llm` referring to the exac
70.
by enether 1y ago
See timestamp 42:20 at https://youtu.be/NXehLy7IiPM?si=QQEOMCt7kOBTMaGK The way it’s worded makes me understand that’s what scheme they’re using. Curious to hear what you know
71.
by enether 1y ago
Author of the 2minutestreaming blog here. Good point! I'll add this as a reference at the end. I loved that piece. My goal was to be more concise and focus on the HDD aspect
72.
by enether 1y ago
It's inherently a chicken and egg problem. If HackerNews didn't exist and the Nostr community created it - it'd be filled with the same content. Network effects are everything. The tech can be good but the product may not be
73.
by enether 1y ago
It's still pretty affordable and not-hard to run your own Lightning node; The pseudo-bank hosted wallets people use (e.g Wallet of Satoshi) is purely out of convenience. The real lesson is that most people don't care enough about
74.
by enether 1y ago
If there is a big enough market for 1), shouldn't it exist? The problem in my eyes seems to be that there isn't enough capital interested to sufficiently fund 1) to compete and create a comparable product. Thus, at best, we end up
75.
Why Kafka and Iceberg Will Define the Next Decade of Data Infrastructure(blog.streambased.io)
4 points by enether 1y ago|0 comments
76.
Iceberg Topics for Kafka(aiven.io)
2 points by enether 1y ago|0 comments
77.
by enether 1y ago
I would assume storage varies greatly. I know that LinkedIn quoted an average read fanout ratio of 5.5x in Kafka, meaning each byte was read 5.5x times. Assuming that is still true, we ought to divide by 6.5x to get to the daily write amoun
78.
by enether 1y ago
It's neither. Only the thumbnail background is AI generated. The look of the page -- that's Substack's default UI, you can't control it too much. The other images are created by me. I'm simply curious what parts giv
79.
by enether 1y ago
~197 GB/s ... nice. I believe these companies save literally every ounce of data they can find. Once you have the infra and teams for it, it seems easy to make a case for storing something. Similarly, Uber has shared they push 89 GB&#x
80.
by enether 1y ago
Confluent has shared they've migrated thousands of Kafka clusters (their whole cloud fleet) to KIP-500 https://www.confluent.io/blog/zookeeper-to-kraft-with-conflu...
81.
by enether 1y ago
Where does this 17 PB/day number come from? I didn't quote any numbers directly. Looking at the 2012 paper, it implies a 1.35TB per day (they store 9.5TB across all topics at 7d retention)
82.
by enether 1y ago
Love the story! > Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was a big step forward for web applications at scale. Could you expand on this - when does it drop C? Are you referrin
83.
by enether 1y ago
What do you mean by "generated ripoff"? Are you saying it read like AI?
84.
Why was Apache Kafka created?(bigdata.2minutestreaming.com)
201 points by enether 1y ago|221 comments
85.
by enether 1y ago
Yes, see WarpStream, Confluent Freight and Diskless Kafka for other examples
86.
by enether 1y ago
OpenAI seems to be a heavy user of Kafka, in their last Kafka conference - Confluent Current (previously called Kafka Summit) they shared their Kafka usage grew... 20x in the last year[1][2] I watched the talk live. They didn't exactly
87.
by enether 1y ago
+1! It may not be sexy, but its lindy and worthwhile the time investment. Likely to last 10-20 more years
88.
I Lowered the CO2 in My House(christian.gen.co)
3 points by enether 1y ago|0 comments
89.
by enether 1y ago
On the subreddit /r/ApacheKafka and LinkedIn mainly. We lack a single forum - the subreddit is the closest we have to it. The space is pretty small and we all pretty much know each other (lots of ex CFLT)
90.
by enether 1y ago
Why would you call it a gold standard. It's just extending the existing dollar standard.

More ›