rockwotj

7 ms·

91.
by rockwotj 2y ago
Oh if you care about durability on Kafka vs Redpanda, see https://www.redpanda.com/blog/why-fsync-is-needed-for-data-s... , acks=all does not fsync (by default before acknowledging the write), so it's still not saf
92.
by rockwotj 2y ago
Yeah be data driven and optimize the hot data path, the other code paths optimize for simplicity (and the majority of a system is the latter)
93.
by rockwotj 2y ago
When the founder started it was designed to be two things: * easy to use * more efficient and lower latency than the big resources needed for Kafka The efficiency really matters at scale and low latency yes but the simplicity of deployment
94.
by rockwotj 2y ago
Plenty of people choose Redpanda because it’s the easiest getting started experience for kafka. There is a single binary for the full broker, where I have never seen Apache Kafka as easy to setup. It’s got a great UI as well. The quickstart
95.
by rockwotj 2y ago
Redpanda is much more lean and scales much better for low latency use cases. It does a bunch of kernel bypass and zero copy mechanisms to deliver low latency. Being in C++ means it can fit into much smaller footprints than Apache Kafka for
96.
by rockwotj 2y ago
SQLites alternative in rust is a thing though https://github.com/tursodatabase/limbo
97.
by rockwotj 2y ago
The kafka protocol is a distributed write ahead log. If you want a job queue you need to build something on top of that, it’s a pretty low level primative.
98.
by rockwotj 2y ago
ANN search is usually memory bandwidth limited from a search speed prospective, so it doesn’t surprise me that the CPU has a few extra cycles to decompress without losing much latency
99.
by rockwotj 2y ago
Are you surfing the web on a toaster? I only run into network bottlenecks or pages that suck because it loads tons of ads, which isn’t a javascript issue really. Yeah its not Wasm fast, and JS has some terrible stuff from a language perspec
100.
by rockwotj 2y ago
Honestly for Java I really like Bazel. You should give it a shot. I have a project with a self contained jvm and jars from maven central. Its more explicit than the other options but way less magical IMO.
101.
by rockwotj 2y ago
It’s in the works from my understanding. I helped build the Redpanda Connect one and it’s quite easy to use
102.
by rockwotj 2y ago
BTW I think most the ecosystem has settled on https://github.com/twmb/franz-go being the best and highest performing kafka client for Golang (purego)
103.
by rockwotj 2y ago
https://github.com/artie-labs/reader is another I know of
104.
by rockwotj 2y ago
lots of Go based CDC stuff going on these days. Redpanda Connect (formerly benthos) recently added support for Postgres CDC [1] and MySQL is coming soon too [2]. 1: https://github.com/redpanda-data/connect/pull&#x2
105.
by rockwotj 2y ago
Yeah understood that the internal meta stuff would be too tied up with internal infra to be OSS, it’s the same with a lot of Google’s tooling here. I do wish there was a startup here. There is sourcegraph, which has ok code search (github h
106.
by rockwotj 2y ago
It tries to bypass the kernel as much as possible. It runs in userspace.
107.
by rockwotj 2y ago
I mean memory is completely allocated up front and fragmentation is something for application developers to deal with.
108.
by rockwotj 2y ago
RE ScyllaDB, there will absolutely be no fork and it’s very unlikely they have ever had a meaningful contribution to Scylla OSS (which is not changing, just going to bit rot and the enterprise version which was closed source was moved to so
109.
by rockwotj 2y ago
Is there any UIs for this available openly? Or for glass? I am a former Googler and I know how awesome this kind of tooling is and it’s so hard to achieve with OSS. I would love open source code search. This seems very close but there is no
110.
by rockwotj 2y ago
Swap out FAISS with usearch, you get all the awesome SIMD acceleration (via dynamic dispatch), optional compression. Not affiliated but really cool tech. https://github.com/unum-cloud/usearch
111.
by rockwotj 2y ago
I have never heard someone calling Github code reviews anything close to optimal. Here is where it falls short for me - do you not hit these? - Main view is not the code, nor can I approve from that view, so it’s always half a dozen clicks
112.
by rockwotj 2y ago
I was just going to say this, but you said it better than I would have!
113.
by rockwotj 2y ago
Everything is just assembly!
114.
by rockwotj 2y ago
I mean lambda urls have some kind of masking going om, but the docs still state it might be possible to determine the account ID: > Lambda generates the <url-id> portion of the endpoint based on a number of factors, including your
115.
by rockwotj 2y ago
I always found it interesting that bucket names are a global resource and not per account (so there would be an account ID prefix in the URL)
116.
by rockwotj 2y ago
Probably because there is no RAII semantics in JS and they don’t want to allow forgetting releasing the lock. Although the promise workaround is explicitly opting into this behavior
117.
by rockwotj 2y ago
I believe that streaming changes from SQLite is what https://sqlsync.dev/ is
118.
by rockwotj 2y ago
I worked for a short time on SearchGPT, and I can tell you OpenAI does respect robots.txt , at least when I was there and does now. They are also careful to shard per domain and only crawl each domain at a small rate (~1 qps) as to not ddos
119.
by rockwotj 2y ago
US folks, opt out here: https://www.linkedin.com/mypreferences/m/settings/data-for-a...
120.
by rockwotj 2y ago
> Firebase Realtime can be more efficient, as it can tell if a permission check has passed statically. I am not sure if Firestore works this way. Firestore's rules are also able to prove before the query runs if the query will onl

More ›