jgraettinger1

10 ms·

61.
by jgraettinger1 4y ago
Here's [1] a particularly crazy one which offers synchronous, low-latency (millis) replication of a SQLite database to a distributed recovery log. It does this by instrumenting the file operations SQLite is making and recording their e
62.
by jgraettinger1 4y ago
IIRC there were two reasons for this. One was an information-leak concern whose details I don't fully remember. The other is flow control. A server push doesn't automatically send the entire object. It opens a new stream and fills
63.
by jgraettinger1 4y ago
Keep in mind that the HTTP/2 standardization process really started from SPDY [1]. Before HTTP/2 the standard hadn't changed significantly in _decades_ due to lack of industry consensus and actual data. Google then used its u
64.
by jgraettinger1 4y ago
Hi, I’m the primary author of gazette. Interesting seeing it here, happy to answer questions. We’re continuing to improve it - it’s a core implementation detail of our current project & company Estuary Flow ( https://estuary.d
65.
by jgraettinger1 5y ago
> us invaded afganistan to protect itself ... is a very different thing than ... > occupying a foreign country for 20 years When NATO invaded the towers were still smoking, the war-on-terror intelligence apparatus had yet to be built,
66.
by jgraettinger1 5y ago
> I'd love to be able to have a "relational database" that is also some "NoSQL" databases (since there's a few major useful paradigms there) that just takes care of this for me. I imagine I'd have to de
67.
by jgraettinger1 5y ago
We've used https://github.com/estuary/connectors/pkgs/container/source-... to load data sets in the many terabytes. Caveat that, while it's implemented to Airbyte's spec, we've only u
68.
by jgraettinger1 5y ago
Our own approach is to keep singer.io et al and JSON (flexible! streaming capable!), but deeply invest in JSON Schema to a) statically infer and map into other kinds of schema on your behalf -- TypeScript, Elastic Search, and SQL DDL so far
69.
by jgraettinger1 5y ago
Not for Kafka, but we are building Flow [1] to offer deterministic ordering, even across multiple logical and physical partitions, and regardless of whether you're back-filling over history or processing in real-time. This ends up bein
70.
by jgraettinger1 5y ago
I can't speak to their solution, but when solving an equivalent problem within Gazette, where you desire a distributed transaction that includes both a) published downstream messages, and b) state mutations in a DB, the solution is to
71.
by jgraettinger1 5y ago
If you're in the Go ecosystem, Gazette [0] offers transactional integrations [1] with remote DB's for stateful processing pipelines, as well as local stores for embedded in-process state management. It also natively stores data as
72.
by jgraettinger1 5y ago
A similar strategy we [1] are pursuing is to use `reduce` JSON Schema annotations to make it easy to define generalized reductions over your JSON data types [2]. Taking the examples from the article, equivalent schemas might be: type: o
73.
by jgraettinger1 5y ago
A big difference is the removal of windowing: Flink lets you aggregate or join events only so long as they arrive in the same temporally-bound window. You're required to have a window, and it's core to the semantics of your workfl
74.
by jgraettinger1 5y ago
Yep. You can, for example, have it watch file(s) in S3, and every time a file changes it will flow its records through into a table it creates in your DB, keyed on your (arbitrary) primary key.
75.
by jgraettinger1 5y ago
Estuary Flow [1] may be interesting to those in this space. We're still building, but it's a GitOps workflow tool that tightly integrates schema definition (JSON Schema), captures and materializations from/to your systems &am
76.
Event Streaming Like It's 1978(github.com)
5 points by jgraettinger1 5y ago|0 comments
77.
by jgraettinger1 5y ago
Can you speak more to how updates get pushed out to your UI clients? Are you using Firestore ? Hasura? Polling? Something we're looking at enabling w/ Estuary Flow [1] is materializing to Firestore for this use case (and Flow alre
78.
by jgraettinger1 5y ago
I'm unclear on how this fully helps, though? As I understand it, it unblocks a dataflow to provide answers with respect to a user-time processed as-of a system-time, which is certainly an improvement. But you'd also have unbounded
79.
Filling a Paradigm-Shaped Hole(github.com)
13 points by jgraettinger1 5y ago|0 comments
80.
by jgraettinger1 5y ago
Thank you for this work! Interesting results. There's an aspect of the setup which troubles me: You're building a view over transaction data in order to arrive at current balances. You configure differential dataflow to reject tim
81.
by jgraettinger1 6y ago
In my twenties, I had an amazing ability to run at a problem. Long periods of great focus. The only trouble was how frequently I was running in the wrong direction. The gift of aging for me (late 30's) has been the appreciation that an
82.
by jgraettinger1 6y ago
> My opinion is that FSMs are best implemented on a higher abstraction layer than directly in code itself. I really love Rust async/await for this. Under the hood, the compiler is simply assembling a big FSM, and polling a Future is
83.
by jgraettinger1 6y ago
You write pure-function mappers using any imperative language. We're targeting TypeScript initially for $reasons, followed by ergonomic OpenAPI / WASM support, but anything that can support JSON => JSON over HTTP will do. It&#x
84.
by jgraettinger1 6y ago
Exactly. You can have the amortization benefits of batching without adding extra latency, if you do optimistic pipelining. It's super powerful if your transaction sizes aren't linear, e.g. because you're doing in-memory aggre
85.
by jgraettinger1 6y ago
> I feel like so many of the concepts and patterns I learned about there are being re-expressed today with modern, distributed data tooling So much this. I work on Flow [0] at Estuary [1]; you might be interested in what we're doing
86.
by jgraettinger1 6y ago
Here's another approach to the problem [0]: This package is part of Gazette [1], and uses a gazette journal (known as a "recovery log") to power raw bytestream replication & persistence. On top of journals, there's a
87.
by jgraettinger1 6y ago
We're building a tool, Estuary Flow, which seeks to be an end-to-end realization of practical, configuration driven, and scale-out flow-based programming -- with an important twist. The central concept is a "collection", an a
88.
by jgraettinger1 6y ago
Funny, the package in question exists because _I_ thought I could do better and wanted to DIY.
89.
by jgraettinger1 6y ago
Slightly different trade-offs. This package is emphatically just "for" Etcd, choosing to directly expose MVCC types & concepts from the client. It also doesn't wrap transactions -- you use the etcd client directly for tha
90.
by jgraettinger1 6y ago
I think the "database" label is tripping up the conversation here. What's being talked about here, really, is fast & HA coordination over a (relatively) small amount of shared state by multiple actors within a distributed

More ›