benesch

11 ms·

61.
by benesch 7y ago
To be clear, window functions are definitely on our radar! But you’d be surprised how many folks are delighted just to have more basic SQL features, like fully functional joins. Streaming SQL is surprisingly far behind batch SQL.
62.
by benesch 7y ago
We’re thinking along very similar lines! We’ve got some of our thoughts around evolving how timestamps work in Materialize written down here: https://github.com/MaterializeInc/materialize/issues/1309
63.
by benesch 7y ago
I forgot to mention: differential dataflow is capable of providing much stronger consistency guarantees than Noria is. Differential dataflow is consistency preserving—if your source of truth provides strict serializability, differential can
64.
by benesch 7y ago
Indeed, Materialize is quite similar to Noria, and has the Frank McSherry stamp of awesomeness. [0] We know many of the Noria folks and have a lot of respect for them and their work. I also worked on the Noria project for a summer in colleg
65.
by benesch 7y ago
It’s not, actually. Noria has its own custom dataflow engine.
66.
by benesch 7y ago
I agree wholeheartedly with your take! Window functions are a particular favorite of mine, but we haven’t seen much customer demand for them yet, so they haven’t been officially scheduled on the roadmap. They require some finesse to support
67.
by benesch 7y ago
Similar in concept, but much more powerful in execution. We can incrementally materialize practically any SQL 92 query, with the killer features being joins and correlated subqueries. I’m not super familiar, but TimescaleDB’s continuous agg
68.
by benesch 7y ago
Considerations are completely different in a streaming context. It’s not so much about how fast you can churn through terabytes of data; it’s more about how quickly you can turn around the incremental computation with each new datum. There’
69.
by benesch 7y ago
I’m glad to hear it! One of my favorite ways to view Materialize is as bringing differential dataflow to the masses. Differential dataflow is deep, elegant stuff, but it requires some serious CS chops to grok. SQL is lacking in elegance but
70.
by benesch 7y ago
> I just want to say this is a very dangerous assumption to make. I think we're actually arguing the same points here. It's not that every use case needs single-digit millisecond latencies! There are plenty of use cases that
71.
by benesch 7y ago
Yes, absolutely. Entire dissertations have been written on the topic: https://dash.harvard.edu/bitstream/handle/1/14226097/KATE-DI... Efficiently maintaining views over arbitrarily complex computations i
72.
by benesch 7y ago
> Would it be fair to say this is a more OLAP-oriented approach to what KSqlDB (not KSql, but https://ksqldb.io/ ) does? I'm not sure I'd say it's "more OLAP." ksqlDB is about as OLAP as it gets,
73.
by benesch 7y ago
To add a bit more context to Frank's answer: adding a column to a table will work just fine today, but the data in that column won't propagate through to Materialize. You'll continue to see the latest inserts, updates, and de
74.
by benesch 7y ago
Each combinator knows how to produce the output diff from an input diff; the computation is then just a matter of stringing these combinators together. For example, consider a map combinator. The output diff is just the input diff with a fu
75.
by benesch 7y ago
Frank’s written a tutorial style mdbook on Differential [0] that I highly recommend in addition to his blog posts. There’s also the research paper on the spawned the Timely and Differential projects, Naiad [1]. If you get into it, the libra
76.
by benesch 7y ago
Any program that uses a logging framework, including the stdlib log package, will wind up depending on runtime.Callers at least transitively. That’s probably most Go programs; certainly most of the programs large enough to be worrying about
77.
by benesch 7y ago
The Go language literally requires that pclntab be included in release builds. I'm with you—it seems kind of crazy that this was designed into the language—but there you have it. The reason is that Go's standard library provides f
78.
by benesch 7y ago
No, it's an official Google project. Here is the official Google announcement by a Google employee: https://groups.google.com/forum/#!topic/golang-announce/OW8b...
79.
by benesch 7y ago
This is... strange? It seems go.dev is a enterprisey marketing hub (with "Solutions" and "Case Studies" and "Testimonials"), while golang.org remains the pure, text-based, engineer-focused home of the language
80.
by benesch 7y ago
Thanks for the thoughtful response. I don't mean to imply that there's a conspiracy here. What I mean is that I think the Plan 9 heritage is clouding strategic decisions around the Go toolchain. What may have been the right decisi
81.
by benesch 7y ago
> Remember there already is a mature C compiler alternative for Go: gccgo. There's also already a first-party LLVM based Go toolchain created by Google (gollvm). Whatever hypothetical benefits you might presume would emerge from thi
82.
by benesch 7y ago
I invoked the personal experience only as evidence that I did what (little) I could to push along the project that I think has the most long term potential. I'm not criticizing without at least trying to do what I can. In short, the re
83.
by benesch 7y ago
> You agree and then again go completely sideways by unreasonably expecting that Go team to contribute to stack they neither claim to be expert on, nor particularly like it. Sorry, but I think you’re still missing my point. It’s well doc
84.
by benesch 7y ago
I spent six weeks, unpaid, at the Recurse Center this year hacking on gccgo to see if I could improve the situation. I didn't get very far—it's hard! And I don't have much experience with compilers. So, truly, I did as much a
85.
by benesch 7y ago
> being able to throw money and people at LLVM development doesn't explain why it's still much slower than the Go compiler toolchain. Right, you need to specifically throw money and people at the problem of making LLVM faster,
86.
by benesch 7y ago
Let me refine my point a bit. Apologies; my original post had gotten a bit long. I agree that enthusiasm is important! And indeed, for the Go creators, their particular leanings might have been such that they couldn't get excited about
87.
by benesch 7y ago
What I would give for the developers of the Go toolchain to have spent the last decade improving GCC or LLVM instead of their own bespoke toolchain. In many ways Go seems like an excuse for Google to fund the continued development of Plan 9
88.
by benesch 7y ago
ZooKeeper uses a custom atomic broadcast protocol, Zab, not Paxos: https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab+vs... (Atomic broadcast can be reduced to consensus, but frames the problem differe
89.
by benesch 7y ago
> Should it not be the case that one of the responsibilities of the quorum is to vote new members in or out? I mean, as a first-class feature of the system. Indeed. What you're describing is one of the main motivations for Raft. Pax
90.
by benesch 7y ago
> It affects performance too -- SQL wants storage locality by table, but applications often want locality by user. This problem gets substantially worse in geographically distributed databases, like Spanner or CockroachDB, where joining

More ›