frankmcsherry

8 ms·

31.
by frankmcsherry 6y ago
Your example join output at https://imply.io/post/introducing-apache-druid-0-18-0 has the incorrect answers for the join.
32.
by frankmcsherry 6y ago
> Yeah, I liked how he realized two possibilities for a number that were next to each other, also meant the box around it could be cleared. This struck me as a bit weird (as he's clearly 100% sharp): this is just the application of
33.
by frankmcsherry 6y ago
Yes, I know what it will be, but it isn't really acknowledging the lack of data, is it? It turns NULL input data into a 100% false result.
34.
by frankmcsherry 6y ago
Let's check out SQL's "acknowledged lack of data" in a case statement: create table test(a bool); insert into test(a) values (NULL); select case a when a then true else false end from test; The answer sh
35.
by frankmcsherry 7y ago
Haskell.
36.
by frankmcsherry 7y ago
I don't think it's about giving up, but accepting that churn has costs. Of the first 15 or so Rust releases, 4 of them caused library code I maintain (timely dataflow) to publicly break. They were all minor "paper cut" b
37.
Taking Materialize for a spin on NYC taxi data(materialize.io)
5 points by frankmcsherry 7y ago|0 comments
38.
by frankmcsherry 7y ago
Hi, I work at Materialize! It's early days, but I can confirm that it's a bit hard to know out of the gates whether Materialize can handle such workloads. In particular, unless I missed it the query workload isn't really disc
39.
by frankmcsherry 7y ago
In the event that it helps anyone, the following is how I think of Rust ownership stuff: 1. Ownership-based memory management is statically elided reference counting. 2. Borrowing (shared and mutable) is statically elided reader-writer lock
40.
by frankmcsherry 7y ago
The underlying compute framework, differential dataflow, supports multi-temporal timestamps. The Crux folks were at one point looking at it for Juxt, though not sure what they concluded.
41.
by frankmcsherry 7y ago
DDLs that alter schemas definitely cause things to catch on fire at the moment (Debezium writes them to a side-topic, which isn't obviously synchronized with the data topics). In principle `CREATE` and `DROP` statements are things we c
42.
by frankmcsherry 7y ago
Yes, the term we're using is "Sink", dual to a Source. https://materialize.io/docs/sql/create-sink/ At the moment you can create Kafka sinks, and we'll write the change log out to t
43.
by frankmcsherry 7y ago
It looks like a caching bug somewhere in the web pipeline. It isn't running on Materialize, sadly. There are a pile of known limitations, and our goal is to be 100% clear about them.
44.
by frankmcsherry 7y ago
The main distinction that led to a different name is that (unlike most instances of dynamic or incremental algorithms) differential dataflow allows you to move through a partial order of updates, rather than a sequence of updates. This is i
45.
by frankmcsherry 7y ago
Depending on what you are looking for, DDlog ( https://github.com/vmware/differential-datalog ) is interesting. I'm casually involved, for people weirded out by that, but it supports arbitrary changes to the input f
46.
by frankmcsherry 7y ago
I think you've misread the paper, which is about differentially private RAM. Differential privacy (w/o "RAM") is neither weaker nor stronger than ORAM.
47.
by frankmcsherry 7y ago
I googled "kafka documention", went to https://kafka.apache.org/documentation/ , searched for "queue" and found 29 matches.
48.
by frankmcsherry 7y ago
Absolutely! I read the von Neumann extractor and presumed too much. For fun, you can reduce the numbers asked by getting a batch of k people, and if all numbers are dissimilar you get a draw from k! The OP is silly though; if you know the d
49.
by frankmcsherry 7y ago
Rejection sampling isn't the best way to go. With b bits you are subdividing the (0,1) interval into 2^b regions, and only need more bits if one of the 9 multiples of 0.1 land in the interval you've picked. As b increases this pro
50.
by frankmcsherry 7y ago
Nonsense. It requires roughly 1+log(10) bits in expectation and has a geometric tail.
51.
by frankmcsherry 8y ago
> Nitpick, but you can only sort in linear time if your sort key domain is bounded (e.g. 64-bit integers). It is more subtle than this. Top-down radix sort (which discrimination generalizes) is linear time in the number of input bytes, r
52.
by frankmcsherry 8y ago
<3
53.
by frankmcsherry 8y ago
Moar links! (caveat: I am involved in many things) 1. You can implement some minor flavor of Doop in differential dataflow ( https://github.com/TimelyDataflow/differential-dataflow/tree... ). The fragment came from
54.
by frankmcsherry 8y ago
No worries! Niko's talk should probably be the reference material, but you definitely can deploy new queries interactively, against maintained indexed representations of the historical streams. No recompilation required. Streams are
55.
by frankmcsherry 8y ago
For some context on "what Niko presented at the Conj", https://2018.clojure-conj.org/nikolas-gobel/ which is a reactive Datalog implementation built over differential dataflow. The .. um reported anxie
56.
by frankmcsherry 8y ago
> What am I missing? That it is a multi-value map? Could be more too; could also be just one part in a larger system and not everything is a race.
57.
by frankmcsherry 8y ago
If you want to check out more about high-throughput incremental view maintenance in a Datomic-like system, here's a decent read (caveat: I'm involved): https://www.nikolasgoebel.com/2018/09/13/increm
58.
by frankmcsherry 8y ago
I just watched this, and it was abundantly clear that the presenter believes there is a scaling issue ("scaling these propulsion methods is not easy" at 6:37). They literally say this, and then present the 10,000N/m^2 vs 3N&#
59.
by frankmcsherry 8y ago
> Something I'd like to see is benchmarks of batch-query performance: If we can queue a couple of queries, then trees should gain a lot from processing them in-order (then, the last query has already warmed the cache for the next; t
60.
by frankmcsherry 8y ago
Absolutely worth watching. Cannot un-watch.

More ›