lsuresh

6 ms·

31.
by lsuresh 1y ago
LLVM optimizations are the overwhelming majority of the compilation bottleneck for us over at Feldera. We blogged about some of the challenges we faced here: https://www.feldera.com/blog/cutting-down-rust-compile-times.
32.
by lsuresh 1y ago
(lalith here) -- Whatever ryzyk and gz09 said above. :)
33.
by lsuresh 1y ago
Makes sense. We'd appreciate some more eyeballs here for sure. Between HN and a Reddit thread, there are a few hypotheses floating around. I've shared a repro here for anyone interested: https://github.com/feldera&
34.
by lsuresh 1y ago
> Now it still seems strange (as in it looks like a performance bug) that most times rust was stuck in just one threat (instead of e.g. 8). Agreed, seems like there are some rustc performance bugs at play here.
35.
by lsuresh 1y ago
Yes. We found both cold and incremental builds sped up. The incremental builds were the main win -- small changes to the SQL can sometimes complete in seconds for what used to be a full recompilation.
36.
by lsuresh 1y ago
For any Rust compiler experts interested in taking a look, I've put together a short repro here: https://github.com/feldera/feldera/issues/3882 It will give you a workspace with a bunch of crates that s
37.
by lsuresh 1y ago
The L3 cache angle is one of our hypotheses too. But it doesn't seem like we can do much about it.
38.
by lsuresh 1y ago
We think a JIT compiler is the right approach too. Will be a substantial effort though, so we're waiting to get a bit of bandwidth on that front.
39.
by lsuresh 1y ago
Towards the end, the article says it takes 7s for linking using mold.
40.
by lsuresh 1y ago
That's correct. These aren't external dependencies but a dataflow graph being split into crates.
41.
by lsuresh 2y ago
Thanks! We already use self-hosted runners on physical machines with NVMe drives that we assembled ourselves. I was wondering if there's something else you're doing for the caching.
42.
by lsuresh 2y ago
This sounds promising. What made your Rust builds become that fast? Any repo you could point us to?
43.
by lsuresh 2y ago
This is exactly a big piece of our frustration -- the terrible feedback loop and how much mental space it wastes. OP does talk about this at the end (babysitting the endless "wip" commits till something works).
44.
by lsuresh 2y ago
Related: Verus (verified Rust): https://github.com/verus-lang/verus It's aimed more at full systems verification (been used to build verified filesystems, kubernetes controllers etc...).
45.
by lsuresh 2y ago
Nice to see this here! There's been an interesting discussion on r/rust too: https://www.reddit.com/r/rust/comments/1i2ttwj/billion_cell_...
46.
by lsuresh 2y ago
Happy to see this gem shared here. I've learnt a lot about the JVM going through these. This article about the "stack allocation" misnomer in Java in particular is one of my favorites: https://shipilev.net/jvm
47.
by lsuresh 2y ago
Yeah you definitely need something like Feldera: https://github.com/feldera/feldera
48.
How to Analyze Unbounded Time-Series Data Using Bounded State(feldera.com)
4 points by lsuresh 2y ago|0 comments
49.
by lsuresh 2y ago
Yes, you can write Rust UDFs with Feldera and even use the dbsp crate directly if you'd like.
50.
by lsuresh 2y ago
Your mental model is spot on and described quite well here: https://current.confluent.io/2024-sessions/streaming-queries...
51.
by lsuresh 2y ago
hi YmiYugy. We have a REST API to subscribe to a change stream (basically what the UI uses underneath). Our ad-hoc query interface is about to receive snapshot-and-follow support as well which should do what you want. That way, you don&#x27
52.
by lsuresh 2y ago
They're quite different from DBSP. Given a program execution, Salsa/adapton seem to reuse prior steps from an execution when the input changes. In contrast, DBSP has built-in knowledge of incremental versions of operations and com
53.
by lsuresh 2y ago
We have a small community over at the Feldera Slack channel you could join. We often have have deeply technical discussions about the incremental computation space, papers, concepts etc over there.
54.
by lsuresh 2y ago
This is our paper describing the theory underlying Feldera: https://www.vldb.org/pvldb/vol16/p1601-budiu.pdf
55.
by lsuresh 2y ago
Great question. DBSP should work here -- spreadsheets are by definition incremental (and there's even recursive queries there with cells depending on each other). Note that we use Z-Sets to bridge SQL/tables with DBSP, but Z-Sets
56.
by lsuresh 2y ago
Yes, we do a lot of work with monotonicity detection. It's central to we perform automatic garbage collection based on lateness.
57.
by lsuresh 2y ago
We use monotonicity detection for various things. I believe (can double check) that it's used for max as well. But you're correct that in the general case, max is non-linear, so will need to maintain state. Update from Leonid on c
58.
by lsuresh 2y ago
Thank you jacques_chester! Piping all that credit to my co-founders Mihai and Leonid, the key inventors.
59.
by lsuresh 2y ago
The previous implementation we built at VMware went from datalog -> Rust, and we supported other language bindings using C bindings and FFI. The same ought to work here too.
60.
by lsuresh 2y ago
Thanks for following our journey! There's still room for more language frontends if you'd like to contribute. :)

More ›