necubi

12 ms·

121.
by necubi 2y ago
Many companies have 100k+ of lines of Spark code. It's not trivial to rewrite all of that in another query framework.
122.
by necubi 2y ago
Futhermore, Postgres is an OLTP (transactional) database, designed to efficiently perform updates and deletes on individual rows. OLAP (analytical) databases/query engines like Clickhouse, Presto, Druid, etc. are designed for efficient
123.
Koto 0.14 released – scripting for Rust appplications(koto.dev)
3 points by necubi 2y ago|0 comments
124.
by necubi 3y ago
I’m not sure why you’d say that we’re “moving towards” this sort of build system complexity. This is 1990s autoconf bs that has not yet been excised from the Linux ecosystem. Every modern build system, even the really obtuse ones, are less
125.
by necubi 3y ago
I see so many comments these days bemoaning how slow modern software has gotten, but no one seems to remember/have been alive for the time when just rendering an image would take multiple seconds. Just goes to show that our expectation
126.
by necubi 3y ago
My understanding is that hacspec is the language (a subset of rust), and hax is the tool that compiles it into format proof languages like Coq. Some explanation of the name changes here: https://hacspec.org/blog/posts&#
127.
by necubi 3y ago
There's no PR yet (and I won't link to the branch in case the contributor doesn't want it public yet) but it exists and is being run in production :)
128.
by necubi 3y ago
Currently only Rust UDFs are supported ( https://doc.arroyo.dev/sql/udfs ) but one of the things that Arrow should enable is performant Python integration, as Arrow has a standardized in-memory format that's portabl
129.
by necubi 3y ago
A NATS and Jetstream connector is in development by an Arroyo user, and hopefully will be merged into master soon!
130.
by necubi 3y ago
That's a future direction we're very excited about, particularly being able to run pyarrow-based UDFs on Arroyo state without any serialization overhead.
131.
by necubi 3y ago
> SQL streaming engines really seem to be having a moment. I definitely agree! In the past few years, a bunch of folks (including myself) who had been working with Flink/Spark Streaming/KSQL/etc. at large companies decided
132.
Building a streaming SQL engine with Arrow and DataFusion(arroyo.dev)
112 points by necubi 3y ago|30 comments
133.
by necubi 3y ago
This is so, so cool. Basically the holy grail as a distributed systems engineer. Like the author, I've also avidly consumed every Jepsen report but the effort of actually implementing Jepsen tests for my systems always seemed too high.
134.
by necubi 3y ago
This wasn't a big acquisition, it was an acquihire. The company essentially failed. They raised ~$40M, and it's unlikely they sold for that much. Investors will get some of their money back, and the remaining founder will get some
135.
by necubi 3y ago
One of the coolest things about the hat tile is that it was discovered by a hobbyist playing with puzzle software. There's a great Quanta story about this: https://www.quantamagazine.org/hobbyist-finds-maths-elusive-...
136.
by necubi 3y ago
We have one very slow job (our Rust CI, for example: https://github.com/ArroyoSystems/arroyo/actions/runs/7702793... ) and a bunch of little jobs that take a few seconds (checking lints, etc.). We never b
137.
by necubi 3y ago
In this case, I don't think the issue is due to filesystem performance. Someone from Ubicloud can correct me, but my understanding is that for custom runners Github still stores the cache on their side. So Ubicloud (in Europe) needs to
138.
by necubi 3y ago
We've been using Ubicloud builders for our Rust project [0] for several months, and it's worked very well. We've seen CI times go from 10-15 minutes to 6-7, and our bill has gone from $300/month to $30. One counter-intui
139.
Why Stateful Stream Processing(arroyo.dev)
2 points by necubi 3y ago|0 comments
140.
by necubi 3y ago
There are a few options here, but I agree this is a weakness with existing systems. One option in Flink is to load the entire fact table into the pipeline (using the filesystem source or a custom operator) and join against that. This provid
141.
by necubi 3y ago
Yep. There are always going to be constraints about how well a system like clickhouse can support arbitrary joins. Queries in clickhouse are fast because the data is laid out in such a way that it can minimize how much it needs to read. Par
142.
by necubi 3y ago
This was in 1999. C++ compilers have come a long ways since then. While there are still opportunities for hand-written asm to go and order of magnitude faster than C++, they're mostly around manual vectorization where the auto-vectoriz
143.
by necubi 3y ago
As an open-source maintainer, the simple answer is because that's what the majority of users want. The barrier to join a discord server and ask a question is very low, compared to signing up for a forum, posting, and hoping that somebo
144.
by necubi 3y ago
Indeed, as someone who maintains a helm package it's mind-boggling. When I've been able to build k8s tooling from scratch, I've been reasonably happy with jsonnet [0], which is a constrained programming language designed for
145.
by necubi 3y ago
Unfortunately JSON numbers are 64 bit floats, so if you're standards compliant you have to treat them as such, which gives you 53 bits of precision for integers. Also hey, been a while ;) Edit: I stand corrected, the latest spec (rfc82
146.
by necubi 3y ago
Thanks for your work on this! Are there plans to improve the compilation times? Aws sdk crates are some of the slowest dependencies in our build—which feels odd for what are basically wrappers for http clients.
147.
by necubi 3y ago
Not an attorney, but as someone with a startup incorporated in DE—it's just the default. Delaware has very good and well-understood corporate law and a judicial system that is able to handle complex corporate cases. If you're fund
148.
Can you replace Prometheus with a stream processor?(arroyo.dev)
2 points by necubi 3y ago|0 comments
149.
by necubi 3y ago
And yet today Materialize is distributed ;)
150.
The semantics of streaming SQL(arroyo.dev)
2 points by necubi 3y ago|0 comments

More ›