necubi

16 ms·

151.
by necubi 3y ago
I'm with Arroyo [0] — thanks for the mention! I'd be interested to see someone from Epsio chime in with where exactly they're positioning, but you're right that this is (recently) a very crowded space. I think you can so
152.
by necubi 3y ago
No locking (which provides mutable `Sync` in rust terms) is generally required, but things must be Send (i.e., allowed to move between threads, but not shared). In my experience is mostly an issue when dealing with references, because `&amp
153.
Why we chose Rust for data infra(arroyo.dev)
4 points by necubi 3y ago|0 comments
154.
KIP-932: Queues for Kafka(cwiki.apache.org)
77 points by necubi 3y ago|21 comments
155.
by necubi 3y ago
I did YC in w23 which was a hybrid batch (talks and office hours on zoom with weekly SF events). Speaking for myself, most of the value was in the in-person events and not the zoom talks. Talking to alumni from the fully remote batches, it&
156.
by necubi 3y ago
This is endlessly debated within the community (e.g., https://internals.rust-lang.org/t/blog-post-no-namespaces-in... ), and there's no consensus on whether namespaces would be an improvement
157.
by necubi 3y ago
As someone building my startup [0] on Rust (and a fan of the language since ~2014) I don't think this is a very compelling list of points. > Rust is overhyped People are excited about Rust! Not sure why this is a reason you _shouldn
158.
by necubi 3y ago
That's incorrect, the Send bounds are only there in the multithreaded runtime (because it might need to Send a task across threads). The single threaded runtime will never do that, so its futures don't need to be Send. Somewhat re
159.
Streaming data to S3 is surprisingly hard(arroyo.dev)
2 points by necubi 3y ago|0 comments
160.
by necubi 3y ago
I use a projector instead of a TV. Home theater projectors are still mostly sane devices that just display their inputs without any bloated software in the middle. And with a retractable screen you have a (giant) TV when you want one, and i
161.
by necubi 3y ago
It must be valuation (if there's a SaaS company that's ever gone from $0 -> $100MM ARR -> $0 within five years I would be very surprised). But that makes this whole thing pretty circular—they only would have $100MM valuation
162.
by necubi 3y ago
Do you still provide low-level control over partition subscriptions and offset management? Any plan to support Kafka transactions? That's all required to build exactly-once systems on top of Kafka (like the stateful stream processing e
163.
by necubi 3y ago
For me it's very helpful to print out papers and read them with a pen in hand, away from my computer. Papers tend to be dense and require a level of focus that (I at least) cannot maintain when reading on a screen. It helps as well to
164.
by necubi 3y ago
Congratulations! Exciting to see more companies in the Rust data space.
165.
by necubi 3y ago
Thanks for that link! We've been heavily inspired by the work out of TU Berlin on Scotty ( https://tu-berlin-dima.github.io/scotty-window-processor/ ) but exciting to see more work on this problem. Our efficient win
166.
by necubi 3y ago
This is an interesting point. In Arroyo [0] (the streaming engine I work on) we also use a simple TCP-based backpressure mechanism: we have NxM TCP connections between each operator subtask, behind small in-memory queues. This essentially i
167.
by necubi 3y ago
Yeah, you definitely need backpressure, and doing that well is had. There's a great blog post about how Flink manages this [0] and it involves a complex credit-based flow control system. But in practice the performance advantages of al
168.
by necubi 3y ago
Another advantage to push is that it can be pretty straightforwardly distributed—just replace the "push" mechanism with a queue connected to a network stack, and now your operators can run on different machines. In a batch context
169.
by necubi 3y ago
Back in the day if you wanted to develop for PalmOS you had to pay Metroworks (IIRC) $~300 for their CodeWarrior C++ compiler, so the fact that XCode is free is a pretty big improvement.
170.
by necubi 3y ago
The trend isn't away from drinking, it's away from beer. Seltzers and other malt-based drinks have had huge growth in the past few years, mostly at the expense of beer sales, and liquor sales are up as well. Slate's been doin
171.
by necubi 3y ago
> Letsencrypt's users are almost all in the personal/hobbyist space That may have been true in the early days of Let's encrypt. But today, if you're using any of a number of newer hosting options (Netlify, Vercel, Fly
172.
by necubi 3y ago
Internationally Disney+ gets most of the content that's shown on Hulu in the US (as Hulu is US-only).
173.
by necubi 3y ago
The comedies are so much better performed than on the page. I find the humor gets a bit lost in the archaic language, but a good cast is able to bring it for modern audiences.
174.
by necubi 3y ago
There's a spectrum of streaming systems from simple, stateless processors to complex, stateful distributed processors. I'm not very familiar with Benthos, but it appears to be on the simple, stateless end. Systems like that can be
175.
by necubi 3y ago
Yep! In California where we live, it typically refers to a seasonal desert stream that can range from a trickle to a torrent. We chose it because Arroyo is a stream processor, and it's very good at autoscaling.
176.
by necubi 3y ago
That'd be great! We have versions of most of the nexmark queries and some internal benchmark vs Flink, we'd love to help if you're interested in benchmarking against more systems. Reach out at micah@arroyo.systems.
177.
by necubi 3y ago
We're big fans of Nomad! It's what we use for scheduling in our cloud platform, due to its simplicity and scheduling speed. Although we also have great support for Kubernetes as that's what most folks will be running.
178.
by necubi 3y ago
Yes, that's mostly true. Arroyo is a stream processor like Flink. In Arroyo, you pre-register the queries you are interested in, and they will be compiled into streaming dataflow jobs that continuously execute as events come in. This m
179.
by necubi 3y ago
Yes, Arroyo is an alternative to KSQL, although more in the design-space of Flink. KSQL is pretty simple and easy to run if you already have Kafka, but will be much more expensive and harder to scale due to its reliance on Kafka streams for
180.
by necubi 3y ago
Yeah, currently late-arriving data is dropped but we will be making this more configurable. We're currently working on what we call "update tables," which means being able to emit incremental changes to state as well as final

More ›