sewen

8 ms·

31.
by sewen 2y ago
Never encountered it before, but it looks cool. I think they are trying to solve a related problem. "We can consolidate the work by making a generic log that has networking and syncing built-in. This can be used by developers to make a
32.
by sewen 2y ago
Nice question! Restate is not a log that retains the raw events for a long time - conceptually just until they where processed by the handlers, DB, locking, etc. When you build stateful handlers, the state per key is in the internal DB, and
33.
Every System is a Log: Avoiding coordination in distributed applications(restate.dev)
229 points by sewen 2y ago|155 comments
34.
by sewen 2y ago
A short summary: Complex distributed coordination and orchestration is at the root of what makes many apps brittle and prone to inconsistencies. But we can mitigate much of complexity with a neat trick, building on the fact that every syste
35.
by sewen 2y ago
Stephan here, one of the authors of Restate and that article. Happy to answer questions. This is a first look at the distributed runtime we are building, which combines some nice properties from event-sourcing architectures, with distribute
36.
by sewen 2y ago
Both systems pick different trade-offs: Flink doesn't persist intermediate state synchronously at all. It runs asynchronous global snapshots in the background, which avoid capturing in-flight messages, just store state, aligned through
37.
by sewen 2y ago
Thank you! Yes, Flink Stateful Functions were a first experiment to build a system for the use cases we have here. Specifically in Virtual Objects you can see that legacy. With Stateful Functions, we quickly realized that we needed somethin
38.
by sewen 2y ago
For a many of those values, the answer would be "as much as you like", but with awareness for tradeoffs. You can store a lot of data in Restate (workflow events, steps). Logged events move quickly to an embedded RocksDB, which is
39.
by sewen 2y ago
That blog post should exist, agree. Here is an attempt at a short answer (with the caveat that I am not an expert in Temporal). (1) Restate has latencies that to the best of my knowledge are not achievable with Temporal. Restate's late
40.
by sewen 2y ago
That is true, individual steps should be idempotent or undo-able. But really only each individual one needs to be idempotent, rather than the full sequence, and that makes many situations much easier. For example, you create a new permissio
41.
by sewen 2y ago
Disclaimer, I am not an Airflow expert and even less of a Prefect expert. One difference is that Airflow seems geared towards heavier operations, like in data pipelines. In contrast, would be that Restate is not by default spawning any task
42.
by sewen 2y ago
A few links worth sharing here: - Blog post with an overview of Restate 1.0: https://restate.dev/blog/announcing-restate-1.0-restate-clou... - Restate docs: https://docs.restate.dev/ - Discord, for any
43.
by sewen 2y ago
That's definitely a good question. A few thoughts here (I am one of the authors). The "bring your own data layer" has several goals: (1) it is really helpful in getting good latencies. (2) it makes it self-contained, so easy
44.
Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust(restate.dev)
185 points by sewen 2y ago|109 comments
45.
by sewen 3y ago
To a first approximation, yes. There is some small cost for the workflow function itself, but as this doesn't wait on responses and only really executes the side effects, it is not that much. Especially, given that this has comparable
46.
by sewen 8y ago
Sorry, but that is not correct. We clearly say in all announcements that this is about doubling down on open source. In contrast to all current the efforts everywhere that restrict open use, we plan to do even more work openly. This is best
47.
by sewen 8y ago
The announcement clearly says that there is the same commitment to customers as before. https://www.da-platform.com/blog/data-artisans-alibaba-new-c...
48.
by sewen 9y ago
My feeling is that #2 is the reason. Also, a lot of Apache projects embed RocksDB and would effectively be stopped (until a suitable replacement would come up) had they not changed the license.
49.
An Application Manager for Stream Processing and Apache Flink(data-artisans.com)
15 points by sewen 9y ago|1 comments
50.
by sewen 9y ago
Disclaimer: I am one of the Flink committers. While Flink has in fact no direct SQL entry point right now (and many users simply wrap the API entry points themselves to form a SQL entry point), the other statements are actually not quite ri
51.
by sewen 9y ago
Flink's SQL implementation actually follows ANSI SQL - which I think is very important. There is a way of interpreting streams and tables to make ANSI SQL meaningful in the presence of streams, which we follow [1]. The big advantage is
52.
by sewen 10y ago
The main point is that this new architecture has many interesting advantages over the standard Rails & database approach. By logging all the interaction events and computing application state via a stream processor, you get all the bene
53.
by sewen 11y ago
I disagree on some points. First, it is crucial to distinguish between "exactly-once" semantics with respect to state inside the stream processor (for example an aggregate computed in a window) an exactly-once delivery to extern
54.
by sewen 11y ago
I don't think SS gets higher streaming throughput than Flink. That was an assumption written in the Yahoo! streaming benchmark without an actual experiment.
55.
by sewen 11y ago
True. In most streaming systems, exactly-once refers to the state that is maintained and controlled by the streaming system. For that, Flink can guarantee such semantics. That state can also be exposed externally, see for example here: htt
56.
by sewen 11y ago
You do not need physical exactly once. If you can "roll back" changes, than physical "at least once" is enough to realize logical "exactly once".
57.
by sewen 11y ago
That was the research project where many contributors came from. Most of the streaming tech and developments in Flink are very disconnected from that and have little to do with database tech any more, actually.
58.
by sewen 11y ago
That works a bit differently in Flink and Spark. Inside a data flow program, the scheduler tries to schedule as local as possible. For the inputs to a streaming program (for example Kafka partitions), there is currently no locality consider
59.
by sewen 11y ago
The differences are mainly around batch-centric vs. streaming-centric model and executions. Here is for example a video walkthrough by MapR: https://www.mapr.com/blog/apache-spark-vs-apache-flink-white...
60.
by sewen 12y ago
Use case is computing recommendations at large scale (collaborative filtering, latent factor models)

More ›