21 ms·
Flawless – Durable execution engine for Rust
- nutate 3y agoHow does this handle the machine running out of memory?
- royjacobs 3y agoLooks interesting, I wonder if the method of marking functions as 'having side effects' is going to be easy to make fool proof. For instance, I assume that in the example the random number generation is a side effect because it comes from a RNG provided by flawless itself. Would this have worked with a regular Rust function as well? I assume there is going to be some kind of test harness that allows developers to check their workflows.
- evrimoztamur 3y agoSeeing how both the RNG and the HTTP request are under the flawless namespace, this is going to require you to write 'std::flawless' instead of giving you access to the entire Rust ecosystem. A harness would solve most issues, but I wonder how much of the functionality you can map. So far, this looks more like a scripting runtime which uses Rust.
- bkolobara 3y agoHi! I'm the author of flawless. By using WebAssembly, it's kinda by default fool proof. WebAssemby explicitly requires you to declare host calls inside your modules. If you try to use another host call that is not provided by flawless, your module can't be instantiated. It's also important to notice that there are multiple standardisation efforts going on in the WebAssembly space. For example if you are using the rust `rand` crate and compiling to WebAssembly, it's using the WASI host functions for generating random numbers. While I'm waiting for wasi, wasi-http and others to be standardised, I expose my own interface for now. Obviously, this also has a big downside. You can't compile all Rust code to WebAssembly. However, I prefer this reject by default approach, so that you are guaranteed to never have unintended side effects.
- drakenot 3y agoCould you provide a way to mark function calls as side effects similar to Temporal? I'm a bit worried about the limitation of having to use the flawless namespace functions for everything.
- bkolobara 3y agoFlawless integrates with your existing system. You probably don't need the guarantees that flawless provides for all code, but just specific functions/workflows.
- deleted 3y ago[deleted]
- skavi 3y agoIf I understand correctly, Flawless provides the WASM runtime. If that’s the case why can’t it be entirely hidden to the user? The system provides things like entropy and networking.
- dathinab 3y agoprobably some subtle technical details like "just" WASM as a target doesn't have many of the APIs build in without non trivial "magic" trics and for WASI might be tricky to get the exact form of determinism they choose right like they might need to provide a custom standard library which is viable but not very convenient to use and even less convenient to maintain (if it's just about custom build flags for the "upstream" standard library it's on the other hand quite easy but likely not good enough) through that's purely speculative but they might be lucky there is quite a bit of benefit for rust to have a deterministic wasm build target (for wasm based derives which if combined with cryptographic hashing of inputs form a potential building block for having a shared (potential public) derive binary artifact cache speeding up CI systems and similar and allowing more complicated/advanced derive usage without having to worry about the derive compliation time adding to the initial build time)
- mgdev 3y agoThe problem is the system on the other end usually has timing/causality expectations (e.g., in the case of reentrant calls), and that system is not usually part of your system (e.g., they wouldn't include whatever monotonically increasing counter your system would require to ensure causal consistency).
- dodyg 3y agoHow different is this from Temporal?
- mytydev 3y agoBesides the obvious difference in execution engine which is explained in the link, I noticed the same thing and after doing some research, I found out that Temporal cofounder Samar Abbas was the creator of Amazon Flow Framework in 2009 and then the creator of the Azure Durable Task Framework in 2014, then moved on to Uber and co-created Candence with Maxim Fateev before they both eventually founded Temporal in 2019. https://www.temporal.io/about https://www.temporal.io/about
- mytydev 3y agoHere's a little more context: https://temporal.io/blog/samars-journey https://temporal.io/blog/samars-journey
- rubenfiszel 3y agoLooks like a rust alternative to temporal using wasm as a runtime. Love it! Founder of windmill.dev here which is another durability engine in Rust except it's a lot less elegant since we split our workflows into well defined steps in python/typescript/go/bash and can resume from incomplete steps only by restarting from the last step and storing the result of each step forever in our postgres db (using jsonb). The use-cases are clearly different and I can see flawless being so lightweight that you could use it to model UI flow state and scale it to millions on a small server as pointed out by the site. This is fantastic, hopefully one day rust will power all distributed systems.
- V__ 3y agoI love the animation showing the core principle and how it works, really well done.
- bkolobara 3y agoThank you! I hand coded it with just HTML, CSS and JavaScript and put a lot of effort and love into it. The code is not the prettiest, but the implementation is straight forward if anyone wants to check it out https://flawless.dev/js/how-does-it-work-animation.js https://flawless.dev/js/how-does-it-work-animation.js
- CaptainFever 3y ago// Update the `keyFrame`. keyFrame += 0.5; // This is just a huge state machine progression. switch (keyFrame) { // In the first 4 seconds we just run through the existing log messages. case 0.5: case 1: case 1.5: ... Wow! I love the design. So simple.
- isuckatcoding 3y agoI kinda wanted to see it fail at the http execution phase because that’s much more common due to timeouts when call external endpoints.
- BiteCode_dev 3y agoOur field is getting closer and closer to architecture and medicine now. We've tech like this we can come out of the fiddling age and deep dive into a serious engineering culture. Quick question: how do you prevent persisting the effect of a DOS attack on those systems?
- rewmie 3y ago> Our field is getting closer and closer to architecture and medicine now. We've tech like this we can come out of the fiddling age and deep dive into a serious engineering culture. This comment reads like ml-generated nonsense. Architecture is summed up as subjective opinions bounded by regulatory constraints, and medicine is still empirical knowledge validated through experiments. None of these fields have even a passing resemblance with engineering.
- BiteCode_dev 3y agoThat's why I said architecture and medicine and not rocket science. We have a long way to go. However, I think you are very reductive in your descriptions of those fields. Sorry I meant, as a large language model...
- rewmie 3y ago> That's why I said architecture and medicine and not rocket science. You conflated architecture and medicine to a builduo to "serious engineering culture". There is no way around it: it's nonsense. Even though they are highly specialized fields, neither architecture not medicine has anything to do with the basic principles of engineering. It's just nonsensical word soup.
- someidiot123 3y ago[dead]
- ijustlovemath 3y agoMedicine has incredibly rigorous engineering processes; if anything I'd say the software world at large has a thing or two to learn from medicine (particularly class II+ devices) about building robust, long lasting software that you know works the way you designed it to See: ISO13485, IEC60601, QMS, etc
- Philpax 3y agoVery cool, and the approach demonstrated might be of interest to a similar problem we have in Ambient (our WASM game runtime that has competing processes that may need to retry interactions.) That being said - what’s the relation to Lunatic [0]? Are you still working on Lunatic? Is this a side project? Or is it something completely separate? [0]: https://lunatic.solutions/ https://lunatic.solutions/
- rapnie 3y agoAh, good observation. This is by Bernard Kolobara the CEO and Co-founder of Lunatic. https://kolobara.com https://kolobara.com
- nrabulinski 3y agoHe (Bernard) hasn’t made any commit to lunatic in half a year where before that he was regularly contributing so one can only presume he took what he learned working on lunatic and started a project behind the closed doors
- pygy_ 3y agoIf anything, he made strides in the naming department :-)
- whalesalad 3y ago“Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.” – Virding’s first rule of programming
- BiteCode_dev 3y agoHey... https://codesync.global/media/erlangrt-a-beam-vm-reimplementation-in-rust/ https://codesync.global/media/erlangrt-a-beam-vm-reimplement...
- anonzzzies 3y agoWhile it might have tons of bugs, this one seems better; programming stuff as you normally would and have it distributed and durable automagically. At least that's what I got from it. And that's nice.
- resonious 3y agoThis is how people often describe Erlang and Elixir. But I guess it's fair to say that functional programming is not "programming as you usually would" for most!
- anonzzzies 3y agoWell, with Erlang you really have to plan these things and implement those. Functionality (pure/immutable more-so) helps, but it doesn't pick up everything as in, it's not drop-in; it's very possible, but you have to change habits, even if you were a functional programmer before.
- galangalalgol 3y agoRust isn't exactly the normal style either unless you were used to some subset of modern c++ that probably only gets used in gaming or embedded. While I wouldn't normally argue for rigor or anything tedious, creating computationally expensive abstraction layers to let you develop capabilities faster without having to worry about defects or edge cases seems the opposite of what rust is designed for. I think the language will fight you the whole way. The only aid I can think of is that you can wildly unwrap stuff everywhere not worrying about panics. Which does save some time.
- hades32 3y agoHow will this deal with workflow versioning? IMO the hardest problem in e.g. Temporal/Cadence
- burntcaramel 3y agoIs this the problem where you have workflows live and in progress, yet you want to update the workflow and have things not break? Would you want to have multiple versions at once, or rather some way to migrate the in progress ones to the latest workflow definition?
- jen20 3y agoYou’d likely want either option depending on the lifetime.
- spiffytech 3y agoThe latter. For short lived workflows, you may not care about updating; just let it finish. For longer jobs, you want some way to replace the current logic and either resume from where the job left off, or restart it idempotently. Especially if your workflow spans months or years (which at least some of these systems are designed for). The challenge is these systens shine when you manage the job state in-memory, but they don't "store" the data in a traditional sense. They just replay your logic and replay the original I/O results. So if your logic changes, the replay breaks and your state goes bye-bye. (I think of it similarly to React's "rules of hooks": you can't do anything that makes the function call key APIs in a different order than previous executions) So you either accept that you can never update an in-flight job (in a meaningful way, at least), or you track job state in some other system and throw away the distinguishing feature of these systems. I'm curious how people normally handle this. When I worked with Azure Durable Functions I couldn't find a way around this.
- joshfee 3y agoI think the problem works both ways too, because for an in progress (but long-running) workflow, you may not want to retroactively apply your new business logic for the part that's already run because that would be unexpected. But for the logic that hasn't run yet, you would certainly want the latest and greatest. I wonder if there could be an approach where you have both versions live simultaneously, and introduce some sort of "checkpoint" into the old version that would act similar to a DB migration. When re-computing a workflow you could then start from the latest checkpoint, but any workflows that were created with the old version that haven't reached a checkpoint would continue to run the old code until it does.
- mytydev 3y agoThis deterministic execution pattern reminds me a lot of the approach with Azure Durable Functions and I know I've seen it in other places as well. Curious if anyone knows where that pattern originated.
- indiv0 3y agoWhere is the state for the side effects stored? Say I have an AWS Lambda that I want to make idempotent. Lambdas don’t have local storage that persists across runs (unless you mount EBS volumes or something) so I presume state can be stored in a DB?
- dogleash 3y ago> Where is the state for the side effects stored? That's what you'll be paying for when they release the product.
- tromp 3y agoThis approach should also work well for Haskell where "main" is already a pure computation producing a description of what IO actions to take.
- Thaxll 3y agoOnly rust and web assembly seems very restrictive, what does it brings to the table vs temporal for example which work on any language and is rock solid?
- rob74 3y agoIf it used anything else than Rust, they couldn't call if "Flawless"... (SCNR)
- whateveracct 3y agoReminds me of Dfinity without the crypto part. The killer app of Dfinity was never the crypto after all.
- two_handfuls 3y agoThis looks very cool. I expect you’ll still want the services you talk to to be idempotent, but Flawless still takes a big chunk out of the work - and it seems very flexible too!
- b20000 3y agosmells like java and the jvm
- Dowwie 3y agocheck out Fermyon
- mdaniel 3y agolinky: https://github.com/fermyon/spin#readme https://github.com/fermyon/spin#readme (Apache 2; and while I don't see any CLA, interestingly they do require GPG signed commits: https://developer.fermyon.com/spin/contributing-spin#committing-and-pushing-your-changes https://developer.fermyon.com/spin/contributing-spin#committ... )
- nextaccountic 3y agorequiring signed commits is awesome
- blumomo 3y agoI have wanted something similar for Python: where the execution of a function will be interrupted (i.e. via a dedicated exception) and then I can rerun it to the very same point it previously halted while none of prior side effects occur and the previous state within that function gets restored. While I have an idea how to implement it, now after having read the article and comments here, how is this concept called? Does an implementation for Python exist already?
- bitdivision 3y agoIt's probably overkill depending on your use-case, but temporal has a python client: https://github.com/temporalio/sdk-python https://github.com/temporalio/sdk-python
- blumomo 3y agoThanks, that would solve it! Their „workflows“ provide a nice abstraction. Though I’d rather like to not include an external service. Imagine a simpler implementation for me where any earlier called subfunction would simply return the previous result, up to the point where the function was previously interrupted. Therefore these previous return values need to be stored (in what temporal calls a workflow), i.e. in a database. Sounds simple enough and decorators would probably help designing a good abstraction and keeping everything self-documenting and simple enough to understand.
- kodablah 3y agohttps://temporal.io/blog/durable-distributed-asyncio-event-loop https://temporal.io/blog/durable-distributed-asyncio-event-l... explains how it works
- amalcon 3y agoDepending on what you're trying to do, you likely want either continuations (which aren't supported in Python directly, but which you can emulate by writing your function in continuation-passing style) or a debugger with rewinding (which is fairly cutting edge, but might exist for Python? PyPy blogged about an early version a few years back). The main use cases for a specific point interrupt are covered by these techniques. That said, Temporal is very similar and supports multiple languages, including Python.
- sandGorgon 3y agohow is something like this implemented ? does this hook into the rust compiler ? or does the rust/wasm compiler architecture provide an intermediate step that allows for these kinds of systems to be built on top ?
- paulgb 3y agoWasm itself provides some determinism constraints[1] as an explicit design goal, which is what enables things like this. I made a very simple snapshottable compute executer for a talk last year and it took surprisingly little code if you want a look at how it works[2], but the complexity of something like Flawless is that doing anything useful involves communicating with the outside world, where non-determinism can easily sneak in. I've been following Bernard's work on Lunatic so I'm incredibly excited to see he's tackling these hard problems. [1] https://github.com/WebAssembly/design/blob/main/Nondeterminism.md https://github.com/WebAssembly/design/blob/main/Nondetermini... [2] https://github.com/drifting-in-space/wasmbox/tree/main https://github.com/drifting-in-space/wasmbox/tree/main
- sandGorgon 3y agothis is very cool! but just to be clear - wasm allows for someone to tap into variable declarations, etc ? which is what flawless is doing. that's almost parse tree right ?
- paulgb 3y agoI'm not 100% sure how Flawless works, but you don't need to tap into variable declarations if you can just snapshot the entire stack and heap, so I think that's what it does.
- daddykotex 3y agoRecently, another alternative was teased out: https://www.golem.cloud/ https://www.golem.cloud/ From the folks at Ziverge, who've worked on ZIO in Scala. They use a similar approach I believe. It's discussed in this podcast: https://podcasters.spotify.com/pod/show/happypathprogramming/episodes/85-Scala--Rust--and-Durable-Computing-with-John-De-Goes-e29ca1s https://podcasters.spotify.com/pod/show/happypathprogramming...
- thelittlenag 3y agoWas thinking exactly the same!
- rapnie 3y agoDoesn't look to be open source, only source available. I wonder what the plans for Flawless are in this regard.
- 10000truths 3y agoDoes this determinism extend to floating point computations? This has historically been a pain point with multiplayer games where the client state has to be periodically re-synced with the server state due to slowly accumulating drift in floating point calculations.
- MuffinFlavored 3y agois one option to sync “bignum” structs instead of f64 or is that not realistic in terms of complexity/size?
- kodablah 3y agoWhen I implemented a WASM compiler, the only source of float-based non-determinism I found was in the exact byte representation of NaN. Floating point math is deterministic. See https://webassembly.org/docs/faq/#why-is-there-no-fast-math-mode-with-relaxed-floating-point-semantics https://webassembly.org/docs/faq/#why-is-there-no-fast-math-... and https://github.com/WebAssembly/design/blob/main/Nondeterminism.md https://github.com/WebAssembly/design/blob/main/Nondetermini....
- adgjlsfhk1 3y agoThe other common one is elementary functions from libm (i.e. exp/log etc)
- jcranmer 3y agoDo note that sqrt should return the same results on all libm implementations, as well as functions like nextafter or scalbn.
- ajb 3y agoIt may also be thread nondeterminism, eg relying on the commutativity of addition etc.
- aseipp 3y agoOn that note, Wasmtime has an option to enable NaN canonicalization, which should patch that hole, too: https://docs.rs/wasmtime/latest/wasmtime/struct.Config.html#method.cranelift_nan_canonicalization https://docs.rs/wasmtime/latest/wasmtime/struct.Config.html#...
- abtinf 3y agoI’ve found it surprising that none of the wasm runtimes I’ve seen support dumping/restoring memory state. Would be very valuable.
- Jalad 3y ago> If you would like to be one of the first developers to get access to flawless as part of our private alpha. Join our waitlist.
- isuckatcoding 3y agoI’ve seen a couple of these now. Looks like a lot of them take the “we will take your code and compile it”-approach. That’s not a problem per se but it does affect the ability to debug and see relevant stack traces. Its like how sometimes you see transpiled JS when what you really want is the typescript using source maps.
- tomjakubowski 3y ago> Imagine if you could just start an arbitrary computation and the system guarantees that it will run until completion and all the operations will be performed exactly once. How is this guaranteed? Isn't exactly once delivery in a distributed system impossible?
- jiveturkey 3y agoIt does say, _Imagine_. Not sure what you're quoting, I don't see this text in TFA and your comment is not a child of another.
- tlarkworthy 3y agoExactly once processing is possible with idempotence keys over an at least once delivery system.
- mgdev 3y agoYou've just pushed the problem down to the system that checks idempotence keys.
- brodouevencode 3y agoIf you're referring to the CAP theorem, it needs some clarification: https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed/ https://www.infoq.com/articles/cap-twelve-years-later-how-th...
- hedora 3y agoIt's not impossible. You just pick CP from the CAP theorem (and give up availability). i.e., the message will be delivered exactly once if the system makes progress. If you want an existence proof, NFSv3 had this working back in the 1980's. I doubt it was the first.
- itunpredictable 3y agoThis site is great
- m00x 3y agoThis would be perfect for money transfers. A big part of money transfer reliability is properly resuming from errors.
- raggi 3y agoIs this an explicit feature of the wasm runtime in use here, or might this break with future optimizations in the runtime and the introduction of threads and other similar features?
- senderista 3y agoVery tenuously related, but the idea of logging all nondeterministic effects for idempotence can also be used to make locks lock-free! https://arxiv.org/abs/2201.00813 https://arxiv.org/abs/2201.00813
- emmanueloga_ 3y agoI was thinking of something like this for running queries for sharded data: the execution could be paused and moved to the server that is closer to the data source at a given runtime. > Notice how flawless takes away the burden of persisting the state. Reminds me of the Wisdom of the Well: "You'll never find a programming language that frees you from the burden of clarifying your ideas." [1] Although solving durability of execution for arbitrary code sounds super cool, I suspect that writing the code in a way that could be checkpointed/resumed more naturally would probably be easier to debug and implement, in the end. -- 1: https://xkcd.com/568/ https://xkcd.com/568/
- deleted 3y ago[deleted]
- mleonhard 3y agoThe boring way to do guaranteed execution with retries is to store the work items in Postgres and use a pool of workers. The workers store intermediate results in a cache. Such systems need tooling for diagnosing and fixing problems: metrics, logging, dead-letter queue, inspecting and evicting cached items, retrying dead jobs, adjusting worker settings. Flawless and similar systems will have the same problems and need the same tools.
- stolsvik 3y agoThe “restart from where it failed”-aspect was a big reason for why I made Mats3. It is message-based, async, transactional, staged stateless services, or message-oriented asynchronous RPC. Due to the transactionality, and the “state lives on the wire”, if a flow fails, it can be restarted from where it left off. https://mats3.io https://mats3.io