3 ms·

I thought the idea of "exactly once" was a very questionable claim given what we know about computing, specially distributed. Then somewhere else you see this

by omeid2 3y ago

I thought the idea of "exactly once" was a very questionable claim given what we know about computing, specially distributed. Then somewhere else you see this gem. > Workflows in flawless are written in Rust, in fact they are just regular Rust functions. This means that they can contain arbitrary logic. But instead of native code, the functions are compiled to WebAssembly and executed in a completely deterministic environment. As much as I love Rust, this sounds like, here is a problem, let me throw fancy Rust and WebAssembly and that should fix it.

abrookewood 3y ago
The main page (https://flawless.dev/ https://flawless.dev/) has a diagram/video that shows how it would work and it basically writes any 'side effects' to a log which is used to track their existence. If they exist in the log, you read them; if they don't you run the code that generates them and then log it. It's interesting, but I can't imagine how it is going to work at scale, both in terms of managing state across a very large application and in terms of running thousands of instances concurrently.
- omeid2 3y ago
  As the saying goes, "it is turtles all the way down". How do you ensure the log is "written" and "synced" _exactly once_?
  - twosdai 3y ago
    Just write the action of writing a log to a different log file. Duh. /s
lifthrasiir 3y ago
That can absolutely work... if you are confined into a single machine. (In fact, that's roughly how cloud instances work.) I see no way to generalize that into a distributed system. In my previous job I worked on a game server engine which solves a very limited version of this problem via virtual actors and it was still way too hard.
hedora 3y ago
Exactly Once is basically a solved problem. The CAP theorem says you can either get consistency or availability in a system that has network partitions. If you give up consistency, you end up corrupting data every once in a while. If you don’t, then you can have exactly once, but it might take a long time if there’s a network failure. NFSv3 solved this back in the 1980’s. (V2 and V1 may have, but I don’t know.) It did it without requiring deterministic execution or other programming language innovations, so I share your skepticism about rust and web assembly solving these problems. (I really like rust, and recommend it for pretty much all new systems code. However, it is not a panacea.)
- sausagefeet 3y ago
  This comment has some errors in it: 1. Exactly Once is absolutely not a solved problem. CAP says nothing about 'exactly once', it's about design choices for your data. But you can do all sorts of side-effectful things when it comes setting the data. 2. Giving up consistency does not mean you "corrupt your data every once in awhile". I don't know why you would say that. Choosing availability means you have to make decisions about how your data becomes consistent, but nothing about that means it is corrupted. 3. Choosing consistency says nothing about "exactly once". Consider this basic workflow backed by a consistent database: request comes into service, service sends email, services goes to store that email was sent in consistent database and crashes, user gets crash back, reruns request, email sent again. Oh so send the email AFTER it's stored in the database, well service crashes after saving in database, same problem.
  - hedora 3y ago
    If your system provides strong consistency, it is possible to build exactly once processing over it. NFSv3 is an existence proof, and there are plenty of theoretical results showing it is possible. Roughly speaking, you run the job and install the result in the consistent store iff your output register is null. If you side effect outside of the exactly once system, and the receiver can’t suppress duplicate notifications then you’re screwed, but the system in the article precludes that, as do many existing systems. As for your other points, with eventual consistency you get weird problems like “I wrote X, then Y, but then read X and the system converged to X, except a week later I read Y, but just from half my fleet, and only for a 30 minute window”. Unless you layer consistency on top of that (which is not always possible), then, tautologically, you don’t get consistency. In particular, most intuitive application-level invariants are going to be violated in all sorts of bizarre ways that take many pages to explain. In practice, eventual consistency is closer to corrupting than not corrupting, because most application developers aren’t going to follow this conversation, and will use the database wrong. I’m sure you or I could model our program and the storage in TLA+ and confirm we’re correctly maintaining application state, but that doesn’t help most developers. Also, it’s not economically efficient. We’re talking about “only” getting 6 nines instead of 7 because we chose CP instead of AP, but as a side benefit the system is easier to maintain, and it took less than 1/10th as much to implement, and has orders of magnitude fewer implementation bugs.

by omeid2 3y ago

abrookewood 3y ago

The main page (https://flawless.dev/ https://flawless.dev/) has a diagram/video that shows how it would work and it basically writes any 'side effects' to a log which is used to track their existence. If they exist in the log, you read them; if they don't you run the code that generates them and then log it. It's interesting, but I can't imagine how it is going to work at scale, both in terms of managing state across a very large application and in terms of running thousands of instances concurrently.

omeid2 3y ago
As the saying goes, "it is turtles all the way down". How do you ensure the log is "written" and "synced" _exactly once_?
- twosdai 3y ago
  Just write the action of writing a log to a different log file. Duh. /s

lifthrasiir 3y ago

That can absolutely work... if you are confined into a single machine. (In fact, that's roughly how cloud instances work.) I see no way to generalize that into a distributed system. In my previous job I worked on a game server engine which solves a very limited version of this problem via virtual actors and it was still way too hard.

hedora 3y ago

Exactly Once is basically a solved problem. The CAP theorem says you can either get consistency or availability in a system that has network partitions. If you give up consistency, you end up corrupting data every once in a while. If you don’t, then you can have exactly once, but it might take a long time if there’s a network failure. NFSv3 solved this back in the 1980’s. (V2 and V1 may have, but I don’t know.) It did it without requiring deterministic execution or other programming language innovations, so I share your skepticism about rust and web assembly solving these problems. (I really like rust, and recommend it for pretty much all new systems code. However, it is not a panacea.)

sausagefeet 3y ago
This comment has some errors in it: 1. Exactly Once is absolutely not a solved problem. CAP says nothing about 'exactly once', it's about design choices for your data. But you can do all sorts of side-effectful things when it comes setting the data. 2. Giving up consistency does not mean you "corrupt your data every once in awhile". I don't know why you would say that. Choosing availability means you have to make decisions about how your data becomes consistent, but nothing about that means it is corrupted. 3. Choosing consistency says nothing about "exactly once". Consider this basic workflow backed by a consistent database: request comes into service, service sends email, services goes to store that email was sent in consistent database and crashes, user gets crash back, reruns request, email sent again. Oh so send the email AFTER it's stored in the database, well service crashes after saving in database, same problem.
- hedora 3y ago
  If your system provides strong consistency, it is possible to build exactly once processing over it. NFSv3 is an existence proof, and there are plenty of theoretical results showing it is possible. Roughly speaking, you run the job and install the result in the consistent store iff your output register is null. If you side effect outside of the exactly once system, and the receiver can’t suppress duplicate notifications then you’re screwed, but the system in the article precludes that, as do many existing systems. As for your other points, with eventual consistency you get weird problems like “I wrote X, then Y, but then read X and the system converged to X, except a week later I read Y, but just from half my fleet, and only for a 30 minute window”. Unless you layer consistency on top of that (which is not always possible), then, tautologically, you don’t get consistency. In particular, most intuitive application-level invariants are going to be violated in all sorts of bizarre ways that take many pages to explain. In practice, eventual consistency is closer to corrupting than not corrupting, because most application developers aren’t going to follow this conversation, and will use the database wrong. I’m sure you or I could model our program and the storage in TLA+ and confirm we’re correctly maintaining application state, but that doesn’t help most developers. Also, it’s not economically efficient. We’re talking about “only” getting 6 nines instead of 7 because we chose CP instead of AP, but as a side benefit the system is easier to maintain, and it took less than 1/10th as much to implement, and has orders of magnitude fewer implementation bugs.