4 ms·

What does it mean by "observed instances of eventual consistency"? Shouldn't it be close to 100% given enough time? The claim of zero for 100k operations is jaw

by jayp 12y ago

What does it mean by "observed instances of eventual consistency"? Shouldn't it be close to 100% given enough time? The claim of zero for 100k operations is jaw dropping (and requires more evidence, or simply, a better explanation of the experiment).

azdle 12y ago
"Observed instances of eventual consistency", which as I understand it means, "the number of times that two (or maybe more) operations were able to succeeded which then had to be reconciled after the fact to get consistency between nodes."
jcrites 12y ago
The author ran a test in which he wrote an object and then attempted to read the object. Across many trials, he counted the number of times the object was not immediately available. When the object is not immediately found, it's an occurrence of observed eventual consistency. In a strongly consistent model, that will never happen (by contract). The author generalized the experiment across creating, updating, and deleting objects, followed by either reading or listing objects. Operations on eventually consistent systems can behave in an outwardly similar way to strongly consistent systems in the average or happy case: when they manage to reach consistency faster than your application can follow up with its next request. The consistency model tells you whether this is guaranteed. A number of eventually consistent systems do a pretty good job of providing apparent strong consistency in the typical case, such that eventual consistency only occurs when recovering from a hardware failure or network partition.
- jayp 12y ago
  Thanks Justin. Ahh -- "observed instances of eventual consistency" as in "able to observe poorer than strong consistency". I was imagining it as able to eventually confirm _consistency_. However, was dumbfounded by the lack of information on delay before check. I wonder if the experiment normalizes for the client origin. The farther the client is from the actual server, the more time it takes to do a round trip, and less chance for an observed instance of eventual consistency.
  - jcrites 12y ago
    > However, was dumbfounded by the lack of information on delay before check. Yes, I was wondering about this too. I examined the source code in the GitHub repository and found that all the follow-up operations are immediate. No consideration of or measurement of latency. > I wonder if the experiment normalizes for the client origin. Unfortunately it does not. As you imply, a high round trip time, or just high latency of the storage operations in general, could easily affect the apparent consistency with the current experiment design. In an enhanced version of this experiment I think it would be helpful to have latency measurements, such as total time from beginning of write until end of read (average and percentiles); and as you suggested round trip time measurements. Luckily the code is available, so I suppose it's on us to reproduce the results and extend them! Fortunately the code is also pretty short, since a lot of the storage system specific logic is encapsulated within the Apache JClouds library. (To clarify, I don't question the accuracy of the results they are about what I'd expect. I'm mostly being pedantic in drilling into these concerns, and wondering what more information we might be able to extract from the experiment.)
    - jayp 12y ago
      Yeah, more sophisticated and challenging experiments can be had. However, as they say, reality is relative. If the tests they ran are "realistic" in the sense that most people use the systems in the same way they tested, then it's all good. I suppose we need to agree on what's realistic / acceptable. I would say just minimize client-server RTT and that's a good setup, which they may have.
  - khc 12y ago
    I helped Andrew ran the S3 tests. For us-standard bucket, we used an m3.medium instance in the us-east region, and for us-west bucket, we used a t2.micro instance in the us-west-2 (Oregon) region. It is also possible that S3 somehow tries to pin requests from the same client to the same storage nodes. A better test setup would coordinate writing and reading across multiple clients, but is beyond the scope of this experiment. http://www.researchgate.net/publication/259541556_Eventual_Consistency_How_soon_is_eventual_An_Evaluation_of_Amazon_S3%27s_Consistency_Behavior/links/0deec52c6e04b49921000000.pdf http://www.researchgate.net/publication/259541556_Eventual_C... describes such a system and their results.

by jayp 12y ago

azdle 12y ago

"Observed instances of eventual consistency", which as I understand it means, "the number of times that two (or maybe more) operations were able to succeeded which then had to be reconciled after the fact to get consistency between nodes."

jcrites 12y ago

The author ran a test in which he wrote an object and then attempted to read the object. Across many trials, he counted the number of times the object was not immediately available. When the object is not immediately found, it's an occurrence of observed eventual consistency. In a strongly consistent model, that will never happen (by contract). The author generalized the experiment across creating, updating, and deleting objects, followed by either reading or listing objects. Operations on eventually consistent systems can behave in an outwardly similar way to strongly consistent systems in the average or happy case: when they manage to reach consistency faster than your application can follow up with its next request. The consistency model tells you whether this is guaranteed. A number of eventually consistent systems do a pretty good job of providing apparent strong consistency in the typical case, such that eventual consistency only occurs when recovering from a hardware failure or network partition.

jayp 12y ago
Thanks Justin. Ahh -- "observed instances of eventual consistency" as in "able to observe poorer than strong consistency". I was imagining it as able to eventually confirm _consistency_. However, was dumbfounded by the lack of information on delay before check. I wonder if the experiment normalizes for the client origin. The farther the client is from the actual server, the more time it takes to do a round trip, and less chance for an observed instance of eventual consistency.
- jcrites 12y ago
  > However, was dumbfounded by the lack of information on delay before check. Yes, I was wondering about this too. I examined the source code in the GitHub repository and found that all the follow-up operations are immediate. No consideration of or measurement of latency. > I wonder if the experiment normalizes for the client origin. Unfortunately it does not. As you imply, a high round trip time, or just high latency of the storage operations in general, could easily affect the apparent consistency with the current experiment design. In an enhanced version of this experiment I think it would be helpful to have latency measurements, such as total time from beginning of write until end of read (average and percentiles); and as you suggested round trip time measurements. Luckily the code is available, so I suppose it's on us to reproduce the results and extend them! Fortunately the code is also pretty short, since a lot of the storage system specific logic is encapsulated within the Apache JClouds library. (To clarify, I don't question the accuracy of the results they are about what I'd expect. I'm mostly being pedantic in drilling into these concerns, and wondering what more information we might be able to extract from the experiment.)
  - jayp 12y ago
    Yeah, more sophisticated and challenging experiments can be had. However, as they say, reality is relative. If the tests they ran are "realistic" in the sense that most people use the systems in the same way they tested, then it's all good. I suppose we need to agree on what's realistic / acceptable. I would say just minimize client-server RTT and that's a good setup, which they may have.
- khc 12y ago
  I helped Andrew ran the S3 tests. For us-standard bucket, we used an m3.medium instance in the us-east region, and for us-west bucket, we used a t2.micro instance in the us-west-2 (Oregon) region. It is also possible that S3 somehow tries to pin requests from the same client to the same storage nodes. A better test setup would coordinate writing and reading across multiple clients, but is beyond the scope of this experiment. http://www.researchgate.net/publication/259541556_Eventual_Consistency_How_soon_is_eventual_An_Evaluation_of_Amazon_S3%27s_Consistency_Behavior/links/0deec52c6e04b49921000000.pdf http://www.researchgate.net/publication/259541556_Eventual_C... describes such a system and their results.