8 ms·

Jepsen: Redpanda 21.10.1

193 points by aphyr 4y ago

northstar702 4y ago
The blog for a deeper dive into results, fixes, and discussion on the write behavior in the kafka protocol. https://redpanda.com/blog/redpanda-official-jepsen-report-and-analysis/ https://redpanda.com/blog/redpanda-official-jepsen-report-an...
rystsov 4y ago
Hey folks, I was working with Kyle Kingsbury on this report from the Redpanda side and I'm happy to help if you have questions
- cgaebel 4y ago
  Thanks for working with Jespen. Being willing to subject your product to their testing is a huge boon for Redpanda's credibility. I have two questions: 1. How surprising were the bugs that Jepsen found? 2. Besides the obvious regression tests for bugs that Jepsen found, how did this report change Redpanda's overall approach to testing? Were there classes of tests missing?
  - rystsov 4y ago
    It wasn't a big surprise for us. Redpanda is a complex distributed system with multiple components even at the core level: consensus, idmepotency, transactions so we were ready that something might be off (but we were pleased to find that all the safety issues were with the things which were behind the feature flags at the time). Also we have internal chaos test and by the time partnership with Kyle started we already identified half of the consistency issues and sent PRs with fixes. The issues got in the report because by the time we started the changes weren't released yet. But it is acknowledged in the report > The Redpanda team already had an extensive test suite— including fault injection—prior to our collaboration. Their work found several serious issues including duplicate writes (#3039), inconsistent offsets (#3003), and aborted reads/circular information flow (#3036) before Jepsen encountered them We missed other issues because haven't exercised some scenario. As soon as Kyle found the issues we were able to reproduce them with the in-house chaos tests and fix. This dual testing (jepsen + existing chaos harness) approach was very beneficial. We were able to check the results and give feedback to Kyle if he found a real thing or if it looks more like an expected behavior. We fixed all the consistency (safety) issues, but there are several unresolved availability dips. We'll stick with Jepsen (the framework) until we're sure we fixed then too. But then we probably rely just on the in house tests. Clojure is very powerful language and I was truly amazed how fast Kyle for able to adjust his tests to new information but we don't have clojure expertise and even simple tasks take time. So it's probably wiser to use what we already know even it it a bit more verbose.
- polio 4y ago
  A complete nit, but the testimonial from the CTO of The Seventh Sense on https://redpanda.com/ https://redpanda.com/ spells Redpanda as "Redpand".
  - northstar702 4y ago
    Thank you. Fixed.
dhshshdhdgfff 4y ago
The first half is jepsen team trying to divine some actual testable guarantees from a pile of blog posts and a random Google doc. What a mess.
- excuses_ 4y ago
  I wonder if Redpanda thinks about or offers some alternative protocol that would be better defined in terms of transaction guarantees. At this point it looks like Kafka’s protocol was a nice try but it needs a major refactoring.
  - rystsov 4y ago
    Documentation is a bit confusing: the protocol was evolved over time (new KIPs) and there is mismatch between the database model and kafka model. But we see a lot of potential in the Kafka transactional protocol. At Redpanda we were able to push to 5k distributed transactions cross replicated shard. It's a mind-blowing for a database to achieve the same result. Also Kafka transactional protocol works at low level it's very easy to build systems on top of it. For example, it's very easy to build a Calvin inspired system http://cs.yale.edu/homes/thomson/publications/calvin-sigmod12.pdf http://cs.yale.edu/homes/thomson/publications/calvin-sigmod1...
- jeffbee 4y ago
  Total mess. It’s a real indictment of Kafka, more than it is anything about redpanda in the first half.
- deleted 4y ago
  [deleted]
- rystsov 4y ago
  The mess is mostly the result of the mismatch between the classic database transactional model and kafka transactional model (G0 anomaly). If you read the documentation without the database background it seems ok, but when you notice the differences between the models it becomes hard to understand if it's a bug or property of the Kafka protocol. There is a lot of research happening around this area even in the database world. The list of the isolation levels isn't final and some of the recent developments include PC-PSI and NMSI which also seem to "violate" the order. I hope one day we get the formal academic description of the Kafka model. It looks very promising.
mandevil 4y ago
I was unfamiliar with Redpanda, and now I know and trust it. Whatever marketing budget Redpanda spent to get a Jepsen report was well worth it.
- belter 4y ago
  Agree. Nowadays, I see anything that did not go through Jepsen with suspicion. Forces me to do the triple of technical due diligence.
- hardwaresofton 4y ago
  One of the clearest indications prices for a service should be raised I’ve ever seen. Can we get patio11 in here to say the thing?
  - agallego 4y ago
    interested! :D
  - titanomachy 4y ago
    Do you have any information on what Jepsen charges? For all we know, it could be precisely the right amount.
    - agallego 4y ago
      kyle is very friendly and I recommend reaching out. we can't and wouldn't disclose any pricing that is not public information. would be unethical on my part. all i can say is we wish to continue our work with him indefinitely as long as we keep making progress on the product :)
- debarshri 4y ago
  Thing that got my attention was that it has inline transform functions that can be added as wasm binary
- divan 4y ago
  I happened to know RedPanda founder back in the days he was at Concord.io (as a founder and a main dev). The level of obsession with performance and optimization of this guy was insane. He's not only extremelly skilled with C++, but also very passionate about rethinking large and complex systems and rebuilding them to enable 10-100x speed improvements. It's like his personal hobby – take a piece of software everyone use, and optimize it to the limits of physics, usually by implementing better version from scratch himself :) Plus, he's an excellent communicator. Watching their team working I was always thinking that successful companies can be built only with that level of passion and expertise in a single package.
CJefferson 4y ago
This isn't anything against Redpanda, but I'm always amazed how badly all these distributed databases do in Jepsen. What would one use them for in practice, which wouldn't be better suitable by a (the thing I've used), say postgresql and streaming replication in case the server goes down? (I'm not suggesting there isn't a good application, just I'm not knowledgeable enough to know of one).
- claytonjy 4y ago
  There's a lot of different ways to answer this, but I think about it as a different architectural paradigm. Yes you can do stream-ish things with Postgres but at some level of scale you'd be putting a square peg in a round hole. What opened my eyes to this world is this post from Martin Kleppman on turning the database inside out: https://martin.kleppmann.com/2015/03/04/turning-the-database-inside-out.html https://martin.kleppmann.com/2015/03/04/turning-the-database...
  - astrange 4y ago
    > What opened my eyes to this world is this post from Martin Kleppman on turning the database inside out: https://martin.kleppmann.com/2015/03/04/turning-the-database https://martin.kleppmann.com/2015/03/04/turning-the-database... Databases are only as bad as the filesystem wrt being mutable, but since we do expect them to be a "better filesystem" it's surprising we let them get away with losing data. IMO beyond transactions you should just be able to unwind any writes to a SQL database, including deletes, for at least a day. But if you ask Alan Kay he'd say all programs should have an explicit concept of time and be able to operate on the past and future state of everything.
- agallego 4y ago
  totally different approaches tho. people have tried what you proposed many times before and for some scale succeeded. hard to compare at all when you dig into the details. expect a companion post. this was super fun to partner with kyle on this. +1 would recommend to anyone building a storage system.
- rystsov 4y ago
newman314 4y ago
Redpanda (back when they were VectorizedIO) spammed my work email after I starred one of their repos, denied it after I called them out on it and I just noticed that they had deleted their response to me. Pretty sneaky to go back and delete the tweets first denying and then apologizing. Receipts: https://twitter.com/d11cc3s/status/1447573471152656389 https://twitter.com/d11cc3s/status/1447573471152656389 https://twitter.com/d11cc3s/status/1450906855115354116 https://twitter.com/d11cc3s/status/1450906855115354116
- agallego 4y ago
  hi newman314 - i mentioned in the tweet this was a mistake and offered an apology there, an sdr reached out to you, when i realized that i apologize. no ill intent. feel free to test this with a fake github account. my tweets automatically delete after 6mo, all of them on a rolling window. nothing special about this interaction. there is no sneaky-ness, though feel free to disagree.
- staticassertion 4y ago
  Sounds like you have a personal, singular issue with them that I can't imagine anyone else cares about.
doommius 4y ago
Always great to read this. I preformed a jenkins test on Microsoft internal infra and it's a huge insight. From an academic side it's just as interesting looking into the lack of standards within consistently and the definitions of them.
- rystsov 4y ago
  Cool! What did you test? I've played with Jepsen and Cosmos DB when I was at Microsoft but we had to ditch ssh, write custom agent and inject faults with PowerShell command lets.
gigatexal 4y ago
If your DB doesn't pass the Jepsen tests it's not worth using. Kudos to both teams.
titanomachy 4y ago
The level of intellectual discipline and competence on display here is inspiring. I'd love to take one of the Jepsen courses, but it seems they're offered only as corporate training. Maybe my employeer will agree to bring them in. For now I'll have to satisfy myself with the YouTube videos.
- aphyr 4y ago
  There's been a lot of interest in this, and I keep meaning to put together an open-to-the-public class when I'm less swamped! Might get a chance to do this shortly--I'll post on https://groups.google.com/a/jepsen.io/g/announce https://groups.google.com/a/jepsen.io/g/announce when it happens.
antonmry 4y ago
This report seems to have some wrong insights. Auto-commit offsets doesn't imply dataloss if records are processed synchronously. This is the safest way to test Kafka instead of commit offsets manually
- rystsov 4y ago
  Can you clarify what you mean? AFAIK with manual commit you have the most control over when the commit happens Look at this blog post describing a data loss caused by auto-commit: https://newrelic.com/blog/best-practices/kafka-consumer-config-auto-commit-data-loss https://newrelic.com/blog/best-practices/kafka-consumer-conf... Also there also may be more subtle issues with auto-commit: https://github.com/edenhill/librdkafka/issues/2782 https://github.com/edenhill/librdkafka/issues/2782
  - antonmry 4y ago
    I'm afraid the article is also wrong, this is a typical misconception when working with Kafka. Offsets are committed in the next poll() invocation. If the previous messages weren't processed, a rebalance occurs and messages are processed by other instance. This is an implementation detail of the Java client library but it allows the at-least-once semantic with auto-commit. The book Effective Kafka has a better explanation. librdkafka isn't part of official Kafka so it may have problems with this as it has other limitations. In any case, the report isn't right about this and it doesn't use the safest options. Commit offsets manually is the most flexible way but it isn't easy, being the error more usual to commit offsets individually
    - aphyr 4y ago
      > Offsets are committed in the next poll() invocation. I'm a little surprised by this--not that you're necessarily wrong, but our tests consumed messages synchronously, and IIRC (pardon, it's been 3 months since I was working on Redpanda full time and my time to go get a repro case is a bit limited) did see lost messages with the default autocommit behavior. At some point I'll have to go dig into this again.
- 4y ago
dstroot 4y ago
> A KafkaConsumer, by contrast, will happily connect to a jar of applesauce14 and return successful, empty result sets for every call to consumer.poll. This makes it surprisingly difficult to tell the difference between “everything is fine and I’m up to date” versus “the cluster is on fire”, and led to significant confusion in our tests. This tickled my funny bone. Never expected humor in a Jepsen writeup. Kudos!
- staticassertion 4y ago
  > Never expected humor in a Jepsen writeup Jepsen reports are often pretty funny, some famously so
- cwillu 4y ago
  Wait until you find out why it's called “Jepson”
  - toolz 4y ago
    please tell me it has something to do with carly jepsens song "call me maybe"
    - chipotle_coyote 4y ago
      Well, when you think about distributed databases, you have a bunch of servers "calling" one another. Some of Jepsen's testing is about disrupting that and seeing what happens. Changing "call me" to "call me maybe." If you will.
    - collinvandyck76 4y ago
      indeed :)
    - cwillu 4y ago
      You mean like this? https://aphyr.com/posts/281-call-me-maybe-carly-rae-jepsen-and-the-perils-of-network-partitions https://aphyr.com/posts/281-call-me-maybe-carly-rae-jepsen-a... https://aphyr.com/posts/283-call-me-maybe-redis https://aphyr.com/posts/283-call-me-maybe-redis
doctor_eval 4y ago
I just came here to thank Jepsen for these amazing reports. What a wonderful way to use your intellect to contribute to the wellbeing of the entire community. Also I wanted to say to redpanda: I was on the fence but now I’m convinced. Will definitely be deploying on my next project, which has already kicked off. I only wish I could run it natively on MacOS instead of requiring docker.

193 points by aphyr 4y ago

The blog for a deeper dive into results, fixes, and discussion on the write behavior in the kafka protocol. https://redpanda.com/blog/redpanda-official-jepsen-report-and-analysis/ https://redpanda.com/blog/redpanda-official-jepsen-report-an...

rystsov 4y ago

Hey folks, I was working with Kyle Kingsbury on this report from the Redpanda side and I'm happy to help if you have questions

cgaebel 4y ago
Thanks for working with Jespen. Being willing to subject your product to their testing is a huge boon for Redpanda's credibility. I have two questions: 1. How surprising were the bugs that Jepsen found? 2. Besides the obvious regression tests for bugs that Jepsen found, how did this report change Redpanda's overall approach to testing? Were there classes of tests missing?
- rystsov 4y ago
  It wasn't a big surprise for us. Redpanda is a complex distributed system with multiple components even at the core level: consensus, idmepotency, transactions so we were ready that something might be off (but we were pleased to find that all the safety issues were with the things which were behind the feature flags at the time). Also we have internal chaos test and by the time partnership with Kyle started we already identified half of the consistency issues and sent PRs with fixes. The issues got in the report because by the time we started the changes weren't released yet. But it is acknowledged in the report > The Redpanda team already had an extensive test suite— including fault injection—prior to our collaboration. Their work found several serious issues including duplicate writes (#3039), inconsistent offsets (#3003), and aborted reads/circular information flow (#3036) before Jepsen encountered them We missed other issues because haven't exercised some scenario. As soon as Kyle found the issues we were able to reproduce them with the in-house chaos tests and fix. This dual testing (jepsen + existing chaos harness) approach was very beneficial. We were able to check the results and give feedback to Kyle if he found a real thing or if it looks more like an expected behavior. We fixed all the consistency (safety) issues, but there are several unresolved availability dips. We'll stick with Jepsen (the framework) until we're sure we fixed then too. But then we probably rely just on the in house tests. Clojure is very powerful language and I was truly amazed how fast Kyle for able to adjust his tests to new information but we don't have clojure expertise and even simple tasks take time. So it's probably wiser to use what we already know even it it a bit more verbose.
polio 4y ago
A complete nit, but the testimonial from the CTO of The Seventh Sense on https://redpanda.com/ https://redpanda.com/ spells Redpanda as "Redpand".
- northstar702 4y ago
  Thank you. Fixed.

dhshshdhdgfff 4y ago

The first half is jepsen team trying to divine some actual testable guarantees from a pile of blog posts and a random Google doc. What a mess.

excuses_ 4y ago
I wonder if Redpanda thinks about or offers some alternative protocol that would be better defined in terms of transaction guarantees. At this point it looks like Kafka’s protocol was a nice try but it needs a major refactoring.
- rystsov 4y ago
  Documentation is a bit confusing: the protocol was evolved over time (new KIPs) and there is mismatch between the database model and kafka model. But we see a lot of potential in the Kafka transactional protocol. At Redpanda we were able to push to 5k distributed transactions cross replicated shard. It's a mind-blowing for a database to achieve the same result. Also Kafka transactional protocol works at low level it's very easy to build systems on top of it. For example, it's very easy to build a Calvin inspired system http://cs.yale.edu/homes/thomson/publications/calvin-sigmod12.pdf http://cs.yale.edu/homes/thomson/publications/calvin-sigmod1...
jeffbee 4y ago
Total mess. It’s a real indictment of Kafka, more than it is anything about redpanda in the first half.
deleted 4y ago
[deleted]
rystsov 4y ago
The mess is mostly the result of the mismatch between the classic database transactional model and kafka transactional model (G0 anomaly). If you read the documentation without the database background it seems ok, but when you notice the differences between the models it becomes hard to understand if it's a bug or property of the Kafka protocol. There is a lot of research happening around this area even in the database world. The list of the isolation levels isn't final and some of the recent developments include PC-PSI and NMSI which also seem to "violate" the order. I hope one day we get the formal academic description of the Kafka model. It looks very promising.

mandevil 4y ago

I was unfamiliar with Redpanda, and now I know and trust it. Whatever marketing budget Redpanda spent to get a Jepsen report was well worth it.

belter 4y ago
Agree. Nowadays, I see anything that did not go through Jepsen with suspicion. Forces me to do the triple of technical due diligence.
hardwaresofton 4y ago
One of the clearest indications prices for a service should be raised I’ve ever seen. Can we get patio11 in here to say the thing?
- agallego 4y ago
  interested! :D
- titanomachy 4y ago
  Do you have any information on what Jepsen charges? For all we know, it could be precisely the right amount.
  - agallego 4y ago
    kyle is very friendly and I recommend reaching out. we can't and wouldn't disclose any pricing that is not public information. would be unethical on my part. all i can say is we wish to continue our work with him indefinitely as long as we keep making progress on the product :)
debarshri 4y ago
Thing that got my attention was that it has inline transform functions that can be added as wasm binary
divan 4y ago
I happened to know RedPanda founder back in the days he was at Concord.io (as a founder and a main dev). The level of obsession with performance and optimization of this guy was insane. He's not only extremelly skilled with C++, but also very passionate about rethinking large and complex systems and rebuilding them to enable 10-100x speed improvements. It's like his personal hobby – take a piece of software everyone use, and optimize it to the limits of physics, usually by implementing better version from scratch himself :) Plus, he's an excellent communicator. Watching their team working I was always thinking that successful companies can be built only with that level of passion and expertise in a single package.

CJefferson 4y ago

This isn't anything against Redpanda, but I'm always amazed how badly all these distributed databases do in Jepsen. What would one use them for in practice, which wouldn't be better suitable by a (the thing I've used), say postgresql and streaming replication in case the server goes down? (I'm not suggesting there isn't a good application, just I'm not knowledgeable enough to know of one).

claytonjy 4y ago
There's a lot of different ways to answer this, but I think about it as a different architectural paradigm. Yes you can do stream-ish things with Postgres but at some level of scale you'd be putting a square peg in a round hole. What opened my eyes to this world is this post from Martin Kleppman on turning the database inside out: https://martin.kleppmann.com/2015/03/04/turning-the-database-inside-out.html https://martin.kleppmann.com/2015/03/04/turning-the-database...
- astrange 4y ago
  > What opened my eyes to this world is this post from Martin Kleppman on turning the database inside out: https://martin.kleppmann.com/2015/03/04/turning-the-database https://martin.kleppmann.com/2015/03/04/turning-the-database... Databases are only as bad as the filesystem wrt being mutable, but since we do expect them to be a "better filesystem" it's surprising we let them get away with losing data. IMO beyond transactions you should just be able to unwind any writes to a SQL database, including deletes, for at least a day. But if you ask Alan Kay he'd say all programs should have an explicit concept of time and be able to operate on the past and future state of everything.
agallego 4y ago
totally different approaches tho. people have tried what you proposed many times before and for some scale succeeded. hard to compare at all when you dig into the details. expect a companion post. this was super fun to partner with kyle on this. +1 would recommend to anyone building a storage system.
rystsov 4y ago

newman314 4y ago

Redpanda (back when they were VectorizedIO) spammed my work email after I starred one of their repos, denied it after I called them out on it and I just noticed that they had deleted their response to me. Pretty sneaky to go back and delete the tweets first denying and then apologizing. Receipts: https://twitter.com/d11cc3s/status/1447573471152656389 https://twitter.com/d11cc3s/status/1447573471152656389 https://twitter.com/d11cc3s/status/1450906855115354116 https://twitter.com/d11cc3s/status/1450906855115354116

agallego 4y ago
hi newman314 - i mentioned in the tweet this was a mistake and offered an apology there, an sdr reached out to you, when i realized that i apologize. no ill intent. feel free to test this with a fake github account. my tweets automatically delete after 6mo, all of them on a rolling window. nothing special about this interaction. there is no sneaky-ness, though feel free to disagree.
staticassertion 4y ago
Sounds like you have a personal, singular issue with them that I can't imagine anyone else cares about.

doommius 4y ago

Always great to read this. I preformed a jenkins test on Microsoft internal infra and it's a huge insight. From an academic side it's just as interesting looking into the lack of standards within consistently and the definitions of them.

rystsov 4y ago
Cool! What did you test? I've played with Jepsen and Cosmos DB when I was at Microsoft but we had to ditch ssh, write custom agent and inject faults with PowerShell command lets.

gigatexal 4y ago

If your DB doesn't pass the Jepsen tests it's not worth using. Kudos to both teams.

titanomachy 4y ago

The level of intellectual discipline and competence on display here is inspiring. I'd love to take one of the Jepsen courses, but it seems they're offered only as corporate training. Maybe my employeer will agree to bring them in. For now I'll have to satisfy myself with the YouTube videos.

aphyr 4y ago
There's been a lot of interest in this, and I keep meaning to put together an open-to-the-public class when I'm less swamped! Might get a chance to do this shortly--I'll post on https://groups.google.com/a/jepsen.io/g/announce https://groups.google.com/a/jepsen.io/g/announce when it happens.

antonmry 4y ago

This report seems to have some wrong insights. Auto-commit offsets doesn't imply dataloss if records are processed synchronously. This is the safest way to test Kafka instead of commit offsets manually

rystsov 4y ago
Can you clarify what you mean? AFAIK with manual commit you have the most control over when the commit happens Look at this blog post describing a data loss caused by auto-commit: https://newrelic.com/blog/best-practices/kafka-consumer-config-auto-commit-data-loss https://newrelic.com/blog/best-practices/kafka-consumer-conf... Also there also may be more subtle issues with auto-commit: https://github.com/edenhill/librdkafka/issues/2782 https://github.com/edenhill/librdkafka/issues/2782
- antonmry 4y ago
  I'm afraid the article is also wrong, this is a typical misconception when working with Kafka. Offsets are committed in the next poll() invocation. If the previous messages weren't processed, a rebalance occurs and messages are processed by other instance. This is an implementation detail of the Java client library but it allows the at-least-once semantic with auto-commit. The book Effective Kafka has a better explanation. librdkafka isn't part of official Kafka so it may have problems with this as it has other limitations. In any case, the report isn't right about this and it doesn't use the safest options. Commit offsets manually is the most flexible way but it isn't easy, being the error more usual to commit offsets individually
  - aphyr 4y ago
    > Offsets are committed in the next poll() invocation. I'm a little surprised by this--not that you're necessarily wrong, but our tests consumed messages synchronously, and IIRC (pardon, it's been 3 months since I was working on Redpanda full time and my time to go get a repro case is a bit limited) did see lost messages with the default autocommit behavior. At some point I'll have to go dig into this again.
4y ago

dstroot 4y ago

> A KafkaConsumer, by contrast, will happily connect to a jar of applesauce14 and return successful, empty result sets for every call to consumer.poll. This makes it surprisingly difficult to tell the difference between “everything is fine and I’m up to date” versus “the cluster is on fire”, and led to significant confusion in our tests. This tickled my funny bone. Never expected humor in a Jepsen writeup. Kudos!

staticassertion 4y ago
> Never expected humor in a Jepsen writeup Jepsen reports are often pretty funny, some famously so
cwillu 4y ago
Wait until you find out why it's called “Jepson”
- toolz 4y ago
  please tell me it has something to do with carly jepsens song "call me maybe"
  - chipotle_coyote 4y ago
    Well, when you think about distributed databases, you have a bunch of servers "calling" one another. Some of Jepsen's testing is about disrupting that and seeing what happens. Changing "call me" to "call me maybe." If you will.
  - collinvandyck76 4y ago
    indeed :)
  - cwillu 4y ago
    You mean like this? https://aphyr.com/posts/281-call-me-maybe-carly-rae-jepsen-and-the-perils-of-network-partitions https://aphyr.com/posts/281-call-me-maybe-carly-rae-jepsen-a... https://aphyr.com/posts/283-call-me-maybe-redis https://aphyr.com/posts/283-call-me-maybe-redis

doctor_eval 4y ago

I just came here to thank Jepsen for these amazing reports. What a wonderful way to use your intellect to contribute to the wellbeing of the entire community. Also I wanted to say to redpanda: I was on the fence but now I’m convinced. Will definitely be deploying on my next project, which has already kicked off. I only wish I could run it natively on MacOS instead of requiring docker.