3 ms·

I'm still hoping Apple (or Snowflake) will pay you to do an analysis of FoundationDB…

by jwr 2y ago

tptacek 2y ago
I do too, but doesn't FDB already do a lot of the same kind of testing?
- kasey_junk 2y ago
  They are famous for doing simulation testing. https://antithesis.com/ https://antithesis.com/ Have recently brought to market a simulation testing product.
- SahAssar 2y ago
  I think they do similar testing, and therefore it might be even more interesting to read what Kyle thinks of their different approaches to it.
- jwr 2y ago
  Yes. But going through Jepsen and surviving is different. Gives an entirely new reputation to a database.
  - kasey_junk 2y ago
    I don’t think ayphr would disagree with me when I say that FDB’s testing regime is the gold standard and Jepsen is trying to get there, not the other way around.
    - aphyr 2y ago
      I'm not sure. I've worked on a few projects now which employed simulation testing and passed, only to discover serious bugs using Jepsen. State space exploration and oracle design are hard problems, and I'm not convinced there's a single, ideal path for DB testing that subsumes all others. I prefer more of a "complete breakfast" approach. On another axis: Jepsen isn't "trying to get there [to FDB's testing]" because Jepsen and FDB's tests are solving different problems. Jepsen exists to test arbitrary, third-party databases without their cooperation, or even access to the source. FoundationDB's test suite is designed to test FoundationDB, and they have political and engineering buy-in to design the database from the ground up to cooperate with a deterministic (and, I suspect, protocol-aware) simulation framework. To some extent Antithesis may be able to bridge the gap by rendering arbitrary distributed binaries deterministic. Something I'd like to explore!
      - kasey_junk 2y ago
        Has your opinion changed on that in the last few years? I could have sworn you were on record as saying this about foundation in the past but I couldn’t find it in my links.
        aphyr 2y ago
        I don't think so, but I've said a lot about databases in the last fifteen years haha. Sometimes I look at what people say about FDB and it feels like... folks are putting words in my mouth that I don't recognize. I was very impressed by a short phone conversation with their engineers ~12 years ago. That's good, but that's not, like, a substantive experimental evaluation. That's "I focus my unpaid efforts on databases which seem more likely to yield fun, interesting results".
        kasey_junk 2y ago
        Fair enough.
        jwr 2y ago
        So, will we get an evaluation of FDB one day? Pretty please? :-)
        aphyr 2y ago
        Apple is positively swimming in money! They could pay me! (Hi, Apple ;-))
        DylanSp 2y ago
        Looks like it was an offhand tweet from 2013: https://web.archive.org/web/20220805112242/https://twitter.com/obfuscurity/status/405016890306985984 https://web.archive.org/web/20220805112242/https://twitter.c.... I got that from a comment on the first Antithesis post on HN, https://news.ycombinator.com/item?id=39376195 https://news.ycombinator.com/item?id=39376195.
      - tptacek 2y ago

I'm still hoping Apple (or Snowflake) will pay you to do an analysis of FoundationDB…

by jwr 2y ago

I'm still hoping Apple (or Snowflake) will pay you to do an analysis of FoundationDB…

tptacek 2y ago

I do too, but doesn't FDB already do a lot of the same kind of testing?

kasey_junk 2y ago
They are famous for doing simulation testing. https://antithesis.com/ https://antithesis.com/ Have recently brought to market a simulation testing product.
SahAssar 2y ago
I think they do similar testing, and therefore it might be even more interesting to read what Kyle thinks of their different approaches to it.
jwr 2y ago
Yes. But going through Jepsen and surviving is different. Gives an entirely new reputation to a database.
- kasey_junk 2y ago
  I don’t think ayphr would disagree with me when I say that FDB’s testing regime is the gold standard and Jepsen is trying to get there, not the other way around.
  - aphyr 2y ago
    I'm not sure. I've worked on a few projects now which employed simulation testing and passed, only to discover serious bugs using Jepsen. State space exploration and oracle design are hard problems, and I'm not convinced there's a single, ideal path for DB testing that subsumes all others. I prefer more of a "complete breakfast" approach. On another axis: Jepsen isn't "trying to get there [to FDB's testing]" because Jepsen and FDB's tests are solving different problems. Jepsen exists to test arbitrary, third-party databases without their cooperation, or even access to the source. FoundationDB's test suite is designed to test FoundationDB, and they have political and engineering buy-in to design the database from the ground up to cooperate with a deterministic (and, I suspect, protocol-aware) simulation framework. To some extent Antithesis may be able to bridge the gap by rendering arbitrary distributed binaries deterministic. Something I'd like to explore!
    - kasey_junk 2y ago
      Has your opinion changed on that in the last few years? I could have sworn you were on record as saying this about foundation in the past but I couldn’t find it in my links.
      - aphyr 2y ago
        I don't think so, but I've said a lot about databases in the last fifteen years haha. Sometimes I look at what people say about FDB and it feels like... folks are putting words in my mouth that I don't recognize. I was very impressed by a short phone conversation with their engineers ~12 years ago. That's good, but that's not, like, a substantive experimental evaluation. That's "I focus my unpaid efforts on databases which seem more likely to yield fun, interesting results".
        kasey_junk 2y ago
        Fair enough.
        jwr 2y ago
        So, will we get an evaluation of FDB one day? Pretty please? :-)
        aphyr 2y ago
        Apple is positively swimming in money! They could pay me! (Hi, Apple ;-))
      - DylanSp 2y ago
        Looks like it was an offhand tweet from 2013: https://web.archive.org/web/20220805112242/https://twitter.com/obfuscurity/status/405016890306985984 https://web.archive.org/web/20220805112242/https://twitter.c.... I got that from a comment on the first Antithesis post on HN, https://news.ycombinator.com/item?id=39376195 https://news.ycombinator.com/item?id=39376195.
    - tptacek 2y ago