10 ms·

FaunaDB 2.5.4

263 points by aphyr 8y ago

drej 8y ago
I recommend first reading bits of the Jepsen report, because the company blog paints quite a different picture. > We’re excited to report that FaunaDB has passed: > Additionally, it offers the highest possible level of correctness: > In consultation with Kyle, we’ve fixed many known issues and bugs vs. > However, queries involving indices, temporal queries, or event streams failed to live up to claimed guarantees. We found 19 issues in FaunaDB[.]
- deleted 8y ago
  [deleted]
- deleted 8y ago
  [deleted]
- greggh 8y ago
  Your comment doesnt seem fair at all. The blog post is from after they worked hard to fix everything. The comment from Jepsen that you are quoting is from before any of that fix work was done. You are comparing two very different situations.
  - drej 8y ago
    I know they worked hard, I appreciate that. But I got a wholly different feeling about the state of things from those two sources. Also, does everyone run the very latest version of all their software? What use to me is that my vendor has fixed everything in the newest release that I am not using? Oh and yes, I'm only quoting bits of each (fully knowing you all have links to both and can read it in full), but that's to illustrate the omission from the PR piece. I know that aphyr concludes that work has been done, "By 2.6.0-rc10, Fauna had addressed almost all issues we identified; some minor work around availability and schema changes is still in progress.", but that doesn't change the fact that the blog post doesn't address their past shortcomings.
    - deleted 8y ago
      [deleted]
    - freels 8y ago
      The latest released version, 2.6.0, has most of the fixes as outlined in the post & report. Minor/ergonomic fixes are forthcoming. If you’re running on cloud, we do the upgrade work for you.
    - samcday 8y ago
      Why should they be required to "address their past shortcomings"? Last I checked, when I visit the Windows website it doesn't say "Really stable now but we've had 1,123,432 critical security bugs in the past!". Same for any other product or open source project. Acknowledging current limitations is an absolute must, and posting thoughtful articles that delve into a past issue and how it was addressed are a bonus. Otherwise, there's no need for self-flagellation ;)
      - drej 8y ago
        Um, that simile does not make any sense - because we're not talking about their website, but their blog about Jepsen tests, which showed some shortcomings (and it is not derogatory, it's extremely rare that somebody passes Jepsen with flying colours).
    - jwr 8y ago
      Approach like yours causes companies not to submit to Jepsen testing and try to hide shortcomings. They did submit, they found problems (it is rare for the Jepsen suite not to find any), and they fixed them. This is a fantastic result and definitely one to be proud of. Additionally, I am not sure if you fully appreciate the complexity of Jepsen and distributed databases in general. As for me, I've actually been waiting for this result to recommend the use of FaunaDB in a commercial setting.
      - drej 8y ago
        Oh I do appreciate it - I've read Martin Kleppmann's book cover to cover and then watched all of the speeches Kyle Kingsbury has given in the past three years or so. I love this area, my absolute favourite is deterministic testing of FoundationDB [0]. It's because I appreciate this work that I felt the blog post didn't do it justice. And I know Jepsen hardly ever passes (ZooKeeper, I believe, did). And I don't take FaunaDB's hard work for granted. [0] https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc
- zellyn 8y ago
  First Jepsen report, huh? That was pretty complimentary, as these things go!
  - deleted 8y ago
    [deleted]
evanweaver 8y ago
So excited to see this; this is the culmination of three months of very hard work by both teams. FaunaDB 2.5 passed the core linearizability tests for multi-partition transactions immediately. To my knowledge no other distributed system has done this. Zookeeper was the strongest candidate on initial testing in the early days, but it does not offer multiple partitions at all, as discussed in the FaunaDB report. And Jepsen itself was much less comprehensive at the time. All other issues affecting correctness were fixed in the course of the analysis, and FaunaDB 2.6 is now available with the improvements. We're happy to answer questions along with @aphyr. Our blog post is here: https://fauna.com/blog/faunadbs-official-jepsen-results https://fauna.com/blog/faunadbs-official-jepsen-results
- judofyr 8y ago
  Congratulations; this looks like some seriously nice results! I haven't used FaunaDB, but it's nice seeing your name around. I still remember your personal blog (blue/grey color; with flowers, right?) fondly from way back, and I believe also "Fauna" existed back then, but it was more an umbrella project for your open source work? Anyway, FaunaDB looks like a well-engineered product in a highly competitive space and I hope you will succeed :)
- lobo_tuerto 8y ago
  What programming language did you use to develop FaunaDB?
  - northstar702 8y ago
    Scala/Java
- AtlasBarfed 8y ago
  I greatly appreciate any distributed system that subjects itself to jepsen. It shows real commitment to honesty and a genuine desire to improve a db tech. One thing that seems to fall by the wayside frequently is a followup. Often the problems found are declared solved after a couple point releases in a blog. Elasticsearch did this to my disappointment, and so did Cassandra. Reading your stuff, it appears you scale with complete replicas of a database on individual nodes. Is that still true?
jatsign 8y ago
I hadn't heard of Fauna before. What's the use case? Looks like it's not open source, and the pricing isn't very clear if I want to host it locally. The "Download" page requires you to provide your contact info first. Why should I go through all those hoops?
- jchrisa 8y ago
  TLDR: FaunaDB is a distributed database that offers multi-partition ACID transactions and is optimized for cloud / multi-cloud deployment. This is a big deal because there aren't many other options with this level of data integrity at worldwide scale. Use cases include financial services, retail, user identity, game world state, etc. Basically anything you'd put in an operational database. In addition to the download (free for one node), you can get started with FaunaDB Serverless Cloud for free in moments. It's a fully managed version that is used in production by some major sites and many smaller apps.
  - jatsign 8y ago
    Sorry, I'm not trying to be snarky, but that sounds like "it's good for everything" and doesn't help me determine when I should look at Fauna and when I shouldn't. Perhaps alternately - when is Fauna NOT a good choice?
    - evanweaver 8y ago
      Well, it is a general-purpose database with a document-relational programming model. It's a good choice when correctness, high availability, and flexible data modeling matter, like the core business objects of a service or app. Currently it supports its own native DSL similar to LINQ or an ORM, and if you want compatibility with existing query languages from other databases, we will be rolling those out over time. It's not a good choice for analytics or time series data which is redundant or aggregated, and doesn't need the high availability or performance overhead of transactional isolation.
    - deleted 8y ago
      [deleted]
asien 8y ago
Tried Fauna once with their « Cloud » versions. I was absolutely shocked by the poor performance of the service. In my case I prototype some simple CRUD queries with NodeJS ,within the same datacenter region. Insert took well over a second to complete and reading a simple document with one field took also half a second. I was also unable to make « join » between document because how complex their query language is and their support basically encouraged me not to use « join » but to use « aggregate » like mongo ... Why offer this feature if I can’t use it ? Has it changed since then ? It seems very clear for me that Fauna is entirely focused on Enterprises customers ( after all this is where the money is ) the cloud version seem to be just a gimmick.
- evanweaver 8y ago
  Hmm, when did you try it? FaunaDB and the cloud service have changed a lot in the last few years, and performance is always improving. Typical write latencies in Cloud are in the 100ms range because the data is globally replicated. Typical read latencies are the 1ms-10ms range, because global coordination is not required, discounting the additional latency from the client to the closest datacenter. If you experienced something worse than that recently, maybe there is some other issue going on.
  - asien 8y ago
    I will give it a second shot then. But just to confirm , is doing « join » something that is still not recommended ? Aggregation is tedious and lead to queries that are difficult to read and maintain.
    - freels 8y ago
      It's definitely recommended, depending on what you're trying to get out of the result. Happy to help out with query patterns on our community slack https://publicslack.com/slacks/fauna-community/invites/new https://publicslack.com/slacks/fauna-community/invites/new
georgewfraser 8y ago
Fauna’s writeup heavily emphasizes the fact that it doesn’t rely on atomic clocks. My understanding is that both AWS and GCP use atomic clock based timekeepers since 2017, so it’s not like this is some exotic technology. The primary advantage described in the Calvin papers is that it’s the only distributed transaction protocol that can handle high contention workloads. But Fauna never seems to bring this up. Does that mean that Fauna’s current implementation isn’t fast under contention?
- freels 8y ago
  It does handle contention well, we haven't emphasized that point well enough yet. Writes never contend on conflicting reads, and serialized reads never contend at all. Accurate clocks are not enough... to really get the benefits that Spanner alone enjoys, you have to have a TrueTime equivalent service available, and it has to be rock solid. As well once your system is sensitive to clock skews in the milliseconds, you start having to care about things like the leap-second policies of your clock sources. All in all, the resiliency tradeoffs are a significant downside to relying on clock synchronization, which is why we did not pursue a transaction protocol dependent on it.
  - georgewfraser 8y ago
    My understanding is that AWS TimeSync and the current GCP time daemon implementation are the equivalent of TrueTime, is that incorrect? I guess what I’m saying is it seems like Fauna is using atomic clocks as FUD against Spanner and CockroachDB, when they aren’t really a problem. Based on my reading of the Calvin paper, the main advantage of Calvin style systems is higher throughout under contention. But for some reason the Fauna marketing team has chosen not to emphasize that, which makes me suspicious that maybe Fauna hasn’t yet realized that advantage in its implementation.
    - deleted 8y ago
      [deleted]
    - evanweaver 8y ago
twic 8y ago
FaunaDB uses Calvin, a transaction protocol developed by Daniel Abadi. Their blog post explains it nicely, after a bit of a slow start: https://fauna.com/blog/consistency-without-clocks-faunadb-transaction-protocol https://fauna.com/blog/consistency-without-clocks-faunadb-tr... But in summary: 1. A 'transaction' is a self-contained blob of code which reads input, does deterministic logic, and writes output (so not like a traditional RDBMS transaction, where the application opens a transaction and then interleaves its own logic between reads and writes) 2. When a transaction arrives, the receiving node runs it, and captures the inputs it read, and the outputs it wrote 3. The transaction, with its captured inputs and outputs, is written to a global stream of transactions - this is the only point of synchronisation between the nodes 4. Each node reads the global stream, and writes each transaction into its persistent state; to do that, it repeats all the reads that the transaction did, and checks that they match the captured input - if so, the outputs are committed, and it not, the transaction is aborted, and retried The key idea is that because the process is deterministic, the nodes can write transactions to disk independently without drifting out of sync. It's pretty neat. And it's exactly what Abadi wrote about a couple of months ago: http://dbmsmusings.blogspot.com/2019/01/its-time-to-move-on-from-two-phase.html http://dbmsmusings.blogspot.com/2019/01/its-time-to-move-on-... This is also what VoltDB does (which Abadi worked on, along with Michael Stonebraker): As an operational store, the VoltDB “operations” in question are actually full ACID transactions, with multiple rounds of reads, writes and conditional logic. If the system is going to run transactions to completion, one after another, disk latency isn’t the only stall that must be eliminated; it is also necessary to eliminate waiting on the user mid-transaction. That means external transaction control is out – no stopping a transaction in the middle to make a network round-trip to the client for the next action. The team made a decision to move logic server-side and use stored procedures. https://www.voltdb.com/product/data-architecture/no-wait-design/ https://www.voltdb.com/product/data-architecture/no-wait-des... It's also similar to, although categorically more sophisticated than, the idea of object prevalence, which is now so old and forgotten that i can't find any really good references, but: Clients communicate with the prevalent system by executing transactions, which are implemented by a set of transaction classes. These are examples of the Command design pattern [Gamma 1995]. Transactions are written to a journal when they are executed. If the prevalent system crashes, its state can be recovered by reading the journal and executing the transactions again. [...] Replaying the journal must always give the same result, so transactions must be deterministic. Although clients can have a high degree of concurrency, the prevalent system is single-threaded, and transactions execute to completion. https://web.archive.org/web/20170610140344/http://hillside.net/sugarloafplop/papers/5.pdf https://web.archive.org/web/20170610140344/http://hillside.n...
gigatexal 8y ago
If they added a standard SQL layer I’d be onboard. Interesting project though.
- devDan 8y ago
  They are adding a GraphQL layer in. It's in prototype mode right now, I believe, but we should see it soon.
buremba 8y ago
Looks great but why did you decide to develop your own query language instead of just using SQL? Even no-sql transactional database solutions started to adopt SQL lately and learning a new language is not really easy for the application developers.
- aphyr 8y ago
  Ooh, I'll bite! For one, they're different data models. FaunaDB is a document store, so records are fundamentally trees, whereas SQL is oriented around processing tuples. FaunaDB records have queryable metadata; SQL rows (I think?) don't. You can extend SQL (look at JSON support in Postgres) to deal with arrays and maps as core datatypes, but a special-purpose language can be better suited to the purpose. Second, FaunaDB's transactional model precludes interactive transactions, whereas SQL transactions are designed for interactive use. Imagine if every transaction was a stored procedure--that's the query structure you'd be looking at. It's certainly possible to do, but stored procedures are sort of an imperative language grafted on to the relational algebra of SQL, and support isn't as standardized as SQL's core. Third, FaunaDB is a temporal store--you can ask for the state of any query at any point in time, and even mix temporal scopes in the same query expression. SQL doesn't have a first-class temporal model. In general, using SQL offers advantages, including user familiarity, code reuse, and easier migration from other SQL stores. None of the things FaunaDB does are impossible to express in SQL, and they have been tackled by various DBs' extensions to SQL, but the familiarity+reuse advantages aren't as applicable once you start thinking about the distinct properties of FaunaDB's data model.
- freels 8y ago
  We are always trying to reduce the learning curve of FQL, but we've gotten a lot of good feedback from those who do take the plunge. That being said, I'd love to provide SQL support eventually, but there were a few reasons why we don't support it as the primary query language for FaunaDB: Our core use-case is OLTP, and we wanted to address the shortcomings of SQL in this context, such as query performance being highly unpredictable in the presence or not of indexes, whims of the optimizer, etc. SQL is just not great as an application-level interface: Tables are not a natural fit for many data models (the classic impedance mismatch problem), programmatic composition is difficult. We want to obviate the necessity of an ORM library on top Fauna. SQL the language is not great for writing complex transactions in, and session transactions require a lot of back and forth between the client and database. We wanted to make it easy to write queries which can encode as much business logic as possible, hence FQL's semantics are a lot closer to a regular programming language.
wmsiler 8y ago
From the post, FaunaDB initially had several issues, which they've generally resolved. Jepsen is open source, so I'm curious why a database company wouldn't run Jepsen internally, work out as many problems as they can, and then engage aphyr in order to get the official thumbs up. Given how important data integrity is, I would assume that any database company would be running Jepsen (or something equivalent) regularly in-house. If they are doing that, then how is it that aphyr finds so many previously unknown issues? And if they aren't running Jepsen in-house, why not?
- deleted 8y ago
  [deleted]
- evanweaver 8y ago
  This is a very good question, and to a substantial degree, this is what we did. We have internal QA systems that overlap Jepsen that catch most issues. We also ran our own Jepsen tests on the core product properties last year, fixed some issues and identified others, and reported the results on our blog. However, correctness testing is fundamentally adversarial, like security penetration testing. Building a database is not easy, and testing a database is not easy either. It is a separate skill set, as anomalies that lingered for decades in other databases reveal. The engagement with the Jepsen team is explicitly designed to explore the entire product surface area for faults, not to apply Jepsen as it currently stands. Thus, a lot of custom work ensued on both sides to make sure that the database was both properly testable, and properly tested. The result of that work is what you see in the report. The typical Jepsen report implicates not just implementation bugs, but the entire architecture of the system itself. Jepsen usually identifies anomalies that cannot be prevented even with a perfect implementation, which didn't happen here. Some vendors restrict their engagement with the Jepsen team to only what they have tested themselves already, although those tests are not always valid. This was not our mindset—we wanted to improve our database by taking advantage of Kyle’s expertise, not present a superficially perfect report that failed to actually exercise the potential faults of the system.
  - wmsiler 8y ago
    Thanks for the response. That all makes sense. I assume the FaunaDB devs would have already tested and fixed all the scenarios they could come up with, so it's reasonable you'd want an outside party to come up with even more scenarios to examine.
etaioinshrdlu 8y ago
This actually reminds me a lot of how Ethereum transactions are represented as code as well. Anyone else see a parallel there? Seems like a good idea, overall. One annoying thing that affects pretty much every database with transactions is that the effort of retrying failed transactions is pushed onto the user, by necessity. But if your transactions are airtight chunks of code... then the DB can retry them for you and provide a simpler interface to your app code.
- cakoose 8y ago
  > One annoying thing that affects pretty much every database with transactions is that the effort of retrying failed transactions is pushed onto the user, by necessity. > > But if your transactions are airtight chunks of code... then the DB can retry them for you and provide a simpler interface to your app code. Building an FaunaDB query is more difficult than just writing session-based DB code. But if you are willing to build FaunaDB queries, it should be strictly easier to write session-based DB code as "airtight" chunks that are easy to retry.
jwr 8y ago
I know this is slightly off-topic, but I'd be very interested in Jepsen testing FoundationDB. They claim to have developed the database test-first (starting with simulations), and it would be great to be able to compare the claims to reality using an external (by now becoming an industry standard!) testing tool.
- misframer 8y ago
  This was done a few years ago. See https://web.archive.org/web/20150325003526/http://blog.foundationdb.com/call-me-maybe-foundationdb-vs-jepsen https://web.archive.org/web/20150325003526/http://blog.found....
anentropic 8y ago
That was pretty great. Does anyone have links to tests of FaunaDB write performance?
- evanweaver 8y ago
  We are working on new benchmarks.

263 points by aphyr 8y ago

drej 8y ago

I recommend first reading bits of the Jepsen report, because the company blog paints quite a different picture. > We’re excited to report that FaunaDB has passed: > Additionally, it offers the highest possible level of correctness: > In consultation with Kyle, we’ve fixed many known issues and bugs vs. > However, queries involving indices, temporal queries, or event streams failed to live up to claimed guarantees. We found 19 issues in FaunaDB[.]

deleted 8y ago
[deleted]
deleted 8y ago
[deleted]
greggh 8y ago
Your comment doesnt seem fair at all. The blog post is from after they worked hard to fix everything. The comment from Jepsen that you are quoting is from before any of that fix work was done. You are comparing two very different situations.
- drej 8y ago
  I know they worked hard, I appreciate that. But I got a wholly different feeling about the state of things from those two sources. Also, does everyone run the very latest version of all their software? What use to me is that my vendor has fixed everything in the newest release that I am not using? Oh and yes, I'm only quoting bits of each (fully knowing you all have links to both and can read it in full), but that's to illustrate the omission from the PR piece. I know that aphyr concludes that work has been done, "By 2.6.0-rc10, Fauna had addressed almost all issues we identified; some minor work around availability and schema changes is still in progress.", but that doesn't change the fact that the blog post doesn't address their past shortcomings.
  - deleted 8y ago
    [deleted]
  - freels 8y ago
    The latest released version, 2.6.0, has most of the fixes as outlined in the post & report. Minor/ergonomic fixes are forthcoming. If you’re running on cloud, we do the upgrade work for you.
  - samcday 8y ago
    Why should they be required to "address their past shortcomings"? Last I checked, when I visit the Windows website it doesn't say "Really stable now but we've had 1,123,432 critical security bugs in the past!". Same for any other product or open source project. Acknowledging current limitations is an absolute must, and posting thoughtful articles that delve into a past issue and how it was addressed are a bonus. Otherwise, there's no need for self-flagellation ;)
    - drej 8y ago
      Um, that simile does not make any sense - because we're not talking about their website, but their blog about Jepsen tests, which showed some shortcomings (and it is not derogatory, it's extremely rare that somebody passes Jepsen with flying colours).
  - jwr 8y ago
    Approach like yours causes companies not to submit to Jepsen testing and try to hide shortcomings. They did submit, they found problems (it is rare for the Jepsen suite not to find any), and they fixed them. This is a fantastic result and definitely one to be proud of. Additionally, I am not sure if you fully appreciate the complexity of Jepsen and distributed databases in general. As for me, I've actually been waiting for this result to recommend the use of FaunaDB in a commercial setting.
    - drej 8y ago
      Oh I do appreciate it - I've read Martin Kleppmann's book cover to cover and then watched all of the speeches Kyle Kingsbury has given in the past three years or so. I love this area, my absolute favourite is deterministic testing of FoundationDB [0]. It's because I appreciate this work that I felt the blog post didn't do it justice. And I know Jepsen hardly ever passes (ZooKeeper, I believe, did). And I don't take FaunaDB's hard work for granted. [0] https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc
zellyn 8y ago
First Jepsen report, huh? That was pretty complimentary, as these things go!
- deleted 8y ago
  [deleted]

evanweaver 8y ago

So excited to see this; this is the culmination of three months of very hard work by both teams. FaunaDB 2.5 passed the core linearizability tests for multi-partition transactions immediately. To my knowledge no other distributed system has done this. Zookeeper was the strongest candidate on initial testing in the early days, but it does not offer multiple partitions at all, as discussed in the FaunaDB report. And Jepsen itself was much less comprehensive at the time. All other issues affecting correctness were fixed in the course of the analysis, and FaunaDB 2.6 is now available with the improvements. We're happy to answer questions along with @aphyr. Our blog post is here: https://fauna.com/blog/faunadbs-official-jepsen-results https://fauna.com/blog/faunadbs-official-jepsen-results

judofyr 8y ago
Congratulations; this looks like some seriously nice results! I haven't used FaunaDB, but it's nice seeing your name around. I still remember your personal blog (blue/grey color; with flowers, right?) fondly from way back, and I believe also "Fauna" existed back then, but it was more an umbrella project for your open source work? Anyway, FaunaDB looks like a well-engineered product in a highly competitive space and I hope you will succeed :)
lobo_tuerto 8y ago
What programming language did you use to develop FaunaDB?
- northstar702 8y ago
  Scala/Java
AtlasBarfed 8y ago
I greatly appreciate any distributed system that subjects itself to jepsen. It shows real commitment to honesty and a genuine desire to improve a db tech. One thing that seems to fall by the wayside frequently is a followup. Often the problems found are declared solved after a couple point releases in a blog. Elasticsearch did this to my disappointment, and so did Cassandra. Reading your stuff, it appears you scale with complete replicas of a database on individual nodes. Is that still true?

jatsign 8y ago

I hadn't heard of Fauna before. What's the use case? Looks like it's not open source, and the pricing isn't very clear if I want to host it locally. The "Download" page requires you to provide your contact info first. Why should I go through all those hoops?

jchrisa 8y ago
TLDR: FaunaDB is a distributed database that offers multi-partition ACID transactions and is optimized for cloud / multi-cloud deployment. This is a big deal because there aren't many other options with this level of data integrity at worldwide scale. Use cases include financial services, retail, user identity, game world state, etc. Basically anything you'd put in an operational database. In addition to the download (free for one node), you can get started with FaunaDB Serverless Cloud for free in moments. It's a fully managed version that is used in production by some major sites and many smaller apps.
- jatsign 8y ago
  Sorry, I'm not trying to be snarky, but that sounds like "it's good for everything" and doesn't help me determine when I should look at Fauna and when I shouldn't. Perhaps alternately - when is Fauna NOT a good choice?
  - evanweaver 8y ago
    Well, it is a general-purpose database with a document-relational programming model. It's a good choice when correctness, high availability, and flexible data modeling matter, like the core business objects of a service or app. Currently it supports its own native DSL similar to LINQ or an ORM, and if you want compatibility with existing query languages from other databases, we will be rolling those out over time. It's not a good choice for analytics or time series data which is redundant or aggregated, and doesn't need the high availability or performance overhead of transactional isolation.
  - deleted 8y ago
    [deleted]

asien 8y ago

Tried Fauna once with their « Cloud » versions. I was absolutely shocked by the poor performance of the service. In my case I prototype some simple CRUD queries with NodeJS ,within the same datacenter region. Insert took well over a second to complete and reading a simple document with one field took also half a second. I was also unable to make « join » between document because how complex their query language is and their support basically encouraged me not to use « join » but to use « aggregate » like mongo ... Why offer this feature if I can’t use it ? Has it changed since then ? It seems very clear for me that Fauna is entirely focused on Enterprises customers ( after all this is where the money is ) the cloud version seem to be just a gimmick.

evanweaver 8y ago
Hmm, when did you try it? FaunaDB and the cloud service have changed a lot in the last few years, and performance is always improving. Typical write latencies in Cloud are in the 100ms range because the data is globally replicated. Typical read latencies are the 1ms-10ms range, because global coordination is not required, discounting the additional latency from the client to the closest datacenter. If you experienced something worse than that recently, maybe there is some other issue going on.
- asien 8y ago
  I will give it a second shot then. But just to confirm , is doing « join » something that is still not recommended ? Aggregation is tedious and lead to queries that are difficult to read and maintain.
  - freels 8y ago
    It's definitely recommended, depending on what you're trying to get out of the result. Happy to help out with query patterns on our community slack https://publicslack.com/slacks/fauna-community/invites/new https://publicslack.com/slacks/fauna-community/invites/new

georgewfraser 8y ago

Fauna’s writeup heavily emphasizes the fact that it doesn’t rely on atomic clocks. My understanding is that both AWS and GCP use atomic clock based timekeepers since 2017, so it’s not like this is some exotic technology. The primary advantage described in the Calvin papers is that it’s the only distributed transaction protocol that can handle high contention workloads. But Fauna never seems to bring this up. Does that mean that Fauna’s current implementation isn’t fast under contention?

freels 8y ago
It does handle contention well, we haven't emphasized that point well enough yet. Writes never contend on conflicting reads, and serialized reads never contend at all. Accurate clocks are not enough... to really get the benefits that Spanner alone enjoys, you have to have a TrueTime equivalent service available, and it has to be rock solid. As well once your system is sensitive to clock skews in the milliseconds, you start having to care about things like the leap-second policies of your clock sources. All in all, the resiliency tradeoffs are a significant downside to relying on clock synchronization, which is why we did not pursue a transaction protocol dependent on it.
- georgewfraser 8y ago
  My understanding is that AWS TimeSync and the current GCP time daemon implementation are the equivalent of TrueTime, is that incorrect? I guess what I’m saying is it seems like Fauna is using atomic clocks as FUD against Spanner and CockroachDB, when they aren’t really a problem. Based on my reading of the Calvin paper, the main advantage of Calvin style systems is higher throughout under contention. But for some reason the Fauna marketing team has chosen not to emphasize that, which makes me suspicious that maybe Fauna hasn’t yet realized that advantage in its implementation.
  - deleted 8y ago
    [deleted]
  - evanweaver 8y ago

twic 8y ago

FaunaDB uses Calvin, a transaction protocol developed by Daniel Abadi. Their blog post explains it nicely, after a bit of a slow start: https://fauna.com/blog/consistency-without-clocks-faunadb-transaction-protocol https://fauna.com/blog/consistency-without-clocks-faunadb-tr... But in summary: 1. A 'transaction' is a self-contained blob of code which reads input, does deterministic logic, and writes output (so not like a traditional RDBMS transaction, where the application opens a transaction and then interleaves its own logic between reads and writes) 2. When a transaction arrives, the receiving node runs it, and captures the inputs it read, and the outputs it wrote 3. The transaction, with its captured inputs and outputs, is written to a global stream of transactions - this is the only point of synchronisation between the nodes 4. Each node reads the global stream, and writes each transaction into its persistent state; to do that, it repeats all the reads that the transaction did, and checks that they match the captured input - if so, the outputs are committed, and it not, the transaction is aborted, and retried The key idea is that because the process is deterministic, the nodes can write transactions to disk independently without drifting out of sync. It's pretty neat. And it's exactly what Abadi wrote about a couple of months ago: http://dbmsmusings.blogspot.com/2019/01/its-time-to-move-on-from-two-phase.html http://dbmsmusings.blogspot.com/2019/01/its-time-to-move-on-... This is also what VoltDB does (which Abadi worked on, along with Michael Stonebraker): As an operational store, the VoltDB “operations” in question are actually full ACID transactions, with multiple rounds of reads, writes and conditional logic. If the system is going to run transactions to completion, one after another, disk latency isn’t the only stall that must be eliminated; it is also necessary to eliminate waiting on the user mid-transaction. That means external transaction control is out – no stopping a transaction in the middle to make a network round-trip to the client for the next action. The team made a decision to move logic server-side and use stored procedures. https://www.voltdb.com/product/data-architecture/no-wait-design/ https://www.voltdb.com/product/data-architecture/no-wait-des... It's also similar to, although categorically more sophisticated than, the idea of object prevalence, which is now so old and forgotten that i can't find any really good references, but: Clients communicate with the prevalent system by executing transactions, which are implemented by a set of transaction classes. These are examples of the Command design pattern [Gamma 1995]. Transactions are written to a journal when they are executed. If the prevalent system crashes, its state can be recovered by reading the journal and executing the transactions again. [...] Replaying the journal must always give the same result, so transactions must be deterministic. Although clients can have a high degree of concurrency, the prevalent system is single-threaded, and transactions execute to completion. https://web.archive.org/web/20170610140344/http://hillside.net/sugarloafplop/papers/5.pdf https://web.archive.org/web/20170610140344/http://hillside.n...

gigatexal 8y ago

If they added a standard SQL layer I’d be onboard. Interesting project though.

devDan 8y ago
They are adding a GraphQL layer in. It's in prototype mode right now, I believe, but we should see it soon.

buremba 8y ago

Looks great but why did you decide to develop your own query language instead of just using SQL? Even no-sql transactional database solutions started to adopt SQL lately and learning a new language is not really easy for the application developers.

aphyr 8y ago
Ooh, I'll bite! For one, they're different data models. FaunaDB is a document store, so records are fundamentally trees, whereas SQL is oriented around processing tuples. FaunaDB records have queryable metadata; SQL rows (I think?) don't. You can extend SQL (look at JSON support in Postgres) to deal with arrays and maps as core datatypes, but a special-purpose language can be better suited to the purpose. Second, FaunaDB's transactional model precludes interactive transactions, whereas SQL transactions are designed for interactive use. Imagine if every transaction was a stored procedure--that's the query structure you'd be looking at. It's certainly possible to do, but stored procedures are sort of an imperative language grafted on to the relational algebra of SQL, and support isn't as standardized as SQL's core. Third, FaunaDB is a temporal store--you can ask for the state of any query at any point in time, and even mix temporal scopes in the same query expression. SQL doesn't have a first-class temporal model. In general, using SQL offers advantages, including user familiarity, code reuse, and easier migration from other SQL stores. None of the things FaunaDB does are impossible to express in SQL, and they have been tackled by various DBs' extensions to SQL, but the familiarity+reuse advantages aren't as applicable once you start thinking about the distinct properties of FaunaDB's data model.
freels 8y ago
We are always trying to reduce the learning curve of FQL, but we've gotten a lot of good feedback from those who do take the plunge. That being said, I'd love to provide SQL support eventually, but there were a few reasons why we don't support it as the primary query language for FaunaDB: Our core use-case is OLTP, and we wanted to address the shortcomings of SQL in this context, such as query performance being highly unpredictable in the presence or not of indexes, whims of the optimizer, etc. SQL is just not great as an application-level interface: Tables are not a natural fit for many data models (the classic impedance mismatch problem), programmatic composition is difficult. We want to obviate the necessity of an ORM library on top Fauna. SQL the language is not great for writing complex transactions in, and session transactions require a lot of back and forth between the client and database. We wanted to make it easy to write queries which can encode as much business logic as possible, hence FQL's semantics are a lot closer to a regular programming language.

wmsiler 8y ago

From the post, FaunaDB initially had several issues, which they've generally resolved. Jepsen is open source, so I'm curious why a database company wouldn't run Jepsen internally, work out as many problems as they can, and then engage aphyr in order to get the official thumbs up. Given how important data integrity is, I would assume that any database company would be running Jepsen (or something equivalent) regularly in-house. If they are doing that, then how is it that aphyr finds so many previously unknown issues? And if they aren't running Jepsen in-house, why not?

deleted 8y ago
[deleted]
evanweaver 8y ago
This is a very good question, and to a substantial degree, this is what we did. We have internal QA systems that overlap Jepsen that catch most issues. We also ran our own Jepsen tests on the core product properties last year, fixed some issues and identified others, and reported the results on our blog. However, correctness testing is fundamentally adversarial, like security penetration testing. Building a database is not easy, and testing a database is not easy either. It is a separate skill set, as anomalies that lingered for decades in other databases reveal. The engagement with the Jepsen team is explicitly designed to explore the entire product surface area for faults, not to apply Jepsen as it currently stands. Thus, a lot of custom work ensued on both sides to make sure that the database was both properly testable, and properly tested. The result of that work is what you see in the report. The typical Jepsen report implicates not just implementation bugs, but the entire architecture of the system itself. Jepsen usually identifies anomalies that cannot be prevented even with a perfect implementation, which didn't happen here. Some vendors restrict their engagement with the Jepsen team to only what they have tested themselves already, although those tests are not always valid. This was not our mindset—we wanted to improve our database by taking advantage of Kyle’s expertise, not present a superficially perfect report that failed to actually exercise the potential faults of the system.
- wmsiler 8y ago
  Thanks for the response. That all makes sense. I assume the FaunaDB devs would have already tested and fixed all the scenarios they could come up with, so it's reasonable you'd want an outside party to come up with even more scenarios to examine.

etaioinshrdlu 8y ago

This actually reminds me a lot of how Ethereum transactions are represented as code as well. Anyone else see a parallel there? Seems like a good idea, overall. One annoying thing that affects pretty much every database with transactions is that the effort of retrying failed transactions is pushed onto the user, by necessity. But if your transactions are airtight chunks of code... then the DB can retry them for you and provide a simpler interface to your app code.

cakoose 8y ago
> One annoying thing that affects pretty much every database with transactions is that the effort of retrying failed transactions is pushed onto the user, by necessity. > > But if your transactions are airtight chunks of code... then the DB can retry them for you and provide a simpler interface to your app code. Building an FaunaDB query is more difficult than just writing session-based DB code. But if you are willing to build FaunaDB queries, it should be strictly easier to write session-based DB code as "airtight" chunks that are easy to retry.

jwr 8y ago

I know this is slightly off-topic, but I'd be very interested in Jepsen testing FoundationDB. They claim to have developed the database test-first (starting with simulations), and it would be great to be able to compare the claims to reality using an external (by now becoming an industry standard!) testing tool.

misframer 8y ago
This was done a few years ago. See https://web.archive.org/web/20150325003526/http://blog.foundationdb.com/call-me-maybe-foundationdb-vs-jepsen https://web.archive.org/web/20150325003526/http://blog.found....

anentropic 8y ago

That was pretty great. Does anyone have links to tests of FaunaDB write performance?

evanweaver 8y ago
We are working on new benchmarks.