8 ms·

FoundationDB 6.0 released, featuring multi-region support

185 points by davelester 8y ago

throwaway5752 8y ago
Anyone know how they're using it at Apple vs other distributed databases?
- jared2501 8y ago
  A mate of mine is ex-apple and said they use it for icloud.
  - throwaway5752 8y ago
    Yes, but I understand they use Cassandra heavily https://www.techrepublic.com/article/apples-secret-nosql-sauce-includes-a-hefty-dose-of-cassandra/ https://www.techrepublic.com/article/apples-secret-nosql-sau...), and I was curious about why they use FoundationDB in some settings vs Cassandra in others. I can imagine a few good technical reasons to use one or the other depending on the context, but figured I'd ask in case anyone knew.
monstrado 8y ago
Congrats to the team on the release. Using FoundationDB has been one of the most rock solid NoSQL experiences I've ever had, and I've used a lot. After having a few months to hammer my cluster with fairly low level atomic operations, I can confidently say this thing holds up to pretty much anything you want to throw at it. Coming from the land of HBase and DynamoDB, It's ability to automatically (and intelligently) repartition data based on write throughput has been an administrative breakthrough for me. Looking forward to additional use cases I can throw at this beast of a system. Kudos to you guys!
- jinqueeny 8y ago
  Any chance of writing up a case study in detail to share your experience and best practice, especially the comparison with other DBs?
- th1nkdifferent 8y ago
  Could you add some details about your scale?
  - monstrado 8y ago
    Hey, I've written some comments regarding the use case in the past you can see here: https://news.ycombinator.com/item?id=18305446 https://news.ycombinator.com/item?id=18305446
ryanworl 8y ago
To anyone who is on the fence about putting FoundationDB into production (or at least evaluating it for their use cases), what is the number one thing you think is missing or you're worried about? i.e. - a SQL interface, - pre-packaged data structure libraries, - monitoring, - limitations of FoundationDB itself, - etc. I'm working on a talk for the upcoming FoundationDB Summit and I'd love to address some real-world questions or issues people have.
- lykr0n 8y ago
  My blocker is that the documentation is confusing and not very straight forward. The tutorials are not very verbose and the code (at least for python) is not written in a way that is easy for someone to understand. I'd be very interested in usage for time-series, but it looks like to get a working example up I'd need to fully parse the documentation and tutorials in order to do it as there is no "Let's build a Timeseries Database on FoundationDB from scratch"
  - monstrado 8y ago
    Have you given https://apple.github.io/foundationdb/time-series.html https://apple.github.io/foundationdb/time-series.html a read? Since FoundationDB is lexicographically sorted it's pretty straight forward to keep things sorted for reading data in chronological order. For example, if you want to read "last 5 minutes" of data, you can keep your timestamp at the end of the key in reverse order (Long.MaxValue - timestamp).
    - lykr0n 8y ago
      Yeah, but that doc assumes you've read and understood everything else in the docs. That document provides no help in understanding the database, it's concepts, or how to implement them. I could try and reconcile that doc with the python tutorial and everything else, but I just want something where I can copy and paste and it works. FoundationDB doesn't have a batteries included documentation like Clickhouse does: https://clickhouse.yandex/tutorial.html https://clickhouse.yandex/tutorial.html
- 8y ago
the_duke 8y ago
So, FoundationDB is a pretty low level distributed key-value store with transactions. Most applications will need something higher level, like a SQL or document db frontend which could be built on top. I'm curios what people have started using FoundationDB for. Any interesting stories to share?
- amirouche 8y ago
  Read on the forums for more (success) stories on forum https://forums.foundationdb.org/ https://forums.foundationdb.org/ tl;dr: mostly timeseries, but (also there is long thread about a distributed task queue using FDB but I am not sure it's in production, yet)
- amirouche 8y ago
  > like a SQL Most people don't mind SQL. RDBMS are good enough. People don't want to learn a new thing that's why it's still here and that's why FDB will have a hard time to compete with CockroachDB or TiDB or Spanner. > document DB frontend which could be built on top A document database is a low hanging fruit in FDB (except if you want 1-to-1 mapping with MongoDB). But my advice is to stay away from MongoDB API. What FDB is offering is much more powerful. Think the ease of use of MongoDB with the power of dynamodb + other niceties likes 'watches' (pgsql-like notify), transactions of course and your favorite language to do the queries. Also, FDB works in standalone mode that is you can start using FDB on day one and grow your business with it. All you need is a good layer. > Any interesting stories to share? I have been dabbling with key-value stores for 5 years now. So I am definitely biased. Simply said, key-value stores open perspectives you can not imagine, FDB, in particular, is a great great idea and it looks like based on the forums interactions that it's a good (if not fabulous) piece of software.
- ex3ndr 8y ago
  We have moved to FDB for our messaging platform. We had several options: a) Rewrite SQL code. In our case we are using Node.JS and all SQL libraries are very very slow. Even replacing one with another is enormous work. b) Rewrite to a new language. It was also an option since querying Postgres can take 1ms, but parsing response can easily take 100ms+. That trashed our event loop and causes awful latency. c) Rewrite to high performance NoSQL database. We picked a last one. In context of a Node.js we were able to write really thin layer on top of FDB that works super-fast and in a way we needed. In my previous startup we eventually ditched all SQL from our codebase too since SQL databases is just too slow for low latency messaging apps. There are no simple way to shard data, there are always random locks around your database (which blocks connections). Locks are really hard to debug sometimes. How to scale single SQL server? All of this is doable, but in FDB it was basically free. We migrated to FDB and got almost x100 improvement in latency/performance. And unlike SQL-code that was very carefully crafted we can do nasty things. Like - "hey, let's just pull this key every 100ms and check for a new value" or "hey, let's do it on 10s of instances at the same time?". In this situations Postgres started to consume all available CPU. You can easily creep out SQL with a single instance of your app. We haven't managed this to do with FDB for 1/2 of the cost. We are often in situation when someone commit something with a bug and, for example, started to pull data every millisecond in N^2 streams where N is number of online users. In this situations we just can't see any impact at all on our platform. Just spikes in monitoring. FDB is wonderful thing - it allows you to forget about optimizing performance of your queries, forget about managing backups and replication. It just works!
ex3ndr 8y ago
If someone want to chat about FoundationDB and want to ask about our (while limited) experience building messaging on top of FDB, please feel free to join our small room: https://app.openland.com/joinChannel/updnSlD https://app.openland.com/joinChannel/updnSlD
Rafuino 8y ago
Shameless plug here, but if anyone wants to benchmark in-memory vs. NVMe NAND SSD vs. NVMe Intel Optane DC SSD performance, we're looking for someone with FoundationDB expertise to give it a shot and share their learnings with the community. Make a request for a server by posting a new issue at our Github page [1]. Basically, I'm curious to know how FDB's memory engine performs compared to the SSD engine with a standard NAND SSD and an Intel Optane DC SSD. Something along the lines of the throughput per core and latency results on the FDB performance page [2]. [1]: https://github.com/AccelerateWithOptane/lab/issues https://github.com/AccelerateWithOptane/lab/issues [2]: https://apple.github.io/foundationdb/performance.html https://apple.github.io/foundationdb/performance.html Disclosure: I'm working at Intel and help manage our open source lab with our friends at Packet.
- whitepoplar 8y ago
  Only tangentially related, but are there any public Postgres benchmarks for systems running high-CPU, high-memory, p4800x optane drives?
  - Rafuino 8y ago
    Would something like the TSBS [1] help with this? It's TimescaleDB but they're built on Postgres. They have built-in high-CPU queries, but I haven't seen high-memory before. Can you point me in the right direction? Otherwise, we've had some Postgres people use the lab and are waiting on their decision whether to share publicly. [1]: https://github.com/timescale/tsbs https://github.com/timescale/tsbs
withhighprod 8y ago
this is awesome
thefounder 8y ago
Is there any high level api/lib like we have for Google Datastore?
discoball 8y ago
How does FDB compare to Spanner as far as the Consistency model and trade offs?
- ryanworl 8y ago
  FoundationDB and Spanner both offer external consistency. Spanner does this through synchronized clocks. FoundationDB has a similar clock called TimeKeeper, which is not a clock per se but a counter which advances approximately 1M times per second. Transactions are ordered based on this timestamp.
  - discoball 8y ago
    With a Lamport Clock (counter, logical clock) you could end up with the following due to dependence on conflict resolution aka optimistic MVCC rather than Wall Time (copies do from another HN): “it is possible for a transaction C to be aborted because it conflicts with another transaction B, but transaction B is also aborted because it conflicts with A (on another resolver), so C "could have" been committed. When Alec Grieser was an intern at FoundationDB he did some simulations showing that in horrible worst cases this inaccuracy could significantly hurt performance. But in practice I don't think there have been a lot of complaints about it.”
    - ryanworl 8y ago
      Yes, I think is fairly well known in the optimistic family of concurrency control algorithms you can get into situations where aborts are not necessary. https://db.cs.cmu.edu/papers/2016/yu-sigmod2016.pdf https://db.cs.cmu.edu/papers/2016/yu-sigmod2016.pdf Section 3.4 of this paper covers another example of this.
romed 8y ago
The upgrade instructions are firmly in the lolwut category. The database is just not available during the transition. I think they should focus on fixing their protocol such that multiple versions can run at the same time, and rollbacks are possible.

185 points by davelester 8y ago

throwaway5752 8y ago

Anyone know how they're using it at Apple vs other distributed databases?

jared2501 8y ago
A mate of mine is ex-apple and said they use it for icloud.
- throwaway5752 8y ago
  Yes, but I understand they use Cassandra heavily https://www.techrepublic.com/article/apples-secret-nosql-sauce-includes-a-hefty-dose-of-cassandra/ https://www.techrepublic.com/article/apples-secret-nosql-sau...), and I was curious about why they use FoundationDB in some settings vs Cassandra in others. I can imagine a few good technical reasons to use one or the other depending on the context, but figured I'd ask in case anyone knew.

monstrado 8y ago

Congrats to the team on the release. Using FoundationDB has been one of the most rock solid NoSQL experiences I've ever had, and I've used a lot. After having a few months to hammer my cluster with fairly low level atomic operations, I can confidently say this thing holds up to pretty much anything you want to throw at it. Coming from the land of HBase and DynamoDB, It's ability to automatically (and intelligently) repartition data based on write throughput has been an administrative breakthrough for me. Looking forward to additional use cases I can throw at this beast of a system. Kudos to you guys!

jinqueeny 8y ago
Any chance of writing up a case study in detail to share your experience and best practice, especially the comparison with other DBs?
th1nkdifferent 8y ago
Could you add some details about your scale?
- monstrado 8y ago
  Hey, I've written some comments regarding the use case in the past you can see here: https://news.ycombinator.com/item?id=18305446 https://news.ycombinator.com/item?id=18305446

ryanworl 8y ago

To anyone who is on the fence about putting FoundationDB into production (or at least evaluating it for their use cases), what is the number one thing you think is missing or you're worried about? i.e. - a SQL interface, - pre-packaged data structure libraries, - monitoring, - limitations of FoundationDB itself, - etc. I'm working on a talk for the upcoming FoundationDB Summit and I'd love to address some real-world questions or issues people have.

lykr0n 8y ago
My blocker is that the documentation is confusing and not very straight forward. The tutorials are not very verbose and the code (at least for python) is not written in a way that is easy for someone to understand. I'd be very interested in usage for time-series, but it looks like to get a working example up I'd need to fully parse the documentation and tutorials in order to do it as there is no "Let's build a Timeseries Database on FoundationDB from scratch"
- monstrado 8y ago
  Have you given https://apple.github.io/foundationdb/time-series.html https://apple.github.io/foundationdb/time-series.html a read? Since FoundationDB is lexicographically sorted it's pretty straight forward to keep things sorted for reading data in chronological order. For example, if you want to read "last 5 minutes" of data, you can keep your timestamp at the end of the key in reverse order (Long.MaxValue - timestamp).
  - lykr0n 8y ago
    Yeah, but that doc assumes you've read and understood everything else in the docs. That document provides no help in understanding the database, it's concepts, or how to implement them. I could try and reconcile that doc with the python tutorial and everything else, but I just want something where I can copy and paste and it works. FoundationDB doesn't have a batteries included documentation like Clickhouse does: https://clickhouse.yandex/tutorial.html https://clickhouse.yandex/tutorial.html
8y ago

the_duke 8y ago

So, FoundationDB is a pretty low level distributed key-value store with transactions. Most applications will need something higher level, like a SQL or document db frontend which could be built on top. I'm curios what people have started using FoundationDB for. Any interesting stories to share?

amirouche 8y ago
Read on the forums for more (success) stories on forum https://forums.foundationdb.org/ https://forums.foundationdb.org/ tl;dr: mostly timeseries, but (also there is long thread about a distributed task queue using FDB but I am not sure it's in production, yet)
amirouche 8y ago
> like a SQL Most people don't mind SQL. RDBMS are good enough. People don't want to learn a new thing that's why it's still here and that's why FDB will have a hard time to compete with CockroachDB or TiDB or Spanner. > document DB frontend which could be built on top A document database is a low hanging fruit in FDB (except if you want 1-to-1 mapping with MongoDB). But my advice is to stay away from MongoDB API. What FDB is offering is much more powerful. Think the ease of use of MongoDB with the power of dynamodb + other niceties likes 'watches' (pgsql-like notify), transactions of course and your favorite language to do the queries. Also, FDB works in standalone mode that is you can start using FDB on day one and grow your business with it. All you need is a good layer. > Any interesting stories to share? I have been dabbling with key-value stores for 5 years now. So I am definitely biased. Simply said, key-value stores open perspectives you can not imagine, FDB, in particular, is a great great idea and it looks like based on the forums interactions that it's a good (if not fabulous) piece of software.
ex3ndr 8y ago
We have moved to FDB for our messaging platform. We had several options: a) Rewrite SQL code. In our case we are using Node.JS and all SQL libraries are very very slow. Even replacing one with another is enormous work. b) Rewrite to a new language. It was also an option since querying Postgres can take 1ms, but parsing response can easily take 100ms+. That trashed our event loop and causes awful latency. c) Rewrite to high performance NoSQL database. We picked a last one. In context of a Node.js we were able to write really thin layer on top of FDB that works super-fast and in a way we needed. In my previous startup we eventually ditched all SQL from our codebase too since SQL databases is just too slow for low latency messaging apps. There are no simple way to shard data, there are always random locks around your database (which blocks connections). Locks are really hard to debug sometimes. How to scale single SQL server? All of this is doable, but in FDB it was basically free. We migrated to FDB and got almost x100 improvement in latency/performance. And unlike SQL-code that was very carefully crafted we can do nasty things. Like - "hey, let's just pull this key every 100ms and check for a new value" or "hey, let's do it on 10s of instances at the same time?". In this situations Postgres started to consume all available CPU. You can easily creep out SQL with a single instance of your app. We haven't managed this to do with FDB for 1/2 of the cost. We are often in situation when someone commit something with a bug and, for example, started to pull data every millisecond in N^2 streams where N is number of online users. In this situations we just can't see any impact at all on our platform. Just spikes in monitoring. FDB is wonderful thing - it allows you to forget about optimizing performance of your queries, forget about managing backups and replication. It just works!

ex3ndr 8y ago

If someone want to chat about FoundationDB and want to ask about our (while limited) experience building messaging on top of FDB, please feel free to join our small room: https://app.openland.com/joinChannel/updnSlD https://app.openland.com/joinChannel/updnSlD

Rafuino 8y ago

Shameless plug here, but if anyone wants to benchmark in-memory vs. NVMe NAND SSD vs. NVMe Intel Optane DC SSD performance, we're looking for someone with FoundationDB expertise to give it a shot and share their learnings with the community. Make a request for a server by posting a new issue at our Github page [1]. Basically, I'm curious to know how FDB's memory engine performs compared to the SSD engine with a standard NAND SSD and an Intel Optane DC SSD. Something along the lines of the throughput per core and latency results on the FDB performance page [2]. [1]: https://github.com/AccelerateWithOptane/lab/issues https://github.com/AccelerateWithOptane/lab/issues [2]: https://apple.github.io/foundationdb/performance.html https://apple.github.io/foundationdb/performance.html Disclosure: I'm working at Intel and help manage our open source lab with our friends at Packet.

whitepoplar 8y ago
Only tangentially related, but are there any public Postgres benchmarks for systems running high-CPU, high-memory, p4800x optane drives?
- Rafuino 8y ago
  Would something like the TSBS [1] help with this? It's TimescaleDB but they're built on Postgres. They have built-in high-CPU queries, but I haven't seen high-memory before. Can you point me in the right direction? Otherwise, we've had some Postgres people use the lab and are waiting on their decision whether to share publicly. [1]: https://github.com/timescale/tsbs https://github.com/timescale/tsbs

withhighprod 8y ago

this is awesome

thefounder 8y ago

Is there any high level api/lib like we have for Google Datastore?

discoball 8y ago

How does FDB compare to Spanner as far as the Consistency model and trade offs?

ryanworl 8y ago
FoundationDB and Spanner both offer external consistency. Spanner does this through synchronized clocks. FoundationDB has a similar clock called TimeKeeper, which is not a clock per se but a counter which advances approximately 1M times per second. Transactions are ordered based on this timestamp.
- discoball 8y ago
  With a Lamport Clock (counter, logical clock) you could end up with the following due to dependence on conflict resolution aka optimistic MVCC rather than Wall Time (copies do from another HN): “it is possible for a transaction C to be aborted because it conflicts with another transaction B, but transaction B is also aborted because it conflicts with A (on another resolver), so C "could have" been committed. When Alec Grieser was an intern at FoundationDB he did some simulations showing that in horrible worst cases this inaccuracy could significantly hurt performance. But in practice I don't think there have been a lot of complaints about it.”
  - ryanworl 8y ago
    Yes, I think is fairly well known in the optimistic family of concurrency control algorithms you can get into situations where aborts are not necessary. https://db.cs.cmu.edu/papers/2016/yu-sigmod2016.pdf https://db.cs.cmu.edu/papers/2016/yu-sigmod2016.pdf Section 3.4 of this paper covers another example of this.

romed 8y ago

The upgrade instructions are firmly in the lolwut category. The database is just not available during the transition. I think they should focus on fixing their protocol such that multiple versions can run at the same time, and rollbacks are possible.