5 ms·

Anyone recommend a good book specifically on distributed databases (not more general distributed systems stuff like e.g. klepmann's DDIA)?

decebalus1 6y ago
Database Internals: A Deep Dive into How Distributed Data Systems Work by Alex Petrov
jkaptur 6y ago
In case someone isn't aware, Designing Data-Intensive Applications is a very good introduction to distributed databases, even if it doesn't specialize in them.
- artificial 6y ago
  I’ll second this. It’s an excellent intro that’s very approachable.
- anonymousDan 6y ago
  Yeah I don't disagree, I've just read it already :)
pcthrowaway 6y ago
Not a book, but the podcast series "Scaling Postgres" looks really good.
throwaway823882 6y ago
Not a book, but I would add to the foreword: "Try not to use them."
- bshipp 6y ago
  This is a valid comment that I'd expand to say "...without exhausting all the tools available." For example, if you're facing lengthy search queries and are looking at partitioning the database to speed them up, many databases include helpful internal tools that should be attempted first. Proper indexing is a textbook all on it's own, and the appropriate design of full-text search dictionaries and queries is another. The one that shocked me a year or two ago was implementing full-text search in a gigantic sqlite DB. I was preparing to migrate it to postgresql or elasticsearch because my traditional "ilike" queries, even indexed, were ridiculously slow, and I was not aware FTS was an included library. Although FTS pretty much doubled the size of the DB, a three minute query dropped to a few seconds. Since read queries are non-blocking in sqlite, this allowed me to continue using the DB without needing a bigger, more complex solution.
- victor106 6y ago
  Genuinely curious: What are some challenges with using distributed databases?
  - biggestdummy 6y ago
    Tons of complexity behind this, but the simple problem is source of truth. When you store data in multiple places, if they become unsynchronized, which is the true answer??? Within a monolithic system, you don't have differing clocks, network partitions that can be very weird, etc. And, a related problem, which is race conditions. What if you are reading one copy of the data while the other replica is being updated. Most complexity, imo, arises from this fundamental problem and the various techniques to deal with it.
  - bordercases 6y ago
    Time is not shared, state is not synchronized, latency exists. What's not to hate?
  - throwaway823882 6y ago
    Distributed [decentralized] databases are basically the most complicated software you can play with that isn't classified/proprietary. Read through all these Jepsen analyses: https://jepsen.io/analyses https://jepsen.io/analyses Then consider that those analyses were done under theoretical environments using failures that tests were written for. It doesn't capture all the garden-variety problems that happen when you are running 40 extra-extra-large instances on shitty non-cloud hardware and serving traffic to thousands of high performance apps on a globe-spanning network. Things that seem stupid or obvious, like just expanding your storage, can be something these databases can't handle. Sometimes there are well-documented problems, like random data corruption, or replication that just stops and never starts again, or an inability to reconfigure a cluster without literally destroying it. So not only do you need to understand consensus algorithms and cutting edge functionality, you also have to become an operational expert on their quirks. Most are practically brand new, making them largely untested in large-scale scenarios, with features that haven't "baked" longer than a month in testing. There's no book to buy, and very few people to get support from. Sometimes there's no getting around it and they are literally your only option to solve your problem. But it's not worth getting involved in if a non-distributed database can solve your problem. On the other hand, if you are just a bored engineer and want to see some very big systems explode in dramatic and obscure ways, definitely use a distributed database.
e12e 6y ago
Not a book... But foundationdb documentation (and source) might be worth looking at? https://apple.github.io/foundationdb/ https://apple.github.io/foundationdb/
convolvatron 6y ago
I got alot of value out of Distributed Databases: Principles and Systems Stefano Ceri, Giuseppe Pelagatti McGraw-Hill, 1985 it adopts a very specific approach, and is obviously missing some newer techniques, but it certainly leaves you with a feeling of 'yeah, i can build this'

decebalus1 6y ago

Database Internals: A Deep Dive into How Distributed Data Systems Work by Alex Petrov

jkaptur 6y ago

In case someone isn't aware, Designing Data-Intensive Applications is a very good introduction to distributed databases, even if it doesn't specialize in them.

artificial 6y ago
I’ll second this. It’s an excellent intro that’s very approachable.
anonymousDan 6y ago
Yeah I don't disagree, I've just read it already :)

pcthrowaway 6y ago

Not a book, but the podcast series "Scaling Postgres" looks really good.

throwaway823882 6y ago

Not a book, but I would add to the foreword: "Try not to use them."

bshipp 6y ago
This is a valid comment that I'd expand to say "...without exhausting all the tools available." For example, if you're facing lengthy search queries and are looking at partitioning the database to speed them up, many databases include helpful internal tools that should be attempted first. Proper indexing is a textbook all on it's own, and the appropriate design of full-text search dictionaries and queries is another. The one that shocked me a year or two ago was implementing full-text search in a gigantic sqlite DB. I was preparing to migrate it to postgresql or elasticsearch because my traditional "ilike" queries, even indexed, were ridiculously slow, and I was not aware FTS was an included library. Although FTS pretty much doubled the size of the DB, a three minute query dropped to a few seconds. Since read queries are non-blocking in sqlite, this allowed me to continue using the DB without needing a bigger, more complex solution.
victor106 6y ago
Genuinely curious: What are some challenges with using distributed databases?
- biggestdummy 6y ago
  Tons of complexity behind this, but the simple problem is source of truth. When you store data in multiple places, if they become unsynchronized, which is the true answer??? Within a monolithic system, you don't have differing clocks, network partitions that can be very weird, etc. And, a related problem, which is race conditions. What if you are reading one copy of the data while the other replica is being updated. Most complexity, imo, arises from this fundamental problem and the various techniques to deal with it.
- bordercases 6y ago
  Time is not shared, state is not synchronized, latency exists. What's not to hate?
- throwaway823882 6y ago
  Distributed [decentralized] databases are basically the most complicated software you can play with that isn't classified/proprietary. Read through all these Jepsen analyses: https://jepsen.io/analyses https://jepsen.io/analyses Then consider that those analyses were done under theoretical environments using failures that tests were written for. It doesn't capture all the garden-variety problems that happen when you are running 40 extra-extra-large instances on shitty non-cloud hardware and serving traffic to thousands of high performance apps on a globe-spanning network. Things that seem stupid or obvious, like just expanding your storage, can be something these databases can't handle. Sometimes there are well-documented problems, like random data corruption, or replication that just stops and never starts again, or an inability to reconfigure a cluster without literally destroying it. So not only do you need to understand consensus algorithms and cutting edge functionality, you also have to become an operational expert on their quirks. Most are practically brand new, making them largely untested in large-scale scenarios, with features that haven't "baked" longer than a month in testing. There's no book to buy, and very few people to get support from. Sometimes there's no getting around it and they are literally your only option to solve your problem. But it's not worth getting involved in if a non-distributed database can solve your problem. On the other hand, if you are just a bored engineer and want to see some very big systems explode in dramatic and obscure ways, definitely use a distributed database.

e12e 6y ago

Not a book... But foundationdb documentation (and source) might be worth looking at? https://apple.github.io/foundationdb/ https://apple.github.io/foundationdb/

convolvatron 6y ago

I got alot of value out of Distributed Databases: Principles and Systems Stefano Ceri, Giuseppe Pelagatti McGraw-Hill, 1985 it adopts a very specific approach, and is obviously missing some newer techniques, but it certainly leaves you with a feeling of 'yeah, i can build this'