7 ms·

Designing Data-Intensive Applications

598 points by wooola 9y ago

bglazer 9y ago
I really enjoyed this book. I was unfamiliar with quite a lot of the material, so I can't really evaluate it in terms of other references. As an introduction, it's fantastic. Nice coverage of practical, modern tools. Theory stuff is covered at a high level, but with a ton of references for further study. The author is pretty opinionated about a lot of the more hype driven "web-scale" marketing claims. I really appreciated that, as there's typically some kernel of truth behind the marketing fluff, and Kleppman brings these out with moderation and context. Great book if you'd like to finally understand Jepsen.io articles.
- mshenfield 9y ago
  Came here to say the same thing. If you want to develop an intuition about when and why to use different data technologies this is the book for you. Each chapter is relatively self contained, so you don't have to read the whole thing to benefit from it.
teej 9y ago
If you're trying to decide if this book is worth picking up, a number of HN commenters recommended it the last time it was brought up - https://news.ycombinator.com/item?id=15185663 https://news.ycombinator.com/item?id=15185663
seanmcdirmid 9y ago
I was a bit dismayed that this was about the technical design of data-intensive applications, not about their UX design. There still seems to be a huge gap in the latter.
- diggan 9y ago
  Hah, I just thought the same! I've ended up in situations where I needed some UX patterns for data that updates often with many updates per second, and wondering how to show that the best way. Guess one example is log viewers with debug output from some program. Would very much appreciate if someone has some resources to link or case studies.
- joshtynjala 9y ago
  Agreed. I was expecting some interesting data visualizations and other frontend topics.
- hakanito 9y ago
  Take a look at Tufte's book The Visual Display of Quantitative Information
  - seanmcdirmid 9y ago
    Interaction works both ways (input and output). Tufte's work is generally in just one direction (visualization as output). It isn't very applicable to application as much as it is to presentation (think: how did we figure out what to present in that PowerPoint inthe first place?).
- lioeters 9y ago
  Probably you interpreted the word "design" with its narrower meaning of visual(ly-oriented) design, rather than, say architecting data-intensive applications. As with another poster, Edward Tufte's books came to mind - though it's about visual presentation of information, not user interface/experience design. I've also felt that there's an unmet demand for books that provide a thorough overview of UI/UX design patterns, especially the way this book (Designing Data-Intensive Applications) does for its domain.
  - seanmcdirmid 9y ago
    Design isn't just about visuals, but also interaction. It is an overloaded term for sure, but the title of the book caught my attention, it is only when I read the article that I realized it was talking about the design of application implementations, not the UX.
topstriker515 9y ago
I'm actually midway through this book and I definitely recommend it. The content manages to be both approachable and enlightening. I'm a backend software engineer with the latitude to architect systems at my company and the content so far has given me a stronger foundation for choosing how and where to manage our data. I really enjoy the mini-dives into the structures and decisions supporting the common databases you see today (B-trees, LSM trees, etc.) and discussing the trade offs between them. Now I find myself better equipped to evaluate the tools at our disposal for a given job. I can imagine it may not dive deep enough for people who really understand the internals of a given data store and the content is probably available elsewhere. However this book is a thoughtful and engaging curation of a lot of information that I may have missed otherwise
- lioeters 9y ago
  "..better equipped to evaluate the tools at our disposal for a given job" That's a great review - I learned about the book recently, and it sounds like exactly what I need right now, to make a more informed decision about database choices.
- leethargo 9y ago
  For the deep dive, there are always the numerous references at the end of each chapter. Some of these link to software project documentation, but must to recent papers or conference presentations.
veritas3241 9y ago
Picked this book up a few weeks ago and starting diving into a yesterday! As somebody who spends more time on the data generation and analysis side, but is looking to move more towards the data engineering side, it's been great. Helps build out that "tree trunk" of knowledge to really grok what's going on. Edit: I picked it up based off the recommendation of another HN commenter who said they picked it up after listening to Martin on the SWE Daily Podcast[0] [0] https://softwareengineeringdaily.com/2017/05/02/data-intensive-applications-with-martin-kleppmann/ https://softwareengineeringdaily.com/2017/05/02/data-intensi...
djhworld 9y ago
I got my work to get me this book, while I appreciate them doing so I wish O'Reilly put a code in or something to let you redeem a PDF copy too. I've seen other publishers do this (e.g. Manning) I tend to read on the tube a lot and lugging a hefty O'Reilly tome around on my commute isn't ideal. Currently the book is just sitting on a shelf, I'll get round to it one day!
g___ 9y ago
I've read it and highly recommend it. Does anyone know books that are similar in style? (conceptual, showcasing different solutions to problems and their tradeoffs, high signal-to-noise)
- teacpde 9y ago
  I have the same question, I thoroughly enjoy the book and would love to see similar recommendations. Maybe it could be a good ask HN post.
- jiujieti 9y ago
  Same question here. I am still reading this book but the way the author combines the concepts with the practices and the contents are structured really inspire me to keep reading. (Usually I gave up easily)
- Erwin 9y ago
  A little more theoretical, but for programming language, there's "Programming Language Pragmatics" which covers imperative, functional, logical PLs and everything needed to make them run from runtimes, linking, virtual machines etc. It's not as demanding or in-depth as e.g. the "Dragon book". A few shorter books have come out that try to touch different approaches that I've liked: "Seven Languages in Seven weeks" -- and the series has also gained 7 databases, concurrency models and web frameworks. Finally, there's this anthology where OSS authors described what they did in their applications, so there's a ton of practical information http://aosabook.org/en/index.html http://aosabook.org/en/index.html
mshenfield 9y ago
In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc. The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds). The architecture seems to solve two big problems * Scaling RDBMS (there are solutions like Cloud Spanner but they rely heavily on Google's proprietary network for low latency and are expensive as balls) * Keeping downstream systems in sync. A lot of companies have gone with a "Tail the RDBMS" of some kind, e.g. writing the MySQL binlog to a Kafka queue and having downstream consumers read from that, but this seems like a more elegant solution. Are there any examples or experiences of people working with systems like this? What are some downsides, challenges, and actual benefits?
- criloz2 9y ago
  HI I am working in one is called omniql, I am lone developer, so probably it will take time, but there are much of the what they said in the book planned for omniql, is based in observable pattern so it will have live queries by default, it will use system like kafka, nats, Redis or any message system, for communicate the partitions of the distributed databases and do computation that can be merged hierarchically, the computation are done by `serverless function` in any language, is a model that can be extended to ui interface to and create a framework very similar to mobx , but more lightwave, and ui components that can automatically query their data requirement to the server just when they are observed, I am creating a binary protocol of communication for this called delta https://github.com/nebtex/delta https://github.com/nebtex/delta, that is perfect for this system an incremental computing, is a really a big project, but I hope to have something working for the next 6 months.
- lobster_johnson 9y ago
  We're building an open source database like this. It's document-oriented and relies on a transaction log core (currently using Postgres, but it's pluggable), that feeds into a query subsystem (currently layered on top of Elasticsearch, but also pluggable). The transaction log encourages small logical "patches" (set a field, increment a number, replace a substring, move an array element, etc.) that are applied in sequence but can be disentangled by clients to generate a consistent UI, and also used to resolve conflicts between multiple distributed writers. You can also follow the log through the gRPC and HTTP APIs, and you can register "watches" on queries that cause matching changes to trigger an event. While the transaction log is the underlying data model, we also maintain a consistent view of the current version so that you can use it as a document database with classical CRUD operations. So on the surface it's a lot like Firebase or CouchDB, except you get things like joins and schemas. Drop me an email (see profile) and I can send you some links.
nindalf 9y ago
I'm halfway through this book and I highly recommend it. I'd say its required reading for anyone who wants to be an architect.
dswalter 9y ago
It seems most of my comments these days are singing the praises of this book. It should be practically mandatory reading for anyone in the field. It ties concepts together and builds understanding in a way that doesn't rely on specific technologies. I wish it had been written and that I could have done a whole course on this in grad school.
wenc 9y ago
This book is a modern survey on practical distributed systems. I knew bits of pieces of the material going in, but the way it was brought together was just masterful. It will not appeal to the absolute novice to be sure. But for anyone else who has worked on systems for moving data (ETL, streams) and storing data (databases and other data stores), this book will show you how things (probably stuff you've done bits of pieces of) fit together and expound on the few foundational big ideas that makes everything cohere. Once you've understood that, you are on your way to designing data systems that are much cleaner and more scalable. My experience reading this book is a bit like that of a tradesperson going back to school to learn theory, and after being enlightened, coming away with a new understanding of how to put together theory and practice to better his craft. I chanced upon this book through an excellent interview Martin Kleppmann did on Software Engineering Daily podcast. If you want the talk-show-host cliff notes version of what the book is about, you should listen to this particular episode: https://softwareengineeringdaily.com/2017/05/02/data-intensive-applications-with-martin-kleppmann/ https://softwareengineeringdaily.com/2017/05/02/data-intensi...
drej 9y ago
I know a lot of people here already praise the book... but I have to do that as well. It's a great overview and it explains all the relevant concepts really nicely. Thanks, Martin!
dm03514 9y ago
I've been massively recommending this book. I think it's very unlikely a backend engineer can avoid having their applications on multiple servers. This book does a wonderfully clear and practical description of the issues with distributed systems, and tools and techniques to address those issues. It's written in a way that is very accessible to someone without a traditional comp sci background. I was excited about this book because there is a gap in distributed systems books. I feel like there's are a large amount of blogs but most of the books available on amazon are text books and/or include heavy math.
sbpayne 9y ago
I was very impressed with the quality and depth of the book. I also appreciated how unbiased it felt. I feel that so many books tout approach/technology X as the best approach, but Martin really did a great job at explaining trade offs of various approaches and possible solutions to the problems they introduce. Highly recommended--especially considering getting the Amazon price point (~ $25).
chajath 9y ago
Hands down the best book in this area. Took me some time to read the book cover to cover but it is definitely worth it
muramira 9y ago
Really loving the book. Definitely a must read for the data engineers. I would add that redbook.io is another great read.
blaisio 9y ago
Good book! It's only at the introductory level, but I liked that it lacks bias and does a survey of the field.
- criloz2 9y ago
  do you have a reference with more advanced level? will be nice if you share it. thanks
  - pritambaral 9y ago
    A book that tries talk broadly necessarily has less room to go into detail. For advanced level references, one will have to drill down to sources that are more specific. Thankfully, this book cites its sources and extensively documents references, and the author even maintains the reference links[0]. [0]: https://github.com/ept/ddia-references https://github.com/ept/ddia-references
agadabanka 9y ago
I am a backend engineer who has been working in the mobile gaming space for many years now. Most of the focus today for mobile gaming backends is to scale to millions of players while offering low latency and real-time interactions. Have been following the development of this book through its beta and having read it now, I think it is fair to say that this book is worth its weight in gold. I can relate directly to all the challenges faced over the years in implementing a real time distributed database _like_ functionality with a high read _and_ write throughput, both custom and off the shelf. Extremely well written.
mehra 9y ago
I highly recommend this book. Its a must read book IMHO. Especially useful when trying to select a database.
deleted 9y ago
[deleted]
yoshuaw 9y ago
I'm about 40% through this book, and it's been a stellar read so far. Using clear, well thought-out language it gets straight to business – chapter after chapter. Highly recommended!
ninjakeyboard 9y ago
I have this book as well. As a more seasoned systems engineer, I think it covers a lot of the groundwork needed for people newer to the distributed disciplines.
horia141 9y ago
This book has a very high ROI and I recommend it whole-heartedly. I can't honestly name any computer science book where I've gained so much in such a small amount of time. I wrote a more detailed review at http://horia141.com/designing-data-intensive-applications-review.html http://horia141.com/designing-data-intensive-applications-re... for the interested.

598 points by wooola 9y ago

I really enjoyed this book. I was unfamiliar with quite a lot of the material, so I can't really evaluate it in terms of other references. As an introduction, it's fantastic. Nice coverage of practical, modern tools. Theory stuff is covered at a high level, but with a ton of references for further study. The author is pretty opinionated about a lot of the more hype driven "web-scale" marketing claims. I really appreciated that, as there's typically some kernel of truth behind the marketing fluff, and Kleppman brings these out with moderation and context. Great book if you'd like to finally understand Jepsen.io articles.

mshenfield 9y ago
Came here to say the same thing. If you want to develop an intuition about when and why to use different data technologies this is the book for you. Each chapter is relatively self contained, so you don't have to read the whole thing to benefit from it.

teej 9y ago

If you're trying to decide if this book is worth picking up, a number of HN commenters recommended it the last time it was brought up - https://news.ycombinator.com/item?id=15185663 https://news.ycombinator.com/item?id=15185663

seanmcdirmid 9y ago

I was a bit dismayed that this was about the technical design of data-intensive applications, not about their UX design. There still seems to be a huge gap in the latter.

diggan 9y ago
Hah, I just thought the same! I've ended up in situations where I needed some UX patterns for data that updates often with many updates per second, and wondering how to show that the best way. Guess one example is log viewers with debug output from some program. Would very much appreciate if someone has some resources to link or case studies.
joshtynjala 9y ago
Agreed. I was expecting some interesting data visualizations and other frontend topics.
hakanito 9y ago
Take a look at Tufte's book The Visual Display of Quantitative Information
- seanmcdirmid 9y ago
  Interaction works both ways (input and output). Tufte's work is generally in just one direction (visualization as output). It isn't very applicable to application as much as it is to presentation (think: how did we figure out what to present in that PowerPoint inthe first place?).
lioeters 9y ago
Probably you interpreted the word "design" with its narrower meaning of visual(ly-oriented) design, rather than, say architecting data-intensive applications. As with another poster, Edward Tufte's books came to mind - though it's about visual presentation of information, not user interface/experience design. I've also felt that there's an unmet demand for books that provide a thorough overview of UI/UX design patterns, especially the way this book (Designing Data-Intensive Applications) does for its domain.
- seanmcdirmid 9y ago
  Design isn't just about visuals, but also interaction. It is an overloaded term for sure, but the title of the book caught my attention, it is only when I read the article that I realized it was talking about the design of application implementations, not the UX.

topstriker515 9y ago

I'm actually midway through this book and I definitely recommend it. The content manages to be both approachable and enlightening. I'm a backend software engineer with the latitude to architect systems at my company and the content so far has given me a stronger foundation for choosing how and where to manage our data. I really enjoy the mini-dives into the structures and decisions supporting the common databases you see today (B-trees, LSM trees, etc.) and discussing the trade offs between them. Now I find myself better equipped to evaluate the tools at our disposal for a given job. I can imagine it may not dive deep enough for people who really understand the internals of a given data store and the content is probably available elsewhere. However this book is a thoughtful and engaging curation of a lot of information that I may have missed otherwise

lioeters 9y ago
"..better equipped to evaluate the tools at our disposal for a given job" That's a great review - I learned about the book recently, and it sounds like exactly what I need right now, to make a more informed decision about database choices.
leethargo 9y ago
For the deep dive, there are always the numerous references at the end of each chapter. Some of these link to software project documentation, but must to recent papers or conference presentations.

veritas3241 9y ago

Picked this book up a few weeks ago and starting diving into a yesterday! As somebody who spends more time on the data generation and analysis side, but is looking to move more towards the data engineering side, it's been great. Helps build out that "tree trunk" of knowledge to really grok what's going on. Edit: I picked it up based off the recommendation of another HN commenter who said they picked it up after listening to Martin on the SWE Daily Podcast[0] [0] https://softwareengineeringdaily.com/2017/05/02/data-intensive-applications-with-martin-kleppmann/ https://softwareengineeringdaily.com/2017/05/02/data-intensi...

djhworld 9y ago

I got my work to get me this book, while I appreciate them doing so I wish O'Reilly put a code in or something to let you redeem a PDF copy too. I've seen other publishers do this (e.g. Manning) I tend to read on the tube a lot and lugging a hefty O'Reilly tome around on my commute isn't ideal. Currently the book is just sitting on a shelf, I'll get round to it one day!

g___ 9y ago

I've read it and highly recommend it. Does anyone know books that are similar in style? (conceptual, showcasing different solutions to problems and their tradeoffs, high signal-to-noise)

teacpde 9y ago
I have the same question, I thoroughly enjoy the book and would love to see similar recommendations. Maybe it could be a good ask HN post.
jiujieti 9y ago
Same question here. I am still reading this book but the way the author combines the concepts with the practices and the contents are structured really inspire me to keep reading. (Usually I gave up easily)
Erwin 9y ago
A little more theoretical, but for programming language, there's "Programming Language Pragmatics" which covers imperative, functional, logical PLs and everything needed to make them run from runtimes, linking, virtual machines etc. It's not as demanding or in-depth as e.g. the "Dragon book". A few shorter books have come out that try to touch different approaches that I've liked: "Seven Languages in Seven weeks" -- and the series has also gained 7 databases, concurrency models and web frameworks. Finally, there's this anthology where OSS authors described what they did in their applications, so there's a ton of practical information http://aosabook.org/en/index.html http://aosabook.org/en/index.html

mshenfield 9y ago

In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc. The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds). The architecture seems to solve two big problems * Scaling RDBMS (there are solutions like Cloud Spanner but they rely heavily on Google's proprietary network for low latency and are expensive as balls) * Keeping downstream systems in sync. A lot of companies have gone with a "Tail the RDBMS" of some kind, e.g. writing the MySQL binlog to a Kafka queue and having downstream consumers read from that, but this seems like a more elegant solution. Are there any examples or experiences of people working with systems like this? What are some downsides, challenges, and actual benefits?

criloz2 9y ago
HI I am working in one is called omniql, I am lone developer, so probably it will take time, but there are much of the what they said in the book planned for omniql, is based in observable pattern so it will have live queries by default, it will use system like kafka, nats, Redis or any message system, for communicate the partitions of the distributed databases and do computation that can be merged hierarchically, the computation are done by `serverless function` in any language, is a model that can be extended to ui interface to and create a framework very similar to mobx , but more lightwave, and ui components that can automatically query their data requirement to the server just when they are observed, I am creating a binary protocol of communication for this called delta https://github.com/nebtex/delta https://github.com/nebtex/delta, that is perfect for this system an incremental computing, is a really a big project, but I hope to have something working for the next 6 months.
lobster_johnson 9y ago
We're building an open source database like this. It's document-oriented and relies on a transaction log core (currently using Postgres, but it's pluggable), that feeds into a query subsystem (currently layered on top of Elasticsearch, but also pluggable). The transaction log encourages small logical "patches" (set a field, increment a number, replace a substring, move an array element, etc.) that are applied in sequence but can be disentangled by clients to generate a consistent UI, and also used to resolve conflicts between multiple distributed writers. You can also follow the log through the gRPC and HTTP APIs, and you can register "watches" on queries that cause matching changes to trigger an event. While the transaction log is the underlying data model, we also maintain a consistent view of the current version so that you can use it as a document database with classical CRUD operations. So on the surface it's a lot like Firebase or CouchDB, except you get things like joins and schemas. Drop me an email (see profile) and I can send you some links.

nindalf 9y ago

I'm halfway through this book and I highly recommend it. I'd say its required reading for anyone who wants to be an architect.

dswalter 9y ago

It seems most of my comments these days are singing the praises of this book. It should be practically mandatory reading for anyone in the field. It ties concepts together and builds understanding in a way that doesn't rely on specific technologies. I wish it had been written and that I could have done a whole course on this in grad school.

wenc 9y ago

This book is a modern survey on practical distributed systems. I knew bits of pieces of the material going in, but the way it was brought together was just masterful. It will not appeal to the absolute novice to be sure. But for anyone else who has worked on systems for moving data (ETL, streams) and storing data (databases and other data stores), this book will show you how things (probably stuff you've done bits of pieces of) fit together and expound on the few foundational big ideas that makes everything cohere. Once you've understood that, you are on your way to designing data systems that are much cleaner and more scalable. My experience reading this book is a bit like that of a tradesperson going back to school to learn theory, and after being enlightened, coming away with a new understanding of how to put together theory and practice to better his craft. I chanced upon this book through an excellent interview Martin Kleppmann did on Software Engineering Daily podcast. If you want the talk-show-host cliff notes version of what the book is about, you should listen to this particular episode: https://softwareengineeringdaily.com/2017/05/02/data-intensive-applications-with-martin-kleppmann/ https://softwareengineeringdaily.com/2017/05/02/data-intensi...

drej 9y ago

I know a lot of people here already praise the book... but I have to do that as well. It's a great overview and it explains all the relevant concepts really nicely. Thanks, Martin!

dm03514 9y ago

I've been massively recommending this book. I think it's very unlikely a backend engineer can avoid having their applications on multiple servers. This book does a wonderfully clear and practical description of the issues with distributed systems, and tools and techniques to address those issues. It's written in a way that is very accessible to someone without a traditional comp sci background. I was excited about this book because there is a gap in distributed systems books. I feel like there's are a large amount of blogs but most of the books available on amazon are text books and/or include heavy math.

sbpayne 9y ago

I was very impressed with the quality and depth of the book. I also appreciated how unbiased it felt. I feel that so many books tout approach/technology X as the best approach, but Martin really did a great job at explaining trade offs of various approaches and possible solutions to the problems they introduce. Highly recommended--especially considering getting the Amazon price point (~ $25).

chajath 9y ago

Hands down the best book in this area. Took me some time to read the book cover to cover but it is definitely worth it

muramira 9y ago

Really loving the book. Definitely a must read for the data engineers. I would add that redbook.io is another great read.

blaisio 9y ago

Good book! It's only at the introductory level, but I liked that it lacks bias and does a survey of the field.

criloz2 9y ago
do you have a reference with more advanced level? will be nice if you share it. thanks
- pritambaral 9y ago
  A book that tries talk broadly necessarily has less room to go into detail. For advanced level references, one will have to drill down to sources that are more specific. Thankfully, this book cites its sources and extensively documents references, and the author even maintains the reference links[0]. [0]: https://github.com/ept/ddia-references https://github.com/ept/ddia-references

agadabanka 9y ago

I am a backend engineer who has been working in the mobile gaming space for many years now. Most of the focus today for mobile gaming backends is to scale to millions of players while offering low latency and real-time interactions. Have been following the development of this book through its beta and having read it now, I think it is fair to say that this book is worth its weight in gold. I can relate directly to all the challenges faced over the years in implementing a real time distributed database _like_ functionality with a high read _and_ write throughput, both custom and off the shelf. Extremely well written.

mehra 9y ago

I highly recommend this book. Its a must read book IMHO. Especially useful when trying to select a database.

deleted 9y ago

[deleted]

yoshuaw 9y ago

I'm about 40% through this book, and it's been a stellar read so far. Using clear, well thought-out language it gets straight to business – chapter after chapter. Highly recommended!

ninjakeyboard 9y ago

I have this book as well. As a more seasoned systems engineer, I think it covers a lot of the groundwork needed for people newer to the distributed disciplines.

horia141 9y ago

This book has a very high ROI and I recommend it whole-heartedly. I can't honestly name any computer science book where I've gained so much in such a small amount of time. I wrote a more detailed review at http://horia141.com/designing-data-intensive-applications-review.html http://horia141.com/designing-data-intensive-applications-re... for the interested.