Designing Data-Intensive Applications

What is this book about?

Published by O'Reilly Media in 2017, this book was written while Martin Kleppmann was a researcher in distributed systems at the University of Cambridge, after several years building large-scale data infrastructure at LinkedIn and earlier startups. Twelve chapters fall into three parts. Part I starts from three goals, reliability, scalability and maintainability, and compares data models, storage engines and the evolution of encoding formats. Part II moves beyond a single machine to replication, partitioning, transactions, the failure models of distributed systems, and consistency and consensus. Part III treats derived data, sets batch and stream processing side by side, and closes on correctness and the ethical use of data. The author's own site is explicit about the scope: the book compares the fundamental ideas and limitations behind a wide variety of systems, and does not go into the details of configuring any one tool. In practice it turns the question of which database to pick into the harder question of which failure modes one is willing to pay for. The simplified Chinese edition, 数据密集型应用系统设计, was published by China Electric Power Press in September 2018, translated by Zhao Junping, Li Sanping, Lü Yunsong and Geng Yu.

Why read it?

Because it turns technology selection from a matter of brand preference into a set of trade-offs that can be itemised. The counter-intuitive claim is that no system is better on every axis and that stronger consistency is not automatically more professional: linearizability, isolation levels and consensus protocols each carry a price, usually paid in latency, availability and operational complexity. Two frameworks transfer. Use reliability, scalability and maintainability as the coordinate system rather than ranking products by benchmark, and ask where each component has relocated complexity, since adopting a simpler middleware commonly moves complexity into the infrastructure instead of removing it. The position is also open to objection. The insistence that engineers should understand underlying mechanisms rather than choose according to the marketing of managed services conflicts directly with the widespread view that cloud platforms have abstracted such decisions away. The first edition barely addresses the ethics of data use; the second edition, co-written with Chris Riccomini and published in 2026, adds that material, which is itself an acknowledgement of the first edition's limits. What changes is how you ask: from which database should we use to which guarantees must survive which failures, and how far can the rest be relaxed. Evidence of standing: (1) the Chinese edition 数据密集型应用系统设计, published by China Electric Power Press in September 2018, ISBN 9787519821968, holds a Douban rating of 9.6 from 1,947 ratings, above the 8.5 / 1,000 threshold; (2) the Goodreads record 23463279 shows an average of 4.69 from 11.4 thousand ratings and 1,015 reviews, above the 4.0 / 5,000 threshold. These are two distinct source types, a Chinese reader platform and an international one. Wikipedia has no standalone article for the book, so none is counted; Google Scholar citation counts, adoption as required reading at two or more leading universities, substantial reviews in the designated review press and WeRead recommendation values were not obtained and are not cited. On conflicting sources: the Goodreads detail page says First published April 25, 2015 while O'Reilly edition information and Chinese reference works give March 2017; following the rule that publisher information takes precedence and contested details are avoided, the site records year 2017 with no month or day. Chinese edition details follow the Douban record. The author's name has no consistent Chinese transliteration across official channels, so the English form is retained rather than inventing one; Wikipedia has no author article and his own site does not state a nationality, while Chinese sources describe a researcher at Cambridge and one retailer labels him American, so the nationality field is recorded as not verified rather than inferred. First published in 2017, more than nine years before this recommendation.

Core Ideas

• Three non-functional goals form the coordinate system for selection. Reliability, scalability and maintainability are not performance metrics but prior statements of what counts as a system working at all. Without them, benchmark comparisons lack a shared question.
• There is no best system, only trade-offs that have been made explicit. Every tool relocates some class of complexity, either out of application code into infrastructure or back again; the real question is which class of complexity one is prepared to own.
• Failure in a distributed system is the normal condition, not an exception. Networks are unreliable, clocks are unreliable, processes pause without warning, and truth is therefore defined by the majority as an engineering premise rather than a metaphor. Any discussion of consistency has to begin there.
• Consistency is a priced set of guarantees, not a virtue. Linearizability, snapshot isolation, serializability and consensus each buy something and pay in latency, availability and operational burden. Stronger has to be able to say what it gave up.
• Derived data gives a unified view. Databases, caches, indexes, batch jobs and stream processors can be read as different materialisations of one dataflow, which turns multi-system integration from an assembly problem into a problem that can be reasoned about.

What questions does this book try to answer?

• When a given failure actually occurs, which guarantee must hold, and how far can the others be relaxed?
• A stronger consistency level pays in latency, availability and operational complexity: what exactly was exchanged, and who bears the cost?
• Once a managed service has abstracted the underlying mechanisms away, how deep does an engineer's understanding still need to go before making an architectural judgement?
• Does treating batch and stream processing as different materialisations of one dataflow conceal real differences in correctness and replay semantics?

Who should read it?

Engineers and architects who design or maintain data pipelines, technical leads choosing among storage, compute and messaging components, and product managers who have been persuaded that swapping a database would solve the problem. It is not for a reader after configuration checklists or tuning recipes: the author's site states plainly that the book does not cover the details of using a particular tool, and offers instead a framework for judgement and a model of failure.

How to Read It

Start with Chapter 1 in Part I and treat reliability, scalability and maintainability as the coordinate system for everything that follows. Chapters 8 and 9 in Part II, on the trouble with distributed systems and on consistency and consensus, deserve the slowest reading: the first sets the physical premises for every trade-off, the second breaks consistency into guarantees with prices. Part III can be sampled according to the project at hand, batch or stream. Carry one question through every chapter: where has this mechanism moved the complexity? Use the 2018 China Electric Power Press translation by Zhao Junping and others. When reading alongside the documentation of a particular system, keep in mind that this book compares mechanisms rather than release features; managed services have changed considerably since the first edition, so current parameters should come from current documentation.