4 ms·

If you've done complex system design for a long time then it is possible to breeze through these interviews almost on autopilot. The key is not the "design" per

If you've done complex system design for a long time then it is possible to breeze through these interviews almost on autopilot. The key is not the "design" per se but asking questions that rapidly constrain the design space and understanding the implications of second-order effects. If you do this regularly, like I do, it feels like a straightforward mechanical process rather than something that requires hard thinking. Muscle memory. System design is largely independent of a specific app or tech stack. I tend to burn through these interviews so quickly that we have time leftover. It was always the easiest part of the FAANG interview process for me.

anonymousDan 5y ago
Can you give an example of what you mean by 'understanding the implications of second order effects'?
- sixstringtheory 5y ago
  That comment made me think of things like data relations that would require indexing or lookups on foreign keys/IDs. Or bottlenecks in components that perform hefty transformations due to the design of the data structures or unbalanced data sources, like where you need to merge infrequent but large chunks of data with smaller but much more numerous chunks from another async source.
- jandrewrogers 5y ago
  Computers are closed finite systems, use of resources is a zero-sum game. You cannot optimize aspects of the system design in a vacuum because they are all competing for the same set of resources. Resources applied to the optimization of one thing is the pessimization of something else. You have to apply them where they make the biggest net impact. A concrete example is cache replacement algorithms for storage. The ideal metric is cache hit rate i.e. the percentage of the time that the storage you were looking for is in the cache -- higher the better. Literature is full of academic algorithms that focus on improving cache hit rates under a variety of workloads. In many real systems, the cache replacement algorithm is in the hot path. Most "efficient" algorithms in literature either have poor CPU cache locality or thrash the CPU cache, causing significantly worse performance than is offset by the marginal gains in cache hit rates. Knowing this, good cache replacement algorithms are explicitly designed to minimize CPU cache locality/thrashing problems, at the cost of slightly lower cache hit rates because it has higher throughput in real systems. In this case, allowing the CPU cache to work efficiently is more important than a better disk cache hit rate. In complex systems, the optimal algorithm in isolation is almost never the optimal algorithm in a system. There is a global resource budget. By "second-order effects", I mostly meant understanding how subtle design choices for one part of the system will tacitly interact with the performance of other parts of the system by virtue of how they use global resources. More broadly, this is the discipline of accounting for the hardware resources available to the software at any point in time and identifying conflicts for those resources. Hopefully that gives you a sense of what I meant.
  - anonymousDan 5y ago
    Yes thanks, I took it to mean understanding what are likely to be the bottlenecks in a particular system and how your design assumptions might shift those bottlenecks, which seems to dovetail with what you said above. Having said that your example above is interesting, I hadn't thought about conflicts between the cache algorithm itself and the cached data in memory.
bluejellybean 5y ago
The thought of these questions, and engineering more generally, boiling down to intelligent dialog trees hit me like a ton of bricks a few months ago. I've been investing time into deeply thinking about these trees, and it has been one of those things that have made me feel like I've 'leveled up' in my career journey. It's very validating to read your comment, and it makes me feel like I'm on the right track. Aside from plenty of practice, do you have any good reference material or advice to share on improving this skill set?
- jandrewrogers 5y ago
  Yeah, dialog trees is a good way of describing it. Every answer to a question directs the subsequent questions. After a long while, you've more or less memorized the common branches of the tree. I benefitted, accidentally, from working in wildly different systems engineering environments, which gave me intuition for large parts of the phase space e.g. exotic silicon supercomputers, massively parallel clusters, federated systems, cheap-ass computers, etc for various workloads. It gives you a better appreciation for the tradeoffs. This also requires understanding how silicon actually behaves, not idealized abstractions. The other complementary skill is resource accounting -- knowing what everything costs in terms of bandwidth, latency, storage, compute, etc and how they interact. This allows you to look at any combination of hardware system and software workload, and quickly identify the resource bottlenecks. Again, if you do it long enough it becomes very intuitive. Old hands can accurately predict the performance characteristics of a software design on given hardware before a single line of code has been written, even if the design is novel, just through resource accounting. The application of these resources involves tradeoffs i.e. you can trade an excess of one kind of resource for another resource that is scarce with clever algorithms and architecture (e.g. classic space/time tradeoffs but more so). Unfortunately, I don't have any reference material. I learned by doing over a very long time.

anonymousDan 5y ago

Can you give an example of what you mean by 'understanding the implications of second order effects'?

sixstringtheory 5y ago
That comment made me think of things like data relations that would require indexing or lookups on foreign keys/IDs. Or bottlenecks in components that perform hefty transformations due to the design of the data structures or unbalanced data sources, like where you need to merge infrequent but large chunks of data with smaller but much more numerous chunks from another async source.
jandrewrogers 5y ago
Computers are closed finite systems, use of resources is a zero-sum game. You cannot optimize aspects of the system design in a vacuum because they are all competing for the same set of resources. Resources applied to the optimization of one thing is the pessimization of something else. You have to apply them where they make the biggest net impact. A concrete example is cache replacement algorithms for storage. The ideal metric is cache hit rate i.e. the percentage of the time that the storage you were looking for is in the cache -- higher the better. Literature is full of academic algorithms that focus on improving cache hit rates under a variety of workloads. In many real systems, the cache replacement algorithm is in the hot path. Most "efficient" algorithms in literature either have poor CPU cache locality or thrash the CPU cache, causing significantly worse performance than is offset by the marginal gains in cache hit rates. Knowing this, good cache replacement algorithms are explicitly designed to minimize CPU cache locality/thrashing problems, at the cost of slightly lower cache hit rates because it has higher throughput in real systems. In this case, allowing the CPU cache to work efficiently is more important than a better disk cache hit rate. In complex systems, the optimal algorithm in isolation is almost never the optimal algorithm in a system. There is a global resource budget. By "second-order effects", I mostly meant understanding how subtle design choices for one part of the system will tacitly interact with the performance of other parts of the system by virtue of how they use global resources. More broadly, this is the discipline of accounting for the hardware resources available to the software at any point in time and identifying conflicts for those resources. Hopefully that gives you a sense of what I meant.
- anonymousDan 5y ago
  Yes thanks, I took it to mean understanding what are likely to be the bottlenecks in a particular system and how your design assumptions might shift those bottlenecks, which seems to dovetail with what you said above. Having said that your example above is interesting, I hadn't thought about conflicts between the cache algorithm itself and the cached data in memory.

bluejellybean 5y ago

The thought of these questions, and engineering more generally, boiling down to intelligent dialog trees hit me like a ton of bricks a few months ago. I've been investing time into deeply thinking about these trees, and it has been one of those things that have made me feel like I've 'leveled up' in my career journey. It's very validating to read your comment, and it makes me feel like I'm on the right track. Aside from plenty of practice, do you have any good reference material or advice to share on improving this skill set?

jandrewrogers 5y ago
Yeah, dialog trees is a good way of describing it. Every answer to a question directs the subsequent questions. After a long while, you've more or less memorized the common branches of the tree. I benefitted, accidentally, from working in wildly different systems engineering environments, which gave me intuition for large parts of the phase space e.g. exotic silicon supercomputers, massively parallel clusters, federated systems, cheap-ass computers, etc for various workloads. It gives you a better appreciation for the tradeoffs. This also requires understanding how silicon actually behaves, not idealized abstractions. The other complementary skill is resource accounting -- knowing what everything costs in terms of bandwidth, latency, storage, compute, etc and how they interact. This allows you to look at any combination of hardware system and software workload, and quickly identify the resource bottlenecks. Again, if you do it long enough it becomes very intuitive. Old hands can accurately predict the performance characteristics of a software design on given hardware before a single line of code has been written, even if the design is novel, just through resource accounting. The application of these resources involves tradeoffs i.e. you can trade an excess of one kind of resource for another resource that is scarce with clever algorithms and architecture (e.g. classic space/time tradeoffs but more so). Unfortunately, I don't have any reference material. I learned by doing over a very long time.