3 ms·

I think intuition goes downhill as your system increases in complexity. It's harder to develop deep experience or regularity in distributed systems than it is i

I think intuition goes downhill as your system increases in complexity. It's harder to develop deep experience or regularity in distributed systems than it is in your monolith that you know inside and out. Feedback can get less immediate, too.

zwieback 8y ago
So true. I spent the bulk of my early SW development career in distributed systems that were complex enough that they felt like they had a mind of their own. What I learned back then is that my intuition was worthless. Maybe that's just me but the key to success was always to set aside your expectation of what your code does and instead spend more time instrumenting and collecting data on what was actually going on. This is especially true for completely asynchronous systems like networks or systems that integrate multiple independent subsystems. Things just don't happen in the order you expected when timeouts, aborted operations and retries bubble up and down your OS and application stack. After 30 years I consider myself pretty capable in debugging such systems but one thing I never use is "intuition" as defined in the beginning of the article.
paulsutter 8y ago
As the system increases in complexity, you need more and more measurement/monitoring in order to gain intuition for how the system works. For example, watching Ganglia graphs of mapreduce clusters make it 10x easier to tune system parameters.
onemoresoop 8y ago
Intuition can give you a sense that the code may be bad, but not necessarily pinpoint the problem in the code
wenc 8y ago
Agreed. Also, in complex systems, there are emergent properties that are hard to predict, so any intuitions about system behavior from previous observations aren't always valid. Intuition is a form of pattern-matching on implicit/tacit variables. It only works if there is a pattern to match.

3 ms·

I think intuition goes downhill as your system increases in complexity. It's harder to develop deep experience or regularity in distributed systems than it is i

by tunesmith 8y ago

zwieback 8y ago
So true. I spent the bulk of my early SW development career in distributed systems that were complex enough that they felt like they had a mind of their own. What I learned back then is that my intuition was worthless. Maybe that's just me but the key to success was always to set aside your expectation of what your code does and instead spend more time instrumenting and collecting data on what was actually going on. This is especially true for completely asynchronous systems like networks or systems that integrate multiple independent subsystems. Things just don't happen in the order you expected when timeouts, aborted operations and retries bubble up and down your OS and application stack. After 30 years I consider myself pretty capable in debugging such systems but one thing I never use is "intuition" as defined in the beginning of the article.
paulsutter 8y ago
As the system increases in complexity, you need more and more measurement/monitoring in order to gain intuition for how the system works. For example, watching Ganglia graphs of mapreduce clusters make it 10x easier to tune system parameters.
onemoresoop 8y ago
Intuition can give you a sense that the code may be bad, but not necessarily pinpoint the problem in the code
wenc 8y ago
Agreed. Also, in complex systems, there are emergent properties that are hard to predict, so any intuitions about system behavior from previous observations aren't always valid. Intuition is a form of pattern-matching on implicit/tacit variables. It only works if there is a pattern to match.