The criteria for discoveries is that they solve real customer issues and solve them fast. This may be direct or indirect, elegant or inelegant, simple or comple

The criteria for discoveries is that they solve real customer issues and solve them fast. This may be direct or indirect, elegant or inelegant, simple or complex. Last night I helped analyze an issue on Manta regarding job run time, and used DTrace to measure all process run time from all servers. It was a lot of data, but easily awk'd (in this case, not enough data to warrant map reduce), at which point I found run times from a certain set of processes that were equivalent to the job latency - taking us towards the fix. That's just last night. A large number of real fixes have come from being able to quickly gather and process performance data, and from that spot latency outliers, unexpected workloads, and patterns over time. I have spent time modeling systems. I wouldn't say I was left feeling skeptical of data-driven analysis, rather, I was left appreciating how complex real computer systems are, which defy accurate modeling. That's not to say that modeling is inferior or not worth doing -- it's another tool. I'll use whatever I can to fix my customer issues the fastest.