4 ms·
"I’m looking forward to new discoveries now that I can store and process large amounts of performance data." There it is. The central dogma of "big data". Bu
by rsync 13y ago
"I’m looking forward to new discoveries now that I can store and process large amounts of performance data."
There it is. The central dogma of "big data".
But there is a dissenting view...
Jaron Lanier said, and I paraphrase: A computer will pass the Turing Test when the human running it becomes dumb enough to be fooled.
I have been very skeptical that the "discoveries" that big data presents to us will only be useful inasmuch as we dumb down our criteria for what discoveries are.
I suspect that anyone that has tried to model data and find real insight with it for any amount of time shares this skepticism.
- deleted 13y ago[deleted]
- brendangregg 13y agoThe criteria for discoveries is that they solve real customer issues and solve them fast. This may be direct or indirect, elegant or inelegant, simple or complex. Last night I helped analyze an issue on Manta regarding job run time, and used DTrace to measure all process run time from all servers. It was a lot of data, but easily awk'd (in this case, not enough data to warrant map reduce), at which point I found run times from a certain set of processes that were equivalent to the job latency - taking us towards the fix. That's just last night. A large number of real fixes have come from being able to quickly gather and process performance data, and from that spot latency outliers, unexpected workloads, and patterns over time. I have spent time modeling systems. I wouldn't say I was left feeling skeptical of data-driven analysis, rather, I was left appreciating how complex real computer systems are, which defy accurate modeling. That's not to say that modeling is inferior or not worth doing -- it's another tool. I'll use whatever I can to fix my customer issues the fastest.