7 ms·
Any sort of maths, really. When my cow-orkers want to look for correlations in graphs they quite literally print them all, lay them out on a meeting room table
by gaius 16y ago
Any sort of maths, really. When my cow-orkers want to look for correlations in graphs they quite literally print them all, lay them out on a meeting room table and stare at them. I just run Spearman and Pearson coefficients on every possible combination, then I look. I look for regular patterns using Fourier. I optimize systems using Linear Programming. My self-taught colleagues are good at their jobs day-to-day but seriously, I am being driven up the wall by alerts and reports that trigger on some dumb threshold (e.g. loadavg 10) and not on some change in the behavior or response of the system...
- awj 16y agoAll that, and yet you refer to the people you work with as "cow-orkers". At least in that one narrow respect you could stand to be more clever, if not more polite.
- onan_barbarian 16y agoI'm not sure that I would brag about running statistical methods on data before I looked at a plot of the data; especially not multiple ones. Aside from the tree-murdering, your cow-orkers (nice reference) are doing the right thing, at least initially. You should always eyeball things before going to statistical tests.
- gaius 16y agoWe're talking 30-40 sheets of A4 at a time here...
- onan_barbarian 16y agoOuch. Still, my overall point stands - eyeball first, formal tests later. This has saved me from much statistical foolishness in my time.
- hessenwolf 16y agoI am ambiguous on the eyeball-first requirement, but I agree that you definitely always have to check the visuals. Thirty or forty sheets of correlations might be made slightly prettier and on one sheet by just using 'plot' on a data frame in R. What context is this in?
- onan_barbarian 16y agoThirty or forty sheets of correlations sounds like too much work however you slice it. If they were grouped in some logical way I agree you could make them palatable with 'plot' or with the magnificent ggplot; the latter is one of the few things that I can happily play with all day... Not sure of the poster's original context. Just thought I'd air my statistical opinion out of sheer orneriness.
- hessenwolf 16y agoYeah... but for 8 variables, there are n choose 2 = n(n-1)/2 = 36 covariates. My spidey sense tells me there is one scatter plot per page. ggplot vs base & lattice in your experience? I have used lattice before but not in a few years, and I use base all the time. Don't have time to muck around with
- onan_barbarian 16y agoPlausible. Or it could be 30-40 arbitrary pairings with no variable used twice. ggplot: came for the Tufte-type aethestic, stayed for the graph grammars. I have a lot more experience with ggplot than base&lattice since I switched so early but wiser heads tell me that they are similar in capability. I just find ggplot easier to work with and look at.
- zmonkeyz 16y agoI feel you on the dumb threshold thing. There is plenty of room to improve on monitoring systems available today.