3 ms·
How the heck to you hope to gain any insighfull metrics when you've got a cobbled together mess that you only half understand. For what it's worth you might onl
by j-pb 6y ago
How the heck to you hope to gain any insighfull metrics when you've got a cobbled together mess that you only half understand.
For what it's worth you might only be benchmarking random code layout fluctuations.
I've seen research groups drown in their legacy code base.
The issue of juggling too many balls you describe is one you only have to begin with because the state of the art implementations are so shoddy to begin with.
Research suffers as much as everybody else from feature creep. Good experiments keep the number of new variables low.
- gmueckl 6y agoResearch code is not only written to measure runtime. Reducing the argument to only that aspect is not helping the discussion. And you say it yourself: good experiments change a single variable at a time. So how do you check that a series of potential improvements that you are making is sound?
- tylermw 6y ago> good experiments change a single variable at a time Although this is a tangent from the above conversation, this isn't actually true: well-designed experiments can indeed change multiple variables at the same time. There's an entire field of statistics dedicated to experimental design (google "factorial designs" for more information). One-factor-at-a-time (OFAT) experiments are often the least efficient method of running experiments, although they are conceptually simple. See the following article for a discussion: http://www.engr.mun.ca/~llye/czitrom.pdf http://www.engr.mun.ca/~llye/czitrom.pdf