4 ms·

Fractional Factorial Experiment Design: There Are Too Many Experiments to Do

102 points by ingve 3y ago

interleave 3y ago
Wow. As an experiment-design nerd, this may be one of my personal top favorite HN posts of all time. I don't 100% get the generation part yet, but I believe that's just me needing practice. A long time ago back in New York, Grace and Trevor from Javelin (Née Lean Startup Machine) taught me the scientific method over the course of a weekend workshop. Still, to this day, I use their experiment design process and I am grateful for it. https://www.youtube.com/watch?v=F-5Iyj9A1MU https://www.youtube.com/watch?v=F-5Iyj9A1MU Thank you demofox for sharing!
wenc 3y ago
I studied Design of Experiments (DOE) in college -- it was part of the core curriculum in my major. The content feels like it ought to be very useful (it's basically techniques on how to conduct experiments efficiently and reducing the number of runs, thus reducing overall cost).... yet I've never had occasion to ever apply it. One area where I feel it might be relevant to my work is in hyperparameter tuning, especially when function evals are expensive. Instead of doing grid search (which is a brute-force exhaustive search), fractional factorial designs can search the multidimensional space with way less function evals. Yet I don't do it because it's easier to write code to do brute force and let the computer chug away over night. In the world of web analytics, fractional factorial lets you run multivariate experiments efficiently rather than a combinatorial combination of A/B tests. From online articles, I'm guessing people do use it there, but A/B tests still dominate. Fractional factorial is also most effective when the covariates are orthogonal and independent, which is rarely the case. We can get around this by projecting high dimensional covariates into lower dimensional space using PCA, which guarantees orthogonality, but this also seems to not be done so much. Just wondering if anyone is using fractional factorial designs in real life? (or optimal designs like D-optimal designs)
- stdbrouw 3y ago
  > Fractional factorial is also most effective when the covariates are orthogonal and independent, which is rarely the case. We can get around this by projecting high dimensional covariates into lower dimensional space using PCA, which guarantees orthogonality, but this also seems to not be done so much. I guess you're right in that if there were very many interactions between variables, then the aliasing of a fractional design would be an incredible nuisance, but on the other hand if there were no dependence between any of the variables at all then there would be no point to any sort of factorial design as you could just test each variable sequentially. > Just wondering if anyone is using fractional factorial designs in real life? (or optimal designs like D-optimal designs) I have to admit, I have never ever seen anyone calculate a D-optimal or G-optimal design for anything -- that said, I only graduated as a statistician 6 or 7 years ago. As you say, there might be some use to it for continuous variables but for factors, given that latin squares (etc.) are known to be optimal so you can just grab that off the shelf.
- conjectures 3y ago
  I think in many industrial settings Bayesian optimisation is more practical. Since it allows use to refine our design on the fly and put resources into promising areas of the space earlier. Some kind of pre-planned fractional design could be useful for generating an initial set of points to test.
- thereisnospork 3y ago
  I've at times made fairly extensive use of d-optimal designs in formulation type research and development, usually in an iterative process, pruning and adding variables as well as refining ranges. Lot of footguns, but the approach can drastically decrease optimization time. In my experience it needs the right combination of: too many factors to make full factorial reasonable, too much (expected) covariance to trust single-vaeiable optimization, and cheap enough to commit to doing 8+ experiments before a major evaluation.
  - samvher 3y ago
    What are the footguns you've run into with D-optimal designs?
csense 3y ago
The operator defined is associative and commutative and doing it twice gives you the identity. So to me, it seems a strange choice to notate this as some kind of word algebra; it feels much more natural to use binary vector notation. In a later example, the post says ABD*BCE = ACDE. Let's translate this to binary vectors by putting a 1 when the letter is present, 0 when absent, and changing * to +. Then it becomes 11010 + 01101 = 10111. Clearly the * operator being discussed is XOR of binary vectors, which is why I choose to use + instead. Now I'll attempt to translate the post into more standard linear algebra terminology. If you write the table rows as binary vectors, "setting up A and B and leaving C blank" is essentially setting up a 2x3 matrix (in general MxN) and leaving the third column blank, filling in the leftmost 2x2 (MxM) square with an identity matrix. Regardless of how you fill in C, this gives you a rank M matrix in reduced row echelon form. Then "decide on a formula for how to set C’s value" is basically setting the remaining column(s) as some linear function of the original columns. The alias is solving for the nullspace, the large example gives: [1 0 0 1 0][x0] [0] [0 1 0 1 1][x1] [0] [0 0 1 0 1][x2] = [0] [x3] [0] [x4] [0] So you get the system of equations: x0 + x3 = 0 x1 + x3 + x4 = 0 x2 + x4 = 0 You can find all solutions by setting x3, x4 as free variables, and x0, x1, x2 are completely determined: x0 = x3 x1 = x3 + x4 x2 = x4 (This looks like a sign mistake, but because + represents XOR, we can freely switch between negative and positive, as addition and subtraction are the same modulo 2.) You proceed by assigning all possible values to x3 and x4, obtaining a nullspace of {00000, 01101, 11010, 10111}, which matches the first row of the table (I, BCE, ABD and ACDE). The remaining rows of the table are obtained by adding that nullspace to every element of the span of the original matrix. (Of course this hits all 32 possible combinations, there is a fairly easy-to-prove linear algebra theorem guaranteeing this.) These are some abstract computations on binary vectors, why are they relevant to the real world? It mystified me for a bit, but I think the answer is that if two vectors x, y are aliases, we've set things up so that y = x+z for some z in the nullspace of our matrix M, which represents some linear function f(). Then f(y) = f(x+z) = f(x)+f(z) = f(x). The function f() should be related to our experiment and the fact f(y) = f(x) should be related to the idea "our setup can't distinguish between x and y" but it's not 100% clear to me how this conclusion follows. Any stats experts care to chime in? It didn't discuss at all how to pick generators, but I would guess (1) you want all variables to be varied at least once, and (2) you want alias classes to contain preferably at most one "low-entropy" entry (where "low-entropy" means combinatorially a low popcount, because a priori a simpler explanation is likelier than a complex one (Occam's Razor), and possibly some application-specific context.)
dan_mctree 3y ago
Is it just me or does this require background knowledge that isn't widely available? I couldn't get through it as it never seemed to really explain what Aliases or Alias structures are, how determining those aliasing structures works (why multiply by C?), how this primary effect thing works and how any of this actually relates to the experiments in any way. As in, like, ok I do tests 1-4, now how do I turn that into insight with this algebra? If someone could elaborate how this all fits together that would be great, because it does seem like there's some nuggets of insight to be found here
- stdbrouw 3y ago
  Let's say you're testing different formulations for cake but because you don't have time to do the full cartesian product of all possible variations over sugar content x fat content x type of flour, you just pick a handful, and it turns out that every time you're testing a high-sugar variation you are also testing spelt flour -- that's an alias because if you really like these particular cakes there's no way to know whether it was the spelt or the sugar that did it. What experimental design brings to the table is a disciplined way to figure out what variables will be aliased and to make sure that these aliases are mostly harmless, either with the help of subject matter knowledge (you can taste sugar so it's fine if it's aliased with other things) or mathematically (let's try to avoid aliasing sugar content with fat content, but instead alias sugar content with the interaction effect of sugar, fat and flour type because that higher-order interaction is unlikely to matter over and above the first-order effects.) Fractional designs in particular are typically used in agriculture and industrial settings, places where you want to try to optimize over very many factors at once but cannot afford to test every variation. It is not common in web analytics because (1) we usually assume that one particular change to a site or app will be independent of another change elsewhere, so there is no need to test them concurrently to see if a particular combo stands out and (2) if we did want to test combinations of variables concurrently, there's usually enough users or visitors to just test the entire grid and not worry about picking a selection of variations.
MauranKilom 3y ago
Not exactly the same topic, but I feel like this is a good context to bring up https://extremelearning.com.au/unreasonable-effectiveness-of-quasirandom-sequences/ https://extremelearning.com.au/unreasonable-effectiveness-of.... It provides (in some sense) maximally un-aliased parameter choices when trying to cover a (continuous) parameter space. It generalizes to arbitrary dimensions, doesn't require you to choose the number of samples/experiments beforehand, and actually expands trivially to parameters representing classes (including possibly weighting the classes). I think it would be interesting to see how using this sequence fares in such a fractional factorial analysis, i.e. how close to optimal the (e.g. binarized) pseudorandom parameter choices are for different numbers of experiments!
CrimpCity 3y ago
Dang I was hoping the article would have been about non-traditional higher level fractional factorial designs. Like for example you have an experiment with 3 factors and 5, 4, 3 levels each but can only run 20 experiments.
- deleted 3y ago
  [deleted]
esafak 3y ago
Does any experimentation platform support this?
- mtu 3y ago
  JMP is used industrially for design of experiments: https://www.jmp.com/en_us/software/capabilities/design-of-experiments.html https://www.jmp.com/en_us/software/capabilities/design-of-ex... R also seems to have some libraries for things related to design of experiments.
  - nxobject 3y ago
    JMP is also a very criminally underrated and flexible tool to do data analysis with – both exploratory and modelling. It's my preferred tool of choice over R, especially after they added a structural equation modelling platform. It's a pity they got rid of pricy perpetual licensing quite a few years ago for a no less pricy subscription model.
- stupidcar 3y ago
  For biology, Synthace supports designing experimental protocols using DOE, executing them on robotically automated lab equipment, then performing analysis on the results: https://www.synthace.com/ https://www.synthace.com/
- deleted 3y ago
  [deleted]
- deleted 3y ago
  [deleted]
clircle 3y ago
This is a great tutorial.
alright2565 3y ago
By coincidence, I just watched this video on this same topic: https://m.youtube.com/watch?v=5oULEuOoRd0 https://m.youtube.com/watch?v=5oULEuOoRd0 The visualizations and the real problem solving demonstrated made it much easier (for me) to intuitively understand than any amount of reading
- danbruc 3y ago
  This article is actually in response to that video by NightHawkInLight, the method used in the video is however different from the one explained in the article, at least that is what the first paragraphs say, I did not read it completely.
wyldfire 3y ago
NIST has a combinatorial test engine designed for combining feature settings to maximize coverage for N tests. https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software https://csrc.nist.gov/projects/automated-combinatorial-testi...
penteract 3y ago
Something about this doesn't make sense - from the C=A case: > we wouldn’t be able to tell the difference in results between only doing option B (doubling the butter) and doing all three options: adding an egg, doubling the butter, and adding nuts on top. However, the experiments listed include exactly those 2 experiments (Test 1 and Test 3 in the table with a column titled C=A), so unless I'm missing something, "wouldn't be able to tell the difference" doesn't mean what I expect it to.
- kawzeg 3y ago
  Similarly: > Lastly, I = AC means that we can’t tell the difference between doing nothing at all, compared to adding an egg and adding nuts on top. But test 0 and test 2 are doing nothing and adding an egg & nuts respectively. What are we missing here? On further reading, it seems like "doing all three options" should be interpreted to mean "the interaction of all three options"? So we aren't able to tell if an improvement of taste comes from B alone or from the interaction of A, B, and C. I'm mostly guessing though.
- pfedak 3y ago
  As best I can tell from reading https://www.stat.purdue.edu/~yuzhu/stat514s2006/Lecnot/fracfacts2006.pdf https://www.stat.purdue.edu/~yuzhu/stat514s2006/Lecnot/fracf... the unstated assumption here is that we're going to do linear regression where (critically) the "-" case for each condition is -1, and the "+" is +1. This has the surprising-to-me effect of making "I", which looks like it might be the control group based on notation, actually a positive recipient of AC interaction (and any even-order interaction). You can think of this as a change in basis in how you parse out the effects, where we're talking about 1 -1 -1 1 (like a covariance matrix) instead of 0 0 0 1 for an interaction. I have a gut feeling it's done this way mostly because the tools being used expect things to be expressed this way rather than any conscious choice by experimenters. Through this lens, if you test I, B, AC, ABC every experiment has a positive effect from AC interaction, and taking B-I, which we might think of as the effect from B, is in this paradigm also sensitive to the ABC interaction and the AB and BC interactions. The "real" effect from B would be approximated as (B + ABC - AC - I)/2, which is exactly the same as the effect from ABC interaction (which is positive when an odd number of its constituents are positive...). I'm pretty sure this is just a difference in mathematical perspective - you can represent exactly the same data, but the coefficients (i.e. effect values) will change, and there's a different notion of what you know vs don't know. Maybe there's a more convincing reason to do this when you have more than two "levels", but from the presentation in TFA it just feels like overcomplicating things with a confusing prior about how effects work. It also seems like the given example is just bad. If the parameters are numeric and there's not a reasonable "control", this perspective feels much more natural.
noqc 3y ago
I have studied at least a little combinatorics, and the language here is completely inpenetrable to me. Neither the question of "what property is this design supposed to have?" or "why does it have it?" are stated in plain or even mathematical english. The wiki entry is similarly unhelpful.
- acyou 3y ago
  It's Design of Experiments language, DOE being a damned useful tiny little subset of applied statistics.
- usgroup 3y ago
  In a full factorial design you’re able to calculate the main effects and all the interactions between everything and everything else: not just pairwise but all combinations of variables. You may not care for that because you don’t think there are such interactions or perhaps you just don’t have the experimental budget. In fractional designs you can chose to confound some of the interactions so that you can’t tell what’s happening beyond a certain level. Eg just main and pairwise effects, with the rest confounding those effects. You would do this because it requires fewer runs. You can also use it as a sort of screener when you have many variables by confounding things together in such a way that you can at least tell which variables are having the effects even if you can’t calculate the individual effects.
- crispyambulance 3y ago
  For whatever reasons, much of the community that does DOE's is beset with heavy use of opaque jargon and flagrant use of "canned tests" from software packages like minitab. Making statistical inferences from data is nuanced and complicated if the thing you're interested in doesn't just pop-out from simple descriptive statistics treatments (histograms, box-plots, scatterplots, simple linear models). There are some islands of sanity: - NIST has a great resource (https://www.itl.nist.gov/div898/handbook/index.htm https://www.itl.nist.gov/div898/handbook/index.htm). Don't be alarmed by the retro html appearance. It's actually meticulously maintained and contains reproduceable examples with data. The plain language is such a relief compared to other references. - "Statistics for Experimenters" the book by George Box et al, is a great resource that's fairly comprehensive in my opinion. Good, clear writing. It seems to be widely available online (don't know if that's intentional).
acyou 3y ago
Stat-Ease's Design-Expert is the industry standard software in this area. Would have liked to see the author do a little more attribution. These are old ideas and techniques, and certainly not attributable to this author. It's helpful to visualise the response surface. Diagrams are critical in learning this material.
- deleted 3y ago
  [deleted]

102 points by ingve 3y ago

Wow. As an experiment-design nerd, this may be one of my personal top favorite HN posts of all time. I don't 100% get the generation part yet, but I believe that's just me needing practice. A long time ago back in New York, Grace and Trevor from Javelin (Née Lean Startup Machine) taught me the scientific method over the course of a weekend workshop. Still, to this day, I use their experiment design process and I am grateful for it. https://www.youtube.com/watch?v=F-5Iyj9A1MU https://www.youtube.com/watch?v=F-5Iyj9A1MU Thank you demofox for sharing!

wenc 3y ago

I studied Design of Experiments (DOE) in college -- it was part of the core curriculum in my major. The content feels like it ought to be very useful (it's basically techniques on how to conduct experiments efficiently and reducing the number of runs, thus reducing overall cost).... yet I've never had occasion to ever apply it. One area where I feel it might be relevant to my work is in hyperparameter tuning, especially when function evals are expensive. Instead of doing grid search (which is a brute-force exhaustive search), fractional factorial designs can search the multidimensional space with way less function evals. Yet I don't do it because it's easier to write code to do brute force and let the computer chug away over night. In the world of web analytics, fractional factorial lets you run multivariate experiments efficiently rather than a combinatorial combination of A/B tests. From online articles, I'm guessing people do use it there, but A/B tests still dominate. Fractional factorial is also most effective when the covariates are orthogonal and independent, which is rarely the case. We can get around this by projecting high dimensional covariates into lower dimensional space using PCA, which guarantees orthogonality, but this also seems to not be done so much. Just wondering if anyone is using fractional factorial designs in real life? (or optimal designs like D-optimal designs)

stdbrouw 3y ago
> Fractional factorial is also most effective when the covariates are orthogonal and independent, which is rarely the case. We can get around this by projecting high dimensional covariates into lower dimensional space using PCA, which guarantees orthogonality, but this also seems to not be done so much. I guess you're right in that if there were very many interactions between variables, then the aliasing of a fractional design would be an incredible nuisance, but on the other hand if there were no dependence between any of the variables at all then there would be no point to any sort of factorial design as you could just test each variable sequentially. > Just wondering if anyone is using fractional factorial designs in real life? (or optimal designs like D-optimal designs) I have to admit, I have never ever seen anyone calculate a D-optimal or G-optimal design for anything -- that said, I only graduated as a statistician 6 or 7 years ago. As you say, there might be some use to it for continuous variables but for factors, given that latin squares (etc.) are known to be optimal so you can just grab that off the shelf.
conjectures 3y ago
I think in many industrial settings Bayesian optimisation is more practical. Since it allows use to refine our design on the fly and put resources into promising areas of the space earlier. Some kind of pre-planned fractional design could be useful for generating an initial set of points to test.
thereisnospork 3y ago
I've at times made fairly extensive use of d-optimal designs in formulation type research and development, usually in an iterative process, pruning and adding variables as well as refining ranges. Lot of footguns, but the approach can drastically decrease optimization time. In my experience it needs the right combination of: too many factors to make full factorial reasonable, too much (expected) covariance to trust single-vaeiable optimization, and cheap enough to commit to doing 8+ experiments before a major evaluation.
- samvher 3y ago
  What are the footguns you've run into with D-optimal designs?

csense 3y ago

The operator defined is associative and commutative and doing it twice gives you the identity. So to me, it seems a strange choice to notate this as some kind of word algebra; it feels much more natural to use binary vector notation. In a later example, the post says ABD*BCE = ACDE. Let's translate this to binary vectors by putting a 1 when the letter is present, 0 when absent, and changing * to +. Then it becomes 11010 + 01101 = 10111. Clearly the * operator being discussed is XOR of binary vectors, which is why I choose to use + instead. Now I'll attempt to translate the post into more standard linear algebra terminology. If you write the table rows as binary vectors, "setting up A and B and leaving C blank" is essentially setting up a 2x3 matrix (in general MxN) and leaving the third column blank, filling in the leftmost 2x2 (MxM) square with an identity matrix. Regardless of how you fill in C, this gives you a rank M matrix in reduced row echelon form. Then "decide on a formula for how to set C’s value" is basically setting the remaining column(s) as some linear function of the original columns. The alias is solving for the nullspace, the large example gives: [1 0 0 1 0][x0] [0] [0 1 0 1 1][x1] [0] [0 0 1 0 1][x2] = [0] [x3] [0] [x4] [0] So you get the system of equations: x0 + x3 = 0 x1 + x3 + x4 = 0 x2 + x4 = 0 You can find all solutions by setting x3, x4 as free variables, and x0, x1, x2 are completely determined: x0 = x3 x1 = x3 + x4 x2 = x4 (This looks like a sign mistake, but because + represents XOR, we can freely switch between negative and positive, as addition and subtraction are the same modulo 2.) You proceed by assigning all possible values to x3 and x4, obtaining a nullspace of {00000, 01101, 11010, 10111}, which matches the first row of the table (I, BCE, ABD and ACDE). The remaining rows of the table are obtained by adding that nullspace to every element of the span of the original matrix. (Of course this hits all 32 possible combinations, there is a fairly easy-to-prove linear algebra theorem guaranteeing this.) These are some abstract computations on binary vectors, why are they relevant to the real world? It mystified me for a bit, but I think the answer is that if two vectors x, y are aliases, we've set things up so that y = x+z for some z in the nullspace of our matrix M, which represents some linear function f(). Then f(y) = f(x+z) = f(x)+f(z) = f(x). The function f() should be related to our experiment and the fact f(y) = f(x) should be related to the idea "our setup can't distinguish between x and y" but it's not 100% clear to me how this conclusion follows. Any stats experts care to chime in? It didn't discuss at all how to pick generators, but I would guess (1) you want all variables to be varied at least once, and (2) you want alias classes to contain preferably at most one "low-entropy" entry (where "low-entropy" means combinatorially a low popcount, because a priori a simpler explanation is likelier than a complex one (Occam's Razor), and possibly some application-specific context.)

dan_mctree 3y ago

Is it just me or does this require background knowledge that isn't widely available? I couldn't get through it as it never seemed to really explain what Aliases or Alias structures are, how determining those aliasing structures works (why multiply by C?), how this primary effect thing works and how any of this actually relates to the experiments in any way. As in, like, ok I do tests 1-4, now how do I turn that into insight with this algebra? If someone could elaborate how this all fits together that would be great, because it does seem like there's some nuggets of insight to be found here

stdbrouw 3y ago
Let's say you're testing different formulations for cake but because you don't have time to do the full cartesian product of all possible variations over sugar content x fat content x type of flour, you just pick a handful, and it turns out that every time you're testing a high-sugar variation you are also testing spelt flour -- that's an alias because if you really like these particular cakes there's no way to know whether it was the spelt or the sugar that did it. What experimental design brings to the table is a disciplined way to figure out what variables will be aliased and to make sure that these aliases are mostly harmless, either with the help of subject matter knowledge (you can taste sugar so it's fine if it's aliased with other things) or mathematically (let's try to avoid aliasing sugar content with fat content, but instead alias sugar content with the interaction effect of sugar, fat and flour type because that higher-order interaction is unlikely to matter over and above the first-order effects.) Fractional designs in particular are typically used in agriculture and industrial settings, places where you want to try to optimize over very many factors at once but cannot afford to test every variation. It is not common in web analytics because (1) we usually assume that one particular change to a site or app will be independent of another change elsewhere, so there is no need to test them concurrently to see if a particular combo stands out and (2) if we did want to test combinations of variables concurrently, there's usually enough users or visitors to just test the entire grid and not worry about picking a selection of variations.

MauranKilom 3y ago

Not exactly the same topic, but I feel like this is a good context to bring up https://extremelearning.com.au/unreasonable-effectiveness-of-quasirandom-sequences/ https://extremelearning.com.au/unreasonable-effectiveness-of.... It provides (in some sense) maximally un-aliased parameter choices when trying to cover a (continuous) parameter space. It generalizes to arbitrary dimensions, doesn't require you to choose the number of samples/experiments beforehand, and actually expands trivially to parameters representing classes (including possibly weighting the classes). I think it would be interesting to see how using this sequence fares in such a fractional factorial analysis, i.e. how close to optimal the (e.g. binarized) pseudorandom parameter choices are for different numbers of experiments!

CrimpCity 3y ago

Dang I was hoping the article would have been about non-traditional higher level fractional factorial designs. Like for example you have an experiment with 3 factors and 5, 4, 3 levels each but can only run 20 experiments.

deleted 3y ago
[deleted]

esafak 3y ago

Does any experimentation platform support this?

mtu 3y ago
JMP is used industrially for design of experiments: https://www.jmp.com/en_us/software/capabilities/design-of-experiments.html https://www.jmp.com/en_us/software/capabilities/design-of-ex... R also seems to have some libraries for things related to design of experiments.
- nxobject 3y ago
  JMP is also a very criminally underrated and flexible tool to do data analysis with – both exploratory and modelling. It's my preferred tool of choice over R, especially after they added a structural equation modelling platform. It's a pity they got rid of pricy perpetual licensing quite a few years ago for a no less pricy subscription model.
stupidcar 3y ago
For biology, Synthace supports designing experimental protocols using DOE, executing them on robotically automated lab equipment, then performing analysis on the results: https://www.synthace.com/ https://www.synthace.com/
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]

clircle 3y ago

This is a great tutorial.

alright2565 3y ago

By coincidence, I just watched this video on this same topic: https://m.youtube.com/watch?v=5oULEuOoRd0 https://m.youtube.com/watch?v=5oULEuOoRd0 The visualizations and the real problem solving demonstrated made it much easier (for me) to intuitively understand than any amount of reading

danbruc 3y ago
This article is actually in response to that video by NightHawkInLight, the method used in the video is however different from the one explained in the article, at least that is what the first paragraphs say, I did not read it completely.

wyldfire 3y ago

NIST has a combinatorial test engine designed for combining feature settings to maximize coverage for N tests. https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software https://csrc.nist.gov/projects/automated-combinatorial-testi...

penteract 3y ago

Something about this doesn't make sense - from the C=A case: > we wouldn’t be able to tell the difference in results between only doing option B (doubling the butter) and doing all three options: adding an egg, doubling the butter, and adding nuts on top. However, the experiments listed include exactly those 2 experiments (Test 1 and Test 3 in the table with a column titled C=A), so unless I'm missing something, "wouldn't be able to tell the difference" doesn't mean what I expect it to.

kawzeg 3y ago
Similarly: > Lastly, I = AC means that we can’t tell the difference between doing nothing at all, compared to adding an egg and adding nuts on top. But test 0 and test 2 are doing nothing and adding an egg & nuts respectively. What are we missing here? On further reading, it seems like "doing all three options" should be interpreted to mean "the interaction of all three options"? So we aren't able to tell if an improvement of taste comes from B alone or from the interaction of A, B, and C. I'm mostly guessing though.
pfedak 3y ago
As best I can tell from reading https://www.stat.purdue.edu/~yuzhu/stat514s2006/Lecnot/fracfacts2006.pdf https://www.stat.purdue.edu/~yuzhu/stat514s2006/Lecnot/fracf... the unstated assumption here is that we're going to do linear regression where (critically) the "-" case for each condition is -1, and the "+" is +1. This has the surprising-to-me effect of making "I", which looks like it might be the control group based on notation, actually a positive recipient of AC interaction (and any even-order interaction). You can think of this as a change in basis in how you parse out the effects, where we're talking about 1 -1 -1 1 (like a covariance matrix) instead of 0 0 0 1 for an interaction. I have a gut feeling it's done this way mostly because the tools being used expect things to be expressed this way rather than any conscious choice by experimenters. Through this lens, if you test I, B, AC, ABC every experiment has a positive effect from AC interaction, and taking B-I, which we might think of as the effect from B, is in this paradigm also sensitive to the ABC interaction and the AB and BC interactions. The "real" effect from B would be approximated as (B + ABC - AC - I)/2, which is exactly the same as the effect from ABC interaction (which is positive when an odd number of its constituents are positive...). I'm pretty sure this is just a difference in mathematical perspective - you can represent exactly the same data, but the coefficients (i.e. effect values) will change, and there's a different notion of what you know vs don't know. Maybe there's a more convincing reason to do this when you have more than two "levels", but from the presentation in TFA it just feels like overcomplicating things with a confusing prior about how effects work. It also seems like the given example is just bad. If the parameters are numeric and there's not a reasonable "control", this perspective feels much more natural.

noqc 3y ago

I have studied at least a little combinatorics, and the language here is completely inpenetrable to me. Neither the question of "what property is this design supposed to have?" or "why does it have it?" are stated in plain or even mathematical english. The wiki entry is similarly unhelpful.

acyou 3y ago
It's Design of Experiments language, DOE being a damned useful tiny little subset of applied statistics.
usgroup 3y ago
In a full factorial design you’re able to calculate the main effects and all the interactions between everything and everything else: not just pairwise but all combinations of variables. You may not care for that because you don’t think there are such interactions or perhaps you just don’t have the experimental budget. In fractional designs you can chose to confound some of the interactions so that you can’t tell what’s happening beyond a certain level. Eg just main and pairwise effects, with the rest confounding those effects. You would do this because it requires fewer runs. You can also use it as a sort of screener when you have many variables by confounding things together in such a way that you can at least tell which variables are having the effects even if you can’t calculate the individual effects.
crispyambulance 3y ago
For whatever reasons, much of the community that does DOE's is beset with heavy use of opaque jargon and flagrant use of "canned tests" from software packages like minitab. Making statistical inferences from data is nuanced and complicated if the thing you're interested in doesn't just pop-out from simple descriptive statistics treatments (histograms, box-plots, scatterplots, simple linear models). There are some islands of sanity: - NIST has a great resource (https://www.itl.nist.gov/div898/handbook/index.htm https://www.itl.nist.gov/div898/handbook/index.htm). Don't be alarmed by the retro html appearance. It's actually meticulously maintained and contains reproduceable examples with data. The plain language is such a relief compared to other references. - "Statistics for Experimenters" the book by George Box et al, is a great resource that's fairly comprehensive in my opinion. Good, clear writing. It seems to be widely available online (don't know if that's intentional).

acyou 3y ago

Stat-Ease's Design-Expert is the industry standard software in this area. Would have liked to see the author do a little more attribution. These are old ideas and techniques, and certainly not attributable to this author. It's helpful to visualise the response surface. Diagrams are critical in learning this material.

deleted 3y ago
[deleted]