3 ms·

I studied Design of Experiments (DOE) in college -- it was part of the core curriculum in my major. The content feels like it ought to be very useful (it's basi

by wenc 3y ago

I studied Design of Experiments (DOE) in college -- it was part of the core curriculum in my major. The content feels like it ought to be very useful (it's basically techniques on how to conduct experiments efficiently and reducing the number of runs, thus reducing overall cost).... yet I've never had occasion to ever apply it. One area where I feel it might be relevant to my work is in hyperparameter tuning, especially when function evals are expensive. Instead of doing grid search (which is a brute-force exhaustive search), fractional factorial designs can search the multidimensional space with way less function evals. Yet I don't do it because it's easier to write code to do brute force and let the computer chug away over night. In the world of web analytics, fractional factorial lets you run multivariate experiments efficiently rather than a combinatorial combination of A/B tests. From online articles, I'm guessing people do use it there, but A/B tests still dominate. Fractional factorial is also most effective when the covariates are orthogonal and independent, which is rarely the case. We can get around this by projecting high dimensional covariates into lower dimensional space using PCA, which guarantees orthogonality, but this also seems to not be done so much. Just wondering if anyone is using fractional factorial designs in real life? (or optimal designs like D-optimal designs)

stdbrouw 3y ago
> Fractional factorial is also most effective when the covariates are orthogonal and independent, which is rarely the case. We can get around this by projecting high dimensional covariates into lower dimensional space using PCA, which guarantees orthogonality, but this also seems to not be done so much. I guess you're right in that if there were very many interactions between variables, then the aliasing of a fractional design would be an incredible nuisance, but on the other hand if there were no dependence between any of the variables at all then there would be no point to any sort of factorial design as you could just test each variable sequentially. > Just wondering if anyone is using fractional factorial designs in real life? (or optimal designs like D-optimal designs) I have to admit, I have never ever seen anyone calculate a D-optimal or G-optimal design for anything -- that said, I only graduated as a statistician 6 or 7 years ago. As you say, there might be some use to it for continuous variables but for factors, given that latin squares (etc.) are known to be optimal so you can just grab that off the shelf.
conjectures 3y ago
I think in many industrial settings Bayesian optimisation is more practical. Since it allows use to refine our design on the fly and put resources into promising areas of the space earlier. Some kind of pre-planned fractional design could be useful for generating an initial set of points to test.
thereisnospork 3y ago
I've at times made fairly extensive use of d-optimal designs in formulation type research and development, usually in an iterative process, pruning and adding variables as well as refining ranges. Lot of footguns, but the approach can drastically decrease optimization time. In my experience it needs the right combination of: too many factors to make full factorial reasonable, too much (expected) covariance to trust single-vaeiable optimization, and cheap enough to commit to doing 8+ experiments before a major evaluation.
- samvher 3y ago
  What are the footguns you've run into with D-optimal designs?
  - thereisnospork 3y ago
    Mainly around bounding variables, especially when coupled with overly ambitious scope/number of tests. The short version is that its really easy to wind up with say a 80 experiment test where half or more of the combinations are invalid or perform poorly to the degree they don't adequately lend confidence to the prediction in the region of the global maximum. Leads to wasted work and the poor predictive ability doesn't lead well into focused future work. There are also the usual statistical foot guns of p-hacking and such. For a concrete example, consider a bread baking optimization, considering time, temperature, and baking soda %. This gives a cubic design space, for which the naively (ignoring expected covariance of temperature*time) optimal design for 9 experiments is the cube's corners + the center point. If, for instance, your predetermined t_max always results in a briquette instead of a loaf, ~half of your experimental data is going to be worthless. With more nuance, and temp_max is genuinely the highest reasonable temperature there are still two problems: a. covariance is likely to drive combinations into 'invalid' territory (e.g. temp_max + time_max is likely to be invalid, or 2/9ths of your experimental. temp_min / time_min is also likely invalid for another 2/9ths). b. predictive power / linearity of response over the ranges specified. Even if the combinations aren't invalid, if they are all the min/max combinations are poorly performing (due to overly wide, but valid, boundaries) you can wind up with poor predictive performance. Covariance can be accounted for, design space trimmed (e.g. to a cube with a corner or two cut off), and bounds set conservatively but that is all tricky manual intervention that relies on knowledge of the problem domain and scaling factors of the underlying physics. When the problem isn't well understood it is easy to make errors in assumptions, those errors have a high cost, and if the problem was well understood a DOE probably wouldn't be necessary. edit: for a less trivial example of a suitable problem for a d-optimal DOE, but with tricky bounding / underlying physics, consider: a 4 part formulation of fumed silica, cyanoacrylate, isopropanol, and water to make a gap-filling/quick-setting adhesive.
jhsvsmyself 3y ago
I just took this class last semester. I've been trying to find things to do with what I learned but I haven't found anything yet.
- densh 3y ago
  For coffee affectionados out there: picking best coffee setup settings is very similar to a physics experiment, and you can totally use the knowledge of experiment design theory there. Settings: * Kinds of beans at various levels of roasting available to you. * Grind size on a grinder with controllable grind size in steps. * Ratio of ground coffee weight to water weight used for brewing. * For filter coffee: preinfusion time, brew time or flow rate (see V60 brewer). * For espresso coffee: preinfusion time & pressure, brew time & pressure or flow rate (see Flair Espresso). * Water temperature. * Water mineral composition. * For milk-based coffee drinks: kinds of milk, milk percentage as ratio of the total drink volume, steaming duration, final milk temperature. All of these create a huge factorial space of possible configurations, especially if you've go into flow or pressure profiling. If you frame it as an experiment you might isolate some of the variables that make the most impact tailored to your coffee preferences. And here is an example of a scientifically inclined paper in this space if you went to get inspired: https://www.cell.com/matter/fulltext/S2590-2385(19)30410-2?_returnURL=https://linkinghub.elsevier.com/retrieve/pii/S2590238519304102?showall=true https://www.cell.com/matter/fulltext/S2590-2385(19)30410-2?_...
onos 3y ago
Random search beats grid search!
samvher 3y ago
I'm in biotech (as a software engineer/data scientist) and here this stuff is used a lot. Experiments are very hands on and expensive in terms of inputs so you want to be efficient with your use of them. Bayesian approaches (e.g. optimization with Gaussian processes) tend to be very serial in nature: later sampling points are decided from the information you get earlier on. But these biological experiments can take days or weeks so you want to run them in parallel as much as possible. DoE is excellent for that. I'm speccing out a D-optimal design as we speak (though I'm not sure this is what we will use for the particular project I'm on).
cgearhart 3y ago
I used a lot of these techniques as an operations research analyst in the early 2010’s. I also wondered the same thing about DoE for hyperparameter tuning. Always felt like it was a case of grid search/random search/bayes opt being “good enough” and easier. But maybe DoE would be worth it for something like LLMs where training runs for a month on 25k GPUs.

by wenc 3y ago

stdbrouw 3y ago

> Fractional factorial is also most effective when the covariates are orthogonal and independent, which is rarely the case. We can get around this by projecting high dimensional covariates into lower dimensional space using PCA, which guarantees orthogonality, but this also seems to not be done so much. I guess you're right in that if there were very many interactions between variables, then the aliasing of a fractional design would be an incredible nuisance, but on the other hand if there were no dependence between any of the variables at all then there would be no point to any sort of factorial design as you could just test each variable sequentially. > Just wondering if anyone is using fractional factorial designs in real life? (or optimal designs like D-optimal designs) I have to admit, I have never ever seen anyone calculate a D-optimal or G-optimal design for anything -- that said, I only graduated as a statistician 6 or 7 years ago. As you say, there might be some use to it for continuous variables but for factors, given that latin squares (etc.) are known to be optimal so you can just grab that off the shelf.

conjectures 3y ago

I think in many industrial settings Bayesian optimisation is more practical. Since it allows use to refine our design on the fly and put resources into promising areas of the space earlier. Some kind of pre-planned fractional design could be useful for generating an initial set of points to test.

thereisnospork 3y ago

I've at times made fairly extensive use of d-optimal designs in formulation type research and development, usually in an iterative process, pruning and adding variables as well as refining ranges. Lot of footguns, but the approach can drastically decrease optimization time. In my experience it needs the right combination of: too many factors to make full factorial reasonable, too much (expected) covariance to trust single-vaeiable optimization, and cheap enough to commit to doing 8+ experiments before a major evaluation.

samvher 3y ago
What are the footguns you've run into with D-optimal designs?
- thereisnospork 3y ago
  Mainly around bounding variables, especially when coupled with overly ambitious scope/number of tests. The short version is that its really easy to wind up with say a 80 experiment test where half or more of the combinations are invalid or perform poorly to the degree they don't adequately lend confidence to the prediction in the region of the global maximum. Leads to wasted work and the poor predictive ability doesn't lead well into focused future work. There are also the usual statistical foot guns of p-hacking and such. For a concrete example, consider a bread baking optimization, considering time, temperature, and baking soda %. This gives a cubic design space, for which the naively (ignoring expected covariance of temperature*time) optimal design for 9 experiments is the cube's corners + the center point. If, for instance, your predetermined t_max always results in a briquette instead of a loaf, ~half of your experimental data is going to be worthless. With more nuance, and temp_max is genuinely the highest reasonable temperature there are still two problems: a. covariance is likely to drive combinations into 'invalid' territory (e.g. temp_max + time_max is likely to be invalid, or 2/9ths of your experimental. temp_min / time_min is also likely invalid for another 2/9ths). b. predictive power / linearity of response over the ranges specified. Even if the combinations aren't invalid, if they are all the min/max combinations are poorly performing (due to overly wide, but valid, boundaries) you can wind up with poor predictive performance. Covariance can be accounted for, design space trimmed (e.g. to a cube with a corner or two cut off), and bounds set conservatively but that is all tricky manual intervention that relies on knowledge of the problem domain and scaling factors of the underlying physics. When the problem isn't well understood it is easy to make errors in assumptions, those errors have a high cost, and if the problem was well understood a DOE probably wouldn't be necessary. edit: for a less trivial example of a suitable problem for a d-optimal DOE, but with tricky bounding / underlying physics, consider: a 4 part formulation of fumed silica, cyanoacrylate, isopropanol, and water to make a gap-filling/quick-setting adhesive.

jhsvsmyself 3y ago

I just took this class last semester. I've been trying to find things to do with what I learned but I haven't found anything yet.

densh 3y ago
For coffee affectionados out there: picking best coffee setup settings is very similar to a physics experiment, and you can totally use the knowledge of experiment design theory there. Settings: * Kinds of beans at various levels of roasting available to you. * Grind size on a grinder with controllable grind size in steps. * Ratio of ground coffee weight to water weight used for brewing. * For filter coffee: preinfusion time, brew time or flow rate (see V60 brewer). * For espresso coffee: preinfusion time & pressure, brew time & pressure or flow rate (see Flair Espresso). * Water temperature. * Water mineral composition. * For milk-based coffee drinks: kinds of milk, milk percentage as ratio of the total drink volume, steaming duration, final milk temperature. All of these create a huge factorial space of possible configurations, especially if you've go into flow or pressure profiling. If you frame it as an experiment you might isolate some of the variables that make the most impact tailored to your coffee preferences. And here is an example of a scientifically inclined paper in this space if you went to get inspired: https://www.cell.com/matter/fulltext/S2590-2385(19)30410-2?_returnURL=https://linkinghub.elsevier.com/retrieve/pii/S2590238519304102?showall=true https://www.cell.com/matter/fulltext/S2590-2385(19)30410-2?_...

onos 3y ago

Random search beats grid search!

samvher 3y ago

I'm in biotech (as a software engineer/data scientist) and here this stuff is used a lot. Experiments are very hands on and expensive in terms of inputs so you want to be efficient with your use of them. Bayesian approaches (e.g. optimization with Gaussian processes) tend to be very serial in nature: later sampling points are decided from the information you get earlier on. But these biological experiments can take days or weeks so you want to run them in parallel as much as possible. DoE is excellent for that. I'm speccing out a D-optimal design as we speak (though I'm not sure this is what we will use for the particular project I'm on).

cgearhart 3y ago

I used a lot of these techniques as an operations research analyst in the early 2010’s. I also wondered the same thing about DoE for hyperparameter tuning. Always felt like it was a case of grid search/random search/bayes opt being “good enough” and easier. But maybe DoE would be worth it for something like LLMs where training runs for a month on 25k GPUs.