4 ms·

Aside from reproducibility issues in ML, what sort of issue did you have in mind in CS? Most CS work is 90% maths, I don't see how you can have reproducibility

by ArnoVW 5y ago

Aside from reproducibility issues in ML, what sort of issue did you have in mind in CS? Most CS work is 90% maths, I don't see how you can have reproducibility issues?

the_snooze 5y ago
Take, for example, network measurement research: https://conferences.sigcomm.org/imc/2021/accepted/ https://conferences.sigcomm.org/imc/2021/accepted/ One of those papers is about counting the scale and scope of online political advertising during the 2020 election. How does one reproduce that study? The 2020 election is long past, and that data isn't archived anywhere other than what the researchers have already collected. This is a pretty simple empiracal data collection tastk, but you can't just re-measure that today because that study is about a moving target.
tlb 5y ago
I worked on this problem 25 years ago. It hasn't gone away. In general, performance comparisons are hard to reproduce. For instance, when benchmarking network protocols, often a tiny change in configuration makes a big change in a results. You might change the size of a buffer from 150 packets to 151 packets and see performance double (or halve.) Instead of making measurements with some arbitrary choices for parameters, you can take lots of measurements with parameters randomly varied to show a distribution of measurements. It's hard work to track down all the possible parameters and decide on a reasonable range for them, so it's rarely done. I found many 10x variances in network protocol performance (like how fairly competing TCP streams can sharing bandwidth). The big idea was to show that by randomizing some decisions in the protocol (like discarding packets with some probability as the buffer gets full) you can make the performance less sensitive to small changes. ie, more reproducible. Less sensitivity is especially good when you care about the worst-case performance rather than average. It can also make tuning a protocol much easier, since you aren't constantly being fooled by unstable performance. Performance sensitivity analysis is hard work, so most papers are just like "we ran our new thing 3 times and got similar numbers so there you go."

by ArnoVW 5y ago

Aside from reproducibility issues in ML, what sort of issue did you have in mind in CS? Most CS work is 90% maths, I don't see how you can have reproducibility issues?

the_snooze 5y ago

Take, for example, network measurement research: https://conferences.sigcomm.org/imc/2021/accepted/ https://conferences.sigcomm.org/imc/2021/accepted/ One of those papers is about counting the scale and scope of online political advertising during the 2020 election. How does one reproduce that study? The 2020 election is long past, and that data isn't archived anywhere other than what the researchers have already collected. This is a pretty simple empiracal data collection tastk, but you can't just re-measure that today because that study is about a moving target.

tlb 5y ago

I worked on this problem 25 years ago. It hasn't gone away. In general, performance comparisons are hard to reproduce. For instance, when benchmarking network protocols, often a tiny change in configuration makes a big change in a results. You might change the size of a buffer from 150 packets to 151 packets and see performance double (or halve.) Instead of making measurements with some arbitrary choices for parameters, you can take lots of measurements with parameters randomly varied to show a distribution of measurements. It's hard work to track down all the possible parameters and decide on a reasonable range for them, so it's rarely done. I found many 10x variances in network protocol performance (like how fairly competing TCP streams can sharing bandwidth). The big idea was to show that by randomizing some decisions in the protocol (like discarding packets with some probability as the buffer gets full) you can make the performance less sensitive to small changes. ie, more reproducible. Less sensitivity is especially good when you care about the worst-case performance rather than average. It can also make tuning a protocol much easier, since you aren't constantly being fooled by unstable performance. Performance sensitivity analysis is hard work, so most papers are just like "we ran our new thing 3 times and got similar numbers so there you go."