3 ms·

I was a researcher at the Large Hadron Collider around the time “Big Data” became a thing. We had one of the use cases where analyzing all the data made sense,

I was a researcher at the Large Hadron Collider around the time “Big Data” became a thing. We had one of the use cases where analyzing all the data made sense, since it boiled down to frequentist statistics, the more data, the better. Yet even with a global network of supercomputers at our disposal, we funnily figured out that fast local storage was better than waiting for huge jobs to finish. So, surprise, surprise, every single grad student managed somehow to boil the relevant data for her analysis down to exactly 1-5 TB, without much loss in analysis flexibility. There must be like a law of convenience here, that rivals Amdahl’s scaling law.

msl09 2y ago
I think that your law of convenience is spot on. One thing that got by talking with commercial systems devs is that they are always under pressure by their clients to make their systems as cheap as possible, reducing the database stored and the size of the computations is one great way to minimize the client's monthly bill.
civilized 2y ago
I think there is a law of convenience, and it also explains why many technologies improve at a consistent exponential rate. People are very good at finding convenient ways to make something a little better each year, but every idea takes some minimal time to execute.
marcosdumay 2y ago
Let me try one: "If you can't do your statistical analysis in 1 to 5 TB of data, your methodology is flawed" This is probably more about human limitations than math. There's a clear ceiling in how much flexibility we can use. That will also change with easier ways to run new kinds of analysis, but it increases with the logarithm of the amount of things we want to do.
kwillets 2y ago
Back in the 80's and 90's NASA built a National Aerodynamic Simulator, which was a big Cray or similar that could crunch FEA simulations (probably a low-range graphics card nowadays). IIRC they found that the queue for that was as long or longer than it took to run jobs on cheaper hardware; MPP systems such as Beowulf grew out of those efforts.

msl09 2y ago

I think that your law of convenience is spot on. One thing that got by talking with commercial systems devs is that they are always under pressure by their clients to make their systems as cheap as possible, reducing the database stored and the size of the computations is one great way to minimize the client's monthly bill.

civilized 2y ago

I think there is a law of convenience, and it also explains why many technologies improve at a consistent exponential rate. People are very good at finding convenient ways to make something a little better each year, but every idea takes some minimal time to execute.

marcosdumay 2y ago

Let me try one: "If you can't do your statistical analysis in 1 to 5 TB of data, your methodology is flawed" This is probably more about human limitations than math. There's a clear ceiling in how much flexibility we can use. That will also change with easier ways to run new kinds of analysis, but it increases with the logarithm of the amount of things we want to do.

kwillets 2y ago

Back in the 80's and 90's NASA built a National Aerodynamic Simulator, which was a big Cray or similar that could crunch FEA simulations (probably a low-range graphics card nowadays). IIRC they found that the queue for that was as long or longer than it took to run jobs on cheaper hardware; MPP systems such as Beowulf grew out of those efforts.