4 ms·

30 minutes is a long time. What if there's a tiny bit of inconsistency in the format, and you spend even more time working around it? Then there's the need to s

30 minutes is a long time. What if there's a tiny bit of inconsistency in the format, and you spend even more time working around it? Then there's the need to spend time to be sure that the transformed data is still correct. By the time you've realized the thing is going to take much longer than you thought, you've already wasted enough time.

mmanfrin 9y ago
Hah, you were spot on. I gave a go at trying to normalize it. I probably spent over an hour getting to this point: https://docs.google.com/spreadsheets/d/1jmdBXVrT6yiXWM3UfWBIJnLtS71WD0prIzb9wtvr858/edit#gid=0 https://docs.google.com/spreadsheets/d/1jmdBXVrT6yiXWM3UfWBI... The tricky bit is extracting the different values out of the price column. I managed to get a decent way there, but need to fix the algorithm to handle multiple (variable) groups of denominations. Once I get there I can likely do plain multiplications by the columns to get them normalized (and then maybe pare it back down to add them up within one giant function). It's definitely a rabbit hole, easily a few more hours of hacking away at it. Would be quicker to do it by hand but that's no fun.
- thaumasiotes 9y ago
  >> Then there's the need to spend time to be sure that the transformed data is still correct. >> By the time you've realized the thing is going to take much longer than you thought, you've already wasted enough time. > I gave a go at trying to normalize it. I probably spent over an hour None of that is something that having the data originally presented in "table form" (whatever that meant... it IS originally presented in table form as far as I understand the term) would have addressed. There are plenty of tables / databases / other collections out there that need cleaning. The issue of "is this data accurate?" is completely unrelated to the issue of "does this data come in a .csv?"

mmanfrin 9y ago

Hah, you were spot on. I gave a go at trying to normalize it. I probably spent over an hour getting to this point: https://docs.google.com/spreadsheets/d/1jmdBXVrT6yiXWM3UfWBIJnLtS71WD0prIzb9wtvr858/edit#gid=0 https://docs.google.com/spreadsheets/d/1jmdBXVrT6yiXWM3UfWBI... The tricky bit is extracting the different values out of the price column. I managed to get a decent way there, but need to fix the algorithm to handle multiple (variable) groups of denominations. Once I get there I can likely do plain multiplications by the columns to get them normalized (and then maybe pare it back down to add them up within one giant function). It's definitely a rabbit hole, easily a few more hours of hacking away at it. Would be quicker to do it by hand but that's no fun.

thaumasiotes 9y ago
>> Then there's the need to spend time to be sure that the transformed data is still correct. >> By the time you've realized the thing is going to take much longer than you thought, you've already wasted enough time. > I gave a go at trying to normalize it. I probably spent over an hour None of that is something that having the data originally presented in "table form" (whatever that meant... it IS originally presented in table form as far as I understand the term) would have addressed. There are plenty of tables / databases / other collections out there that need cleaning. The issue of "is this data accurate?" is completely unrelated to the issue of "does this data come in a .csv?"