3 ms·

I don't doubt this guy's assertion that the data could be formatted better, but if I were working on something like this I'd just be glad that I could access th

I don't doubt this guy's assertion that the data could be formatted better, but if I were working on something like this I'd just be glad that I could access the data at all. And 64 lines of code... bfd? That's much better than the bad old days of custom binary data formats where you have to write a thousand line data loader in C, worrying about issues like lsb vs msb, whether the number storage is properly following IEEE 754, etc. I guess I'm just...old, because my first response to 'I had to write 64 lines of Python code to get this data into shape' is to shake my head and think "Kids these days!".

repiret 15y ago
I wish I could parse everything I've been asked to parse with 64 lines of Python. FWIW, I find it ironic that the author is complaining about whitespace-implied structure in his source data, yet uses Python.
- john_horton 15y ago
  My point was not that this was absolutely hard, but rather that's way too hard for the totally mundane thing I was trying to accomplish---namely getting public data from a US-funded statistical agency into a form suitable for further statistical analysis (which is the whole reason they make this data public). Also, I don't see how it's ironic that I used Python---it seems irrelevant. Whitespace in Python has a well-defined, commonly understood purpose; using tabs or spaces to indicate hierarchal data relationships is not at all standard, presumably because it creates messy dependencies across rows of data.
  - lurker19 15y ago
    "Messy dependencies across rows" is exactly how Python whitespace works. That data file exactly Pythonic indentation in the main content section.
    - john_horton 15y ago
      The reason indentation to imply structure doesn't make sense for datasets is that in data analysis you are constantly subsetting and sorting data at the row level---something you generally don't do to the lines of a compter program.

repiret 15y ago

I wish I could parse everything I've been asked to parse with 64 lines of Python. FWIW, I find it ironic that the author is complaining about whitespace-implied structure in his source data, yet uses Python.

john_horton 15y ago
My point was not that this was absolutely hard, but rather that's way too hard for the totally mundane thing I was trying to accomplish---namely getting public data from a US-funded statistical agency into a form suitable for further statistical analysis (which is the whole reason they make this data public). Also, I don't see how it's ironic that I used Python---it seems irrelevant. Whitespace in Python has a well-defined, commonly understood purpose; using tabs or spaces to indicate hierarchal data relationships is not at all standard, presumably because it creates messy dependencies across rows of data.
- lurker19 15y ago
  "Messy dependencies across rows" is exactly how Python whitespace works. That data file exactly Pythonic indentation in the main content section.
  - john_horton 15y ago
    The reason indentation to imply structure doesn't make sense for datasets is that in data analysis you are constantly subsetting and sorting data at the row level---something you generally don't do to the lines of a compter program.