11 ms·

How to share data with a statistician

70 points by shrikant 13y ago

imahboob 13y ago
or just use R tools yourself
compare 13y ago
How does everyone feel about a JSON dicts separated by newlines format? (Like the JSON format that mongoimport can accept.) Each sample of the type of data that I'm often dealing with tends to be nested in nature. Yes, I do have a script that can flatten out the nested dicts into a regular table, but that always results in a blowup into hundreds of columns. Nice suggestion to share the raw data. I've never seen a researcher do that, I think many don't even save the raw data to disk before extracting what they want, but I always try to.
- edraferi 13y ago
  I've been using this format recently and appreciate the additional fidelity from simple rows. The flexibility makes it good for the raw stage, but I usually have to extract tidy subsets for real analytic work. I have written many little scripts to pull arbitrarily deep keys out of these structures and produce tidy tables for further analysis. actually, this format is also nice because iterating over the lines of a file is very similar to running through a mongo cursor. that makes it easy to reprise choose to work with both inputs.
- edraferi 13y ago
  I've been using this format recently and appreciate the additional fidelity from simple rows. The flexibility makes it good for the raw stage, but I usually have to extract tidy subsets for real analytic work. I have written many little scripts to pull arbitrarily deep keys out of these structures and produce tidy tables for further analysis. actually, this format is also nice because iterating over the lines of a file is very similar to running through a mongo cursor. that makes it easy to reprise choose to work with both inputs.
- pallandt 13y ago
  It's rare (unfortunately), but Dr. Jeffrey Leek isn't the 1st proponent of sharing raw data, or otherwise a believer in reproducible results. Dr. Eamonn Keogh has made very important contributions in data mining and is also a huge advocate of reproducibility. See for example 'Why the lack of reproducibility is crippling research in data mining and what you can do about it' dated 2007 @ http://dl.acm.org/citation.cfm?id=1341922 http://dl.acm.org/citation.cfm?id=1341922
loup-vaillant 13y ago
Before all that, you need to choose your statistician. My advice is: resuscitate E.T. Jaynes. Failing that, find one of his disciples. Failing that, find a Bayesian. Failing that, read Probability Theory: the Logic of Science, and just do the analysis yourself. Failing that, maybe a Frequentist statistician will do. Maybe.
- pja 13y ago
  And your Bayesian statistician will acquire their priors how exactly? Bayesian statistics can be very powerful, but it would be a terrible idea to prefer Bayesian approaches to Frequentist ones in all situations.
  - loup-vaillant 13y ago
    > And your Bayesian statistician will acquire their priors how exactly? As If Frequentists somehow didn't need priors. Everyone starts with prior knowledge. We might as well use it. Or do you advocate not using every scrap of knowledge available to you? That would be stupid. Sure, prior knowledge can be shaky, or difficult to justify. But at least, a Bayesian will be explicit about it, instead of, like, sweeping normal probability distribution assumptions under the linear regression rug. > it would be a terrible idea to prefer Bayesian approaches to Frequentist ones in all situations. Name three examples that doesn't involve the Frequentist using better prior information than the Bayesian. By the way, Bayesians know that using probability theory correctly is sometimes intractable (combinatorial explosion and all that). In those cases, they will use approximations. But at least, they will know it's an approximation. --- You really should read chapters 1 and 2 of Probability Theory: the Logic of Science. They give a good feel of why Bayesians are correct as a simple matter of fact.
    - rfergie 13y ago
      Those chapters show that any consistent (in some sense) statistical reasoning system must result in the same answers as a Bayesian system. They don't show that the frequentist approach is wrong. If both methods result in the same answer and frequentist methods are easier to use then what is the problem?
bermanoid 13y ago
I'd like to follow this up with a plea to statisticians to make sure you're not sending data back to programmers with bizarre newline formats - certain versions of Mac Excel save with /r newlines, which haven't been used in over a decade and pretty much break everything. If in doubt, it's probably best to save in Windows newline mode from MS software, at least most utilities are used to dealing with that.
- sampo 13y ago
  I once debugged a case like that. A NumPy script expected to read in a table from a file, but the file used /r newlines, so the NumPy script read it in as a very long single line. Then later references to table lines >1 broke the script. The strange part is, the file and the script were provided by the same person.
- FigBug 13y ago
  And a plea to programmers, handle \n, \r\n and \r. And utf8, utf16, and all the other common text encodings. It's sad that in 2013 it'sstill hard to read in a text file, so the burden is shifted to the user.
tel 13y ago
The Leek group is doing some fantastic things, pushing for better transparency and coding integrity from statisticians. I met with Jeff a little while back and was just really enthused by his desire to change the way traditional statistics/biostatistics is viewed and interacted with.
croisillon 13y ago
Bit of a side-question but I have been looking for such a thing without really knowing what to look for: is there some kind of small and easy JS and/or PHP programm allowing to do some easy work on a tidy database? Pretty much like you would have a full excel table in a browser where you can add/modify/remove rows and columns? No need for all the excel formulae and such... Thanks very much for any help.
- alanctgardner2 13y ago
  Most databases have a companion GUI tool that will allow you to "edit" the database visually; what DB are you using? pgAdmin (Postgres) - http://www.pgadmin.org/ http://www.pgadmin.org/ MySQL Workbench (MySQL) - http://dev.mysql.com/downloads/tools/workbench/ http://dev.mysql.com/downloads/tools/workbench/ SQL Developer (Oracle) - http://www.oracle.com/technetwork/developer-tools/sql-developer/overview/index.html http://www.oracle.com/technetwork/developer-tools/sql-develo... I strongly recommend learning some SQL though; this will give you the ability to bulk edit columns and apply formulae.
  - croisillon 13y ago
    I was thinking TSV in plain text format, not SQL. Hence the option of a pure JS solution?
    - alanctgardner2 13y ago
      A TSV is typically referred to as a "flat file". You're going to get a lot of confused looks if you talk about a "TSV database". That said, you can: - create an external table in an RDBMS (any of the above will work), which allows you to work on a flat file in place: http://www.fromdual.com/csv-storage-engine http://www.fromdual.com/csv-storage-engine - import the TSV to an RDBMS, work with it, and export it again. I'm sure someone has written what you're asking for, but I don't get the appeal. RDBMS aren't scary, you can install MySQL and work with it for free, and if you use the GUI tool you don't even have to touch SQL. edit: Excel is a tool that can edit TSV files, and as a bonus it looks and works exactly like Excel. What exactly do you want?
jtleek 13y ago
If you liked my guide to sharing data, you may also like my guides: For writing R packages: https://github.com/jtleek/rpackages https://github.com/jtleek/rpackages For writing scientific reviews: https://github.com/jtleek/reviews https://github.com/jtleek/reviews Or on the future of stats: https://github.com/jtleek/futureofstats https://github.com/jtleek/futureofstats

70 points by shrikant 13y ago

or just use R tools yourself

How does everyone feel about a JSON dicts separated by newlines format? (Like the JSON format that mongoimport can accept.) Each sample of the type of data that I'm often dealing with tends to be nested in nature. Yes, I do have a script that can flatten out the nested dicts into a regular table, but that always results in a blowup into hundreds of columns. Nice suggestion to share the raw data. I've never seen a researcher do that, I think many don't even save the raw data to disk before extracting what they want, but I always try to.

edraferi 13y ago
I've been using this format recently and appreciate the additional fidelity from simple rows. The flexibility makes it good for the raw stage, but I usually have to extract tidy subsets for real analytic work. I have written many little scripts to pull arbitrarily deep keys out of these structures and produce tidy tables for further analysis. actually, this format is also nice because iterating over the lines of a file is very similar to running through a mongo cursor. that makes it easy to reprise choose to work with both inputs.
edraferi 13y ago
I've been using this format recently and appreciate the additional fidelity from simple rows. The flexibility makes it good for the raw stage, but I usually have to extract tidy subsets for real analytic work. I have written many little scripts to pull arbitrarily deep keys out of these structures and produce tidy tables for further analysis. actually, this format is also nice because iterating over the lines of a file is very similar to running through a mongo cursor. that makes it easy to reprise choose to work with both inputs.
pallandt 13y ago
It's rare (unfortunately), but Dr. Jeffrey Leek isn't the 1st proponent of sharing raw data, or otherwise a believer in reproducible results. Dr. Eamonn Keogh has made very important contributions in data mining and is also a huge advocate of reproducibility. See for example 'Why the lack of reproducibility is crippling research in data mining and what you can do about it' dated 2007 @ http://dl.acm.org/citation.cfm?id=1341922 http://dl.acm.org/citation.cfm?id=1341922

loup-vaillant 13y ago

Before all that, you need to choose your statistician. My advice is: resuscitate E.T. Jaynes. Failing that, find one of his disciples. Failing that, find a Bayesian. Failing that, read Probability Theory: the Logic of Science, and just do the analysis yourself. Failing that, maybe a Frequentist statistician will do. Maybe.

pja 13y ago
And your Bayesian statistician will acquire their priors how exactly? Bayesian statistics can be very powerful, but it would be a terrible idea to prefer Bayesian approaches to Frequentist ones in all situations.
- loup-vaillant 13y ago
  > And your Bayesian statistician will acquire their priors how exactly? As If Frequentists somehow didn't need priors. Everyone starts with prior knowledge. We might as well use it. Or do you advocate not using every scrap of knowledge available to you? That would be stupid. Sure, prior knowledge can be shaky, or difficult to justify. But at least, a Bayesian will be explicit about it, instead of, like, sweeping normal probability distribution assumptions under the linear regression rug. > it would be a terrible idea to prefer Bayesian approaches to Frequentist ones in all situations. Name three examples that doesn't involve the Frequentist using better prior information than the Bayesian. By the way, Bayesians know that using probability theory correctly is sometimes intractable (combinatorial explosion and all that). In those cases, they will use approximations. But at least, they will know it's an approximation. --- You really should read chapters 1 and 2 of Probability Theory: the Logic of Science. They give a good feel of why Bayesians are correct as a simple matter of fact.
  - rfergie 13y ago
    Those chapters show that any consistent (in some sense) statistical reasoning system must result in the same answers as a Bayesian system. They don't show that the frequentist approach is wrong. If both methods result in the same answer and frequentist methods are easier to use then what is the problem?

bermanoid 13y ago

I'd like to follow this up with a plea to statisticians to make sure you're not sending data back to programmers with bizarre newline formats - certain versions of Mac Excel save with /r newlines, which haven't been used in over a decade and pretty much break everything. If in doubt, it's probably best to save in Windows newline mode from MS software, at least most utilities are used to dealing with that.

sampo 13y ago
I once debugged a case like that. A NumPy script expected to read in a table from a file, but the file used /r newlines, so the NumPy script read it in as a very long single line. Then later references to table lines >1 broke the script. The strange part is, the file and the script were provided by the same person.
FigBug 13y ago
And a plea to programmers, handle \n, \r\n and \r. And utf8, utf16, and all the other common text encodings. It's sad that in 2013 it'sstill hard to read in a text file, so the burden is shifted to the user.

tel 13y ago

The Leek group is doing some fantastic things, pushing for better transparency and coding integrity from statisticians. I met with Jeff a little while back and was just really enthused by his desire to change the way traditional statistics/biostatistics is viewed and interacted with.

croisillon 13y ago

Bit of a side-question but I have been looking for such a thing without really knowing what to look for: is there some kind of small and easy JS and/or PHP programm allowing to do some easy work on a tidy database? Pretty much like you would have a full excel table in a browser where you can add/modify/remove rows and columns? No need for all the excel formulae and such... Thanks very much for any help.

alanctgardner2 13y ago
Most databases have a companion GUI tool that will allow you to "edit" the database visually; what DB are you using? pgAdmin (Postgres) - http://www.pgadmin.org/ http://www.pgadmin.org/ MySQL Workbench (MySQL) - http://dev.mysql.com/downloads/tools/workbench/ http://dev.mysql.com/downloads/tools/workbench/ SQL Developer (Oracle) - http://www.oracle.com/technetwork/developer-tools/sql-developer/overview/index.html http://www.oracle.com/technetwork/developer-tools/sql-develo... I strongly recommend learning some SQL though; this will give you the ability to bulk edit columns and apply formulae.
- croisillon 13y ago
  I was thinking TSV in plain text format, not SQL. Hence the option of a pure JS solution?
  - alanctgardner2 13y ago
    A TSV is typically referred to as a "flat file". You're going to get a lot of confused looks if you talk about a "TSV database". That said, you can: - create an external table in an RDBMS (any of the above will work), which allows you to work on a flat file in place: http://www.fromdual.com/csv-storage-engine http://www.fromdual.com/csv-storage-engine - import the TSV to an RDBMS, work with it, and export it again. I'm sure someone has written what you're asking for, but I don't get the appeal. RDBMS aren't scary, you can install MySQL and work with it for free, and if you use the GUI tool you don't even have to touch SQL. edit: Excel is a tool that can edit TSV files, and as a bonus it looks and works exactly like Excel. What exactly do you want?

jtleek 13y ago

If you liked my guide to sharing data, you may also like my guides: For writing R packages: https://github.com/jtleek/rpackages https://github.com/jtleek/rpackages For writing scientific reviews: https://github.com/jtleek/reviews https://github.com/jtleek/reviews Or on the future of stats: https://github.com/jtleek/futureofstats https://github.com/jtleek/futureofstats