4 ms·

If the institution is public, data should be public as long as individual PII is removed. No exceptions. And FOIA requests should be able to make this data avai

If the institution is public, data should be public as long as individual PII is removed. No exceptions. And FOIA requests should be able to make this data available to anyone filing for an access within a reasonable amount of time. Period.

kayodelycaon 3y ago
Not when there is an existing law limiting it. FERPA specifically protects student records.
- galangalalgol 3y ago
  Even anonymous records? That would seem to preclude studying the effectiveness of the education system. And if they weren't anonymous, what possible conclusions could you draw compared to anonymous records that would warrant that access?
- rovr138 3y ago
  > as long as individual PII is removed https://www2.ed.gov/policy/gen/guid/fpco/ferpa/index.html https://www2.ed.gov/policy/gen/guid/fpco/ferpa/index.html, "The Family Educational Rights and Privacy Act (FERPA) (20 U.S.C. § 1232g; 34 CFR Part 99) is a Federal law that protects the privacy of student education records" Now the question is if aggregated data with no PII is protected.
  - jasonlotito 3y ago
    Data aggregation does not guarantee privacy.
    - prepend 3y ago
      Of course not, but there are tests that can be applied to determine if privacy is protected. It’s not possible to just aggregate and be done. But it is possible to set some privacy threshold and then insure that all records conform to that acceptable risk level.
- jrochkind1 3y ago
  If FERPA makes it illegal to share the data with researchers, then certainly it shouldn't have been shared. If FERPA allows sharing the data with researchers, is it right/proper/legal to share only on the condition it can't be used to harm the schools in court? Presumably that part is not in FERPA. (And to be clear, California here didn't just say they couldn't use the data in court, they said the researchers could not testify in any court case at all against the state. But we're talking hypothetically)
tmpz22 3y ago
> as long as individual PII is removed The devil is in the details. These records were likely not designed to be shared and I'd assume the entire system contains vulnerabilities that could create leakage. Leakage that could be used to harm individuals in a variety of ways - from discriminating future prospects to harassment and much much more. I agree in principle to what you're saying but we need to be truthful about what these current systems are capable of.
maximinus_thrax 3y ago
> data should be public as long as individual PII is removed This is one of those things when ideology doesn't match the real world. If the amount of data is large enough and with enough parameters, removing PII doesn't do anything to protect privacy. What about medical records? What about protected classes? What about data about vulnerable people or victims? Student data is protected by another layer of regulation and for good reason. Also, the judiciary is a 'public' institution in general. Should we not seal records for minors? 'No exceptions' - year right..
- godelski 3y ago
  > If the amount of data is large enough and with enough parameters, removing PII doesn't do anything to protect privacy. Honestly it doesn't have to be that large. We see this all the time with data websites or apps collect. Sure, you remove John Smith's name, but you still have his GPS coordinates. For the school, you remove Professor Smith's name, but you have a professor who teaches CS 123 and has 4 graduate students. You bet you can guess who that is. I really do support open data, especially about public institutions, but at the same time we are in an era where this information is quite powerful. Seems to make a case for something like homeomorphic encryption or something, but will that even stop these collisions?
  - CaptainNegative 3y ago
    The appropriate notion here seems to be Differential Privacy, which is a mathematical definition informally saying "a scrambling of the dataset that is information theoretically indistinguishable from that where one arbitrary person is added or removed". It's a surprisingly deep topic, with entire (very good!) textbooks dedicated to it. PDF (entirely legal): https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf
- nradov 3y ago
  We're getting a little off topic here, but the Federal government has published specific guidance on de-identification of medical records. You can construct some artificial scenarios where re-identification might be theoretically possible through record linkage with other data sources but in practice it's unlikely. In principle a similar approach could be used for student data, although I'm not familiar with the legal issues. https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/index.html https://www.hhs.gov/hipaa/for-professionals/privacy/special-... But all of this is orthogonal to the core issue of whether a state government should be allowed to prevent researchers from participating in lawsuits. There is no student privacy issue involved there. Witnesses in a civil suit still aren't allowed to violate student privacy laws regardless of the data they have access to, so it makes no sense to conflate those issues.
jeroenhd 3y ago
Removing PII would probably also involve removing individual grades and other information that's necessary for any research to be effective. Thanks to predatory data collection practices on the internet, we know how little information you actually need to deanonimize someone. The problem only becomes worse when we're talking about kids. That said, research that cannot be reproduced is useless. There's a balance to be struck here, and it's somewhere between "make all data public" and "lock the data in a vault".
dragonwriter 3y ago
> If the institution is public, data should be public as long as individual PII is removed. PII is much broader than most people understand because reidentification of what amateurs would see as deidentified data is easy (often trivial), and, as a consequence, to be useful for research data is often not fully deidentified. EDIT: As an example, the HIPAA safe harbor deidentification standard requires removing 18 kinds of identifiers, including, as one of them: All geographic subdivisions smaller than a state, including street address, city, county, precinct, ZIP code, and their equivalent geocodes, except for the initial three digits of the ZIP code if, according to the current publicly available data from the Bureau of the Census: (1) The geographic unit formed by combining all ZIP codes with the same three initial digits contains more than 20,000 people; and (2) The initial three digits of a ZIP code for all such geographic units containing 20,000 or fewer people is changed to 000
- Zetice 3y ago
  To add to this, PII isn't always even clear. Different jurisdictions identify PII differently, there isn't One Master Definition that you pass a unit of data through, upon which an authoritative "THIS IS PII" or "THIS ISNT PII" is returned.
  - orzig 3y ago
    +1 We had a multi-month project to get a subset of our data considered 'clean', and it required a consultant, a stats PhD and many dev hours. It was healthcare, so on the high end of paranoia (justifiably) but nowhere it is as simple as dropping the "name" column
- prepend 3y ago
  HIPAA also allows for “expert determination” [0] for deidentification that differs from safe harbor and can allow for all sorts of things since there’s no definition of what an expert is. And reidentification risk can be as high as even 1% and still be acceptable for hipaa. In a dataset of a million people that’s 10,000 people identified and still be “acceptable.” But hipaa doesn’t apply to these CA data, it’s just the clearest example of deidentification regulations I know of. But it’s totally possible to deidentify data suitable for release to these researchers. It’s just what CA considers deidentified and if it’s still useful enough to these researchers. For the topic they are researching it should be pretty straightforward to remove PII enough to protect individuals and only remove some really unique characteristics (ie, only a single 20 year old or a particular race and ethnicity). But I’m guessing age groups by race and gender and socioeconomic are possible to preserve without tying back to an individual. Id go so far to say as it would be non-trivial, but pretty easy, for CA to produce this for the researchers, if not to the general public. [0] https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/index.html#standard https://www.hhs.gov/hipaa/for-professionals/privacy/special-...
nineplay 3y ago
You'd have to remove so much PII as to make any examination worthless "A student of age <redacted> and gender <redacted> at school <redacted> has a GPA of <redacted>". As little information as "a 16 year old black male at Main Street High School" can be narrowed down to a handful of possible candidates at a lot of CA schools.
- prepend 3y ago
  I disagree as it depends on how many 16 year old black males there are in that high school. It’s pretty simple to apply k-anonymity to control for an acceptable risk level. And add in generalization of age into groups and many questions can be answered. I think you could definite answer race x gender x grade but it will be harder when you factor in more unique characteristics like household income or vaccination status, etc.
ke88y 3y ago
I did some consulting on this WAAAAY back in the day. In the case of education, good enough anonymization often isn't really possible. A lot of information about school students is public -- yearbooks are gold mines, as are results from extra-curricular activities such as sports, class notes about whether/where they went to college or first jobs after college, etc. Still more can be purchased. As yet more can be inferred from public data (eg home address, rough estimates at parental income, etc.). This was back in the day. I'm sure now it's much worse. Most of the questions you want to ask about education are about treatments and outcomes. If these treatments (eg extra-curriculars) and outcomes (eg attended college, graduated HS, etc) are public then you can often figure out which student corresponds to each supposedly anonymous data-point. Maybe not perfectly. But way more than you would think. It's like the statistics version of those little logic puzzles from grade school -- "Four people have red hair. Five are girls and two are boys. Billy and Sally are chewing gum. Boy gum chewers have brown hair. No one over 5 has red hair. Sally is 4. etc. etc. etc. Match each person's name with their hair color.". You sort of figure out a small set of data points, then look at the results of the paper and reverse engineer some statistical calculations, and then a surprising amount of the others start falling into place. We didn't have a name for it when I did this work, but the basic point was "if you publish the dataset everyone will know little Johnny Table's test scores and GPA". Today this is called a reidentification attack. (I don't really know enough about the topic to have an opinion on the article per se, but "just publish everything" is definitely not a workable solution :))
- mrangle 3y ago
  Provide proof by de-anonymizing one data set from research within the past five years.
  - ke88y 3y ago
    LOL. I'm not going to deanonymize datasets for an internet stranger. Certainly not for free, but also not at any hourly rate unless I know that the organization employing me either has the original non-anonymized dataset or at least have very strict internal controls about how deanonymized data will be handled. (I also haven't done this work in a LONG time, and there's now a whole lot of academic work on the topic that didn't exist back then, so there are probably much better consultants for legitimate organizations looking to hire for this work.) If you want proof, you can use google to find LOTS of papers along these lines analyzing real datasets. I think https://arxiv.org/pdf/cs/0610105.pdf https://arxiv.org/pdf/cs/0610105.pdf is a fairly typical example.
tensor 3y ago
I very very strongly disagree. It's important that researchers get access to diverse data and industry collaboration is often crucial for this. If companies are required to make all their data public they will be far less willing to collaborate with research. It's already hard enough as is to convince corporations that it's worth their time.
karaterobot 3y ago
In theory this makes sense, and I agree with the spirit. In practice, you can often re-identify individuals even in an anonymized dataset. For example, if you're dealing with a very rare disease, or a small minority group, you can usually figure out an anonymized row of data is referring to if you really try. So, it's not so simple, and the responsible thing to do is not have a blanket policy that takes human judgment and accountability out of the loop.

kayodelycaon 3y ago

Not when there is an existing law limiting it. FERPA specifically protects student records.

galangalalgol 3y ago
Even anonymous records? That would seem to preclude studying the effectiveness of the education system. And if they weren't anonymous, what possible conclusions could you draw compared to anonymous records that would warrant that access?
rovr138 3y ago
> as long as individual PII is removed https://www2.ed.gov/policy/gen/guid/fpco/ferpa/index.html https://www2.ed.gov/policy/gen/guid/fpco/ferpa/index.html, "The Family Educational Rights and Privacy Act (FERPA) (20 U.S.C. § 1232g; 34 CFR Part 99) is a Federal law that protects the privacy of student education records" Now the question is if aggregated data with no PII is protected.
- jasonlotito 3y ago
  Data aggregation does not guarantee privacy.
  - prepend 3y ago
    Of course not, but there are tests that can be applied to determine if privacy is protected. It’s not possible to just aggregate and be done. But it is possible to set some privacy threshold and then insure that all records conform to that acceptable risk level.
jrochkind1 3y ago
If FERPA makes it illegal to share the data with researchers, then certainly it shouldn't have been shared. If FERPA allows sharing the data with researchers, is it right/proper/legal to share only on the condition it can't be used to harm the schools in court? Presumably that part is not in FERPA. (And to be clear, California here didn't just say they couldn't use the data in court, they said the researchers could not testify in any court case at all against the state. But we're talking hypothetically)

tmpz22 3y ago

> as long as individual PII is removed The devil is in the details. These records were likely not designed to be shared and I'd assume the entire system contains vulnerabilities that could create leakage. Leakage that could be used to harm individuals in a variety of ways - from discriminating future prospects to harassment and much much more. I agree in principle to what you're saying but we need to be truthful about what these current systems are capable of.

maximinus_thrax 3y ago

> data should be public as long as individual PII is removed This is one of those things when ideology doesn't match the real world. If the amount of data is large enough and with enough parameters, removing PII doesn't do anything to protect privacy. What about medical records? What about protected classes? What about data about vulnerable people or victims? Student data is protected by another layer of regulation and for good reason. Also, the judiciary is a 'public' institution in general. Should we not seal records for minors? 'No exceptions' - year right..

godelski 3y ago
> If the amount of data is large enough and with enough parameters, removing PII doesn't do anything to protect privacy. Honestly it doesn't have to be that large. We see this all the time with data websites or apps collect. Sure, you remove John Smith's name, but you still have his GPS coordinates. For the school, you remove Professor Smith's name, but you have a professor who teaches CS 123 and has 4 graduate students. You bet you can guess who that is. I really do support open data, especially about public institutions, but at the same time we are in an era where this information is quite powerful. Seems to make a case for something like homeomorphic encryption or something, but will that even stop these collisions?
- CaptainNegative 3y ago
  The appropriate notion here seems to be Differential Privacy, which is a mathematical definition informally saying "a scrambling of the dataset that is information theoretically indistinguishable from that where one arbitrary person is added or removed". It's a surprisingly deep topic, with entire (very good!) textbooks dedicated to it. PDF (entirely legal): https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf
nradov 3y ago
We're getting a little off topic here, but the Federal government has published specific guidance on de-identification of medical records. You can construct some artificial scenarios where re-identification might be theoretically possible through record linkage with other data sources but in practice it's unlikely. In principle a similar approach could be used for student data, although I'm not familiar with the legal issues. https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/index.html https://www.hhs.gov/hipaa/for-professionals/privacy/special-... But all of this is orthogonal to the core issue of whether a state government should be allowed to prevent researchers from participating in lawsuits. There is no student privacy issue involved there. Witnesses in a civil suit still aren't allowed to violate student privacy laws regardless of the data they have access to, so it makes no sense to conflate those issues.

jeroenhd 3y ago

Removing PII would probably also involve removing individual grades and other information that's necessary for any research to be effective. Thanks to predatory data collection practices on the internet, we know how little information you actually need to deanonimize someone. The problem only becomes worse when we're talking about kids. That said, research that cannot be reproduced is useless. There's a balance to be struck here, and it's somewhere between "make all data public" and "lock the data in a vault".

dragonwriter 3y ago

> If the institution is public, data should be public as long as individual PII is removed. PII is much broader than most people understand because reidentification of what amateurs would see as deidentified data is easy (often trivial), and, as a consequence, to be useful for research data is often not fully deidentified. EDIT: As an example, the HIPAA safe harbor deidentification standard requires removing 18 kinds of identifiers, including, as one of them: All geographic subdivisions smaller than a state, including street address, city, county, precinct, ZIP code, and their equivalent geocodes, except for the initial three digits of the ZIP code if, according to the current publicly available data from the Bureau of the Census: (1) The geographic unit formed by combining all ZIP codes with the same three initial digits contains more than 20,000 people; and (2) The initial three digits of a ZIP code for all such geographic units containing 20,000 or fewer people is changed to 000

Zetice 3y ago
To add to this, PII isn't always even clear. Different jurisdictions identify PII differently, there isn't One Master Definition that you pass a unit of data through, upon which an authoritative "THIS IS PII" or "THIS ISNT PII" is returned.
- orzig 3y ago
  +1 We had a multi-month project to get a subset of our data considered 'clean', and it required a consultant, a stats PhD and many dev hours. It was healthcare, so on the high end of paranoia (justifiably) but nowhere it is as simple as dropping the "name" column
prepend 3y ago
HIPAA also allows for “expert determination” [0] for deidentification that differs from safe harbor and can allow for all sorts of things since there’s no definition of what an expert is. And reidentification risk can be as high as even 1% and still be acceptable for hipaa. In a dataset of a million people that’s 10,000 people identified and still be “acceptable.” But hipaa doesn’t apply to these CA data, it’s just the clearest example of deidentification regulations I know of. But it’s totally possible to deidentify data suitable for release to these researchers. It’s just what CA considers deidentified and if it’s still useful enough to these researchers. For the topic they are researching it should be pretty straightforward to remove PII enough to protect individuals and only remove some really unique characteristics (ie, only a single 20 year old or a particular race and ethnicity). But I’m guessing age groups by race and gender and socioeconomic are possible to preserve without tying back to an individual. Id go so far to say as it would be non-trivial, but pretty easy, for CA to produce this for the researchers, if not to the general public. [0] https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/index.html#standard https://www.hhs.gov/hipaa/for-professionals/privacy/special-...

nineplay 3y ago

You'd have to remove so much PII as to make any examination worthless "A student of age <redacted> and gender <redacted> at school <redacted> has a GPA of <redacted>". As little information as "a 16 year old black male at Main Street High School" can be narrowed down to a handful of possible candidates at a lot of CA schools.

prepend 3y ago
I disagree as it depends on how many 16 year old black males there are in that high school. It’s pretty simple to apply k-anonymity to control for an acceptable risk level. And add in generalization of age into groups and many questions can be answered. I think you could definite answer race x gender x grade but it will be harder when you factor in more unique characteristics like household income or vaccination status, etc.

ke88y 3y ago

I did some consulting on this WAAAAY back in the day. In the case of education, good enough anonymization often isn't really possible. A lot of information about school students is public -- yearbooks are gold mines, as are results from extra-curricular activities such as sports, class notes about whether/where they went to college or first jobs after college, etc. Still more can be purchased. As yet more can be inferred from public data (eg home address, rough estimates at parental income, etc.). This was back in the day. I'm sure now it's much worse. Most of the questions you want to ask about education are about treatments and outcomes. If these treatments (eg extra-curriculars) and outcomes (eg attended college, graduated HS, etc) are public then you can often figure out which student corresponds to each supposedly anonymous data-point. Maybe not perfectly. But way more than you would think. It's like the statistics version of those little logic puzzles from grade school -- "Four people have red hair. Five are girls and two are boys. Billy and Sally are chewing gum. Boy gum chewers have brown hair. No one over 5 has red hair. Sally is 4. etc. etc. etc. Match each person's name with their hair color.". You sort of figure out a small set of data points, then look at the results of the paper and reverse engineer some statistical calculations, and then a surprising amount of the others start falling into place. We didn't have a name for it when I did this work, but the basic point was "if you publish the dataset everyone will know little Johnny Table's test scores and GPA". Today this is called a reidentification attack. (I don't really know enough about the topic to have an opinion on the article per se, but "just publish everything" is definitely not a workable solution :))

mrangle 3y ago
Provide proof by de-anonymizing one data set from research within the past five years.
- ke88y 3y ago
  LOL. I'm not going to deanonymize datasets for an internet stranger. Certainly not for free, but also not at any hourly rate unless I know that the organization employing me either has the original non-anonymized dataset or at least have very strict internal controls about how deanonymized data will be handled. (I also haven't done this work in a LONG time, and there's now a whole lot of academic work on the topic that didn't exist back then, so there are probably much better consultants for legitimate organizations looking to hire for this work.) If you want proof, you can use google to find LOTS of papers along these lines analyzing real datasets. I think https://arxiv.org/pdf/cs/0610105.pdf https://arxiv.org/pdf/cs/0610105.pdf is a fairly typical example.

tensor 3y ago

I very very strongly disagree. It's important that researchers get access to diverse data and industry collaboration is often crucial for this. If companies are required to make all their data public they will be far less willing to collaborate with research. It's already hard enough as is to convince corporations that it's worth their time.

karaterobot 3y ago

In theory this makes sense, and I agree with the spirit. In practice, you can often re-identify individuals even in an anonymized dataset. For example, if you're dealing with a very rare disease, or a small minority group, you can usually figure out an anonymized row of data is referring to if you really try. So, it's not so simple, and the responsible thing to do is not have a blanket policy that takes human judgment and accountability out of the loop.