5 ms·

If the project is truly "free" - build it in a way that can be passed down through generations: easily indexed with semantic data and without proprietary logins

by ugh123 8y ago

If the project is truly "free" - build it in a way that can be passed down through generations: easily indexed with semantic data and without proprietary logins or fiddle-some navigation. archive.org would be a good long term home or some other long lasting and perpetually funded service, beyond what Harvard would spend its billions of alumni bucks on.

bpchaps 8y ago
No kidding. After looking at this project for a bit longer, I'm much less optimistic about this project. Frankly, it really depresses me that they require research agreements to download all information. Developers of this project: Please make the information freely available in a way that doesn't require agreements with a giant for-profit company. For now, I'm convinced that this project is nothing more than a veiled advertisement for lexis nexis.
- wheaties 8y ago
  Unless you can provide them with the information, it's not the developers who are making the rules. There's a reason Lexus is as profitable as they are.
- paulgb 8y ago
  My reading of this is that it's something they're contractually bound to for the near term but not forever: > Access limitations on full text and bulk data are a component of Harvard’s collaboration agreement with Ravel Law, Inc. (now part of Lexis-Nexis). These limitations will end, at the latest, in March of 2024. Hopefully this means no logins, also, but that's less clear. The blame on this should really fall on the courts that allow private companies to paywall access to the rule of law. At least some states have started to do it right: > Once a jurisdiction transitions from print-first publishing to digital-first publishing, these limitations cease. Thus far, Illinois and Arkansas have made this important and positive shift and, as a result, all historical cases from these jurisdictions are freely available to the public without restriction. We hope many other jurisdictions will follow their example soon.
  - JackC 8y ago
    > Hopefully this means no logins, also, but that's less clear. The only things we have behind logins are what we are contractually required to, yes. This gets pretty fine-grained -- if you do a logged-out search across jurisdictions, requesting full text, the json contains error fields for the specific fields we aren't allowed to share without a login yet.
    - paulgb 8y ago
      Awesome, thanks for the clarification!
JackC 8y ago
Definitely! I'll sleep better when we have all the bulk data shipped to everyone who wants it. That will happen by March 2024, or earlier for any states that switch to official digital publishing. (I mean, Harvard isn't a bad home for this -- I work in a building with books that predate the printing press, and I work on stuff like Creative Commons-licensed forkable textbooks. Libraries are cool places. But Harvard definitely shouldn't be the only place that preserves this data set.) As far as preservation-friendly formats, our bulk data download format is xzipped jsonlines, which is tuned for NLP (highly compressed, parseable in a few lines of python with low memory requirements) rather than preservation: https://case.law/bulk/download/ https://case.law/bulk/download/ Internally we have a preservation format where each volume is stored as a bagit bag containing METS XML for the OCR and case-level data, plus color and black and white images of each case. These are much harder to work with, so it's not a focus to share them right now, but we can definitely share if someone makes a case for it.

by ugh123 8y ago

bpchaps 8y ago

No kidding. After looking at this project for a bit longer, I'm much less optimistic about this project. Frankly, it really depresses me that they require research agreements to download all information. Developers of this project: Please make the information freely available in a way that doesn't require agreements with a giant for-profit company. For now, I'm convinced that this project is nothing more than a veiled advertisement for lexis nexis.

wheaties 8y ago
Unless you can provide them with the information, it's not the developers who are making the rules. There's a reason Lexus is as profitable as they are.
paulgb 8y ago
My reading of this is that it's something they're contractually bound to for the near term but not forever: > Access limitations on full text and bulk data are a component of Harvard’s collaboration agreement with Ravel Law, Inc. (now part of Lexis-Nexis). These limitations will end, at the latest, in March of 2024. Hopefully this means no logins, also, but that's less clear. The blame on this should really fall on the courts that allow private companies to paywall access to the rule of law. At least some states have started to do it right: > Once a jurisdiction transitions from print-first publishing to digital-first publishing, these limitations cease. Thus far, Illinois and Arkansas have made this important and positive shift and, as a result, all historical cases from these jurisdictions are freely available to the public without restriction. We hope many other jurisdictions will follow their example soon.
- JackC 8y ago
  > Hopefully this means no logins, also, but that's less clear. The only things we have behind logins are what we are contractually required to, yes. This gets pretty fine-grained -- if you do a logged-out search across jurisdictions, requesting full text, the json contains error fields for the specific fields we aren't allowed to share without a login yet.
  - paulgb 8y ago
    Awesome, thanks for the clarification!

JackC 8y ago

Definitely! I'll sleep better when we have all the bulk data shipped to everyone who wants it. That will happen by March 2024, or earlier for any states that switch to official digital publishing. (I mean, Harvard isn't a bad home for this -- I work in a building with books that predate the printing press, and I work on stuff like Creative Commons-licensed forkable textbooks. Libraries are cool places. But Harvard definitely shouldn't be the only place that preserves this data set.) As far as preservation-friendly formats, our bulk data download format is xzipped jsonlines, which is tuned for NLP (highly compressed, parseable in a few lines of python with low memory requirements) rather than preservation: https://case.law/bulk/download/ https://case.law/bulk/download/ Internally we have a preservation format where each volume is stored as a bagit bag containing METS XML for the OCR and case-level data, plus color and black and white images of each case. These are much harder to work with, so it's not a focus to share them right now, but we can definitely share if someone makes a case for it.