7 ms·

Something is either public record - in which case it should be on a government website for free, and the AI companies should be free to scrape to their hearts d

Something is either public record - in which case it should be on a government website for free, and the AI companies should be free to scrape to their hearts desire... Or it should be sealed for X years and then public record. Where X might be 1 in cases where you don't want to hurt an ongoing investigation, or 100 if it's someone's private affairs. Nothing that goes through the courts should be sealed forever. We should give up with the idea of databases which are 'open' to the public, but you have to pay to access, reproduction isn't allowed, records cost pounds per page, and bulk scraping is denied. That isn't open.

nonethewiser 8mo ago
>We should give up with the idea of databases which are 'open' to the public, but you have to pay to access, reproduction isn't allowed, records cost pounds per page, and bulk scraping is denied. That isn't open. How about rate limited?
- londons_explore 8mo ago
  No. Open is open. Beyond DDoS protections, there should be no limits. If load on the server is a concern, make the whole database available as a torrent. People who run scrapers tend to prefer that anyway. This isn't someone's hobby project run from a $5 VPS - they can afford to serve 10k qps of readonly data if needed, and it would cost far less than the salary of 1 staff member.
  - delichon 8mo ago
    Rate limiting is a DDoS protection.
    - bastawhiz 8mo ago
      Pedantically: rate limiting is DoS prevention, not DDoS prevention. If you rate limit per IP, you're not mounting effective protection against a distributed attack. If you're rate limiting globally, you're taking your service offline for everyone.
  - tchalla 8mo ago
    > Open is open. I’d then ask OpenAI to be open too since open is open.
  - Gud 8mo ago
    The world is not black and white.
  - wang_li 8mo ago
    You're talking about a tragedy of the commons situation. There is an organic query rate of this based on the amount of public interest. Then there is the inorganic vacuuming of the entire dataset by someone who wants to exploit public services for private profit. There is zero reason why the public should socialize the cost of serving the excess capacity caused by private parties looking to profit from the public data. I could have my mind changed if the public policy is that any public data ingested into an AI system makes that AI system permanently free to use at any degree of load. If a company thinks that they should be able to put any load they want on public services for free, they should be willing to provide public services at any load for free.
- hyperpape 8mo ago
  If the rate limit is reasonable (allows full download of the entire set of data within a feasible time-frame), that could be acceptable. Otherwise, no.
- BillinghamJ 8mo ago
  Systems running core government functions should be set up to be able to efficiently execute their functions at scale, so I'd say it should only restrict extreme load, ie DoS attacks
- alberto467 8mo ago
  The issue with that is people can then flood everything with huge piles of documents, which is bad enough if it's all clean OCR'd digital data that you can quickly download in its entirety, but if you're stuck having to wait between downloading documents, you'll never find out what they don't want you to find out. It's like having you search through sand, it's bad enough while you can use a sift, but then they tell you that you can only use your bare hands, and your search efforts are made useless. This is not a new tactic btw and pretty relevant to recent events...
whizzter 8mo ago
Open to research yes. Free to ingest and make someones crimes a permanent part of AI datasets resulting in forever-convictions? No thanks. AI firms have shown themselves to be playing fast and loose with copyrighted works, a teenager shouldn't have their permanent AI profile become "shoplifter" because they did a crime at 15 yo that would otherwise have been expunged after a few years.
- evolve2k 8mo ago
  Fully agree. The AI companies have broken the basic pacts of public open data. Their ignoring of robots.txt files is but one example of their lack of regard. With the open commons being quickly pillaged we’ll end up in a “community member access only model”. A shift from grab any books here you like just get them back in a month; to you’ll need to register as a library member before you can borrow. I see that’s where we’ll end up. Public blogs and websites will suffer and respond first is my prediction.
- harel 8mo ago
  Between not delivering the data to AI companies, and barring it altogether is a fair distance. As far as I know, the MoJ is in talks with openAI themselves (https://www.ukauthority.com/articles/ministry-of-justice-reaches-agreement-with-openai https://www.ukauthority.com/articles/ministry-of-justice-rea...).
- spacebanana7 8mo ago
  The names of minors should never be released in public (with a handful of exceptions). But why shouldn't a 19 year old shoplifter have that on their public record? Would you prevent newspapers from reporting on it, or stop users posting about it on public forums?
  - criddell 8mo ago
    Would you want the first thing to show up after somebody googles your name to be an accusation for improper conduct around a child? In theory, people could dig deeper and find out you won in court and were acquitted, but people here should know that nobody ever reads the article...
bschmidt800 8mo ago
[dead]
tchalla 8mo ago
> Something is either public record - in which case it should be on a government website for free, and the AI companies should be free to scrape to their hearts desire...Or it should be sealed for X years and then public record. OR it should be allowed for humans to access the public record but charge fees for scrapers
dgxyz 8mo ago
Yes. This should be held by the London Archives in theory with the rest of the paper records of that sort. They have ability to seal documents until set dates and deal with digital archival and retrieval. I suspect some of this is it's a complete shit show and they want to bury it quickly or avoid having to pay up for an expensive vendor migration.
DrScientist 8mo ago
The story is about a tool that allows journalists to get advanced warning of court proceedings so them can choose to cover things of public interest. It's not about any post-case information.
- alberto467 8mo ago
  Then it would be even worse if this ends up affecting post-case information.
- sefrost 8mo ago
  We should remember that local journalism has been dead for a decade in most of the UK, largely due to social media. Any tool like this that can help important stories be told, by improving journalist access to data and making the process more efficient, must be a good thing.
bloak 8mo ago
I don't know what the particular issue is in this case but I've read about what happens with Freedom of Information (FOI) requests in England: apparently most of the requests are from male journalists/writers looking for salacious details of sex crimes against women, and the authorities are constantly using the mental health of family members as an argument for refusing to disclose material. Obviously there are also a few journalists using the FOI system to investigate serious political matters such as human rights and one wouldn't want those serious investigations to be hampered but there is a big problem with (what most people would call) abuse of the system. There _might_ perhaps be a similar issue with this court reporting database. England has a genuinely independent judiciary. Judges and court staff do not usually attempt to hide from journalists stuff that journalists ought to be investigating. On the other hand, if it's something like an inquest into the death of a well-known person which would only attract the worst kind of journalist they sometimes do quite a good job of scheduling the "public" hearing in such a way that only family members find out about it in time. A world government could perhaps make lots of legal records public while making it illegal for journalists to use that material for entertainment purpose but we don't have a world government: if the authorities in one country were to provide easy access to all the details of every rape and murder in that country then so-called "tech" companies in another country would use that data for entertainment purposes. I'm not sure what to do about that, apart, obviously, from establishing a world government (which arguably we need anyway in order to handle pollution and other things that are a "tragedy of the commons" but I don't see it happening any time soon).
- neversupervised 8mo ago
  Without numbers this sounds made up
  - bloak 8mo ago
    I should clarify that I was talking about the FOI requests submitted to a particular authority: I think it was the National Archives or some subsection thereof. If you're talking about all FOI requests submitted to all authorities then probably most of them don't relate in any way to criminal cases. I think we don't really need precise numbers to observe that public access to judicial data can be abused, which is all I wanted to say, really. I wrote too many words.
newsclues 8mo ago
I want information to be free. I don't think all information should be easily accessible. Some information should be in libraries, held for the public to access, but have that access recorded. If a group of people (citizens of a country) have data stored, they ought to be able to access it, but others maybe should pay a fee. There is data in "public records" that should be very hard to access, such as evidence of a court case involving the abuse of minors that really shouldn't be public, but we also need to ensure that secrets are not kept to protect wrongdoing by those in government or in power.
- LeifCarrotson 8mo ago
  Totally agreed! This is yet another example of reduced friction due to improved technology breaking a previously functional system without really changing the qualities it had before. I don't understand why this isn't obvious to more people. It's been said that "quantity has a quality all its own", and this is even more true when that quantity approaches infinity. Yes, license plates are public, and yes, a police officer could have watched to see whether or not a suspect vehicle went past. No, that does not mean that it's the same thing to put up ALPRs and monitor the travel activity of every car in the country. Yes, court records should be public, no, that doesn't mean an automatic process is the same as a human process. I don't want to just default to the idea that the way society was organized when I was a young person is the way it should be organized forever, but the capacity for access and analysis when various laws were passed and rights were agreed upon are completely different from the capacity for access and analysis with a computer.
Aerroon 8mo ago
One of the problems with open access to these government DBs is that it gives out a lot of information that spammers and scammers use. Eg if you create a business then that email address/phone number is going to get phished and spammed to hell and back again. It's all because the government makes that info freely accessible online. You could be a one man self-employed business and the moment you register you get inundated with spam.
adastra22 8mo ago
> Nothing that goes through the courts should be sealed forever. What about family law?
- pixl97 8mo ago
  Yep, these are commonly sealed records and having worked with family law lawyers there are things that happened to the victims that should never be unsealed.
  - adastra22 8mo ago
    Even outside of family law there are many justifiable reasons for sealing and even expunging (deletion) of records. I’m a believer that under the correct circumstances criminal records should be sealed & even in some cases expunged as well. People deserve a second chance. Family law is just the most obvious and unarguable example.
tw04 8mo ago
The idea that an individual can look up and case they want is the same thing as a bot being able to scrape and archive an entire dataset forever is just silly. One individual could spend their entire life going through one by one recording cases and never get through the whole dataset. A bot farm could sift through it in an hour. They are not the same thing.
croes 8mo ago
>and the AI companies should be free to scrape to their hearts desire... Why? They generate massive traffic, why should they get access for free?
forgetfreeman 8mo ago
Spoken like someone who's never spent thousands of dollars and literal years struggling to get online records corrected to reflect an expungement. Fuck anything that makes that process even more difficult which AI companies certainly will.
cyanydeez 8mo ago
I think the right balance is to air gap a database and allow access to the public by your standard: show up somewhere with a USB. I think it's right to prevent random drive by scraping by bots/AI/scammers. But it shouldnt inhibit consumers who want to use it to do their civic duties.
__MatrixMan__ 8mo ago
This is a good use case for a blockchain. AI companies can run their own nodes so they're not bashing infra that they don't pay for. Concerned citizens can run their own nodes so they know that the government isn't involved in any 1984-type shenanigans. In the sealed-for-X-years case, the government can publish a hash of the blocks that they intend to publish in X years so that when the time comes, people can prove that nobody tampered with the data in the interim. The government can decide to stop paying for the infra, but the only way to delete something that was once public record should be for all interested parties to also stop their nodes.
- madeofpalk 8mo ago
  I like the part where you created a system where if someone has enough resources they can just alter the judicial record.
  - __MatrixMan__ 8mo ago
    It seems like you're assuming we'd use proof-of-work. That would be crazy. The consensus mechanism would be: block is good if it has a judge's signature (or some other combination of signatures from other elected officials, depending on how the laws work where you are). Or are you proposing that somebody out there is prepared to subvert the signatures by computing a hash collision or some other herculean task?
pyrale 8mo ago
The issue is that the ease of access to information and the ease of proagating it can be transformative with regards to the effects of information to the public. > We should give up with the idea of databases which are 'open' to the public, but you have to pay to access, reproduction isn't allowed, records cost pounds per page, and bulk scraping is denied. That isn't open. I really don't see why. Adding friction to how available information is may be a way to preserve the ability for the public to access information, while also avoiding the pitfalls of unrestricted information access and processing.
protocolture 8mo ago
>We should give up with the idea of databases which are 'open' to the public, but you have to pay to access, reproduction isn't allowed, records cost pounds per page, and bulk scraping is denied. That isn't open. Eeehh I agree in principle. Its just that, a lot of people would want those databases heavily redacted if this was the case, which would ruin their utility. The financial disincentive that a 25 dollar access fee creates, reduces the amount of spam that can be directed at people listed in these databases by several orders of magnitude. Land title searches are already difficult (in my locality), because a few people can afford to have their solicitors or agents receive and read all their mail. Open that right up, and everyone will either hide behind an agent selling a trash all service, or try and sneak the wrong data in there. Poisoning it.
bsder 8mo ago
> We should give up with the idea of databases which are 'open' to the public, but you have to pay to access, reproduction isn't allowed, records cost pounds per page, and bulk scraping is denied. That isn't open. I disagree. Even if you simply made the database no cost but such that an actual human has to show up at an office with a signed request, that is fine. That's still open. The problem isn't the openness; it's the aggregation.

nonethewiser 8mo ago

londons_explore 8mo ago
No. Open is open. Beyond DDoS protections, there should be no limits. If load on the server is a concern, make the whole database available as a torrent. People who run scrapers tend to prefer that anyway. This isn't someone's hobby project run from a $5 VPS - they can afford to serve 10k qps of readonly data if needed, and it would cost far less than the salary of 1 staff member.
- delichon 8mo ago
  Rate limiting is a DDoS protection.
  - bastawhiz 8mo ago
    Pedantically: rate limiting is DoS prevention, not DDoS prevention. If you rate limit per IP, you're not mounting effective protection against a distributed attack. If you're rate limiting globally, you're taking your service offline for everyone.
- tchalla 8mo ago
  > Open is open. I’d then ask OpenAI to be open too since open is open.
- Gud 8mo ago
  The world is not black and white.
- wang_li 8mo ago
  You're talking about a tragedy of the commons situation. There is an organic query rate of this based on the amount of public interest. Then there is the inorganic vacuuming of the entire dataset by someone who wants to exploit public services for private profit. There is zero reason why the public should socialize the cost of serving the excess capacity caused by private parties looking to profit from the public data. I could have my mind changed if the public policy is that any public data ingested into an AI system makes that AI system permanently free to use at any degree of load. If a company thinks that they should be able to put any load they want on public services for free, they should be willing to provide public services at any load for free.
hyperpape 8mo ago
If the rate limit is reasonable (allows full download of the entire set of data within a feasible time-frame), that could be acceptable. Otherwise, no.
BillinghamJ 8mo ago
Systems running core government functions should be set up to be able to efficiently execute their functions at scale, so I'd say it should only restrict extreme load, ie DoS attacks
alberto467 8mo ago
The issue with that is people can then flood everything with huge piles of documents, which is bad enough if it's all clean OCR'd digital data that you can quickly download in its entirety, but if you're stuck having to wait between downloading documents, you'll never find out what they don't want you to find out. It's like having you search through sand, it's bad enough while you can use a sift, but then they tell you that you can only use your bare hands, and your search efforts are made useless. This is not a new tactic btw and pretty relevant to recent events...

whizzter 8mo ago

Open to research yes. Free to ingest and make someones crimes a permanent part of AI datasets resulting in forever-convictions? No thanks. AI firms have shown themselves to be playing fast and loose with copyrighted works, a teenager shouldn't have their permanent AI profile become "shoplifter" because they did a crime at 15 yo that would otherwise have been expunged after a few years.

evolve2k 8mo ago
Fully agree. The AI companies have broken the basic pacts of public open data. Their ignoring of robots.txt files is but one example of their lack of regard. With the open commons being quickly pillaged we’ll end up in a “community member access only model”. A shift from grab any books here you like just get them back in a month; to you’ll need to register as a library member before you can borrow. I see that’s where we’ll end up. Public blogs and websites will suffer and respond first is my prediction.
harel 8mo ago
Between not delivering the data to AI companies, and barring it altogether is a fair distance. As far as I know, the MoJ is in talks with openAI themselves (https://www.ukauthority.com/articles/ministry-of-justice-reaches-agreement-with-openai https://www.ukauthority.com/articles/ministry-of-justice-rea...).
spacebanana7 8mo ago
The names of minors should never be released in public (with a handful of exceptions). But why shouldn't a 19 year old shoplifter have that on their public record? Would you prevent newspapers from reporting on it, or stop users posting about it on public forums?
- criddell 8mo ago
  Would you want the first thing to show up after somebody googles your name to be an accusation for improper conduct around a child? In theory, people could dig deeper and find out you won in court and were acquitted, but people here should know that nobody ever reads the article...

bschmidt800 8mo ago

[dead]

tchalla 8mo ago

> Something is either public record - in which case it should be on a government website for free, and the AI companies should be free to scrape to their hearts desire...Or it should be sealed for X years and then public record. OR it should be allowed for humans to access the public record but charge fees for scrapers

dgxyz 8mo ago

Yes. This should be held by the London Archives in theory with the rest of the paper records of that sort. They have ability to seal documents until set dates and deal with digital archival and retrieval. I suspect some of this is it's a complete shit show and they want to bury it quickly or avoid having to pay up for an expensive vendor migration.

DrScientist 8mo ago

The story is about a tool that allows journalists to get advanced warning of court proceedings so them can choose to cover things of public interest. It's not about any post-case information.

alberto467 8mo ago
Then it would be even worse if this ends up affecting post-case information.
sefrost 8mo ago
We should remember that local journalism has been dead for a decade in most of the UK, largely due to social media. Any tool like this that can help important stories be told, by improving journalist access to data and making the process more efficient, must be a good thing.

bloak 8mo ago

I don't know what the particular issue is in this case but I've read about what happens with Freedom of Information (FOI) requests in England: apparently most of the requests are from male journalists/writers looking for salacious details of sex crimes against women, and the authorities are constantly using the mental health of family members as an argument for refusing to disclose material. Obviously there are also a few journalists using the FOI system to investigate serious political matters such as human rights and one wouldn't want those serious investigations to be hampered but there is a big problem with (what most people would call) abuse of the system. There _might_ perhaps be a similar issue with this court reporting database. England has a genuinely independent judiciary. Judges and court staff do not usually attempt to hide from journalists stuff that journalists ought to be investigating. On the other hand, if it's something like an inquest into the death of a well-known person which would only attract the worst kind of journalist they sometimes do quite a good job of scheduling the "public" hearing in such a way that only family members find out about it in time. A world government could perhaps make lots of legal records public while making it illegal for journalists to use that material for entertainment purpose but we don't have a world government: if the authorities in one country were to provide easy access to all the details of every rape and murder in that country then so-called "tech" companies in another country would use that data for entertainment purposes. I'm not sure what to do about that, apart, obviously, from establishing a world government (which arguably we need anyway in order to handle pollution and other things that are a "tragedy of the commons" but I don't see it happening any time soon).

neversupervised 8mo ago
Without numbers this sounds made up
- bloak 8mo ago
  I should clarify that I was talking about the FOI requests submitted to a particular authority: I think it was the National Archives or some subsection thereof. If you're talking about all FOI requests submitted to all authorities then probably most of them don't relate in any way to criminal cases. I think we don't really need precise numbers to observe that public access to judicial data can be abused, which is all I wanted to say, really. I wrote too many words.

newsclues 8mo ago

I want information to be free. I don't think all information should be easily accessible. Some information should be in libraries, held for the public to access, but have that access recorded. If a group of people (citizens of a country) have data stored, they ought to be able to access it, but others maybe should pay a fee. There is data in "public records" that should be very hard to access, such as evidence of a court case involving the abuse of minors that really shouldn't be public, but we also need to ensure that secrets are not kept to protect wrongdoing by those in government or in power.

LeifCarrotson 8mo ago
Totally agreed! This is yet another example of reduced friction due to improved technology breaking a previously functional system without really changing the qualities it had before. I don't understand why this isn't obvious to more people. It's been said that "quantity has a quality all its own", and this is even more true when that quantity approaches infinity. Yes, license plates are public, and yes, a police officer could have watched to see whether or not a suspect vehicle went past. No, that does not mean that it's the same thing to put up ALPRs and monitor the travel activity of every car in the country. Yes, court records should be public, no, that doesn't mean an automatic process is the same as a human process. I don't want to just default to the idea that the way society was organized when I was a young person is the way it should be organized forever, but the capacity for access and analysis when various laws were passed and rights were agreed upon are completely different from the capacity for access and analysis with a computer.

Aerroon 8mo ago

One of the problems with open access to these government DBs is that it gives out a lot of information that spammers and scammers use. Eg if you create a business then that email address/phone number is going to get phished and spammed to hell and back again. It's all because the government makes that info freely accessible online. You could be a one man self-employed business and the moment you register you get inundated with spam.

adastra22 8mo ago

> Nothing that goes through the courts should be sealed forever. What about family law?

pixl97 8mo ago
Yep, these are commonly sealed records and having worked with family law lawyers there are things that happened to the victims that should never be unsealed.
- adastra22 8mo ago
  Even outside of family law there are many justifiable reasons for sealing and even expunging (deletion) of records. I’m a believer that under the correct circumstances criminal records should be sealed & even in some cases expunged as well. People deserve a second chance. Family law is just the most obvious and unarguable example.

tw04 8mo ago

The idea that an individual can look up and case they want is the same thing as a bot being able to scrape and archive an entire dataset forever is just silly. One individual could spend their entire life going through one by one recording cases and never get through the whole dataset. A bot farm could sift through it in an hour. They are not the same thing.

croes 8mo ago

>and the AI companies should be free to scrape to their hearts desire... Why? They generate massive traffic, why should they get access for free?

forgetfreeman 8mo ago

Spoken like someone who's never spent thousands of dollars and literal years struggling to get online records corrected to reflect an expungement. Fuck anything that makes that process even more difficult which AI companies certainly will.

cyanydeez 8mo ago

I think the right balance is to air gap a database and allow access to the public by your standard: show up somewhere with a USB. I think it's right to prevent random drive by scraping by bots/AI/scammers. But it shouldnt inhibit consumers who want to use it to do their civic duties.

__MatrixMan__ 8mo ago

This is a good use case for a blockchain. AI companies can run their own nodes so they're not bashing infra that they don't pay for. Concerned citizens can run their own nodes so they know that the government isn't involved in any 1984-type shenanigans. In the sealed-for-X-years case, the government can publish a hash of the blocks that they intend to publish in X years so that when the time comes, people can prove that nobody tampered with the data in the interim. The government can decide to stop paying for the infra, but the only way to delete something that was once public record should be for all interested parties to also stop their nodes.

madeofpalk 8mo ago
I like the part where you created a system where if someone has enough resources they can just alter the judicial record.
- __MatrixMan__ 8mo ago
  It seems like you're assuming we'd use proof-of-work. That would be crazy. The consensus mechanism would be: block is good if it has a judge's signature (or some other combination of signatures from other elected officials, depending on how the laws work where you are). Or are you proposing that somebody out there is prepared to subvert the signatures by computing a hash collision or some other herculean task?

pyrale 8mo ago

The issue is that the ease of access to information and the ease of proagating it can be transformative with regards to the effects of information to the public. > We should give up with the idea of databases which are 'open' to the public, but you have to pay to access, reproduction isn't allowed, records cost pounds per page, and bulk scraping is denied. That isn't open. I really don't see why. Adding friction to how available information is may be a way to preserve the ability for the public to access information, while also avoiding the pitfalls of unrestricted information access and processing.

protocolture 8mo ago

>We should give up with the idea of databases which are 'open' to the public, but you have to pay to access, reproduction isn't allowed, records cost pounds per page, and bulk scraping is denied. That isn't open. Eeehh I agree in principle. Its just that, a lot of people would want those databases heavily redacted if this was the case, which would ruin their utility. The financial disincentive that a 25 dollar access fee creates, reduces the amount of spam that can be directed at people listed in these databases by several orders of magnitude. Land title searches are already difficult (in my locality), because a few people can afford to have their solicitors or agents receive and read all their mail. Open that right up, and everyone will either hide behind an agent selling a trash all service, or try and sneak the wrong data in there. Poisoning it.

bsder 8mo ago

> We should give up with the idea of databases which are 'open' to the public, but you have to pay to access, reproduction isn't allowed, records cost pounds per page, and bulk scraping is denied. That isn't open. I disagree. Even if you simply made the database no cost but such that an actual human has to show up at an office with a signed request, that is fine. That's still open. The problem isn't the openness; it's the aggregation.