5 ms·

Hey HN, Over the past year, I’ve been working on building the Open Australian Legal Corpus, the largest open database of Australian law. I started this project

Hey HN, Over the past year, I’ve been working on building the Open Australian Legal Corpus, the largest open database of Australian law. I started this project when I realised there were no open databases of Australian law I could use to train an LLM on. In this article, I run through the entire process of how I built my database, from months-long negotiations with governments to reverse engineering ancient web technologies to hacking together a multitude of different solutions for extracting text from documents. My hope is that the next time someone like me is interested in training an LLM to solve legal problems, they won't have to go down a year-long journey of trying to find the right data! You can find my database on HuggingFace (https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus https://huggingface.co/datasets/umarbutler/open-australian-l...) and the code used to create it on GitHub (https://github.com/umarbutler/open-australian-legal-corpus-creator https://github.com/umarbutler/open-australian-legal-corpus-c...).

deleted 3y ago
[deleted]
nextworddev 3y ago
Awesome work!
benn0 3y ago
Fantastic work, and really appreciate the write up. It's quite timely for me - I'm from a tech background and have just started studying Australian law, and was thinking about doing exactly this - so you are years ahead of me :). Just one note - the link in your Github readme to https://umarbutler.com/open-australian-legal-corpus https://umarbutler.com/open-australian-legal-corpus doesn't seem to go anywhere. For someone interested in using the data (and help out with bugs/issues), where would you suggest starting?
- ubutler 3y ago
  > Just one note - the link in your Github readme to https://umarbutler.com/open-australian-legal-corpus https://umarbutler.com/open-australian-legal-corpus doesn't seem to go anywhere. Thanks for the heads up! I've fixed that now. > For someone interested in using the data (and help out with bugs/issues), where would you suggest starting? I think the best place to start is by downloading the Corpus (visit https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus https://huggingface.co/datasets/umarbutler/open-australian-l... , and then click "Files and versions" and then "corpus.jsonl"). You can then use my Python library orjsonl to parse the dataset (you'd run, `corpus = orjsonl.load('corpus.jsonl')`). At that point, there's any number of applications you could use the dataset for. You could pretrain a model like BERT, ELECTRA, etc... and share it on HuggingFace. You could connect the dataset to GPT and do RAG over it. Etc...
SgtBastard 3y ago
Hey Umar, Fantastic work here! I was griping to my team just last night how painful developing a chunking strategy for Australian Legislation is that while there's (generally) layout consistency within a piece of legislation, that's not true across pieces of legislation... so I can imagine the pain of trying to collate legislation across jurisdictions. I've reach out via your LinkedIn profile - would be great if there was an opportunity to collaborate.
- ubutler 3y ago
  > Fantastic work here! I was griping to my team just last night how painful developing a chunking strategy for Australian Legislation is that while there's (generally) layout consistency within a piece of legislation, that's not true across pieces of legislation... so I can imagine the pain of trying to collate legislation across jurisdictions. Absolutely, there's a lack of consistency even within the same jurisdiction and document type. It only gets worse once you want to add multiple jurisdictions and different types of documents. My best strategy so far has been to use recursive chunking where you begin chunking at the largest section of newlines. Ideally though you want some form of semantic chunking where you already know what parts of the document represent Parts, Divisions, Schedules, Sections, Sub-sections, etc... > I've reach out via your LinkedIn profile - would be great if there was an opportunity to collaborate. Great! Always happy to connect.
  - triggercut 3y ago
    Yes, this. I've been trying to find a general way to automatically semantically chunk various legislation for a while now. Partly so as to diff various versions/amendments, but also to graph connections to other referenced legislation. Most of the time I end up having to just take half an hour to manually regex and format plain text. A particular case I have is where there is a draft bill put out for industry/community consultation. Quickly diffing the releases is the goal but for now usually relies on one (preferably two) subject matter experts to read the whole thing top to bottom to build an understanding. I don't think these would be available via the means you've secured. They are usually hosted on a relevant government entities website as PDFs One last question/comment, have you considered adding some additional reference info like the federal list of entities?[1] [1] https://www.finance.gov.au/government/managing-commonwealth-resources/structure-australian-government-public-sector/pgpa-act-flipchart-and-list https://www.finance.gov.au/government/managing-commonwealth-...
duttonw 3y ago
Are you open to pushing one of your Work in Progress(WIP) models to https://ollama.ai/library https://ollama.ai/library to show off and let others try it out and provide feedback?
- ubutler 3y ago
  Unfortunately, I haven't been able to train a model on my database just yet (although I have seen great results using it as a RAG data source for GPT-4). I'll keep ollama in mind once I have something to share.
triggercut 3y ago
Just FYI my work's network has blocked your site as "Malicious" (Symantec Endpoint Protection chrome extension)
- ubutler 3y ago
  Weird. Thanks for the heads up. I'm using SiteGround to host it which is shared hosting so perhaps that's why? I'll have to investigate.
kennethologist 3y ago
How can you be contacted? I would like to sponsor a project that does the same for smaller jurisdictions. Can email me at my username at yahoo.com

deleted 3y ago

[deleted]

nextworddev 3y ago

Awesome work!

benn0 3y ago

Fantastic work, and really appreciate the write up. It's quite timely for me - I'm from a tech background and have just started studying Australian law, and was thinking about doing exactly this - so you are years ahead of me :). Just one note - the link in your Github readme to https://umarbutler.com/open-australian-legal-corpus https://umarbutler.com/open-australian-legal-corpus doesn't seem to go anywhere. For someone interested in using the data (and help out with bugs/issues), where would you suggest starting?

ubutler 3y ago
> Just one note - the link in your Github readme to https://umarbutler.com/open-australian-legal-corpus https://umarbutler.com/open-australian-legal-corpus doesn't seem to go anywhere. Thanks for the heads up! I've fixed that now. > For someone interested in using the data (and help out with bugs/issues), where would you suggest starting? I think the best place to start is by downloading the Corpus (visit https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus https://huggingface.co/datasets/umarbutler/open-australian-l... , and then click "Files and versions" and then "corpus.jsonl"). You can then use my Python library orjsonl to parse the dataset (you'd run, `corpus = orjsonl.load('corpus.jsonl')`). At that point, there's any number of applications you could use the dataset for. You could pretrain a model like BERT, ELECTRA, etc... and share it on HuggingFace. You could connect the dataset to GPT and do RAG over it. Etc...

SgtBastard 3y ago

Hey Umar, Fantastic work here! I was griping to my team just last night how painful developing a chunking strategy for Australian Legislation is that while there's (generally) layout consistency within a piece of legislation, that's not true across pieces of legislation... so I can imagine the pain of trying to collate legislation across jurisdictions. I've reach out via your LinkedIn profile - would be great if there was an opportunity to collaborate.

ubutler 3y ago
> Fantastic work here! I was griping to my team just last night how painful developing a chunking strategy for Australian Legislation is that while there's (generally) layout consistency within a piece of legislation, that's not true across pieces of legislation... so I can imagine the pain of trying to collate legislation across jurisdictions. Absolutely, there's a lack of consistency even within the same jurisdiction and document type. It only gets worse once you want to add multiple jurisdictions and different types of documents. My best strategy so far has been to use recursive chunking where you begin chunking at the largest section of newlines. Ideally though you want some form of semantic chunking where you already know what parts of the document represent Parts, Divisions, Schedules, Sections, Sub-sections, etc... > I've reach out via your LinkedIn profile - would be great if there was an opportunity to collaborate. Great! Always happy to connect.
- triggercut 3y ago
  Yes, this. I've been trying to find a general way to automatically semantically chunk various legislation for a while now. Partly so as to diff various versions/amendments, but also to graph connections to other referenced legislation. Most of the time I end up having to just take half an hour to manually regex and format plain text. A particular case I have is where there is a draft bill put out for industry/community consultation. Quickly diffing the releases is the goal but for now usually relies on one (preferably two) subject matter experts to read the whole thing top to bottom to build an understanding. I don't think these would be available via the means you've secured. They are usually hosted on a relevant government entities website as PDFs One last question/comment, have you considered adding some additional reference info like the federal list of entities?[1] [1] https://www.finance.gov.au/government/managing-commonwealth-resources/structure-australian-government-public-sector/pgpa-act-flipchart-and-list https://www.finance.gov.au/government/managing-commonwealth-...

duttonw 3y ago

Are you open to pushing one of your Work in Progress(WIP) models to https://ollama.ai/library https://ollama.ai/library to show off and let others try it out and provide feedback?

ubutler 3y ago
Unfortunately, I haven't been able to train a model on my database just yet (although I have seen great results using it as a RAG data source for GPT-4). I'll keep ollama in mind once I have something to share.

triggercut 3y ago

Just FYI my work's network has blocked your site as "Malicious" (Symantec Endpoint Protection chrome extension)

ubutler 3y ago
Weird. Thanks for the heads up. I'm using SiteGround to host it which is shared hosting so perhaps that's why? I'll have to investigate.

kennethologist 3y ago

How can you be contacted? I would like to sponsor a project that does the same for smaller jurisdictions. Can email me at my username at yahoo.com