Have a look at the datasets library [1], but as a shortcut, you can just create a file named "my_code.json" in jsonlines format with one line per source file th

Have a look at the datasets library [1], but as a shortcut, you can just create a file named "my_code.json" in jsonlines format with one line per source file that looks like: {"text": "contents_of_source_file_1"} {"text": "contents_of_source_file_2"} ... And then pass that my_code.json as the dataset name. [1] https://github.com/huggingface/datasets https://github.com/huggingface/datasets