I've not done pre-training for LLMs, but years ago I generated a completely synthetic dataset for table recognition using an off the shelf document segmentation

I've not done pre-training for LLMs, but years ago I generated a completely synthetic dataset for table recognition using an off the shelf document segmentation model, raw TeX, a random table generator, a discriminator and an evolutionary algorithm to generate different styles of tables. The project got killed due to management but I still got results on that dataset better than state of the art in 2023 with no human annotation. The Venn diagram of people who know TeX well enough to write a modular script for table generation with metadata and people who know how to train LLMs has an intersection of a dozen people I imagine.