The PDFs didn't have headers/footers/line numbers chopped off, which makes chunking tough and embeddings less effective. I preprocessed the text a lot before ev

The PDFs didn't have headers/footers/line numbers chopped off, which makes chunking tough and embeddings less effective. I preprocessed the text a lot before even beginning to generate embeddings in my own pet project. Then metadata extraction (speaker tags, NER from transcript including dates and GPEs, etc.) to add a rudimentary feature set that could be presented and matched by a GPT3.5 call upon the initial query in order to trim down the vector search area. I feel like a lot was left on the table here. 360 pages of court transcripts (large font, generous margins and spacing, etc.) isn't massive at all, and it seems like there was too much reliance on the embeddings and the LLM to magically figure it all out.