I tried this as a project for a masters course a few months ago. I found the performance to be decent but nowhere near what humans could do. I wonder where that

I tried this as a project for a masters course a few months ago. I found the performance to be decent but nowhere near what humans could do. I wonder where that 53% figure came from. When I tested the code master I wrote against the guesser, because they used the same embeddings it was able to get near a perfect score. I ended up creating a few overlapping datasets and adding some variation in order to better simulate real humans.