It looks like very few of your photos have captions, meaning that the algorithm doesn't have a lot of text to work with, and among those that do have text there

It looks like very few of your photos have captions, meaning that the algorithm doesn't have a lot of text to work with, and among those that do have text there are a few there which contain keywords that are probably heavily waited toward female such as "pink". The algorithm could probably be improved to also take the instagram name into consideration. Someone named "arthur" is very unlikely to be female.