The trouble is, this is a very difficult position for the organizers to defend as it is easy to attack them with "So you claim that the other sex is not capable

The trouble is, this is a very difficult position for the organizers to defend as it is easy to attack them with "So you claim that the other sex is not capable for skilled presentations, how dare you!" -comments. And likely this is something that resonates with enough people to create a storm in social media where you as the organizer end up in the middle. It is much easier to put up a more equal lineup of speakers and probably picking up few less insightful presentations. While somebody could also attack this approach, it would be much more difficult as you would need to argue that somebody who was left out due to their gender would have given much better presentation than somebody who was included. This would be so complicated that it is not likely to create the angry mob effect.