You don't challenge every idiot that insults you to a duel. Since duels are extremely risky, you would only demand one when something more important than your

You don't challenge every idiot that insults you to a duel. Since duels are extremely risky, you would only demand one when something more important than your life is at stake. Like your honor, or that of your beloved. To put honor at stake, the insults must be issued in public, with a specific goal of that besmirching. Even then most reasonable people will take steps to allay the situation before it gets too bad. And that was only something the nobility did. Everybody else just started a fight. Taking a few swings provides everyone around with entertainment while letting you work out your anger in a relatively safe way.