Ask HN: Is it hard to host your own podcast?

If I understand correctly It's just RSS Feed correct? Then I feel I should be able to host one easily on my own website. Are there any gotchas that I'm missing. Like hosting an email server? Email servers has problems like - getting a good ranking ip - filtering out spam - managing security updates etc